Monitoring & Events

Watch your fleet instead of polling it. Subscribe to file changes, dispatch a command across many nodes at once, or be told when a machine comes back online — then read the results as they arrive.

Overview

A subscription is a standing request: "tell me when this happens on these nodes". It has three parts, and they're the same whatever you subscribe to:

1. Subscribe — fs_subscribe, command_subscribe, presence_subscribe, or fs_grep_subscribe. Each returns a subscription_id.
2. Read — event_tail on that id. Pass wait_seconds so it blocks until something arrives instead of spinning.
3. Unsubscribe — event_unsubscribe when you're done.

Two properties are worth knowing up front, because they're what make subscriptions cheap:

• Reading keeps a subscription alive. It expires about 10 idle minutes after the last event_tail.
• Unsubscribing stops the work, not just the delivery. Watchers are torn down on every node, and any command still running under the subscription is killed. An abandoned subscription keeps that work running until it expires — so end it deliberately.

What a node allows

Filesystem, search, and command subscriptions run on the node, so the node's owner has the final say over what runs there. Three keys in the agent's profile config (/etc/nobgp/default.yml on Linux) govern it, and all three are permissive by default — a freshly installed agent serves all of them, so a subscription works without touching the box:

allow-tools:        # capability domains this node serves
  - fs
  - command
allow-roots:        # where the fs domain may reach
  - /
allow-admin: true   # may a caller ask for elevated execution?

• allow-tools is the allowlist of capability domains — fs and command. One answer per node: it governs event sources and the direct tools, so dropping command refuses both command_subscribe and the command tool on that machine. Search rides the fs domain, so dropping fs takes the file watches, the fs_* tools, fs_grep and fs_grep_subscribe with it. An explicit allow-tools: [] serves neither. These are domains, not tool names: a value that is neither (fs_read, commands) is ignored with a log line, and if nothing valid remains the permissive default comes back — a typo can't quietly lock a machine out of its own remote remedy.
• allow-roots bounds where the fs domain may reach. Narrow it to e.g. [/etc, /var/log] and anything outside those trees is refused — that is what stops a single recursive watch on /, or one fleet-wide search, from walking a peer's whole filesystem. On Windows the roots are drive-qualified (C:\ProgramData\app); the whole-filesystem value is / or \, and both are accepted on either platform, so the default reaches everything on a Windows node too. One tree is excluded whatever the value: the agent's own configuration directory (/etc/nobgp, /usr/local/etc/nobgp, C:\ProgramData\nobgp), which holds the node's key and the config these keys are read from.
• allow-admin vetoes elevated execution and elevated file access. With allow-admin: false, anything that would run as uid 0 is refused rather than silently downgraded — including an unelevated request on a node with no user configured, where root is all there is. Everything else runs as the profile's user. The fs_* tools are a real privilege drop now, not an authorization check standing in for one: the agent drops privilege before opening the file, so a file written unelevated is owned by that account.

Paths are checked with symlinks resolved on both sides, so a link inside a root can't smuggle a watch out of it. Changes take effect without a restart — the agent watches its own config and reloads itself. To see what a node is applying right now, read the allow block of nobgp status on the box.

They govern every remote path onto the node

All three bound what a node_grant reaches — the MCP tools and the event-bus sources — and, since agent 0.4.33, a published terminal service as well. Services were exempt before then; the exemption meant a service published to run as root handed out root on a node whose owner had set allow-admin: false, so it is gone.

sudo nobgp config --allow-tools= and friends warn at the moment you save when the result leaves a node serving no commands or no file access, and --allow-admin=false asks for confirmation when it would turn remote execution off entirely. Take the warning seriously: without command, the browser terminal is refused too, and the way back is at the machine itself. The same warning appears in the allow block of nobgp status.

allow-admin: false on Windows

Windows has no user switching, so there is no unprivileged identity to drop to. Since agent 0.4.33 the setting is honoured there as the only answer the platform can give — it refuses remote execution on that node entirely. Earlier agents logged that they could not apply it and kept admin enabled. On a Windows node you want to keep operable, narrow what reaches it with the router-side node_grant instead.

Renamed from event-sources / event-watch-roots

Earlier agents called these event-sources and event-watch-roots, and both were empty by default. The keys were renamed and the defaults flipped to permissive; an agent upgrading from the old names migrates them without tightening anything the node already served. A refusal now means the owner narrowed something, not that they never opted in.

Presence subscriptions need none of this — the router publishes those, and the nodes run nothing.

matched_nodes is who the work was sent to

Not who accepted it. A node that won't serve a source says so with a refused event naming the reason, rather than going quiet.

Choosing a cohort

Every subscribe tool takes the same node selector, so you address a set of machines once instead of looping over them:

Field	Restricts to
network_id / network_name	The network to act in (one or the other, not both)
node_ids / node_names	Specific nodes, by id or by name
platform / os / arch	What the nodes report about themselves (debian, linux, arm64…). arch takes the same spellings the rest of the product uses — arm64, amd64, armv7, armv6, 386 — and accepts the kernel synonyms (aarch64, x86_64) too
labels	Nodes carrying all of these labels

All fields are ANDed, and a subscription only ever matches nodes that are currently online — an offline machine can't start a watcher or run a dispatched command. Omit every node-narrowing field to cover the whole network.

The durable way to name a cohort is labels: tag once, then scope by meaning.

Label the Raspberry Pis in production with role=edge
Now watch /etc/nginx on everything labelled role=edge

Because filesystem and presence subscriptions re-evaluate their selector continuously, a node you enroll or relabel later joins the running subscription on its own — no re-subscribing. (The two dispatches are the deliberate exception: command_subscribe and fs_grep_subscribe pin their cohort when you subscribe, so a machine that appears a minute later never picks up a migration that was already sent, or a search that has already been handed out.)

Every subscribe call reports what your selector actually matched — matched_nodes plus the resolved nodes list — so you never have to create a throwaway subscription to find out. For a dispatch that list is the fixed cohort; for a file or presence subscription it is a snapshot, since the selector keeps being re-evaluated and later joiners won't appear in it. A selector that matches nothing right now still creates a file or presence subscription (it picks nodes up as they match) but says so plainly rather than returning a success-shaped zero; the dispatches — command_subscribe and fs_grep_subscribe — return not_found instead, because work sent to nobody is nothing.

Watching files

Watch /var/log/app for writes on every node labelled role=edge,
and watch /etc/nginx/nginx.conf for everything

Each entry in watches is one path with its own recursive flag and ops filter (create, write, remove, rename, chmod):

{
  "labels": { "role": "edge" },
  "watches": [
    { "path": "/var/log/app", "recursive": true, "ops": ["write"] },
    { "path": "/etc/nginx/nginx.conf" }
  ]
}

Paths are literal — there is no glob syntax, and a path containing *, ?, or [ is refused. To watch a tree, name the directory and set recursive.

Events carry paths and operations, never file content:

{
  "source_id": "9f1c…",
  "at": "2026-07-29T10:15:04Z",
  "fs": { "changes": [{ "path": "/etc/nginx/nginx.conf", "op": "write", "timestamp_ms": 1785060904123 }] }
}

To see what actually changed, read the file with fs_read on the node named by source_id.

Narrow the ops

Filtering happens on the node, so anything you exclude costs nothing. On a busy tree, chmod noise from a package manager can overflow a subscription that only ever cared about writes.

Each platform's native watcher has its own habits, and they show through:

• macOS (FSEvents) reports paths with symlinks resolved — a watch on /tmp/x reports events under /private/tmp/x — so don't prefix-match against the path you subscribed. A create followed immediately by writes can also coalesce into the create alone, which means an ops: ["write"] filter may miss a file that was created and written in one go.
• Linux (inotify) reports the distinct operations, on the paths as you gave them.

Dispatching a command across a fleet

command_subscribe runs a command once on each matching node, detached, and collects a report from every one of them. It returns immediately — nothing holds a session open.

Run the backup script on every node labelled role=db and tell me how each one went

{
  "labels": { "role": "db" },
  "command": "/usr/local/bin/backup.sh",
  "max_seconds": 1800
}

The response lists exactly which nodes the work went to. Each node reports back:

{
  "source_id": "9f1c…",
  "status": "completed",
  "at": "2026-07-29T10:41:22Z",
  "command": {
    "lines": [{ "line": "backup finished: 4.2 GB", "stderr": false }],
    "exit_code": 0,
    "duration_ms": 184320
  }
}

exit_code appears on the event that ends a run — including when it is 0 — and is absent from the streaming output events in between, where there is no exit status yet.

event_tail also returns a progress block for a dispatch:

{ "dispatched": 12, "accepted": 11, "settled": 12, "outstanding": 0 }

• accepted below dispatched means some nodes never took the work — offline since, or refusing the source.
• outstanding: 0 means every node has finished. That is how you know the dispatch is done.

Every run is bounded (max_seconds, default 300, maximum 3600) and the dispatch always ends. There is no repeating mode: a command that should keep running is a service, not a subscription. event_unsubscribe kills whatever is still running.

Who the command runs as

Execution identity matches the command tool field for field. A node offers exactly two identities and one boolean picks between them: admin.

• admin omitted or false — each node's own configured user, which is not uniformly root (typically the installing account). On a node with none configured — a container, a bare-root install — there is nothing lesser to fall back to, so this still runs as root.
• admin: true — the node's ambient identity. A node whose owner set allow-admin: false reports refused rather than running the work unprivileged. The gate is on the resolved uid 0, not a spelling.

A cohort is rarely uniform: network_directory reports each node's user and allow_admin, and the same dispatch can run as a login on one node and as root on the next. Windows has one identity, so both values land on LocalSystem there.

To run as some third account, use su inside the command itself.

Same authority as command

A dispatched command runs through each node's shell with the same authority the command tool gives you. Scope the selector deliberately.

Progress from inside the command

A dispatched command sees $NOBGP_SUBSCRIPTION in its environment, so a long script can publish its own progress back into the same subscription you're reading — with nobgp notify, which reads that variable itself:

#!/bin/sh
nobgp notify phase=start
pg_dump mydb | gzip > /backups/db.sql.gz
nobgp notify phase=done bytes="$(stat -c %s /backups/db.sql.gz)"

key=value arguments become a typed JSON object; a plain string or a JSON document is sent as written. Publishing when nobody is listening exits 0 — a script under the bus shouldn't fail because the reader went away. Publishing nothing is different: no payload and no --status fails with exit 1, because it almost always means a pipe is missing its - (./thing | nobgp notify - reads stdin; ./thing | nobgp notify does not).

Your own client can do the same with event_publish — for example to drop a marker into the stream before starting a phase.

On the node itself, nobgp events shows what that machine is publishing, which is the half you can't see from the subscriber side. It's the quickest way to confirm a source is actually reporting.

Searching a fleet

fs_grep_subscribe runs one RE2 regex search across every matching node and delivers the matches as events, instead of making you wait on one big response per machine. It is the async half of fs_grep — same walk, same skipping of binary files and noise directories (.git, node_modules, .venv, target, dist, …), aimed at a cohort.

Search /etc/nginx for "ssl_certificate" on everything labelled role=edge

{
  "labels": { "role": "edge" },
  "pattern": "ssl_certificate",
  "paths": ["/etc/nginx"],
  "max_count": 200
}

It returns immediately with a subscription_id. Each match arrives as its own event, and source_id is the node the match is on — so a fleet-wide search is attributable without a second call:

{
  "source_id": "9f1c…",
  "at": "2026-07-31T09:12:44Z",
  "payload": { "path": "/etc/nginx/sites-enabled/api", "line": 42, "text": "  ssl_certificate /etc/ssl/api.crt;" }
}

Each node then reports once more with a terminal status — completed with a summary of how many matches across how many files, or failed / refused with a reason. Like a command dispatch, the walk always ends: event_tail's progress block reaching outstanding: 0 means every node has finished.

max_count is per node (default 1000, hard cap 100 000), and paths must be absolute. Narrow them rather than searching / on a busy fleet — and drain the queue while the walk runs, since only the newest 256 events are held.

Reach for	When
fs_grep	One node, and you want its matches in one response
fs_grep_subscribe	More than one node · a tree big enough that you'd rather read matches as they land · you want to keep working while it runs

It reads, but it isn't marked read-only

The search itself only reads. The call stands up a subscription and starts work on every node in the cohort, which is state — so a client that auto-approves on capability hints is told so rather than waved through.

Waiting for a machine

presence_subscribe reports node lifecycle transitions. The router publishes these, so the nodes run nothing and pay nothing — it's the cheapest thing on the bus and the right way to wait for a machine to come back after a reboot.

Tell me when kitchen-pi comes back online

{
  "source_id": "9f1c…",
  "at": "2026-07-29T10:02:00Z",
  "presence": { "transition": "registered", "node_name": "kitchen-pi", "version": "0.4.15", "platform": "raspbian" }
}

transition is one of:

• online — the node's connection came up
• offline — the node's connection went away
• registered — the node re-registered. This is where version and platform can change, so it's the transition to watch when tracking an agent rollout.

Reading events

event_tail returns pending events oldest first:

{ "subscription_id": "…", "max": 50, "wait_seconds": 30 }

Use wait_seconds (up to 30) when you're waiting on something specific rather than calling in a loop. An empty result is normal.

Reading drains the queue — the next call returns the next events, not the same ones again. If a reply can't be delivered, its batch is put back and redelivered on the following call, so delivery is at-least-once: an event may repeat, but a failed read never swallows it.

Don't read at all — be told

Through a node's local MCP server, a subscription is an MCP resource. Subscribe to nobgp://events/<subscription_id> once and the server pushes notifications/resources/updated as events land, so waiting for a long command or a file change costs no polling and survives between turns. event_tail remains the fallback for clients that don't consume notifications.

Each event may carry a status. Absent (or ok) means ordinary data with more to come; anything else is that node's last word — the other nodes on the subscription carry on:

status	Meaning
completed	The work finished cleanly — a command exited 0, or a node's search walk completed and reason summarises what it found
failed	The work finished badly — a non-zero exit, or a search the node could not complete
timeout	The command hit max_seconds and was killed
cancelled	You unsubscribed and the run was torn down
overflow	The node couldn't keep up and stopped — see reason
refused	The node won't run this source at all — see reason

Refusals and overflow

Both are answers, not errors, and both name a reason:

• refused — the node won't serve this source: the kind isn't in its allow-tools, the path is outside its allow-roots, the run asked for elevation its allow-admin: false vetoes, or the kind isn't supported on its platform. See What a node allows. matched_nodes tells you who the work was sent to, not who accepted it.
• overflow — changes (or output) outran what the node could report, so it stopped rather than delivering a silent fraction. The reason names which side overflowed — the node's own buffer or the kernel's queue. Narrow the watch paths or ops; for a chatty command, filter at the source (grep, tail -n) or redirect to a file and read it with fs_read.

A subscription holds only the newest 256 pending events. Treat events as coarse signals and investigate details with command or fs_read on the node that reported.

Cleaning up

List my subscriptions in production and cancel the file watcher

event_subscriptions lists the subscriptions you created in a network, with each one's id and kind (fs, command, grep, presence). It's the recovery path when an id is lost to a new session or a crashed script — finding the orphan and ending it beats waiting out the idle clock while its watchers keep running.

Limits

Limit	Value
Idle lifetime (refreshed by every event_tail)	10 minutes
Pending events held per subscription	newest 256
Watches per fs_subscribe	64
Watch path / command length	4096 characters
Command runtime	300s default, 3600s maximum
event_publish payload	8192 bytes

A network's active subscriptions also share a budget for their declarations, since those travel to each node in a single frame. An oversized or over-budget subscription is refused with invalid_args and tells you to narrow it or unsubscribe something.

Isolated networks

Isolated networks have no event bus — their members must not observe one another.

Next Steps

• MCP Reference - Full schemas for every event tool
• Local MCP Server - Push delivery, and driving a network from one machine's Claude
• Core Concepts - How subscriptions fit with sessions and services
• Publishing Services - For work that should keep running, rather than a bounded dispatch