Monitoring & Events

Watch your fleet instead of polling it. Subscribe to file changes, dispatch a command across many nodes at once, or be told when a machine comes back online — then read the results as they arrive.

Overview

A subscription is a standing request: "tell me when this happens on these nodes". It has three parts, and they're the same whatever you subscribe to:

1. Subscribe — fs_subscribe, command_subscribe, presence_subscribe, or fs_grep_subscribe. Each returns subscription.id.
2. Read — event_tail with that id as subscription. Pass wait so it blocks until something arrives instead of spinning.

⚠ The argument names changed in router 0.4.201, and the old ones are refused over MCP: the selector fields now live under select, subscription_id is subscription, wait_seconds is wait, keep_seconds is keep, max_seconds is timeout, and an event names its node as node rather than source_id. The full table is on the MCP reference. ⚠ From router 0.4.207 that node is an object, {id, name}, so a fleet-wide stream reads as machine names instead of uuids — name is absent when the router cannot name the node any more, id is always there, and a client that reads event.node as a string needs event.node.id instead. 3. Unsubscribe — event_unsubscribe when you're done.

Two properties are worth knowing up front, because they're what make subscriptions cheap:

• Reading keeps a subscription alive. It expires about 10 idle minutes after the last event_tail — and from router 0.4.91 a file or presence subscription can ask for longer with keep, up to 24 hours.
• Unsubscribing stops the work, not just the delivery. Watchers are torn down on every node, and any command still running under the subscription is killed. An abandoned subscription keeps that work running until it expires — so end it deliberately.

What a node allows

Filesystem, search, and command subscriptions run on the node, so the node's owner has the final say over what runs there. Three keys in the agent's profile config (/etc/nobgp/default.yml on Linux) govern it, and all three are permissive by default — a freshly installed agent serves all of them, so a subscription works without touching the box:

allow-tools:        # capability domains this node serves
  - fs
  - command
  - logs
allow-roots:        # where the fs domain may reach
  - /
allow-root: true    # may a caller ask to run as the superuser?

• allow-tools is the allowlist of capability domains — fs, command, and from agent 0.4.153 logs, which serves node_logs. One answer per node: it governs event sources and the direct tools, so dropping command refuses both command_subscribe and the command tool on that machine. Search rides the fs domain, so dropping fs takes the file watches, the fs_* tools, fs_grep and fs_grep_subscribe with it. An explicit allow-tools: [] serves none of them. These are domains, not tool names: a value that is not one (fs_read, commands) is dropped with a log line. ⚠ From agent 0.4.153 that fails closed — if nothing valid remains the node serves nothing, where earlier agents restored the permissive default. nobgp config --allow-tools refuses an unrecognised value outright rather than letting one be saved, and nobgp status reports allow.tools_invalid for a profile that already holds one.
• allow-roots bounds where the fs domain may reach. Narrow it to e.g. [/etc, /var/log] and anything outside those trees is refused — that is what stops a single recursive watch on /, or one fleet-wide search, from walking a peer's whole filesystem. On Windows the roots are drive-qualified (C:\ProgramData\app); the whole-filesystem value is / or \, and both are accepted on either platform, so the default reaches everything on a Windows node too. One tree is excluded whatever the value: the agent's own configuration directory (/etc/nobgp, /usr/local/etc/nobgp, C:\ProgramData\nobgp), which holds the node's key and the config these keys are read from.
• allow-root vetoes execution and file access as the superuser — the node half of the two gates on it, the other being whether the caller holds root on that node at the router (Execution identity). With allow-root: false, anything that would run as uid 0 is refused rather than silently downgraded. Everything else runs as the profile's user — and where that would itself resolve to uid 0, the node refuses the unelevated request too rather than falling back to root (unelevated never means root), which is why allow-root: false on a node with no usable user is an off switch rather than a narrowing. The fs_* tools are a real privilege drop now, not an authorization check standing in for one: the agent drops privilege before opening the file, so a file written unelevated is owned by that account. ⚠ It was called allow-admin through agent 0.4.152, the old name is still read, and where both keys are set the stricter value wins — see The veto on root was renamed.

Paths are checked with symlinks resolved on both sides, so a link inside a root can't smuggle a watch out of it. Changes take effect without a restart — the agent watches its own config and reloads itself. To see what a node is applying right now, read the allow block of nobgp status on the box.

They govern every remote path onto the node

All three bound what a node_grant reaches — the MCP tools and the event-bus sources — and, since agent 0.4.33, a published terminal service as well. Services were exempt before then; the exemption meant a service published to run as root handed out root on a node whose owner had set allow-root: false, so it is gone.

sudo nobgp config --allow-tools= and friends warn at the moment you save when the result leaves a node serving no commands or no file access, and --allow-root=false asks for confirmation when it would turn remote execution off entirely. Take the warning seriously: without command, the browser terminal is refused too, and the way back is at the machine itself. The same warning appears in the allow block of nobgp status.

allow-root: false on Windows

A Windows node that captured no install account runs processes only as LocalSystem, so there is no unprivileged identity for a dispatched command to drop to. Since agent 0.4.33 the setting is honoured there as the only answer such a node can give — it refuses remote execution on that node entirely. Earlier agents logged that they could not apply it and kept the superuser enabled. On a Windows node you want to keep operable, narrow what reaches it with the router-side node_grant instead.

Since agent 0.4.44 a configured user gives such a node a second identity, and from agent 0.4.46 it covers every source: an fs watch acts as that account and a dispatched command runs as it, so allow-root: false narrows such a node instead of switching it off. On 0.4.44 and 0.4.45 the identity covered file operations only, so the fs sources gained it while the exec source did not.

Renamed from event-sources / event-watch-roots

Earlier agents called these event-sources and event-watch-roots, and both were empty by default. The keys were renamed and the defaults flipped to permissive; an agent upgrading from the old names migrates them without tightening anything the node already served. A refusal now means the owner narrowed something, not that they never opted in.

Presence subscriptions need none of this — the router publishes those, and the nodes run nothing.

matched_nodes is who the work was sent to

Not who accepted it. A node that won't serve a source says so with a refused event naming the reason, rather than going quiet.

Choosing a cohort

Every subscribe tool takes the same node selector, in one object called select from router 0.4.201 (its fields were at the top level of the call before), so you address a set of machines once instead of looping over them:

Field of select	Restricts to
network	The network to act in — its id or its name
nodes	Specific nodes, each an id or a name
name	Nodes whose name matches a glob
platform / os / arch	What the nodes report about themselves (debian, linux, arm64…). arch takes the same spellings the rest of the product uses — arm64, amd64, armv7, armv6, 386, and from router 0.4.199 armv5, mips and riscv64, with mipsle a value of its own from router 0.4.200 — and accepts the kernel synonyms (aarch64, x86_64) too. From router 0.4.200 it matches the agent build a node runs, which network_directory reports as info.build_architecture, and the kernel's architecture only for a node that has not reported one
labels	Nodes carrying all of these labels
owner	Whose nodes: "me", "any", or a member's email. Router 0.4.201+ — see below

All fields are ANDed, and a subscription only ever matches nodes that are currently online — an offline machine can't start a watcher or run a dispatched command. Omit every node-narrowing field to cover the whole cohort.

⚠ presence_subscribe is the exception from router 0.4.204: it matches nodes in any state, because waiting for an offline machine to come back is what it is for, and it delivers that machine's events. Before, a selector naming two rebooting nodes was answered matched_nodes: 0 — matches nothing — about a subscription that was working. Send select.online to narrow it yourself.

⚠ Naming a node that is not in the network you named is refused, from router 0.4.204: failed_precondition, counting them and saying which network each is in, and nothing runs. Make one call per network, or omit select.network. Before, such a node was acted on anyway while the same selector written with labels or a platform left it out. A file or presence subscription works in one network, so those two refuse it even when you named no network at all.

The scope is your organizations, not a network, from router 0.4.201: a selector with no network covers every node your organizations own, in any network or in none, and network narrows it to one. ⚠ The event bus itself is still keyed by one network, so a subscription whose selector names no network takes the one network every node it matched is in — and when those nodes span several networks, or one of them is in none, the call is failed_precondition asking you to name select.network.

A cohort that names no node now takes only your nodes

Router 0.4.201. On command_subscribe, fs_grep_subscribe, fs_subscribe, presence_subscribe and net_reach_many, a select that names no node defaults to the nodes you own. It used to take every node you may use, colleagues' nodes included. Send select.owner: "any" for that behaviour; a select that does name nodes is never narrowed, because naming a node is the intent.

The narrowing is never silent: the result carries owner_filter and not_mine_skipped (how many nodes matched the rest of the selector and were left out), and when the filter leaves nothing the call is not_found naming owner: "any". node_list, node_label, node_grant and node_revoke share the selector but default to any owner — seeing a node is not using it.

⚠ not_mine_skipped counts only the nodes owner: "any" would actually reach, from router 0.4.204 — on command_subscribe, fs_grep_subscribe and net_reach_many. A node you may not use is left out of the count, so following the note and re-sending with owner: "any" gives you exactly the number it named. Before, it counted nodes the wider call would then have refused.

The durable way to name a cohort is labels: tag once, then scope by meaning.

Label the Raspberry Pis in production with role=edge
Now watch /etc/nginx on everything labelled role=edge

Because filesystem and presence subscriptions re-evaluate their selector continuously, a node you enroll or relabel later joins the running subscription on its own — no re-subscribing. (The two dispatches are the deliberate exception: command_subscribe and fs_grep_subscribe pin their cohort when you subscribe, so a machine that appears a minute later never picks up a migration that was already sent, or a search that has already been handed out.)

Every subscribe call reports what your selector actually matched — matched_nodes plus the resolved nodes list — so you never have to create a throwaway subscription to find out. For a dispatch that list is the fixed cohort; for a file or presence subscription it is a snapshot, since the selector keeps being re-evaluated and later joiners won't appear in it. A selector that matches nothing right now still creates a file or presence subscription (it picks nodes up as they match) but says so plainly rather than returning a success-shaped zero; the dispatches — command_subscribe and fs_grep_subscribe — return not_found instead, because work sent to nobody is nothing.

Watching files

Watch /var/log/app for writes on every node labelled role=edge,
and watch /etc/nginx/nginx.conf for everything

Each entry in watches is one path with its own recursive flag and ops filter (create, write, remove, rename, chmod):

{
  "select": { "labels": { "role": "edge" } },
  "watches": [
    { "path": "/var/log/app", "recursive": true, "ops": ["write"] },
    { "path": "/etc/nginx/nginx.conf" }
  ]
}

Paths are literal — there is no glob syntax, and a path containing *, ?, or [ is refused. To watch a tree, name the directory and set recursive.

The path has to exist already

A selector keeps being re-evaluated, so a node enrolled or relabelled later joins the subscription on its own. A path does not: a running watch never re-examines a path the node answered no such file or directory for.

How you find out depends on how many paths you declared. If every path was missing, the node has no watch to run and says so — a refused event naming each path and what it answered — and because nothing is running for it, the node tries the whole watch again the next time the router re-sends the subscription, which a reconnect does. If some path was missing, the watch runs on the rest and nothing reports the one that failed: its events simply never arrive, and an empty read looks exactly like a quiet tree.

To be told when something appears, watch the parent directory with recursive and let the create tell you. To watch a path you are about to make, make it first and subscribe after.

Events carry paths and operations, never file content:

{
  "node": { "id": "9f1c…", "name": "edge-01" },
  "at": "2026-07-29T10:15:04Z",
  "fs": { "changes": [{ "path": "/etc/nginx/nginx.conf", "op": "write", "timestamp_ms": 1785060904123 }] }
}

To see what actually changed, read the file with fs_read on the node node names (an object from router 0.4.207, the id as a bare string before it, and source_id until router 0.4.200).

Narrow the ops

Filtering happens on the node, so anything you exclude costs nothing. On a busy tree, chmod noise from a package manager can overflow a subscription that only ever cared about writes.

A noBGP write arrives as a create, not a write

The file tools write atomically — fs_write, fs_edit, fs_copy and fs_fetch_url all land bytes in a sibling temporary file and rename it onto the destination — so the watcher sees a new file appearing. An ops: ["write"] filter therefore misses every change made by these very tools. Watch for create too.

From agent 0.4.86 the temporary sibling itself is dropped on the node and never reaches you: by the time you read an event about config.yaml.nobgp-write-1274 the name is already gone, so it could be neither read nor acted on and only brought overflow closer. Measured on one node, two writes and a delete produced seven events, four of them siblings. Earlier agents forward them, so on a mixed fleet expect the noise from the nodes that have not upgraded. Only the agent's own scratch names are filtered, and only where the marker ends the filename — a file of your own called report.nobgp-write-final.csv still reports.

Each platform's native watcher has its own habits, and they show through:

• macOS (FSEvents) reports paths with symlinks resolved — a watch on /tmp/x reports events under /private/tmp/x — so don't prefix-match against the path you subscribed. A create followed immediately by writes can also coalesce into the create alone, which means an ops: ["write"] filter may miss a file that was created and written in one go.
• Linux (inotify) reports the distinct operations, on the paths as you gave them.

A node that could not watch any of the paths you gave it reports a refused event rather than sitting there silently, and from agent 0.4.89 that refusal names each path and its own reason — missing, a permissions problem, or the node's 1024-directory watch cap — instead of one sentence covering all three. ⚠ A node that watched some of them runs with partial coverage and does not tell you which ones it dropped, so subscribe to a path you are unsure about on its own if you need to know it is being watched.

Dispatching a command across a fleet

command_subscribe runs a command once on each matching node, detached, and collects a report from every one of them. It returns immediately — nothing holds a session open.

Run the backup script on every node labelled role=db and tell me how each one went

{
  "select": { "labels": { "role": "db" } },
  "command": "/usr/local/bin/backup.sh",
  "timeout": 1800
}

The response lists exactly which nodes the work went to. Each node reports back:

{
  "node": { "id": "9f1c…", "name": "db-02" },
  "status": "completed",
  "at": "2026-07-29T10:41:22Z",
  "command": {
    "output": "backup finished: 4.2 GB\n",
    "duration_ms": 184320
  }
}

The output is text, from router 0.4.201 — the form the command tool already answered in, where this surface used to send a lines array of {line, stderr} objects. output holds every line in the order the command wrote it, stdout and stderr together; stderr holds the stderr lines again, so a reader that needs them apart still has them, and it is absent when the command wrote none. From agent 0.4.180 each line is the command's exact bytes, its terminator included, so a \r\n survives and a last line with no newline stays without one; an older agent sends each line with its terminator stripped and the router appends \n, so a program that wrote \r\n comes back with \n, and a last line with no newline comes back with one. A cohort can span both — read the agent version in network_directory if a byte-exact tail matters.

exit_code appears on the event that ends a run, and only when it is not 0: status: "completed" already says the run exited 0, and failed, timeout and cancelled carry their code. It is absent from the streaming output events in between, where there is no exit status yet. A run the node killed reports 128 + the signal number on Linux and macOS (143 for SIGTERM, 137 for SIGKILL) from agent 0.4.38, so a timed-out or cancelled run is distinguishable from one that chose its own exit code; Windows reports 1.

From router 0.4.84, on nodes running agent 0.4.86 or newer, the event also carries command.signal — the name of the signal that ended the run, never a number — so you no longer have to decode the exit code to find out. It is read from the same wait status the single-node tool reads, so the two surfaces cannot name one kill two different ways; a node on an older agent simply omits it, and a cohort can span both. A Windows node never reports one on a dispatch: it has no signals, and a dispatch is torn down with taskkill /T /F, which you never asked for by name — so there is nothing honest to attribute, and the field is absent rather than guessed from the exit code. It is the same field the single-node command tool reports, and it survives the case that needs it most: a run killed at its deadline reports cancelled with exit_code overwritten to -1, where the name is the whole answer.

event_tail also returns a progress block for a dispatch:

{ "dispatched": 12, "accepted": 11, "settled": 12, "outstanding": 0 }

• accepted below dispatched means some nodes never took the work — offline since, or refusing the source.
• outstanding: 0 means every node has finished. That is how you know the dispatch is done.

Every run is bounded (timeout, default 300, maximum 3600 — max_seconds until router 0.4.200) and the dispatch always ends. There is no repeating mode: a command that should keep running is a service, not a subscription. event_unsubscribe kills whatever is still running.

Sending a script to a whole fleet

From router 0.4.83, command_subscribe takes the same script field command does — a whole script body, real newlines, no shell quoting — plus shell, workdir and env. Provide command (one line) or script (a body), never both.

{
  "select": { "labels": { "role": "db" } },
  "script": "set -e\nsystemctl stop app\n/usr/local/bin/backup.sh --full\nsystemctl start app\n",
  "timeout": 1800
}

⚠ The body is lowered for each node separately, not once for the cohort — verbatim under the login shell on Unix, a temp batch script on Windows, and a temporary .ps1 under shell: "powershell". That is what lets one dispatch cross platforms at all, which is the thing this tool most often has to do: before it, a POSIX one-liner sent to a mixed fleet came back as 'printf' is not recognized from the one Windows node, buried among a dozen successes.

Three per-node refusals follow from it, and each arrives as that node's own refused event while the rest of the cohort runs: a node that has never reported its OS (nothing to lower the body for); a node that reports a non-Windows OS under shell: "powershell", from router 0.4.204, since the lowering is a powershell.exe command line; and a node whose lowered line exceeds its platform's command-line limit — the encodings inflate, so the same body can fit on Linux and not on Windows, and the Windows limit is cmd.exe's real 8,191 characters from router 0.4.204 rather than the 32,000 assumed before. Only a cohort where no node can be served fails the call outright. workdir and env are one setting for the whole cohort, so the directory has to exist on every node the selector matched. Their contents are no longer restricted, from router 0.4.144: a workdir may hold any character, an env value may hold any character, and a name only has to avoid an equals sign and a null byte. The old refusal of a double quote, a percent sign, a newline or a null byte described the way noBGP used to fold these into the command line; they now travel to each node as fields of their own, which nothing parses on the way.

From agent 0.4.85 each node applies those two itself, and a node that cannot refuses instead of running: a workdir that is missing or is not a directory there, or an env name that would inject code into the run (LD_PRELOAD, BASH_ENV, NODE_OPTIONS and the rest of that list), arrives as that node's own refused event with the reason while the cohort carries on. ⚠ A node below agent 0.4.85 now ignores both and reports success, from router 0.4.144 — that is the other half of the same change, since the fold that used to carry them to such an agent is what was retired. It only reaches a node that is not taking upgrades, because one that reconnects installs what its channel offers within about a minute. With no workdir at all the command runs in the agent's own working directory rather than a home directory, so write absolute paths.

Who the command runs as

Execution identity matches the command tool field for field: root (old name admin, which the REST surface still accepts; the MCP tools refuse it from router 0.4.195), which since router 0.4.201 is a JSON boolean — true or false — with the strings "true" and "false" still accepted. Omit it to let each node pick.

• root omitted — each node runs the work at the identity it offers. A node with a configured account runs it as that account (typically the installing account). A node whose unelevated identity would be the superuser — no account on a root-running agent, an account that no longer resolves, or user: root — runs it elevated. So one dispatch covers a cohort that mixes both shapes. The subscribe response says which nodes those were in subscription.root, a list of node ids — one value could not answer for a cohort whose nodes resolve differently. A node absent from it runs as the account its owner configured, or was refused before dispatch and ran nothing; its row in subscription.nodes[] carries the reason, and so does its refused event. It replaces the root_used map keyed by node id (old name admin_used), which earlier routers returned. A caller who may not elevate gets a refused event on each node where the work would run elevated, and the other nodes still run. The call fails with forbidden only when every node is refused. On a node whose owner set allow-root: false, an omitted root runs unelevated.
• root: false — each node's own configured user. A node whose unelevated identity would be the superuser reports a refused event for that node instead of running the work as root (unelevated never means root). Send "false" when the work must not run as root on any node.
• root: true — the node's ambient identity, and gated twice. From router 0.4.179 the router asks root on each node of the cohort, before anything is dispatched. If you do not hold root on one of the nodes, the whole call fails with forbidden, the refusal names those nodes, and nothing runs. So a dispatch is never partly elevated, and the gate is no longer an organization role. Past that, a node whose owner set allow-root: false reports refused as its own event rather than running the work unprivileged, and the rest of the cohort runs. The node's gate is on the resolved uid 0, not a spelling. ⚠ A node you hold no use of does not run: it reports refused as its own event, before any identity question, and the rest of the cohort runs. See Execution identity.

A cohort is rarely uniform: node_list (with view: "full") reports each node's user, allow_root and user_is_admin, and the same dispatch can run as a login on one node, a service account on the next, and be refused on a third. A Windows node refuses an unelevated dispatch where it captured no install account, because LocalSystem reports as uid 0 — and, on agents 0.4.44 and 0.4.45, on one with a second identity too, since that identity covered file operations and not execution. Agent 0.4.46 dispatches as the configured account there, so read this as a per-node answer the refused event reports rather than a rule about the platform.

To run as some third account, use su inside the command itself.

Same authority as command

A dispatched command runs through each node's shell with the same authority the command tool gives you. Scope the selector deliberately.

Progress from inside the command

A dispatched command sees $NOBGP_SUBSCRIPTION in its environment, so a long script can publish its own progress back into the same subscription you're reading — with nobgp notify, which reads that variable itself:

#!/bin/sh
nobgp notify phase=start
pg_dump mydb | gzip > /backups/db.sql.gz
nobgp notify phase=done bytes="$(stat -c %s /backups/db.sql.gz)"

key=value arguments become a typed JSON object; a plain string or a JSON document is sent as written. Publishing when nobody is listening exits 0 — a script under the bus shouldn't fail because the reader went away. Publishing nothing is different: no payload and no --status fails with exit 1, because it almost always means a pipe is missing its - (./thing | nobgp notify - reads stdin; ./thing | nobgp notify does not).

Your own client can do the same with event_publish — for example to drop a marker into the stream before starting a phase, or to write down what you concluded once one has ended.

From router 0.4.88 that second one works: a publish succeeds for as long as the subscription does, including after outstanding: 0. Through router 0.4.87 a marker published into a settled dispatch was dropped — the call reported delivered: false and the event was simply not in the next event_tail, even though the subscription was otherwise alive and readable.

On the node itself, nobgp events shows what that machine is publishing, which is the half you can't see from the subscriber side. It's the quickest way to confirm a source is actually reporting.

Searching a fleet

fs_grep_subscribe runs one RE2 regex search across every matching node and delivers the matches as events, instead of making you wait on one big response per machine. It is the async half of fs_grep — same walk, same skipping of binary files and noise directories (.git, node_modules, .venv, target, dist, …), aimed at a cohort.

Search /etc/nginx for "ssl_certificate" on everything labelled role=edge

{
  "select": { "labels": { "role": "edge" } },
  "pattern": "ssl_certificate",
  "paths": ["/etc/nginx"],
  "limit": 200
}

It returns immediately with the subscription's id. Each match arrives as its own event, and node is the node the match is on — so a fleet-wide search is attributable without a second call:

{
  "node": { "id": "9f1c…", "name": "edge-01" },
  "at": "2026-07-31T09:12:44Z",
  "payload": { "path": "/etc/nginx/sites-enabled/api", "line": 42, "text": "  ssl_certificate /etc/ssl/api.crt;" }
}

Each node then reports once more with a terminal status — completed with a summary of how many matches across how many files, or failed / refused with a reason. Like a command dispatch, the walk always ends: event_tail's progress block reaching outstanding: 0 means every node has finished.

Read that summary before concluding a machine is clean — for two separate reasons, and neither implies the other.

• A file that was there and was too big. An agent skips any file over its 16 MiB per-file cap without reading it, so a node holding a rotated log can report zero matches having never opened the file you were asking about. From agent 0.4.40 the summary says so — 0 match(es) across 0 file(s), 1 file(s) too large to search — and the synchronous fs_grep returns the same count as skipped_files.
• A path the node never looked at. From router 0.4.96 the summary also names paths that could not be read at all, and why — , 1 path(s) NOT SEARCHED: /opt/app (not_found) — counted as unreadable_paths on fs_grep. This is the one that bites a cohort: a path present on some machines and absent on others is the normal case across a fleet, and such a node used to report completed with no matches, indistinguishable from having searched. A node running an agent below 0.4.96 cannot report it, so its silence is still ambiguous.

An empty cohort search is evidence of nothing only on the nodes where both numbers are zero.

From router 0.4.207 you no longer have to read those two numbers out of a sentence: the completion event carries them as fields — skipped_files, unreadable_paths and unreadable[], the same names fs_grep returns — always, zero included, because a zero is the statement that the walk read everything it was pointed at. event_tail with until: "done" copies all three onto that node's row, so one call answers did every machine actually search? for the whole cohort. Details

limit is the matches per node (default 1000, hard cap 100 000; it was max_count until router 0.4.200), and paths must be absolute. Narrow them rather than searching / on a busy fleet — and drain the queue while the walk runs, since only the newest 256 events are held.

From router 0.4.81 the walk takes root, like fs_grep does. Omitted, a node with a configured account searches as that account — which is how a fleet-wide search comes back short and looks complete wherever the files that matter are root-owned. A node that offers only its elevated identity searches elevated. Send root: true to search elevated on every node. With root: true, your authority to elevate is asked on each node of the cohort (router 0.4.179+). If you do not hold root on one of the nodes, the whole call fails with forbidden and names those nodes, so that cohort is never partly elevated. With root omitted, a caller who may not elevate gets a refused event on each node that would search elevated, and the other nodes search unelevated. Either way, a node whose owner refused elevation locally reports that as its own event and the rest of the search carries on.

Reach for	When
fs_grep	One node, and you want its matches in one response
fs_grep_subscribe	More than one node · a tree big enough that you'd rather read matches as they land · you want to keep working while it runs

It reads, but it isn't marked read-only

The search itself only reads. The call stands up a subscription and starts work on every node in the cohort, which is state — so a client that auto-approves on capability hints is told so rather than waved through.

Waiting for a machine

presence_subscribe reports node lifecycle transitions. The router publishes these, so the nodes run nothing and pay nothing — it's the cheapest thing on the bus and the right way to wait for a machine to come back after a reboot. If the wait is longer than ten minutes — an overnight rebuild, a machine someone will power on tomorrow — set keep_seconds so the subscription is still there when it does.

Tell me when kitchen-pi comes back online

{
  "node": { "id": "9f1c…", "name": "kitchen-pi" },
  "at": "2026-07-29T10:02:00Z",
  "presence": { "transition": "registered", "node_name": "kitchen-pi", "version": "0.4.15", "platform": "raspbian" }
}

transition is one of:

• online — the node's connection came up
• ready — that same connection has held for 5 seconds (router 0.4.84+)
• offline — the node's connection went away
• registered — the node re-registered. This is where version and platform can change, so it's the transition to watch when tracking an agent rollout.

Wait for ready, not online, before you use a node

ready is not a debounced online — online keeps its exact meaning, the bridge came up, which is true again after every reconnect. During a provision it is true for a fraction of a second before the node re-registers and drops: measured, the fleet went registered → online → offline → registered → online in 1.5 seconds, and the doomed bridge behind that first online lived 188 ms.

ready is published only when a session has survived 5 seconds, so it's the transition that says the node will still be there when your next call lands. A session that dies inside the window emits no ready at all; its replacement arms its own. Keep watching online when the question is the bridge itself — a gap, a bounce, a flap count.

Subscribe before you cause the thing you're waiting for

There is no replay. A subscription's queue starts empty and holds only what arrives after it exists — event_tail's offset re-reads what that queue already holds, it doesn't reach back past the subscription. Reboot a machine and then subscribe, and the ready you were waiting for has already gone by: event_tail returns empty for as long as you're willing to wait, and an empty tail looks exactly like a node that hasn't come back yet, so the mistake reads as patience.

The order is always:

1. Subscribe — presence_subscribe, scoped to the node or cohort.
2. Do the disruptive thing — reboot, upgrade, provision, restart the agent.
3. event_tail with wait — that call is the wait.

If you also need the state before it changes, read node_list after subscribing rather than before, so nothing can fall between the two calls.

Don't wait with a timer

A guessed sleep, or node_list polled in a loop, is wrong in both directions — it reports a node up before its session has held, and it burns the whole guess when the machine was back in two seconds. node_list answers is it up now; presence answers tell me when that changes.

If you did subscribe too late, the directory's online_at (router 0.4.108+) is the fallback: it's re-stamped on every reconnect, so an online_at later than the moment you rebooted the node is the new session rather than the one you just ended. online: true on its own can't tell those apart.

Reading events

event_tail returns pending events oldest first:

{ "subscription": "…", "limit": 50, "wait": 30 }

Use wait (up to 100) when you're waiting on something specific rather than calling in a loop. An empty result is normal. From router 0.4.201 until: "done" is the shape for a dispatch: it waits until every node of a command_subscribe or fs_grep_subscribe cohort has settled, or until wait runs out, and returns the events and one row per node.

Reading no longer destroys what it returns (router 0.4.84). Each response carries next, and the next call resumes there on its own — so an ordinary loop behaves exactly as it always did. What changed is that a response you never received is recoverable: send its next back as cursor and the same events are delivered again. Before this the queue was drained on read, so an event a lost response was carrying was simply gone.

The newest 256 events are retained. A cursor older than that is served from the oldest event still held, with truncated: true and dropped saying how many were missed — never silence at a different position. A cursor past the end is invalid_args: it belongs to a different subscription.

⚠ Re-reading is not replay. The cursor reaches back only as far as the subscription's own creation — the queue starts empty, so nothing that happened before you subscribed is in it at any offset. See Subscribe before you cause the thing you're waiting for.

⚠ One subscription has one cursor, and every reader shares it. If two readers tail the same subscription — or one of your own calls succeeds and its response never reaches you — the position moves and your next cursor-less call starts later, with nothing flagged: truncated stays false because nothing aged out. Subtract the number of events a response carried from its next to see where that read started, compare it with the next you last held, and ask for the difference back by cursor while it is still retained. See Re-reading events for the two limits on that recovery.

Don't read at all — be told

Through a node's local MCP server, a subscription is an MCP resource. Subscribe to nobgp://events/<subscription_id> once and the server pushes notifications/resources/updated as events land, so waiting for a long command or a file change costs no polling and survives between turns. A read carries the progress block too from agent 0.4.121, so you can tell a finished dispatch from a quiet one without a second call. event_tail remains the fallback for clients that don't consume notifications — and it is the one to use on a node whose agent is below 0.4.121, where this path delivers nothing at all.

Each event may carry a status. Absent (or ok) means ordinary data with more to come; anything else is that node's last word — the other nodes on the subscription carry on:

status	Meaning
completed	The work finished cleanly — a command exited 0, or a node's search walk completed and reason summarises what it found
failed	The work finished badly — a non-zero exit, or a search the node could not complete
timeout	The command hit timeout and was killed
cancelled	You unsubscribed and the run was torn down
overflow	The node couldn't keep up and stopped — see reason
refused	The node won't run this source at all — see reason

Refusals and overflow

Both are answers, not errors, and both name a reason:

• refused — the node won't serve this source: the kind isn't in its allow-tools, the path is outside its allow-roots, the run asked for the superuser its allow-root: false vetoes, or the kind isn't supported on its platform. A caller who was not entitled to ask for elevation never gets this far — the subscribe call itself returns forbidden and nothing is dispatched. See What a node allows. matched_nodes tells you who the work was sent to, not who accepted it.
• overflow — changes (or output) outran what the node could report, so it stopped rather than delivering a silent fraction. The reason names which side overflowed — the node's own buffer or the kernel's queue. Narrow the watch paths or ops; for a chatty command, filter at the source (grep, tail -n) or redirect to a file and read it with fs_read.

A subscription holds only the newest 256 pending events. Treat events as coarse signals and investigate details with command or fs_read on the node that reported.

Staying subscribed while you are away

A subscription dies about ten idle minutes after the last event_tail, which is fine for a loop and wrong for the thing you most often want a watcher for: tell me if this changes while I am doing something else. From router 0.4.91, fs_subscribe and presence_subscribe take keep — the idle lifetime, in seconds, and keep_seconds until router 0.4.200.

{
  "select": { "nodes": ["kitchen-pi"] },
  "keep": 14400
}

Omit it for the default 600 (10 minutes); the maximum is 86400 (24 hours). Reading still resets the clock — this changes how long the subscription tolerates silence, not how it is kept alive.

• A value over the maximum is clamped rather than refused, and ttl on the subscribe response is the lifetime that was actually applied (ttl_seconds until router 0.4.200). That is the only place a clamp shows, so read it back rather than assuming you got what you asked for.
• A value below the default is raised to it. keep is an extension: sending 30 must not give you a subscription that dies sooner than sending nothing.
• The two dispatches don't take it. command_subscribe and fs_grep_subscribe are bound to work, not to idleness — they close when every node has settled — so there is no idle clock to extend and sending the field is invalid_args.

It keeps the watchers, not the events

The queue still holds the newest 256 events and drops the oldest. A client away for twenty hours comes back to what fits, not to what it missed — event_tail says how many were dropped, but it cannot hand back what the queue no longer holds. If you need every event, keep reading; keep_seconds is what stops the watchers being torn down while you are gone.

Cleaning up

List my subscriptions in production and cancel the file watcher

event_subscriptions lists the subscriptions you created in a network, with each one's id and kind (fs, command, grep, presence). It's the recovery path when an id is lost to a new session or a crashed script — finding the orphan and ending it beats waiting out the idle clock while its watchers keep running.

Limits

Limit	Value
Idle lifetime (refreshed by every event_tail)	10 minutes default · up to 24 hours with keep on fs_subscribe / presence_subscribe
Pending events held per subscription	newest 256
Watches per fs_subscribe	64
Watch path / command length	4096 characters
Command runtime	300s default, 3600s maximum
event_publish payload	8192 bytes

A network's active subscriptions also share a budget for their declarations, since those travel to each node in a single frame. An oversized or over-budget subscription is refused with invalid_args and tells you to narrow it or unsubscribe something.

Isolated networks

Isolated networks have no event bus — their members must not observe one another.

Next Steps

• MCP Reference - Full schemas for every event tool
• Local MCP Server - Push delivery, and driving a network from one machine's Claude
• Core Concepts - How subscriptions fit with sessions and services
• Publishing Services - For work that should keep running, rather than a bounded dispatch