From 52d919e48be9cd9b2a240729491fd38836b409bd Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Fri, 23 Oct 2026 04:42:12 +0000 Subject: [PATCH 1/7] =?UTF-8?q?#67=20docs:=20tailnet=20flaps=20=E2=80=94?= =?UTF-8?q?=20the=20rule,=20the=202026-10-22=20signature,=20and=20how=20to?= =?UTF-8?q?=20tell=20it=20apart?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The operator's rule (2026-10-23) replaces "a second flap gets an issue": a flap gets an issue only if it delays a command, pages, or outlasts liveness's 900 s grace; others are logged on the epic. The 2026-10-22 flap was host-side: virtio_balloon "Out of puff" at 04:13:26-38Z, then tailscaled's rebind-reason=[time-jumped(33s),...]: the VM most likely paused about 33 s. The 2026-10-10 flap was path churn with neither. exe.dev's docs explain neither. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/DEPLOYMENT.md | 42 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index e931d31..37baf1f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -100,7 +100,7 @@ scripts/deploy.sh [] # ship: CI green, then build a release ## Detail Docs - [docs/specs/2026-09-29-processor-service-design.md](docs/specs/2026-09-29-processor-service-design.md) — the spec: decisions, grants, runtime, failure table, versioning, cutover, testing, contract quick reference -- [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — prerequisites, the broker credential handoff, `/etc/processor/`, releases and `scripts/deploy.sh`, the drift check, liveness, containment, install and rollback, operate, co-core bumps, TLS trust, Tailscale DNS, Node.js +- [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — prerequisites, the broker credential handoff, `/etc/processor/`, releases and `scripts/deploy.sh`, the drift check, liveness, containment, install and rollback, operate, co-core bumps, TLS trust, Tailscale DNS, tailnet flaps (signature, rule), Node.js - [docs/plans/](docs/plans/) — implementation plans - [docs/SKILLS.md](docs/SKILLS.md) — vendored agent skills: inventory, overrides (none), the refresh hook, worktrees - [docs/SOCRATICODE.md](docs/SOCRATICODE.md) — semantic search: tools, the shared store, link stubs, the pin, the memory decision diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 2e63c6b..bf9bc2b 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -548,6 +548,48 @@ The backup is gone afterwards (measured 2026-10-01), so there is nothing left to After this, `broker` stops resolving. Until DNS is back on, put the broker's address from `tailscale status` into `CO_PROCESSOR_BUS_URL` and restart the service. +## Tailnet flaps + +**The rule** (operator, 2026-10-23, #67): "A tailnet flap gets an issue only if it delays a command, pages, or outlasts liveness's 900 s grace. Any other flap is logged on the epic with its window and signature." + +A flap is the tailnet path to `broker` going away for minutes while tailscaled keeps running. The consumer backs off and retries, with no restart (§ Liveness). It has ridden out both flaps so far: + +| Window | What it was | Processor | +|---|---|---| +| 2026-10-10 02:26–04:12Z | Path churn between direct (peers at the shared NAT `16.145.19.221`) and DERP. No `LinkChange` and no clock jump | 19 WARNINGs, 14 `open-conn-track` timeouts, no command affected | +| 2026-10-22 04:13:38–04:15:30Z | A host-side event: memory-balloon pressure, then the VM most likely paused about 33 s | 6 WARNINGs, 1 failed check-in, no command delayed, nothing paged | + +**The 2026-10-22 signature,** in order: + +- kernel: `virtio_balloon virtio5: Out of puff! Can't get 1 pages` ×26, 04:13:26–38Z. The host was inflating the balloon, and the guest had no pages to give back. These were the only balloon messages on this boot (since 2026-10-03). +- `processor`: `consumer loop error; backing off`, `error` `TimeoutError: Timeout reading from broker:6379`, then `Timeout connecting to server` until the path is back (backoff 1, 2, 4 … s). +- The system journal: silent for about 68 s (04:13:38–04:14:46Z). +- tailscaled: + - `magicsock: endpoints changed`, with a second STUN address beside the usual one; + - `LinkChange: major, rebinding … rebind-reason=[time-jumped(33s),ips-changed]`. The wall clock jumped 33 s, which is what a paused VM looks like from inside; + - `closing connection to derp-10 (rebind-ping-fail)`, then a reconnect; + - `open-conn-track` timeouts. +- `processor`: possibly one `liveness check-in failed` (the 10 s bound). The next check-in lands inside the 900 s grace, so nothing pages. +- Recovery on its own: `NRestarts=0`, and `ack: complete` resumes. A command issued during the gap stays in the stream and is acked once the path is back. + +exe.dev's docs don't describe maintenance, host migration, ballooning or pauses. So the cause is a host-side event they don't explain. Whether other cohort VMs saw the same windows is recorded on #67. + +**Telling it apart:** + +- **A broker-side problem** (a Redis restart, the broker's patch window, broker#91): this VM's tailscaled logs no `LinkChange` or `time-jumped`. The broker's patch run (2026-10-13 04:17–04:22Z) left backoff WARNINGs at its Redis restart and at its reboot, and `open-conn-track` timeouts during the reboot. +- **A tailscaled restart here** (#65, 2026-10-22 23:40:29Z): tailscaled gets a new `MainPID` and logs `linkChange: in state NoState`. The tailnet is down for 2.3 s, and the broker socket survives with no WARNING. +- **A flap:** this VM's tailscaled logs `endpoints changed` and `LinkChange` (and, on 2026-10-22, `time-jumped`), or peers go stale or relayed in `tailscale status`. + +```bash +journalctl -u processor -o cat --since | jq -cR 'fromjson? | select(.level != "INFO") | {timestamp, logger, message, error}' +journalctl -u tailscaled -o short-iso --since | grep -E 'LinkChange|endpoints changed|rebind-reason|closing connection to derp|open-conn-track' +journalctl -k -o short-iso --since | grep -E 'Out of puff|balloon' +systemctl show processor -p NRestarts +tailscale status +``` + +Log a flap on the epic as: the window (first WARNING to the first `ack: complete` after it), the counts above, and which kind it was. + ## Node.js (agent tooling only) Node 24 LTS from NodeSource's apt repo, for `using-mayfly-chat` and SocratiCode (`npx`). The `processor` unit never runs it. The cohort survey and the reasons for NodeSource and for 24 are in #6. From 71874b66edbe009bd6543c4bdd0da091dc2ccec2 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Fri, 23 Oct 2026 04:51:53 +0000 Subject: [PATCH 2/7] =?UTF-8?q?#67=20docs:=20the=20cohort=20saw=202026-10-?= =?UTF-8?q?22=20too=20=E2=80=94=20a=20host=20pause=20behind=20one=20egress?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mayfly questionnaire answers (synthesis on #67): co-broker and co-replicator, behind 32.185.164.108 like co-processor, each logged balloon pressure then a 31-33 s time jump; archiver, provisioner and watcher, on other egress, only lost the broker. Replaces "silent 68 s" (true only excluding tailscaled and Processor) with the 32.7 s hole. Co-Authored-By: Claude Opus 5.5 --- docs/DEPLOYMENT.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index bf9bc2b..72b876e 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -557,13 +557,13 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k | Window | What it was | Processor | |---|---|---| | 2026-10-10 02:26–04:12Z | Path churn between direct (peers at the shared NAT `16.145.19.221`) and DERP. No `LinkChange` and no clock jump | 19 WARNINGs, 14 `open-conn-track` timeouts, no command affected | -| 2026-10-22 04:13:38–04:15:30Z | A host-side event: memory-balloon pressure, then the VM most likely paused about 33 s | 6 WARNINGs, 1 failed check-in, no command delayed, nothing paged | +| 2026-10-22 04:13:38–04:15:30Z | A host-side event on the VMs behind egress `32.185.164.108` (broker, processor, replicator): memory-balloon pressure, then each VM paused about 31–33 s | 6 WARNINGs, 1 failed check-in, no command delayed, nothing paged | **The 2026-10-22 signature,** in order: +- The journal: no entry at all 04:12:53–04:13:26Z (32.7 s, the length of the jump). An idle VM can be that quiet, so this suggests the pause rather than proving it. - kernel: `virtio_balloon virtio5: Out of puff! Can't get 1 pages` ×26, 04:13:26–38Z. The host was inflating the balloon, and the guest had no pages to give back. These were the only balloon messages on this boot (since 2026-10-03). - `processor`: `consumer loop error; backing off`, `error` `TimeoutError: Timeout reading from broker:6379`, then `Timeout connecting to server` until the path is back (backoff 1, 2, 4 … s). -- The system journal: silent for about 68 s (04:13:38–04:14:46Z). - tailscaled: - `magicsock: endpoints changed`, with a second STUN address beside the usual one; - `LinkChange: major, rebinding … rebind-reason=[time-jumped(33s),ips-changed]`. The wall clock jumped 33 s, which is what a paused VM looks like from inside; @@ -572,11 +572,17 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k - `processor`: possibly one `liveness check-in failed` (the 10 s bound). The next check-in lands inside the 900 s grace, so nothing pages. - Recovery on its own: `NRestarts=0`, and `ack: complete` resumes. A command issued during the gap stays in the stream and is acked once the path is back. -exe.dev's docs don't describe maintenance, host migration, ballooning or pauses. So the cause is a host-side event they don't explain. Whether other cohort VMs saw the same windows is recorded on #67. +**The cohort saw the same event** (a Mayfly questionnaire, 2026-10-23; synthesis on #67, the broker's timeline on CannObserv/broker#118): +- co-broker and co-replicator, behind the same egress, each logged the first balloon pressure of their boot (04:13:07Z and 04:13:39Z), then a 31–33 s time jump; +- co-registrar (archiver), co-provisioner and co-watcher, each on its own egress, logged neither; they only lost `broker` for about 2 min. + +The broker paused too, so every bus client timed out at once. Whether the pauses were simultaneous or staggered isn't settled. + +exe.dev's docs don't describe maintenance, host migration, ballooning or pauses. The cause is a host-side memory reclaim they don't explain. Asking exe.dev is the operator's step (CannObserv/broker#118). **Telling it apart:** -- **A broker-side problem** (a Redis restart, the broker's patch window, broker#91): this VM's tailscaled logs no `LinkChange` or `time-jumped`. The broker's patch run (2026-10-13 04:17–04:22Z) left backoff WARNINGs at its Redis restart and at its reboot, and `open-conn-track` timeouts during the reboot. +- **A broker-side problem** (a Redis restart, the broker's patch window, broker#91, or a host pause of the broker alone, which is how archiver, provisioner and watcher saw 2026-10-22): this VM's tailscaled logs no `LinkChange` or `time-jumped`. The broker's patch run (2026-10-13 04:17–04:22Z) left backoff WARNINGs at its Redis restart and at its reboot, and `open-conn-track` timeouts during the reboot. - **A tailscaled restart here** (#65, 2026-10-22 23:40:29Z): tailscaled gets a new `MainPID` and logs `linkChange: in state NoState`. The tailnet is down for 2.3 s, and the broker socket survives with no WARNING. - **A flap:** this VM's tailscaled logs `endpoints changed` and `LinkChange` (and, on 2026-10-22, `time-jumped`), or peers go stale or relayed in `tailscale status`. From 8e868e13271c5e802e49b103f5b62f3043fefe94 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Fri, 23 Oct 2026 04:52:41 +0000 Subject: [PATCH 3/7] =?UTF-8?q?#67=20docs:=20notifier=20paused=20too,=2019?= =?UTF-8?q?=20min=20later=20=E2=80=94=20the=20events=20were=20staggered=20?= =?UTF-8?q?per=20VM?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/DEPLOYMENT.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 72b876e..84fd95c 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -557,7 +557,7 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k | Window | What it was | Processor | |---|---|---| | 2026-10-10 02:26–04:12Z | Path churn between direct (peers at the shared NAT `16.145.19.221`) and DERP. No `LinkChange` and no clock jump | 19 WARNINGs, 14 `open-conn-track` timeouts, no command affected | -| 2026-10-22 04:13:38–04:15:30Z | A host-side event on the VMs behind egress `32.185.164.108` (broker, processor, replicator): memory-balloon pressure, then each VM paused about 31–33 s | 6 WARNINGs, 1 failed check-in, no command delayed, nothing paged | +| 2026-10-22 04:13:38–04:15:30Z | A host-side event on the VMs behind egress `32.185.164.108` (broker, processor, replicator, notifier): memory-balloon pressure, then each VM paused 24–33 s, staggered over about 40 min | 6 WARNINGs, 1 failed check-in, no command delayed, nothing paged | **The 2026-10-22 signature,** in order: @@ -574,9 +574,10 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k **The cohort saw the same event** (a Mayfly questionnaire, 2026-10-23; synthesis on #67, the broker's timeline on CannObserv/broker#118): - co-broker and co-replicator, behind the same egress, each logged the first balloon pressure of their boot (04:13:07Z and 04:13:39Z), then a 31–33 s time jump; +- notifier, behind it too, had the same 19 min later: balloon pressure at 04:32:48Z, then a 24 s pause logged at 04:51:10Z; - co-registrar (archiver), co-provisioner and co-watcher, each on its own egress, logged neither; they only lost `broker` for about 2 min. -The broker paused too, so every bus client timed out at once. Whether the pauses were simultaneous or staggered isn't settled. +The broker paused too, so every bus client timed out at once. The events were staggered per VM, not one simultaneous pause. exe.dev's docs don't describe maintenance, host migration, ballooning or pauses. The cause is a host-side memory reclaim they don't explain. Asking exe.dev is the operator's step (CannObserv/broker#118). From 6e3ffb174cdaba8086c41e402d5c8860d97c6f18 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Fri, 23 Oct 2026 04:53:32 +0000 Subject: [PATCH 4/7] #67 docs: the bus outage matches the broker's balloon reclaim, not its later pause Co-Authored-By: Claude Opus 5.5 --- docs/DEPLOYMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 84fd95c..5668d78 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -577,7 +577,7 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k - notifier, behind it too, had the same 19 min later: balloon pressure at 04:32:48Z, then a 24 s pause logged at 04:51:10Z; - co-registrar (archiver), co-provisioner and co-watcher, each on its own egress, logged neither; they only lost `broker` for about 2 min. -The broker paused too, so every bus client timed out at once. The events were staggered per VM, not one simultaneous pause. +The events were staggered per VM, not one simultaneous pause. Every bus client lost `broker` at once, from about 04:13:18 to 04:15:33Z. That window matches the broker's balloon reclaim, not its own pause, which its clock places about 04:16:14–43Z (CannObserv/broker#118). exe.dev's docs don't describe maintenance, host migration, ballooning or pauses. The cause is a host-side memory reclaim they don't explain. Asking exe.dev is the operator's step (CannObserv/broker#118). From 44a0a767b4e0d76ac6e2825b60ecfa07dbe29c38 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Fri, 23 Oct 2026 04:55:03 +0000 Subject: [PATCH 5/7] #67 docs: co-status had the event too; the balloon reaches earlyoom on small VMs Co-Authored-By: Claude Opus 5.5 --- docs/DEPLOYMENT.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 5668d78..fc184d8 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -557,7 +557,7 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k | Window | What it was | Processor | |---|---|---| | 2026-10-10 02:26–04:12Z | Path churn between direct (peers at the shared NAT `16.145.19.221`) and DERP. No `LinkChange` and no clock jump | 19 WARNINGs, 14 `open-conn-track` timeouts, no command affected | -| 2026-10-22 04:13:38–04:15:30Z | A host-side event on the VMs behind egress `32.185.164.108` (broker, processor, replicator, notifier): memory-balloon pressure, then each VM paused 24–33 s, staggered over about 40 min | 6 WARNINGs, 1 failed check-in, no command delayed, nothing paged | +| 2026-10-22 04:13:38–04:15:30Z | A host-side event on the VMs behind egress `32.185.164.108` (broker, processor, replicator, status, notifier): memory-balloon pressure, then each VM paused, staggered over about 40 min | 6 WARNINGs, 1 failed check-in, no command delayed, nothing paged | **The 2026-10-22 signature,** in order: @@ -574,7 +574,9 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k **The cohort saw the same event** (a Mayfly questionnaire, 2026-10-23; synthesis on #67, the broker's timeline on CannObserv/broker#118): - co-broker and co-replicator, behind the same egress, each logged the first balloon pressure of their boot (04:13:07Z and 04:13:39Z), then a 31–33 s time jump; -- notifier, behind it too, had the same 19 min later: balloon pressure at 04:32:48Z, then a 24 s pause logged at 04:51:10Z; +- co-status (moved onto that egress during the event) had balloon pressure at 04:14:04Z and a pause at about 04:17:07–24Z; +- notifier, behind it too, had the same 19 min later: balloon pressure at 04:32:48Z, then a pause logged at 04:51:10Z; +- on two small VMs without swap (status, notifier), the balloon pushed memory to earlyoom's floor, and earlyoom killed an agent-tooling process; - co-registrar (archiver), co-provisioner and co-watcher, each on its own egress, logged neither; they only lost `broker` for about 2 min. The events were staggered per VM, not one simultaneous pause. Every bus client lost `broker` at once, from about 04:13:18 to 04:15:33Z. That window matches the broker's balloon reclaim, not its own pause, which its clock places about 04:16:14–43Z (CannObserv/broker#118). From ed5148bed59e11d59615a6ed9604a61ae6c82b1c Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Fri, 23 Oct 2026 04:55:20 +0000 Subject: [PATCH 6/7] #67 docs: earlyoom's victims were user processes, not services Co-Authored-By: Claude Opus 5.5 --- docs/DEPLOYMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index fc184d8..8281195 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -576,7 +576,7 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k - co-broker and co-replicator, behind the same egress, each logged the first balloon pressure of their boot (04:13:07Z and 04:13:39Z), then a 31–33 s time jump; - co-status (moved onto that egress during the event) had balloon pressure at 04:14:04Z and a pause at about 04:17:07–24Z; - notifier, behind it too, had the same 19 min later: balloon pressure at 04:32:48Z, then a pause logged at 04:51:10Z; -- on two small VMs without swap (status, notifier), the balloon pushed memory to earlyoom's floor, and earlyoom killed an agent-tooling process; +- on two small VMs without swap (status, notifier), the balloon pushed memory to earlyoom's floor, and earlyoom killed a user process (not a service); - co-registrar (archiver), co-provisioner and co-watcher, each on its own egress, logged neither; they only lost `broker` for about 2 min. The events were staggered per VM, not one simultaneous pause. Every bus client lost `broker` at once, from about 04:13:18 to 04:15:33Z. That window matches the broker's balloon reclaim, not its own pause, which its clock places about 04:16:14–43Z (CannObserv/broker#118). From a23970b9b3ccedf218562ee92b7ab32d321c11f1 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Fri, 23 Oct 2026 12:39:15 +0000 Subject: [PATCH 7/7] #67 docs: co-index, behind the same egress, didn't pause on 10-22 Co-Authored-By: Claude Opus 5.5 --- docs/DEPLOYMENT.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 8281195..6574228 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -557,7 +557,7 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k | Window | What it was | Processor | |---|---|---| | 2026-10-10 02:26–04:12Z | Path churn between direct (peers at the shared NAT `16.145.19.221`) and DERP. No `LinkChange` and no clock jump | 19 WARNINGs, 14 `open-conn-track` timeouts, no command affected | -| 2026-10-22 04:13:38–04:15:30Z | A host-side event on the VMs behind egress `32.185.164.108` (broker, processor, replicator, status, notifier): memory-balloon pressure, then each VM paused, staggered over about 40 min | 6 WARNINGs, 1 failed check-in, no command delayed, nothing paged | +| 2026-10-22 04:13:38–04:15:30Z | A host-side event on five of the six VMs behind egress `32.185.164.108` (broker, processor, replicator, status, notifier): memory-balloon pressure, then each VM paused, staggered over about 40 min | 6 WARNINGs, 1 failed check-in, no command delayed, nothing paged | **The 2026-10-22 signature,** in order: @@ -577,6 +577,7 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k - co-status (moved onto that egress during the event) had balloon pressure at 04:14:04Z and a pause at about 04:17:07–24Z; - notifier, behind it too, had the same 19 min later: balloon pressure at 04:32:48Z, then a pause logged at 04:51:10Z; - on two small VMs without swap (status, notifier), the balloon pushed memory to earlyoom's floor, and earlyoom killed a user process (not a service); +- co-index, behind the same egress, logged neither. It has room to spare, and `Out of puff` appears only when the guest *can't* give pages back, so a quiet balloon there isn't ruled out. It did pause on its own on 2026-10-18 around 04:06Z; co-processor didn't; - co-registrar (archiver), co-provisioner and co-watcher, each on its own egress, logged neither; they only lost `broker` for about 2 min. The events were staggered per VM, not one simultaneous pause. Every bus client lost `broker` at once, from about 04:13:18 to 04:15:33Z. That window matches the broker's balloon reclaim, not its own pause, which its clock places about 04:16:14–43Z (CannObserv/broker#118).