Skip to content

Record a worker's heartbeat so the dashboard tells a busy worker from a stuck one - #492

Merged
el-bri-ga merged 3 commits into
mainfrom
fix/issue-482
Nov 20, 2026
Merged

el-bri-ga merged 3 commits into
mainfrom
fix/issue-482

Conversation

@el-bri-ga

Copy link
Copy Markdown
Contributor

Fixes #482.

Once a minute the supervisor's kbl-mcp records which command workers' processes are still running: new record worker_alive {workers} (format 26), written through POST /v1/work/workers/alive with the supervisor's own token. The fold keeps it as the worker's alive_ms. The dashboard reads that field only:

  • Workers dialog: process running under the state while the last beat is under 3 minutes old, no heartbeat for 7 min (warning colour) after, nothing for a worker with no beat (a hosted one).
  • A holder on a card, a list row and the item's Now line gets the same words next to last tracker write 6 min ago; silence turns the Now line to the warning tone.

Tests: core format 26 log (golden pinned), tracker unit and HTTP tests, supervisor heartbeat test, dashboard beat/workingLine/nowLine/fold tests, and a browser check for the dialog. Each new test was run with its fix removed and failed (supervisor beat, fold's own-worker check, card line). Dashboard TS tests that need the wasm build, and the screenshots (SHOT_MATRIX in the dashboard build job), run in CI only.

Review notes

  • Decided: a new record kind and format bump, because a heartbeat has to be on the log to be a recorded field. No Upgrade-Break: a fresh log starts below 26, the heartbeat is refused with format and the supervisor ignores that.
  • Decided: one note a minute per supervisor, naming up to 64 workers, all or none in the fold; the tracker drops names that stopped since the supervisor looked so a race does not refuse the note.
  • Decided: only command workers get a heartbeat. A hosted worker has no process, so it shows nothing rather than a false silence. The dashboard never infers liveness.
  • Decided: 'process exited' needs no new field: the supervisor stops a worker whose process ended in the same round, so it shows Stopped.
  • Decided: renamed 'last active' / 'last activity' to 'last tracker write' on cards and the Now line, since the old wording is what made a quiet step look dead.
  • Threat model: new route and record rows; the note is a supervisor's claim about its own workers and gates nothing.

@el-bri-ga
el-bri-ga enabled auto-merge (squash) November 20, 2026 06:03
@sonarqubecloud

Copy link
Copy Markdown

@el-bri-ga
el-bri-ga merged commit 8707794 into main Nov 20, 2026
10 checks passed
@el-bri-ga
el-bri-ga deleted the fix/issue-482 branch November 20, 2026 06:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dashboard can't tell a busy worker from a stuck one

1 participant