https://live.setkernel.com: what setkernel's four AI engineers (Vega, Rigel, Lyra and Orion) are doing, about a minute behind. The page is "Day" (setkernel-live#48; the design is in docs/design/v3/day/, chosen in setkernel/hq#177): one sentence on top, and one chart under it.
-
The sentence says who is doing what right now, naming every engineer, with the "Latest" line under it. The masthead says "About a minute behind · updated 6 s ago" (events leave the office 60 s late), or only how long ago when the office has gone quiet.
-
The chart is the last 24 hours, one lane per engineer, time running left to right to now: a quiet activity band (Claude requests and chat posts per ten minutes) and the stage line through it (Building, In review heavy, Fixing red and dashed), with two marks, ● merged and ✕ changes asked for. Polaris, the director, is one row of merge ticks, with its three Claude sessions (Kochab, Pherkad and Yildun, the guardians) folded into a count. CI is one shared row: a quiet tick for a pass, a red bar for a failure. The key lists only what the day draws.
-
The Now column at the right edge says what each lane is doing and its own stage since when. Scrubbing (hover, click, or the arrow keys on the focused chart; Shift for an hour,
[]for the previous or next action, Esc for now) turns it into "what each lane was doing at 10:20", and a polite live region reads it out. -
"The day as a table" lists every action and CI run, newest first, for screen readers and for anyone who wants the detail.
-
Today is the Halifax day: the counts reset at midnight in Halifax, and the chart marks it as "Today" on the visitor's clock. Times are in the visitor's own zone.
-
An engineer whose VM stopped reporting (sk-obs01 sends
SILENT, setkernel-live#54) leads the sentence ("Orion stopped reporting at 14:05."), reads "Not reporting" in red in the Now column, and their lane is hatched red from the last run. When the VM comes back, the hatch goes. -
?demo: a sample day (labelled as such), so the page can be shown any time.?demo=quietis a quiet day,?demo=offlinethe office gone quiet 47 minutes ago,?demo=downthe busy day with Orion's VM stopped 2 h 35 min ago,?demo=waitingnothing yet. -
?wall: the same view on a fixed 1440×810 canvas, scaled whole to any screen, in one fixed dark theme. It fills a 16:9 screen edge to edge and never scrolls or clips on any other. -
Phones: the Now list on top, then the lanes running down the page, newest first.
| 1440×900 | 375×812 | ?wall, 1920×1080 |
|---|---|---|
![]() |
![]() |
![]() |
The screenshots are the built page on ?demo, in America/Halifax time
(docs/design/v3/day/built/).
sk-obs01 (office) Cloudflare visitors
wall service ── filtered events ──▶ /ingest (Access: service token only) ──▶ Durable Object ──▶ /ws
every 5 s, holds the picture,
60 s late fans out batches
- The office never accepts connections from the internet. sk-obs01 pushes; the page only reads. If the pushes stop, the page says the office is offline.
- What may appear is an allowlist, checked twice: on sk-obs01 (setkernel/hq
observability/wall/public.py) and again here (src/logic.js). Agents by first name, the kind of work, a dot per Claude request and per team-chat post, a short list of GitHub actions, CI results (pass or fail; a private repo's run is just "a private project"), and a repo's name only when the repo is public. No code, conversations, titles, numbers, usernames, hostnames, branches, commit messages, run links or customer data. /ingestsits behind a Cloudflare Access app that admits only thelive-ingestservice token (setkernel/setkernel-cloudflare, access stack). The Worker checks the Access token again.
The Deploy workflow (.github/workflows/deploy.yml) runs on every push to main, docs-only ones
too, so a merge deploys. It follows the org's deploy contract (setkernel/hq
templates/repo/.github/workflows/deploy.yml, checked by hq tools/check-deploy-contract.py): jobs
check (npm run validate), deploy, smoke, rollback, smoke-again and report. Only
deploy and rollback can get the deploy token; the checks and the smoke run without it. Runs
never overlap (concurrency: deploy-setkernel-live); a waiting run is replaced by a newer one. To
redeploy main without a change, start the workflow by hand (workflow_dispatch). Never deploy from
a laptop. Each deploy is one Worker version with the message deploy <commit>, carrying the two
Worker secrets from Infisical. /api/state carries version, the VERSION var from
wrangler.jsonc: dev unless the deploy sets the commit with wrangler deploy --var VERSION:<commit>.
node scripts/smoke.mjs --expect <commit> first polls /api/state until its version is that
commit. The field comes from the Durable Object's own code, and for a few minutes after a deploy
the object can still run the previous version, so the check waits up to 10 minutes. Then it checks
each route once: / 200, /api/state 200 with every key the page uses, /api/replay?h=1 200,
GET /github/webhook 405 (POST only), GET /github/queue 401 (Access answers first) and
GET /usage/data refused (401, 403, 503 or a 302 whose Location host is exactly
setkernel.cloudflareaccess.com, Access's login; a 200 or a 302 anywhere else fails it). It prints
status codes, that host, key names and the version, never a response body, and exits 1 on failure (2 on bad
arguments).
Before it deploys, deploy records the version serving all traffic and the commit its
deploy <commit> message names. If the smoke fails, rollback runs wrangler rollback to that
version (message "rollback from "), and smoke-again runs the smoke with
--expect that commit. Any failed job, the checks included, opens one issue titled "Deploy failed
for ", labelled agent-ready, with each job's result and the run link; the smoke
output is in the run's log. It also gets owner when the rollback failed, no version was live, or
the version rolled back to failed its smoke too (or names no commit, as a version from before
setkernel-live#83 does): then nothing rolls back again. main is unchanged either way: fix the
change and merge it again.
Read the output from the top:
FAIL the Durable Object still serves version X: the new code never reached the object in 10 minutes.version unknownmeans/api/statenever answered with a version at all; thewaiting:lines show its status (404 or 5xx is old code or a crash).version devmeans the deploy didn't setVERSION.FAIL GET /api/state: 200, version X, want Y: the version changed after the wait had seen the right one. Either the Durable Object is still switching between old and new code, or someone rolled back or deployed by hand. The Deployments list shows which. A later Deploy run can't overlap this one (concurrency: deploy-setkernel-live).FAIL GET /api/state: 200, missing keys: …: the page's data changed shape;view()inpublic/lib/state.jsandSTATE_KEYSinscripts/smoke.mjsmust agree (the worker tests check).FAIL GET /usage/data: 200, want …: the route answered without a login, so its Access app is missing or doesn't cover/usage; the Worker's token check (USAGE_AUD) isn't the only gate it should have.302 to <host>, with a host that isn'tsetkernel.cloudflareaccess.com: something other than Access redirected it.FAIL GET /…: 404, want 200and the like: that route is broken in the new code.0 (TimeoutError)means no answer within 15 s.GET /github/queue: 403, want 401or a 302: Access no longer sits in front of the queue; check the "live queue" Access app (setkernel-cloudflare) before anything else.GET /usage/data: 200, want 401 or 403 or 503: the private usage route answered without a token. The rollback has already happened; find why before merging anything else.
The automatic rollback covers a failed smoke; for anything the smoke misses, roll back by hand.
Rollbacks use the deploy token, which lives in Infisical and never on a laptop, so use the
dashboard.
In the Cloudflare dashboard: Workers & Pages, setkernel-live, Deployments, pick the last good
version and "Rollback". The Durable Object again takes a few minutes to follow; check with
node scripts/smoke.mjs --expect <that version's commit>. Then revert the bad change in a PR.
Cloudflare won't roll back across a Durable Object migration: if the bad deploy added one, fix
forward.
Rolling back to a version from before the pole stars (setkernel-live#33): that code doesn't pad
the shorter replay rows saved since, so some counts in the minutes around the rollback become
null. The page and the hourly chart read them as 0, and they age out within 24 hours.
GitHub (org webhook) ── workflow_run, deployment_status, ──▶ /github/webhook ──▶ queue in the
pull_request, pull_request_review,
issues, issue_comment
Durable Object
sk-obs01 ◀── pulls ── /github/queue?since=<cursor> (Access: live-queue service token only) ◀──┘
The office still accepts no connections: GitHub calls the Worker, and sk-obs01 pulls from it.
-
POST /github/webhookis public, because GitHub calls it. The Worker caps the body at 512 KB, checksX-Hub-Signature-256againstGITHUB_WEBHOOK_SECRET, ignores repeatedX-GitHub-Deliveryids, and accepts only completedworkflow_run, finished (success,failure,error)deployment_status,pull_request(opened, andclosedwhen merged),pull_request_review(submitted, approved or changes requested),issues(opened) andissue_comment(created) events. Every other event is ignored (202 "Ignored"). -
Each run or deployment is one queued event; a re-run replaces it. Only the repo name, whether the repo is private (anything unclear counts as private), the workflow name, the conclusion and the time are kept. Never the payload, commit messages, branch names, URLs, actors or environment names. The queue keeps the latest 500 events.
-
A pull request event keeps
kind: "pr", the repo name,private, the action (opened_pr,merged,approved,requested_changes), the time and anactor. The actor is a name, never the login. For an open or a merge the actor is the pull request's author; for a review, the reviewer.- A pull request merged from the director's account (
merged_byisrealRaghavGupta) also queuesmerged_bywith the actor Polaris, right after the author'smerged, from the same delivery: "Polaris merged a pull request". A merge has no body, so it carries no marker: one a Claude makes from that account shows as Polaris too. When the director merges its own pull request (no marker), onlymerged_byis queued, so it isn't counted twice. vega-setkernel,rigel-setkernel,lyra-setkernelandorion-setkernelbecome Vega, Rigel, Lyra and Orion.- The director's account (
realRaghavGupta) is shared by its Claude sessions, so a hidden marker picks the star. On a pull request, the head branch:claude/…is Pherkad,raghav-dev/…Yildun,lead/…Kochab. Otherwise, and always for a review (whose head branch is the author's, not the reviewer's), the body's last line, when it is exactly a signature:— Claude Cloudis Pherkad,— raghav-devYildun,— Claude LeadKochab. Trailing blank lines and footers (🤖 Generated with [Claude Code](…),Co-Authored-By: …,_Automated: …_) are skipped first (setkernel/agent-plugins#17). With no marker it is Polaris. The branch and body are read only to find the marker; neither is kept. - Anyone else is dropped.
Never a title, number, body, branch name, URL or login. Each open, merge and review is one queued event. sk-obs01 (setkernel/hq#163) accepts exactly these eight actor names.
- A pull request merged from the director's account (
-
A request (setkernel-live#35) keeps
kind: "request", the repo name,private, the action (opened_issueorcommented), the time and anactor, mapped exactly as for pull requests: the agents by login; from the director's account, the body's signature line picks Kochab, Pherkad or Yildun, else Polaris (a comment's own author and body, never the issue's); anyone else is dropped. Never a title, body, number, label or login. A comment on a pull request is a request too. Each issue and each comment is one queued event. -
GET /github/queue?since=<cursor>returns{ events, cursor, more, gap }: up to 200 events after the cursor, oldest first, each with its owncursor. Send the returnedcursornext time.moremeans pull again now;gapmeans events after your cursor were dropped by the cap. It sits behind its own Access app (setkernel-cloudflare#54) and the Worker checks the token again againstQUEUE_AUD. UntilQUEUE_AUDis set it answers 503. -
This queue doesn't reach the public page by itself. What the page may show of it is decided on sk-obs01 and in
src/logic.js(setkernel/hq#123, setkernel-live#14). For requests,/ingestaccepts{ t: "msg", kind: "request", agent, action: "opened_issue" | "commented", repo?, ts }for any of the eight stars, andmerged_byas a GitHub action for Polaris only (setkernel/hq#164 sends them). A request is counted apart from work: it is in no star's PR, review or Claude tally, moves no card on the line and is not in the hourly chart. Yildun alone may also sendclaudemessages, from raghav-dev's own telemetry.
For a few minutes after a deploy, the new Worker can reach a Durable Object still running the old code, which doesn't know the newer routes and answers 404. The Worker retries those calls for about 7.5 s (inside GitHub's 10 s delivery timeout), and answers 503 if the old code is still there, so sk-obs01's puller tries again and GitHub's delivery log shows a server error. GitHub never retries a failed delivery on its own: after a deploy, check the webhook's recent deliveries (github.com/organizations/setkernel/settings/hooks) and redeliver any that failed.
- Generate a webhook secret (for example
openssl rand -hex 32) and store it in Infisical, projectsetkernel-live, environmentprod, asGITHUB_WEBHOOK_SECRET. Do this before the next deploy: the deploy fails while it is missing. - Run the Deploy workflow on
main. It sets the secret on the Worker; until then/github/webhookanswers 503. - Create the org webhook (github.com/organizations/setkernel/settings/hooks): payload URL
https://live.setkernel.com/github/webhook, content typeapplication/json, the same secret, SSL verification on, and "Let me select individual events" with only Workflow runs and Deployment statuses ticked. GitHub's ping should get200 pong. Once the pull request receiver (setkernel-live#33) is deployed, also tick Pull requests and Pull request reviews. Once the request receiver (setkernel-live#35) is deployed, also tick Issues and Issue comments. Before each deploy the receiver answers 202 "Ignored" to them. - After setkernel-cloudflare#54 is applied, put its audience tag in
QUEUE_AUDinwrangler.jsonc(a PR) and deploy again.
Private, and kept apart from the page (setkernel-live#66, setkernel/hq#721). A second Durable
Object, Usage, holds hourly buckets in SQLite (buckets, one row per hour, seat, model and
surface); the Hub never sees them, so /ws, /api/state and /api/replay can't return them.
Hours older than 400 days are deleted on each push.
POST /ingest/usagesits behind the same Access app and token as/ingest, and the Worker checksINGEST_AUDagain. Body (up to 256 KiB):{ "buckets": [{ "hour": "2026-10-14T15", "seat": "vega", "model": "claude-opus-5-5", "surface": "vm", "input": 0, "output": 0, "cache_read": 0, "cache_write": 0, "usd_micros": 0, "requests": 0 }] }.houris a UTC hour; seat is an employee insrc/hq-registry.json(below), polaris, codex or other; surface one of vm, desktop, cli, cloud, codex; the counts are whole numbers from 0. One bad bucket makes the whole push 400. A bucket pushed again replaces the stored one, so a pusher can resend an hour's running totals. The answer is{ stored, days }: what is now stored for the hours pushed, per Halifax day and seat.POST /ingest/limits(setkernel-live#88) takes each seat's newest 5-hour and weekly limit readings, behind the same Access app andINGEST_AUDcheck:{ "vega": { "five_hour": { "utilization": 0.84, "resets_at": "…", "observed_at": "…" }, "seven_day": { … }, "event": { "status": "rejected", "rate_limit_type": "five_hour", "resets_at": "…", "received_at": "…" } } }. The seats are the registry copy's employees and polaris (Codex has no Claude limits);utilizationis a fraction from 0; times are ISO 8601 with a zone (resets_atmay be null: unknown);statusis allowed, allowed_warning or rejected; any part may be null or left out, and changes nothing then. Other fields are dropped; a bad seat, value or time makes the whole push 400. The store (seat_limits, one row per seat and part) keeps a part only if it is as new as or newer than the one it has, and answers{ limits }: every seat's as now stored.- The employees are a copy of setkernel/hq's
registry/people.csv(setkernel-live#96): each employee who isn't removed, as the wall gives them a lane, handles only. A person added or removed there needs no code edit here, only a fresh copy:node scripts/copy-hq-registry.mjs --hq <hq checkout>, then commitsrc/hq-registry.json(--checkexits 1 when the committed copy differs). Never edit it by hand. GET /usage/data?month=YYYY-MMneeds a token for its own Access app, checked againstUSAGE_AUD(the "live usage" Access app, setkernel-cloudflare#119; its tag is inwrangler.jsonc). With no login Access answers 302 to its login page; ifUSAGE_AUDwere ever empty the Worker would answer 503 to everyone. It returns{ month, days: { "2026-10-14": { seat: sums } }, totals: { seat: sums } }withCache-Control: no-store, and for the current Halifax month, once any reading is stored,limits:{ seat: { five_hour, seven_day, event } }, each part null until it has a reading. A past month has none: limits are read live. Days are Halifax days (homeDay), so an hour's usage counts on the day it was in Halifax: 03:00 UTC is 23:00 the day before in winter.
The read is open behind Access: the app for live.setkernel.com/usage exists (setkernel-cloudflare#119).
Everything under /usage (the page Raghav reads it on, setkernel-live#71, and its files in
public/usage/) needs the same USAGE_AUD token as /usage/data, not only Access in front:
wrangler.jsonc runs the Worker first for /usage and /usage/* (without it the assets are served
before the Worker sees the request). /usage/data names the newest hour stored, in any month, in a
Usage-Newest-Hour header (absent while nothing is stored); the body keeps its shape.
public/usage/table.js turns a month into the page's totals, seats, rows and trends: fresh tokens
are input + output + cache writes, and a day or seat with no data is a dash, not zero. The seats are
the ones the data names. public/usage/limits.js reads each seat's 5-hour and weekly windows from a
limits object in the answer, with the wall's rules (alerts at 80% and 70%, a limit hit at 100% or on
a rejected request, a reading from before its window reset is stale). Until a reading is stored,
/usage/data sends no limits, so the page says it can't check them and shows no figure. public/usage/index.html,
usage.css (its own palette and the fonts in public/usage/fonts/) and page.js draw it
(setkernel-live#78, v5 in #86): ?month=YYYY-MM picks the month, the page also reads the two months
before it for the 30-day trend, the newest hour is the header's, and a 403, 503 or expired login shows a
message. docs/design/usage/shoot.mjs takes its shots, beside it, with the ones from before v5 in before/.
A Cron Trigger asks the Durable Object every 5 minutes when the last batch arrived. After more than 5 minutes without one, the Worker posts to Buzz #infra; when batches come back, it posts once more with how long they were gone. One alert per outage. A post Buzz doesn't take is tried again on the next check, 5 minutes later (so, rarely, a post that timed out after Buzz stored it appears twice). Buzz runs in the office, so during a full office outage the alert can't get through; the recovery message is then the first report, and says how long it lasted.
The check runs every 5 minutes, not every minute, to keep the Durable Object's request count down (setkernel/hq#243). What that costs:
-
The alert comes 5-10 minutes after the last update (with a check every minute it was 5-6).
-
The recovery comes up to 5 minutes after updates resume (it was 1). Its end time is the last update the check saw, and its duration can read up to 5 minutes too long.
-
A gap of up to about 10 minutes can pass with no alert and no recovery (it was about 6): if updates are back before the first check that finds them more than 5 minutes stale, neither is posted.
-
Mute for planned maintenance: set
ALERTS_MUTEDto"true"inwrangler.jsoncand deploy; set it back to"false"afterwards. While muted no alert is posted, and an outage that ends inside the mute is never reported. One still going when the mute ends is alerted then. The recovery for an alert posted before the mute still goes out. -
Identity: the Worker posts as its own Buzz member, "live.setkernel.com watcher". Its key is the Worker secret
BUZZ_PRIVATE_KEY, which the deploy workflow brings from Infisical (projectsetkernel-live, prod). It is never in this repo or in logs. The relay and the channel are plain vars inwrangler.jsonc.
The deploy fails until BUZZ_PRIVATE_KEY is in Infisical, so do this before the next deploy. It
is the same procedure as the Grafana alerts identity in setkernel/hq observability/README.md;
the private details (hosts, logins) are there, not in this public repo. Only the public key is
ever printed.
- On sk-obs01, make the key in a new private file with hq's
buzz/agent-identity.sh, withSK_BUZZ_CONFset to that file andSK_BUZZ_CHANNELto the #infra channel id fromwrangler.jsonc. It prints only the public key (hex). - Make that public key a relay member (
buzzctl add-member <hex>, hqbuzz/README.md). - With that file's variables loaded (
set -a; . <file>; set +a), name the identity (buzz users set-profile --name "live.setkernel.com watcher") and join #infra (buzz channels join --channel "$BUZZ_CHANNEL"). - Put the key in Infisical as
BUZZ_PRIVATE_KEY(projectsetkernel-live, environment prod, path/) by piping it, never through a clipboard or the web UI. From setkernel-cloudflare:ssh <obs host> "sed -n 's/^BUZZ_PRIVATE_KEY=//p' <file>" | scripts/infisical-put.py setkernel-live BUZZ_PRIVATE_KEY prod(it prints only the key name, the project and the HTTP status; see "Credentials for a repo" in setkernel/hqdocs/repo-standard.md). shred -u <file>, so Infisical holds the only copy.- Run the Deploy workflow. To check the whole path, stop sk-obs01's publisher for 11 minutes: an alert should arrive in #infra, and a recovery message within 5 minutes of restarting it.
To retire the identity: remove its relay member, delete the Infisical secret, and in one PR take
the cron out of wrangler.jsonc and BUZZ_PRIVATE_KEY out of the deploy workflow.
Tracking: setkernel/hq#105.


