From ac8e3e54f0cbc31e59fd41721dda9325469131ad Mon Sep 17 00:00:00 2001 From: Raghav Gupta Date: Mon, 9 Nov 2026 09:12:16 -0400 Subject: [PATCH 1/3] feat(meter): sk-meter moves here as workers/meter, deployed by deploy.yml's meter job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The billing guard's code (src, allowances, tests, baseline and offline drill) moves from the retiring control repository at c1e3253, plan v6 F3. Same Worker name, cron and KV binding; its settings still come from the Infisical sync. deploy.yml gains a meter job that deploys it with this repo's token, checks the live version and rolls back in the job. Refs setkernel/hq#1449 โ€” Claude Lead Co-Authored-By: Claude Opus 5.5 --- .github/workflows/deploy.yml | 70 +++++++- AGENTS.md | 11 +- package.json | 5 +- workers/meter/README.md | 104 +++++++++++ workers/meter/allowances.json | 119 +++++++++++++ workers/meter/src/flags.js | 49 +++++ workers/meter/src/index.js | 57 ++++++ workers/meter/src/meter.js | 176 ++++++++++++++++++ workers/meter/src/notify.js | 33 ++++ workers/meter/src/usage.js | 243 +++++++++++++++++++++++++ workers/meter/src/windows.js | 35 ++++ workers/meter/test/baseline.test.js | 167 +++++++++++++++++ workers/meter/test/fakes.js | 76 ++++++++ workers/meter/test/flags.test.js | 54 ++++++ workers/meter/test/meter.test.js | 202 +++++++++++++++++++++ workers/meter/test/notify.test.js | 69 ++++++++ workers/meter/test/retry.test.js | 108 +++++++++++ workers/meter/test/run.test.js | 266 ++++++++++++++++++++++++++++ workers/meter/tools/baseline.mjs | 78 ++++++++ workers/meter/tools/drill.mjs | 37 ++++ workers/meter/tools/schema.mjs | 97 ++++++++++ workers/meter/wrangler.toml | 33 ++++ 22 files changed, 2083 insertions(+), 6 deletions(-) create mode 100644 workers/meter/README.md create mode 100644 workers/meter/allowances.json create mode 100644 workers/meter/src/flags.js create mode 100644 workers/meter/src/index.js create mode 100644 workers/meter/src/meter.js create mode 100644 workers/meter/src/notify.js create mode 100644 workers/meter/src/usage.js create mode 100644 workers/meter/src/windows.js create mode 100644 workers/meter/test/baseline.test.js create mode 100644 workers/meter/test/fakes.js create mode 100644 workers/meter/test/flags.test.js create mode 100644 workers/meter/test/meter.test.js create mode 100644 workers/meter/test/notify.test.js create mode 100644 workers/meter/test/retry.test.js create mode 100644 workers/meter/test/run.test.js create mode 100644 workers/meter/tools/baseline.mjs create mode 100644 workers/meter/tools/drill.mjs create mode 100644 workers/meter/tools/schema.mjs create mode 100644 workers/meter/wrangler.toml diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index f3b2f04..81ab650 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -7,7 +7,7 @@ name: Deploy # smokes that, and a failed run opens an agent-ready issue with the run's URL. # - One concurrency group, never cancelled: a newer push replaces the pending run, so the newest main # always deploys and no deploy stops half way. -# - Only `deploy` and `rollback` can get the deploy token (id-token: write), and they run no app code: +# - Only `deploy`, `rollback` and `meter` can get the deploy token (id-token: write), and they run no app code: # install scripts are off and wrangler runs without the OIDC request variables. The checks and the # smoke run in jobs that have no token; only `report` can write issues. # - Rollback: `deploy` records the live version, and the commit its "deploy " message names, before @@ -22,6 +22,10 @@ name: Deploy # the two Worker secrets from Infisical into the same version, so each deploy is one version. # - The smoke may wait 10 minutes for the Durable Object to run the new code, so its jobs get 15, and # scripts/smoke.mjs has no dependencies, so they install nothing. +# - The `meter` job deploys the second Worker, sk-meter (workers/meter, setkernel/hq#1449), with the same +# token after the same `check`. It has no route, so it checks the live deployment itself and rolls back +# in the same job; like `deploy` and `rollback` it installs no scripts and runs nothing but wrangler. +# It doesn't wait for the page's deploy, and the page's smoke doesn't wait for it. on: push: branches: [main] @@ -202,11 +206,70 @@ jobs: fi node scripts/smoke.mjs --expect "$EXPECT" + # sk-meter (workers/meter/README.md, "Deploy"): one hourly cron and no route, so nothing can be asked over + # HTTP. The job records the version serving all traffic, deploys, and passes only if this run's version + # ("deploy ") now serves all of it. Otherwise it rolls back to the recorded version and fails. + meter: + needs: check + runs-on: setkernel-linux + timeout-minutes: 10 + permissions: + contents: read + id-token: write # OIDC login to Infisical + env: + WRANGLER_SEND_METRICS: "false" + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - run: npm ci --ignore-scripts --no-audit --no-fund + - name: Fetch the deploy token from Infisical + uses: Infisical/secrets-action@d2e351f16c6ca20d17c85e6c992e04bdeb64e87d # v1.0.18 + with: + method: oidc + identity-id: 37b06681-c931-4bad-b3cf-4e76bfb633dc # setkernel-live-deploy + oidc-audience: https://github.com/setkernel + domain: https://secrets-api.setkernel.net + project-slug: ${{ github.event.repository.name }} + env-slug: prod + secret-path: / + export-type: env + extra-headers: | + CF-Access-Client-Id: ${{ secrets.CF_ACCESS_CLIENT_ID }} + CF-Access-Client-Secret: ${{ secrets.CF_ACCESS_CLIENT_SECRET }} + # Only CLOUDFLARE_API_TOKEN reaches wrangler: env -i drops the OIDC request variables and the + # page's two Worker secrets, which sk-meter never gets. Its settings come from the Infisical sync. + - name: Deploy sk-meter, check it, roll back if it isn't live + run: | + wrangler() { + env -i PATH="$PATH" HOME="$HOME" CI=true WRANGLER_SEND_METRICS=false \ + CLOUDFLARE_API_TOKEN="$CLOUDFLARE_API_TOKEN" \ + npx --no-install wrangler "$@" --config workers/meter/wrangler.toml + } + live() { wrangler deployments status --json | jq -r '[.versions[]? | select(.percentage == 100) | .version_id][0] // empty'; } + before=$(live) || before= + wrangler deploy --message "deploy ${GITHUB_SHA}" + now=$(live) || now= + message=$(wrangler versions view "$now" --json | jq -r '.annotations["workers/message"] // empty') || message= + if [ "$message" = "deploy ${GITHUB_SHA}" ]; then + echo "sk-meter serves ${now} (deploy ${GITHUB_SHA}) to all traffic" + exit 0 + fi + echo "::error::sk-meter's live version ${now:-(none)} is '${message}', not 'deploy ${GITHUB_SHA}'" + if [ -z "$before" ]; then + echo "::error::no version was live before, so there is nothing to roll back to" + exit 1 + fi + wrangler rollback "$before" --message "rollback from ${GITHUB_SHA::7}" --yes + [ "$(live)" = "$before" ] && echo "::error::rolled back to ${before}, which serves all traffic again" || + echo "::error::rolled back to ${before}, but it doesn't serve all traffic" + exit 1 + # Any failure above, the checks included, opens one issue for the agents, and for the owner too when # a failed smoke could not be rolled back or failed again after it. The runner image has curl and jq # but no gh. report: - needs: [check, deploy, smoke, rollback, smoke-again] + needs: [check, deploy, smoke, rollback, smoke-again, meter] if: failure() runs-on: setkernel-linux timeout-minutes: 5 @@ -222,6 +285,7 @@ jobs: SMOKE: ${{ needs.smoke.result }} ROLLBACK: ${{ needs.rollback.result }} SMOKE_AGAIN: ${{ needs.smoke-again.result }} + METER: ${{ needs.meter.result }} VERSION: ${{ needs.deploy.outputs.previous-version }} PREVIOUS: ${{ needs.deploy.outputs.previous-sha }} run: | @@ -242,7 +306,7 @@ jobs: *) rolled="Nothing was rolled back: no smoke failed after a deploy." ;; esac - body="The deploy of ${GITHUB_SHA} failed. Jobs: check ${CHECK}, deploy ${DEPLOY}, smoke ${SMOKE}, rollback ${ROLLBACK}, smoke-again ${SMOKE_AGAIN}. + body="The deploy of ${GITHUB_SHA} failed. Jobs: check ${CHECK}, deploy ${DEPLOY}, smoke ${SMOKE}, rollback ${ROLLBACK}, smoke-again ${SMOKE_AGAIN}, meter ${METER} (sk-meter; its log says whether it rolled back). ${rolled} diff --git a/AGENTS.md b/AGENTS.md index acb897a..fe6edf2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,6 +8,8 @@ secrets, branches and commits, the end-of-run report) come from setkernel's cent ## What this repo is The public page at https://live.setkernel.com: a Cloudflare Worker with static assets and one Durable Object. sk-obs01 pushes filtered, delayed events to it; visitors watch over a WebSocket. +It also holds `workers/meter`, sk-meter, the Cloudflare billing guard (an hourly cron, no route; +`workers/meter/README.md`), moved here on 2026-11-09 (setkernel/hq#1449). Public repo: everything here is visible to anyone. ## Commands @@ -21,8 +23,10 @@ Public repo: everything here is visible to anyone. # its poster (the artifact, the pull request, the shots branch, the comment), # and repo names against GitHub's public list, live) # worker: npm run test:worker (vitest in workerd: the real Worker and Hub, see test/worker/) +# meter: npm run test:meter (node --test: sk-meter's guard, usage queries, alert, baseline) # lint: npm run check (node --check on every script) -# validate: npm run validate (check + test + test:worker + wrangler deploy --dry-run; what CI runs) +# validate: npm run validate (check + test + test:worker + test:meter + a wrangler deploy --dry-run +# of each Worker; what CI runs) ``` ## Deploy @@ -37,6 +41,8 @@ reads `CLOUDFLARE_API_TOKEN`, `GITHUB_WEBHOOK_SECRET` and `BUZZ_PRIVATE_KEY` fro workflow on `main`, in the `deploy` and `rollback` jobs only. The token itself is `setkernel-live-deploy` in setkernel-cloudflare `stacks/services/api_tokens.tf`. Never deploy from a laptop. +The same workflow's `meter` job deploys sk-meter (`workers/meter/wrangler.toml`) with that token, +checks that its version serves all traffic and rolls it back in the job if not. ## Layout - `src/worker.js`: routes (`/ws`, `/api/state`, `/api/replay`, `/ingest`, `/ingest/usage`, `/ingest/limits`, `/usage/data`, `/usage/`, `/github/webhook`, `/github/queue`) and the `Hub` and `Usage` Durable Objects. @@ -91,6 +97,9 @@ a laptop. put text from the artifact in the comment, and never write any ref but `visual-diff-shots`. The poster reads only the files main's `SHOTS` (in `scripts/visual-diff.mjs`) names, so a pull request that changes `SHOTS` gets no comment ("unexpected file in the artifact") until it merges. +- `workers/meter/`: sk-meter, its own Worker (`wrangler.toml`, not `wrangler.jsonc`): `src/` the hourly + run and the guard flags, `allowances.json`, `tools/` the baseline and the offline drill, `test/` its + `node --test` suites. Its settings come from the Infisical -> Workers sync (`keep_vars = true`). - `test/`: `node --test` suites for the pure modules and the page (`day.test.mjs` for Day). - `test/worker/`: vitest suites run in workerd by `@cloudflare/vitest-plugin` (`vitest.config.mjs` reads `wrangler.jsonc`). They fake Access's certs with a throwaway key and drive every route diff --git a/package.json b/package.json index 926179d..cdcc6a4 100644 --- a/package.json +++ b/package.json @@ -7,8 +7,9 @@ "scripts": { "test": "node --test test/*.test.mjs", "test:worker": "vitest run", - "check": "for f in src/*.js public/*.js public/lib/*.js public/usage/*.js scripts/*.mjs; do node --check \"$f\" || exit 1; done", - "validate": "npm run check && npm test && npm run test:worker && wrangler deploy --dry-run --outdir .wrangler/dry-run" + "test:meter": "node --test workers/meter/test/*.test.js", + "check": "for f in src/*.js public/*.js public/lib/*.js public/usage/*.js scripts/*.mjs workers/meter/src/*.js workers/meter/tools/*.mjs; do node --check \"$f\" || exit 1; done", + "validate": "npm run check && npm test && npm run test:worker && npm run test:meter && wrangler deploy --dry-run --outdir .wrangler/dry-run && wrangler deploy --dry-run --config workers/meter/wrangler.toml --outdir .wrangler/dry-run-meter" }, "dependencies": { "@noble/secp256k1": "3.2.0" diff --git a/workers/meter/README.md b/workers/meter/README.md new file mode 100644 index 0000000..4eb666a --- /dev/null +++ b/workers/meter/README.md @@ -0,0 +1,104 @@ +# sk-meter: the Cloudflare billing guard + +Cloudflare's budget alerts email a day late and don't cap usage, and overage on Workers Paid +bills automatically. sk-meter keeps setkernel inside the $5 plan. Every hour (cron `7 * * * *`) +it reads the account's usage through the GraphQL Analytics API, totals each metric over its +window, compares it with the plan's included allowance and sets three KV flags. + +| Share of any allowance | KV key | What happens | +|---|---|---| +| 50% | `guard:alert` | an alert to Raghav | +| 80% | `guard:degrade` | readers run with `DEGRADED` (`src/flags.js`) | +| 95% | `guard:refuse` | readers refuse non-essential requests until the window resets | + +Cloudflare's own $10 budget alert stays on as the backstop. + +It moved here from a private repository on 2026-11-09 (setkernel/hq#1449, plan v6 F3). Its +history up to then is that repository's `meter/` at commit `c1e3253`. Not moved, and retired +with that repository: the Workers Logs `readings` tool and the `baseline` and live `drill` +workflows (each read Infisical through an identity bound to a workflow there). + +## Files +- `allowances.json`: the included allowance for each metric, its window (`month` from the + billing day, `day` from 00:00 UTC for Workers AI, `gauge` for storage), and the pricing page + it came from. Change it only from those pages. +- `src/usage.js`: the GraphQL queries, one per dataset. +- `src/meter.js`: shares, levels, flags, the pro-rated projection, the alert text. Pure. +- `src/flags.js`: the contract for every reader: `readGuard(env.GUARD)`, + `refusal(guard, { essential })` and `DEGRADED`. +- `src/index.js`: the hourly run. `src/notify.js`: the alert webhook. +- `tools/baseline.mjs`: a read-only report of current usage, also the schema check. +- `tools/drill.mjs`: the offline drill (below). + +## How it decides +- Each metric's total is recomputed from the start of its window every hour, so a missed run + loses nothing. +- A flag is on while any metric is at or past its threshold, and lists those metrics and when + the last of them resets (`until`; `null` when storage holds it, because only deleting data + brings that down). +- A query that fails keeps the metric's last value from the same window, so an outage of the + analytics API never lowers a flag; the failure is in the alert. +- Pro-rated: once 10% of a window has passed, a metric on pace to pass its allowance before the + reset is named in the alert. Flags act on actual use, not the projection. +- A reader that can't reach KV treats it as degrade, not refuse. +- The Workers and Durable Object queries also group by `scriptName` in the same query, so a + metric projected to reach 50% names its top script in the alert (`topScripts` in the report). +- An alert goes out when the flags, errors, on-pace list or Workers AI gateways change, and is + retried each hour until delivered. + +## What it costs +Its own use of the plan (https://developers.cloudflare.com/workers/platform/pricing/), a month: + +| Metric | sk-meter's use | Included | Share | +|---|---|---|---| +| Workers requests (cron runs) | 720 | 10,000,000 | 0.01% | +| Workers CPU time | estimate: a few ms a run, well under 36,000 ms | 30,000,000 ms | <0.2% | +| KV reads | 1,440 (report + drill each run) | 10,000,000 | 0.01% | +| KV writes | 720 (report) + a few on flag changes | 1,000,000 | <0.1% | +| GraphQL Analytics subrequests | ~11 a run; up to ~44 when every dataset needs its 3 retries | not billed | | + +## Commands +```sh +npm run test:meter # the tests (node --test); npm run validate runs them too +node workers/meter/tools/drill.mjs # the offline drill: in-memory KV, fake analytics API, nothing leaves the machine +CF_ACCOUNT_ID=โ€ฆ CF_ANALYTICS_TOKEN=โ€ฆ BILLING_DAY=13 node workers/meter/tools/baseline.mjs # live, read-only +``` + +## Deploy +Only from CI: the `meter` job in `.github/workflows/deploy.yml`, on every push to `main` and by +hand, after the same `check` job as the page. It reads the repo's deploy token (Infisical +project `setkernel-live`, identity `setkernel-live-deploy`: Workers Editor on the account, which +deploys an existing Worker and binds an existing KV namespace). sk-meter has no route, so the +job checks the live deployment itself: the version serving all traffic must be this run's +(`deploy `). If not, it rolls back to the version that was live before, and the job fails, +so `report` opens an issue. Nothing sets vars or secrets on deploy (`keep_vars = true`). + +### The Worker's settings +All in Infisical `sk-ctl-workers/prod`, reaching the Worker through the Infisical -> Cloudflare +Workers secret sync (target Worker `sk-meter`). The sync is by Worker name, so the move changed +nothing there. + +| Key | Value | +|---|---| +| `BILLING_DAY` | `13` (cycle starts on the 13th; required, 1-28) | +| `CF_ACCOUNT_ID` | the account to meter | +| `CF_ANALYTICS_TOKEN` | `sk-meter-analytics-read`: Account Analytics Read, nothing else (setkernel-cloudflare tofu) | +| `ALERT_WEBHOOK_URL` | `https://alerts.setkernel.net/meter` (https only; anything else is refused) | +| `ALERT_WEBHOOK_TOKEN` | bearer the receiver checks | +| `ALERT_ACCESS_CLIENT_ID`, `ALERT_ACCESS_CLIENT_SECRET` | the `sk-meter-alerts` Access service token | + +The KV namespace `sk-meter` (`f79ca12aeb9347f9851f5dfd53313f39`) is tofu-owned in +setkernel-cloudflare: this repo binds it and never creates or deletes it. + +## The baseline +`tools/baseline.mjs` introspects the GraphQL schema for every dataset, field and filter key in +`src/usage.js`, then reads the current usage the way the hourly run does, and prints both. Any +"Can't read" line fails it. A dataset still answering "try again later" after three retries is +an "Unknown" line: its metric shows as `?` and the run passes. It writes nothing and prints no +secret. + +## The drill +Proves the degrade end to end without extra spend: `meter:drill` in the namespace lowers one +metric's allowance below what is already used, until its `expiresAt`. A drill can only lower an +allowance. `tools/drill.mjs` runs the steps offline and fails if any step doesn't see what it +should. diff --git a/workers/meter/allowances.json b/workers/meter/allowances.json new file mode 100644 index 0000000..9864ad8 --- /dev/null +++ b/workers/meter/allowances.json @@ -0,0 +1,119 @@ +{ + "plan": "Workers Paid, $5 a month; no spend above it (setkernel/hq docs/decisions/0007-model-v4c.html)", + "checked": "2026-10-15", + "thresholds": { "alert": 0.5, "degrade": 0.8, "refuse": 0.95 }, + "metrics": { + "workers_requests": { + "label": "Workers requests", + "allowance": 10000000, + "unit": "requests", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "workers_cpu_ms": { + "label": "Workers CPU time", + "allowance": 30000000, + "unit": "CPU ms", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "do_requests": { + "label": "Durable Objects requests", + "allowance": 1000000, + "unit": "requests", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "do_duration_gbs": { + "label": "Durable Objects duration", + "allowance": 400000, + "unit": "GB-s", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "workflows_steps": { + "label": "Workflows steps", + "allowance": 500000, + "unit": "steps", + "window": "month", + "source": "https://developers.cloudflare.com/workflows/reference/pricing/" + }, + "queues_ops": { + "label": "Queues operations", + "allowance": 1000000, + "unit": "operations", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "d1_rows_read": { + "label": "D1 rows read", + "allowance": 25000000000, + "unit": "rows", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "d1_rows_written": { + "label": "D1 rows written", + "allowance": 50000000, + "unit": "rows", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "r2_storage_bytes": { + "label": "R2 storage", + "allowance": 10000000000, + "unit": "bytes (10 GB-month, counted as 10^9-byte GB so the guard errs low)", + "window": "gauge", + "source": "https://developers.cloudflare.com/r2/pricing/" + }, + "r2_class_a": { + "label": "R2 Class A operations", + "allowance": 1000000, + "unit": "requests", + "window": "month", + "source": "https://developers.cloudflare.com/r2/pricing/" + }, + "r2_class_b": { + "label": "R2 Class B operations", + "allowance": 10000000, + "unit": "requests", + "window": "month", + "source": "https://developers.cloudflare.com/r2/pricing/" + }, + "kv_reads": { + "label": "KV reads", + "allowance": 10000000, + "unit": "operations", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "kv_writes": { + "label": "KV writes", + "allowance": 1000000, + "unit": "operations", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "kv_deletes": { + "label": "KV deletes", + "allowance": 1000000, + "unit": "operations", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "kv_lists": { + "label": "KV lists", + "allowance": 1000000, + "unit": "operations", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "ai_neurons": { + "label": "Workers AI neurons", + "allowance": 10000, + "unit": "neurons", + "window": "day", + "source": "https://developers.cloudflare.com/workers-ai/platform/pricing/" + } + } +} diff --git a/workers/meter/src/flags.js b/workers/meter/src/flags.js new file mode 100644 index 0000000..26d3332 --- /dev/null +++ b/workers/meter/src/flags.js @@ -0,0 +1,49 @@ +// The guard's contract with every Worker that reads it. sk-meter writes these KV keys in the GUARD +// namespace; every other component binds the same namespace read-only in spirit and imports this +// module. A key is present while its threshold is passed and deleted when usage falls back (the +// window resets). Each value is JSON: { since, metrics: [ids], until: ISO time or null }. +// +// guard:alert 50% of any allowance: Buzz alert to Raghav (sk-meter sends it) +// guard:degrade 80%: run with DEGRADED below +// guard:refuse 95%: refuse non-essential requests until the window resets +// +// The static status page is served as static assets, which run before any Worker code, so it +// keeps serving whatever these flags say. + +export const KEYS = { alert: "guard:alert", degrade: "guard:degrade", refuse: "guard:refuse" }; +export const REPORT_KEY = "meter:report"; +export const DRILL_KEY = "meter:drill"; + +// What a component does while guard:degrade is on. +export const DEGRADED = Object.freeze({ + heartbeatSeconds: 60, + retentionDays: 7, + bundlesPaused: true, + newLeases: false, + workersAi: false, +}); + +// Read the three flags. KV caches each for a minute at the edge, so a busy Worker costs a few +// reads a minute, not one per request. If KV can't be read the guard can't be confirmed, so the +// answer is degrade (stop starting new spend) but not refuse (keep serving). +export async function readGuard(kv, { cacheTtl = 60 } = {}) { + try { + const [alert, degrade, refuse] = await Promise.all( + [KEYS.alert, KEYS.degrade, KEYS.refuse].map((k) => kv.get(k, { type: "json", cacheTtl })), + ); + // degrade is implied by refuse, alert by both: a reader never sees refuse without degrade. + return { alert: Boolean(alert || degrade || refuse), degrade: Boolean(degrade || refuse), refuse: Boolean(refuse), detail: { alert, degrade, refuse } }; + } catch (e) { + return { alert: true, degrade: true, refuse: false, error: String(e?.message ?? e) }; + } +} + +// For a Worker's fetch handler: a 503 to send instead of doing the work, or null to carry on. +// Essential requests (the status page, health checks, the guard's own readers) always carry on. +export function refusal(guard, { essential = false } = {}) { + if (!guard.refuse || essential) return null; + const until = guard.detail?.refuse?.until; + const headers = { "content-type": "text/plain; charset=utf-8" }; + if (until) headers["retry-after"] = String(Math.max(60, Math.ceil((Date.parse(until) - Date.now()) / 1000))); + return new Response("Paused: setkernel is holding Cloudflare usage inside its plan until the usage window resets.\n", { status: 503, headers }); +} diff --git a/workers/meter/src/index.js b/workers/meter/src/index.js new file mode 100644 index 0000000..4c0ca06 --- /dev/null +++ b/workers/meter/src/index.js @@ -0,0 +1,57 @@ +// sk-meter: Cloudflare's budget alerts arrive a day late and stop nothing, and overage on Workers +// Paid bills automatically. Every hour this reads the account's usage, compares it with the +// plan's included allowances (allowances.json) and sets the guard flags (flags.js) that the +// Workers reading them obey. +// +// Its own cost: 720 cron runs a month, ~11 GraphQL subrequests each (up to ~44 when every dataset +// needs all its retries), two KV reads, one KV write for the report and a write only when a flag +// changes. + +import allowances from "../allowances.json" with { type: "json" }; +import { evaluate, message, signature, FLAGS } from "./meter.js"; +import { KEYS, REPORT_KEY, DRILL_KEY } from "./flags.js"; +import { notify } from "./notify.js"; +import { readUsage } from "./usage.js"; +import { billingDay } from "./windows.js"; + +export async function run(env, now, { fetchImpl = fetch, table = allowances, retry } = {}) { + const kv = env.GUARD; + const day = billingDay(env.BILLING_DAY); + const [prev, drill] = await Promise.all([kv.get(REPORT_KEY, "json"), kv.get(DRILL_KEY, "json")]); + const usage = await readUsage(env, now, day, fetchImpl, retry); + const report = evaluate({ allowances: table, usage, now, day, prev, drill }); + + // Flags first: they are what stops spend. Write only on change; on the first run (no previous + // report) clear any flag left over from before. + for (const name of FLAGS) { + const was = prev ? prev.flags?.[name] ?? null : undefined; + const is = report.flags[name]; + if (is && JSON.stringify(is) !== JSON.stringify(was)) await kv.put(KEYS[name], JSON.stringify(is)); + else if (!is && was !== null) await kv.delete(KEYS[name]); + } + + // Alert when the state changes, and retry each hour until one is delivered. The first run has + // nothing to compare with, so it always reports: that is the baseline. + const sig = signature(report); + report.notified = prev?.notified ?? null; + if (sig !== report.notified) { + try { + if (await notify(env, message(report, { full: !prev }), fetchImpl)) report.notified = sig; + } catch (e) { + report.notifyError = e.message; + console.error(`sk-meter: alert failed: ${e.message}`); + } + } + + await kv.put(REPORT_KEY, JSON.stringify(report)); + const on = FLAGS.filter((f) => report.flags[f]); + console.log(`sk-meter: flags ${on.join(",") || "none"}; errors ${report.errors.join(",") || "none"}; unknown ${report.unknown.join(",") || "none"}`); + return report; +} + +export default { + // Awaited, so a run that throws shows as a failed cron event (sk-watch watches for those). + async scheduled(controller, env) { + await run(env, new Date(controller.scheduledTime)); + }, +}; diff --git a/workers/meter/src/meter.js b/workers/meter/src/meter.js new file mode 100644 index 0000000..715cd36 --- /dev/null +++ b/workers/meter/src/meter.js @@ -0,0 +1,176 @@ +// Turns usage into the guard's state: a share of the allowance per metric, the three flags, and +// the alert to send. Pure: no I/O, so every threshold is tested with synthetic usage. + +import { windowFor, elapsed } from "./windows.js"; + +export const LEVELS = ["ok", "alert", "degrade", "refuse"]; +export const FLAGS = ["alert", "degrade", "refuse"]; + +// Projection is noisy early in a window; it only speaks after this share has passed. +const PACE_AFTER = 0.1; + +export function levelOf(share, thresholds) { + if (share == null) return "unknown"; + if (share >= thresholds.refuse) return "refuse"; + if (share >= thresholds.degrade) return "degrade"; + if (share >= thresholds.alert) return "alert"; + return "ok"; +} + +// A drill lowers one metric's allowance until it expires, so real usage crosses a threshold with +// no extra spend. It can only make the guard stricter: a drill that would raise an allowance, or +// names an unknown metric, or has expired, is ignored. +export function activeDrill(drill, allowances, now) { + if (!drill || typeof drill !== "object") return null; + const pinned = allowances.metrics[drill.metric]?.allowance; + const a = Number(drill.allowance); + const until = Date.parse(drill.expiresAt); + if (!pinned || !(a > 0) || a >= pinned || !(until > now.getTime())) return null; + return { metric: drill.metric, allowance: a, expiresAt: new Date(until).toISOString() }; +} + +// The script using most of a metric that is projected to reach the alert level (50%) of its +// allowance, for any metric the run read per script. Ties go to the name that sorts first. +function topScripts(metrics, scripts, th) { + const out = {}; + for (const [id, by] of Object.entries(scripts ?? {})) { + const m = metrics[id]; + if (!m || m.projected == null || m.projected / m.allowance < th.alert) continue; + const rows = Object.entries(by).sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1)); + const sum = rows.reduce((t, [, v]) => t + v, 0); + if (!rows.length || !(sum > 0)) continue; + out[id] = { script: rows[0][0], used: rows[0][1], share: rows[0][1] / sum }; + } + return out; +} + +export function evaluate({ allowances, usage, now, day = 1, prev = null, drill = null }) { + const th = allowances.thresholds; + const d = activeDrill(drill, allowances, now); + const metrics = {}; + for (const [id, spec] of Object.entries(allowances.metrics)) { + const w = windowFor(spec.window, now, day); + const windowStart = w.start.toISOString(); + const allowance = d?.metric === id ? d.allowance : spec.allowance; + let used = usage.values[id] ?? null; + let stale = false; + const error = usage.errors[id]; + const unknown = usage.unknown?.[id]; + if (used == null) { + // Unreadable: keep the last value from the same window, so a failing query can never + // lower a flag. A gauge's window moves every hour, so match on the window kind instead. + const last = prev?.metrics?.[id]; + const sameWindow = last && (spec.window === "gauge" || last.windowStart === windowStart); + if (sameWindow && last.used != null) { used = last.used; stale = true; } + } + const share = used == null ? null : used / allowance; + const f = spec.window === "gauge" ? null : elapsed(w, now); + const projected = used == null || f == null || f < PACE_AFTER ? null : used / f; + metrics[id] = { + label: spec.label, unit: spec.unit, window: spec.window, used, allowance, share, + level: levelOf(share, th), + projected, + windowStart, + resetsAt: spec.window === "gauge" ? null : w.end.toISOString(), + ...(stale && { stale }), + ...(error && { error }), + ...(unknown && { unknown }), + ...(d?.metric === id && { drill: d.expiresAt }), + }; + } + + const flags = {}; + for (const name of FLAGS) { + const rank = LEVELS.indexOf(name); + const hit = Object.entries(metrics).filter(([, m]) => LEVELS.indexOf(m.level) >= rank); + if (!hit.length) { flags[name] = null; continue; } + const resets = hit.map(([, m]) => m.resetsAt); + flags[name] = { + since: prev?.flags?.[name]?.since ?? now.toISOString(), + metrics: hit.map(([id]) => id), + // When the last metric holding the flag resets; null while storage holds it (deleting + // objects is the only way down). + until: resets.includes(null) ? null : resets.sort().at(-1), + }; + } + + const errors = Object.keys(metrics).filter((id) => metrics[id].error).sort(); + // Still failing with a "try again" after the retries: not an error, but nothing is known this run. + const unknown = Object.keys(metrics).filter((id) => metrics[id].unknown).sort(); + const overPace = Object.keys(metrics) + .filter((id) => metrics[id].projected != null && metrics[id].projected >= metrics[id].allowance) + .sort(); + return { + at: now.toISOString(), + metrics, + flags, + errors, + unknown, + overPace, + drill: d, + gateways: usage.gateways ?? null, + topScripts: topScripts(metrics, usage.scripts, th), + }; +} + +// What the alert is about. A new alert goes out when this changes, and keeps being retried each +// hour until one is delivered. +export function signature(report) { + return JSON.stringify({ + flags: FLAGS.filter((f) => report.flags[f]).map((f) => [f, report.flags[f].metrics]), + errors: report.errors, + // A blip with a last known value changes nothing and sends nothing; a metric that has never + // been read is unguarded, so that stays in the alert. + unknown: (report.unknown ?? []).filter((id) => report.metrics[id].used == null), + overPace: report.overPace, + drill: report.drill?.metric ?? null, + workersAi: report.gateways?.workersAi ?? [], + // Not report.topScripts: the top script moves as usage does, and would re-send the alert. + }); +} + +const n = (x) => Math.round(x).toLocaleString("en-US"); +const pct = (x) => `${Math.round(x * 100)}%`; +const when = (iso) => iso.replace("T", " ").replace(/:00\.000Z$/, " UTC"); + +// A script name is the account owner's, but it goes into a chat message: keep it to name characters. +const scriptName = (name) => `\`${String(name).replace(/[^\w.\-/() ]/g, "?").slice(0, 64)}\``; + +function line(m) { + const per = m.window === "day" ? " a day" : m.window === "month" ? " this month" : ""; + const reset = m.resetsAt ? `, resets ${when(m.resetsAt)}` : ", stays until data is deleted"; + return `- ${m.label}: ${n(m.used)} of ${n(m.allowance)} ${m.unit.split(" (")[0]}${per} (${pct(m.share)})${reset}${m.stale ? " [last known]" : ""}`; +} + +// The alert text. `full` lists every metric, for the baseline report on the first run. +export function message(report, { full = false } = {}) { + const top = [...FLAGS].reverse().find((f) => report.flags[f]); + const head = { + refuse: "๐Ÿ”ด **sk-meter: REFUSE** (95%): non-essential requests are refused until the window resets.", + degrade: "๐ŸŸ  **sk-meter: DEGRADE** (80%): heartbeats 60 s, retention 7 days, bundles paused, no new leases, Workers AI off.", + alert: "๐ŸŸก **sk-meter: 50%** of an allowance is used.", + }[top] ?? "โœ… **sk-meter**: every metric is under 50% of its allowance."; + const lines = [(report.drill ? "[DRILL] " : "") + head]; + for (const m of Object.values(report.metrics)) { + if (m.used != null && (full || m.level !== "ok")) lines.push(line(m)); + } + for (const [id, t] of Object.entries(report.topScripts ?? {})) { + lines.push(`Top script for ${report.metrics[id].label}: ${scriptName(t.script)}, ${n(t.used)} ${report.metrics[id].unit.split(" (")[0]} (${pct(t.share)} of the account's).`); + } + const gw = report.gateways; + if (gw?.workersAi?.length) lines.push(`AI Gateways sending requests to Workers AI: ${gw.workersAi.join(", ")}.`); + else if (full && gw?.all) lines.push(`AI Gateways with traffic this month: ${gw.all.length}; none send requests to Workers AI.`); + else if (full && gw?.transient) lines.push(`AI Gateway traffic is unknown this run (${gw.error}).`); + else if (full && gw?.error) lines.push(`Can't read AI Gateway traffic (${gw.error}).`); + if (report.overPace.length) { + lines.push(`On pace to pass the allowance before it resets: ${report.overPace.map((id) => report.metrics[id].label).join(", ")}.`); + } + if (report.errors.length) { + lines.push(`Can't read ${report.errors.map((id) => report.metrics[id].label).join(", ")} (${report.metrics[report.errors[0]].error}); flags keep their last known values.`); + } + if (report.unknown?.length) { + lines.push(`Unknown this run (the API said to try again later): ${report.unknown.map((id) => report.metrics[id].label).join(", ")} (${report.metrics[report.unknown[0]].unknown}); flags keep their last known values.`); + } + if (report.drill) lines.push(`Drill: ${report.drill.metric} allowance lowered to ${n(report.drill.allowance)} until ${when(report.drill.expiresAt)}.`); + return { text: lines.join("\n"), mention: Boolean(report.flags.degrade || report.flags.refuse) }; +} diff --git a/workers/meter/src/notify.js b/workers/meter/src/notify.js new file mode 100644 index 0000000..d8f09c8 --- /dev/null +++ b/workers/meter/src/notify.js @@ -0,0 +1,33 @@ +// Alert delivery. A Worker can't run the buzz CLI or hold a Buzz identity the way hq's +// observability bridge does, so it POSTs { source, text, mention } to ALERT_WEBHOOK_URL (a Worker +// secret) with ALERT_WEBHOOK_TOKEN as a bearer token, and the receiver sends it to Buzz. The +// receiver is the buzz-alerts bridge's /meter route on sk-obs01 (hq#222, observability/README.md), +// behind Cloudflare Access: ALERT_ACCESS_CLIENT_ID and ALERT_ACCESS_CLIENT_SECRET are the +// sk-meter-alerts service token. A caller may name its own `source`. Without a webhook the alert is logged and the KV flags still act; +// the guard never depends on the alert arriving. + +export async function notify(env, alert, fetchImpl = fetch) { + const url = env.ALERT_WEBHOOK_URL; + if (!url) { + console.warn(`sk-meter: no ALERT_WEBHOOK_URL, alert not delivered: ${alert.text}`); + return false; + } + // Never send the bearer token over plain HTTP. + if (!url.startsWith("https://")) throw new Error("ALERT_WEBHOOK_URL must be https"); + const headers = { "content-type": "application/json" }; + if (env.ALERT_WEBHOOK_TOKEN) headers.authorization = `Bearer ${env.ALERT_WEBHOOK_TOKEN}`; + if (env.ALERT_ACCESS_CLIENT_ID && env.ALERT_ACCESS_CLIENT_SECRET) { + headers["cf-access-client-id"] = env.ALERT_ACCESS_CLIENT_ID; + headers["cf-access-client-secret"] = env.ALERT_ACCESS_CLIENT_SECRET; + } + const res = await fetchImpl(url, { + method: "POST", + headers, + // Access answers a refused token with a redirect to its login page; following it could end in + // a 200 and count the alert as delivered. A redirect is a failure, retried next hour. + redirect: "manual", + body: JSON.stringify({ source: alert.source ?? "sk-meter", text: alert.text, mention: alert.mention }), + }); + if (!res.ok) throw new Error(`alert webhook answered HTTP ${res.status}`); + return true; +} diff --git a/workers/meter/src/usage.js b/workers/meter/src/usage.js new file mode 100644 index 0000000..90ebb4c --- /dev/null +++ b/workers/meter/src/usage.js @@ -0,0 +1,243 @@ +// Account usage from Cloudflare's GraphQL Analytics API. One query per dataset, so a field the API +// rejects costs that dataset's metrics only (they are reported as errors and keep their last +// value), never the whole run. A server-side "try again" (HTTP 5xx or 429, a network error, or a +// GraphQL "try again later") is retried, then reported as unknown for this run; only a rejection +// that retrying can't fix (a missing field, a 403) is an error. Each query covers the metric's whole window, so a missed hour is +// caught up on the next run and nothing drifts. +// +// The two datasets behind the metrics hq#243 can't place (Workers CPU, Durable Object requests) are +// also grouped by script, in the same query, so the totals don't change: no `datetime` dimension, so +// a query returns one row per script. `perScript` maps a metric to what one row contributes to it. +// +// Dataset and field names come from the product docs' analytics pages (2026-10-15). Names marked +// "unconfirmed" were not on those pages; `node tools/baseline.mjs` checks every name here +// against the live schema before the first deploy. + +import { windowFor } from "./windows.js"; + +export const ENDPOINT = "https://api.cloudflare.com/client/v4/graphql"; + +// R2 operation classes, from https://developers.cloudflare.com/r2/pricing/. Anything not Class B +// counts as Class A, the smaller allowance, so an action we don't know about errs high. +export const R2_CLASS_B = new Set([ + "HeadBucket", "HeadObject", "GetObject", "UsageSummary", "GetBucketEncryption", + "GetBucketLocation", "GetBucketCors", "GetBucketLifecycleConfiguration", +]); + +const total = (rows, pick) => rows.reduce((n, r) => n + (Number(pick(r)) || 0), 0); + +export const SOURCES = [ + { + // unconfirmed: sum.cpuTimeUs (the tutorial shows only quantiles of CPU time) and + // dimensions.scriptName on this dataset + dataset: "workersInvocationsAdaptive", window: "month", + fields: "sum { requests cpuTimeUs } dimensions { scriptName }", + metrics: ["workers_requests", "workers_cpu_ms"], + perScript: { + workers_requests: (r) => r.sum?.requests, + workers_cpu_ms: (r) => (Number(r.sum?.cpuTimeUs) || 0) / 1000, + }, + read: (rows) => ({ + workers_requests: total(rows, (r) => r.sum?.requests), + workers_cpu_ms: total(rows, (r) => r.sum?.cpuTimeUs) / 1000, + }), + }, + { + // unconfirmed: dimensions.scriptName on this dataset + dataset: "durableObjectsInvocationsAdaptiveGroups", window: "month", + fields: "sum { requests } dimensions { scriptName }", + metrics: ["do_requests"], + perScript: { do_requests: (r) => r.sum?.requests }, + read: (rows) => ({ do_requests: total(rows, (r) => r.sum?.requests) }), + }, + { + // unconfirmed: sum.activeTime in microseconds. Duration bills 128 MB per active object, so + // GB-s = seconds active x 0.125. + dataset: "durableObjectsPeriodicGroups", window: "month", + fields: "sum { activeTime }", + metrics: ["do_duration_gbs"], + read: (rows) => ({ do_duration_gbs: (total(rows, (r) => r.sum?.activeTime) / 1e6) * 0.125 }), + }, + { + // A billed step is a step run; retries (ATTEMPT_*) and rollbacks don't count. + dataset: "workflowsAdaptiveGroups", window: "month", + filter: 'eventType: "STEP_START"', + fields: "count", + metrics: ["workflows_steps"], + read: (rows) => ({ workflows_steps: total(rows, (r) => r.count) }), + }, + { + dataset: "queueMessageOperationsAdaptiveGroups", window: "month", + fields: "sum { billableOperations }", + metrics: ["queues_ops"], + read: (rows) => ({ queues_ops: total(rows, (r) => r.sum?.billableOperations) }), + }, + { + dataset: "d1AnalyticsAdaptiveGroups", window: "month", + fields: "sum { rowsRead rowsWritten }", + metrics: ["d1_rows_read", "d1_rows_written"], + read: (rows) => ({ + d1_rows_read: total(rows, (r) => r.sum?.rowsRead), + d1_rows_written: total(rows, (r) => r.sum?.rowsWritten), + }), + }, + { + dataset: "r2OperationsAdaptiveGroups", window: "month", + fields: "sum { requests } dimensions { actionType }", + metrics: ["r2_class_a", "r2_class_b"], + read: (rows) => ({ + r2_class_a: total(rows.filter((r) => !R2_CLASS_B.has(r.dimensions?.actionType)), (r) => r.sum?.requests), + r2_class_b: total(rows.filter((r) => R2_CLASS_B.has(r.dimensions?.actionType)), (r) => r.sum?.requests), + }), + }, + { + // The largest size each bucket reached in the last day, summed over buckets. + dataset: "r2StorageAdaptiveGroups", window: "gauge", + fields: "max { payloadSize metadataSize } dimensions { bucketName }", + metrics: ["r2_storage_bytes"], + read: (rows) => ({ + r2_storage_bytes: total(rows, (r) => (Number(r.max?.payloadSize) || 0) + (Number(r.max?.metadataSize) || 0)), + }), + }, + { + // actionType is read, write, delete or list; anything else counts as a write (smallest allowance). + dataset: "kvOperationsAdaptiveGroups", window: "month", + fields: "sum { requests } dimensions { actionType }", + metrics: ["kv_reads", "kv_writes", "kv_deletes", "kv_lists"], + read: (rows) => { + const out = { kv_reads: 0, kv_writes: 0, kv_deletes: 0, kv_lists: 0 }; + const key = { read: "kv_reads", delete: "kv_deletes", list: "kv_lists" }; + for (const r of rows) out[key[r.dimensions?.actionType] ?? "kv_writes"] += Number(r.sum?.requests) || 0; + return out; + }, + }, + { + // unconfirmed: dataset aiInferenceAdaptiveGroups and sum.totalNeurons + dataset: "aiInferenceAdaptiveGroups", window: "day", + fields: "sum { totalNeurons }", + metrics: ["ai_neurons"], + read: (rows) => ({ ai_neurons: total(rows, (r) => r.sum?.totalNeurons) }), + }, +]; + +// Not a metric: which AI Gateways send requests to Workers AI. Any that do spend neurons, which +// the rule allows only inside the free 10,000 a day. +export const GATEWAYS = { + dataset: "aiGatewayRequestsAdaptiveGroups", window: "month", + fields: "count dimensions { gateway provider }", + read: (rows) => { + const all = new Set(); + const workersAi = new Set(); + for (const r of rows) { + const g = r.dimensions?.gateway; + if (!g) continue; + all.add(g); + if (String(r.dimensions?.provider).toLowerCase() === "workers-ai") workersAi.add(g); + } + return { all: [...all].sort(), workersAi: [...workersAi].sort() }; + }, +}; + +// Rows of a source's per-script metrics: { metric: { script: amount } }. A row with no script name +// (the API's null) is kept under NO_SCRIPT so the per-script amounts still add up to the total. +export const NO_SCRIPT = "(no script name)"; +export function scriptsOf(source, rows) { + const out = {}; + for (const [metric, pick] of Object.entries(source.perScript ?? {})) { + const by = {}; + for (const r of rows) { + const name = r.dimensions?.scriptName || NO_SCRIPT; + by[name] = (by[name] ?? 0) + (Number(pick(r)) || 0); + } + out[metric] = by; + } + return out; +} + +export function buildQuery(source) { + const extra = source.filter ? `, ${source.filter}` : ""; + return `query Usage($account: string!, $start: Time!, $end: Time!) { + viewer { accounts(filter: { accountTag: $account }) { + rows: ${source.dataset}(limit: 10000, filter: { datetime_geq: $start, datetime_leq: $end${extra} }) { ${source.fields} } + } } +}`; +} + +// A failure that asking again may fix. Everything else thrown here is a real error. +export class TransientError extends Error {} + +// Waits after the 1st, 2nd and 3rd failure: up to four requests for one dataset, then it is unknown. +export const RETRY_DELAYS_MS = [2000, 5000, 10000]; +const TRY_AGAIN = /try again/i; +const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +async function queryOnce(env, source, window, fetchImpl) { + let res; + try { + res = await fetchImpl(ENDPOINT, { + method: "POST", + headers: { "content-type": "application/json", authorization: `Bearer ${env.CF_ANALYTICS_TOKEN}` }, + body: JSON.stringify({ + query: buildQuery(source), + variables: { account: env.CF_ACCOUNT_ID, start: window.start.toISOString(), end: window.end.toISOString() }, + }), + }); + } catch (e) { + throw new TransientError(`${source.dataset}: ${String(e.message).slice(0, 200)}`); + } + if (!res.ok) { + const Kind = res.status >= 500 || res.status === 429 ? TransientError : Error; + throw new Kind(`${source.dataset}: HTTP ${res.status}`); + } + const body = await res.json(); + if (body.errors?.length) { + const message = String(body.errors[0].message).slice(0, 200); + throw new (TRY_AGAIN.test(message) ? TransientError : Error)(`${source.dataset}: ${message}`); + } + const rows = body.data?.viewer?.accounts?.[0]?.rows; + if (!Array.isArray(rows)) throw new Error(`${source.dataset}: no data in the response`); + if (rows.length >= 10000) throw new Error(`${source.dataset}: 10,000 rows, the total would be cut short`); + return rows; +} + +// Rows of one dataset over one window. Throws with a message that never contains the token: a +// TransientError once the retries are used up, a plain Error at once for anything retrying can't fix. +export async function queryRows(env, source, window, fetchImpl, { delays = RETRY_DELAYS_MS, sleep = wait } = {}) { + for (let attempt = 0; ; attempt++) { + try { + return await queryOnce(env, source, window, fetchImpl); + } catch (e) { + if (!(e instanceof TransientError) || attempt >= delays.length) throw e; + await sleep(delays[attempt]); + } + } +} + +// { values: { metric: number }, errors: { metric: message }, unknown: { metric: message }, +// gateways: { all, workersAi } | null, scripts: { metric: { script: number } } }. `scripts` has +// the metrics of a source with `perScript` that was read this run. `unknown` is a dataset still failing with a "try again" +// after the retries; `errors` is one that can't be fixed by waiting. +export async function readUsage(env, now, day, fetchImpl = fetch, retry = {}) { + const values = {}; + const errors = {}; + const unknown = {}; + const scripts = {}; + // Each window up to now; the API has no data from the future. + const until = (kind) => ({ start: windowFor(kind, now, day).start, end: now }); + await Promise.all(SOURCES.map(async (s) => { + try { + const rows = await queryRows(env, s, until(s.window), fetchImpl, retry); + Object.assign(values, s.read(rows)); + Object.assign(scripts, scriptsOf(s, rows)); + } catch (e) { + for (const m of s.metrics) (e instanceof TransientError ? unknown : errors)[m] = e.message; + } + })); + let gateways; + try { + gateways = GATEWAYS.read(await queryRows(env, GATEWAYS, until(GATEWAYS.window), fetchImpl, retry)); + } catch (e) { + gateways = { error: e.message, ...(e instanceof TransientError && { transient: true }) }; + } + return { values, errors, unknown, gateways, scripts }; +} diff --git a/workers/meter/src/windows.js b/workers/meter/src/windows.js new file mode 100644 index 0000000..09a36de --- /dev/null +++ b/workers/meter/src/windows.js @@ -0,0 +1,35 @@ +// Usage windows. Monthly allowances reset on the account's billing day (BILLING_DAY, 1-28, read +// it from the dashboard's billing page); Workers AI's free neurons reset daily at 00:00 UTC. +// A "gauge" metric (storage) has no reset: it is read over the last day, and what is stored now +// is what the month will bill. + +const DAY = 86400000; + +// Required: with no [vars] in wrangler.toml it arrives through the Infisical sync, and a +// silent default would measure the wrong month. +export function billingDay(value) { + if (value == null || value === "") throw new Error("BILLING_DAY is not set (Infisical sk-ctl-workers)"); + const n = Number(value); + if (!Number.isInteger(n) || n < 1 || n > 28) throw new Error(`BILLING_DAY must be 1-28, got ${value}`); + return n; +} + +// The window containing `now`: { start, end } as Dates, end exclusive. +export function windowFor(kind, now, day = 1) { + if (kind === "gauge") return { start: new Date(now.getTime() - DAY), end: now }; + if (kind === "day") { + const start = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate())); + return { start, end: new Date(start.getTime() + DAY) }; + } + if (kind !== "month") throw new Error(`unknown window ${kind}`); + const m = now.getUTCMonth() - (now.getUTCDate() < day ? 1 : 0); + const start = new Date(Date.UTC(now.getUTCFullYear(), m, day)); + const end = new Date(Date.UTC(start.getUTCFullYear(), start.getUTCMonth() + 1, day)); + return { start, end }; +} + +// Share of the window that has passed, 0..1. +export function elapsed(window, now) { + const f = (now - window.start) / (window.end - window.start); + return Math.min(1, Math.max(0, f)); +} diff --git a/workers/meter/test/baseline.test.js b/workers/meter/test/baseline.test.js new file mode 100644 index 0000000..a790493 --- /dev/null +++ b/workers/meter/test/baseline.test.js @@ -0,0 +1,167 @@ +// The baseline and its schema check, against a fake GraphQL endpoint that also answers +// introspection with a schema built from the names src/usage.js asks for. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { SOURCES, GATEWAYS } from "../src/usage.js"; +import { selectionPaths, filterKeys, checkSchema, introspector } from "../tools/schema.mjs"; +import { baseline } from "../tools/baseline.mjs"; +import { fakeGraphQL, rowsFor, noSleep, ENV } from "./fakes.js"; + +const NOW = new Date("2026-10-15T12:00:00Z"); +const TRY_AGAIN = new Error("unable to execute query, please try again later"); +const ref = (name) => ({ kind: "OBJECT", name, ofType: null }); +const list = (name) => ({ kind: "LIST", name: null, ofType: { kind: "NON_NULL", name: null, ofType: ref(name) } }); + +// Every name usage.js uses, minus `drop` (["dataset", "path.to.field"] or ["dataset"]). +function fakeSchema(drop = []) { + const types = { Query: { fields: [{ name: "viewer", type: ref("viewer") }] }, viewer: { fields: [{ name: "accounts", type: list("account") }] }, account: { fields: [] } }; + const add = (type, name, t) => { types[type] ??= { fields: [] }; if (!types[type].fields.some((f) => f.name === name)) types[type].fields.push({ name, type: t }); }; + for (const s of [...SOURCES, GATEWAYS]) { + if (drop.some(([d, p]) => d === s.dataset && !p)) continue; + const row = `${s.dataset}Row`; + types.account.fields.push({ name: s.dataset, type: list(row), args: [{ name: "filter", type: ref(`${s.dataset}Filter`) }] }); + types[`${s.dataset}Filter`] = { inputFields: filterKeys(s).map((name) => ({ name, type: ref("Time") })) }; + types[row] ??= { fields: [] }; + for (const path of selectionPaths(s.fields)) { + const dropped = drop.some(([d, p]) => d === s.dataset && p === path.join(".")); + let type = row; + for (const [i, step] of path.entries()) { + const last = i === path.length - 1; + const next = last ? "Leaf" : `${type}_${step}`; + if (last && dropped) types[type] ??= { fields: [] }; + else add(type, step, ref(next)); + type = next; + } + } + } + return types; +} + +function withSchema(types, rows = {}) { + const graphql = fakeGraphQL(rows); + return async (url, init) => { + const { query, variables } = JSON.parse(init.body); + if (query.includes("__schema")) return Response.json({ data: { __schema: { queryType: { name: "Query" } } } }); + if (query.includes("__type")) return Response.json({ data: { __type: types[variables.name] ? { name: variables.name, ...types[variables.name] } : null } }); + return graphql(url, init); + }; +} + +test("selectionPaths: nested selections become field paths", () => { + assert.deepEqual(selectionPaths("max { payloadSize metadataSize } dimensions { bucketName }"), + [["max", "payloadSize"], ["max", "metadataSize"], ["dimensions", "bucketName"]]); + assert.deepEqual(selectionPaths("count dimensions { gateway provider }"), [["count"], ["dimensions", "gateway"], ["dimensions", "provider"]]); + assert.deepEqual(filterKeys(SOURCES.find((s) => s.filter)), ["datetime_geq", "datetime_leq", "eventType"]); +}); + +test("schema check: every name present, no problems", async () => { + assert.deepEqual(await checkSchema(introspector(ENV, withSchema(fakeSchema())), [...SOURCES, GATEWAYS]), []); +}); + +test("schema check names a missing field, a missing dataset and a missing filter key, with what is there", async () => { + const types = fakeSchema([["workersInvocationsAdaptive", "sum.cpuTimeUs"], ["aiInferenceAdaptiveGroups"]]); + types.workflowsAdaptiveGroupsFilter.inputFields = types.workflowsAdaptiveGroupsFilter.inputFields.filter((f) => f.name !== "eventType"); + const problems = await checkSchema(introspector(ENV, withSchema(types))); + assert.deepEqual(problems.map((p) => `${p.where}.${p.missing}`).sort(), [ + "aiInferenceAdaptiveGroups.aiInferenceAdaptiveGroups", + "workersInvocationsAdaptive.sum.cpuTimeUs", + "workflowsAdaptiveGroups(filter).eventType", + ]); + assert.deepEqual(problems.find((p) => p.missing === "cpuTimeUs").has, ["requests"]); +}); + +test("baseline: exit 0 and every metric listed when everything reads", async () => { + const out = []; + const code = await baseline({ ...ENV, BILLING_DAY: "13" }, { now: NOW, fetchImpl: withSchema(fakeSchema(), rowsFor("kv_reads", 42)), log: (l) => out.push(l) }); + assert.equal(code, 0); + const text = out.join("\n"); + assert.match(text, /billing day 13/); + assert.match(text, /^kv_reads \| 42 \| 10000000 \|/m); + assert.doesNotMatch(text, /Can't read/); +}); + +test("baseline: CPU-ms and DO requests by script, the top five unless --scripts asks for all", async () => { + const rows = { + workersInvocationsAdaptive: Array.from({ length: 7 }, (_, i) => ({ sum: { requests: 1, cpuTimeUs: (i + 1) * 1000 }, dimensions: { scriptName: `w${i + 1}` } })), + durableObjectsInvocationsAdaptiveGroups: [{ sum: { requests: 40 }, dimensions: { scriptName: "setkernel-ops" } }, { sum: { requests: 2 }, dimensions: { scriptName: "sk-status" } }], + }; + const run = async (opts) => { + const out = []; + const code = await baseline({ ...ENV, CF_ANALYTICS_TOKEN: "s3cret-token-value" }, { now: NOW, fetchImpl: withSchema(fakeSchema(), rows), log: (l) => out.push(l), ...opts }); + return { code, text: out.join("\n") }; + }; + const top = await run(); + assert.equal(top.code, 0); + assert.match(top.text, /^workers_cpu_ms by script \(5 of 7\) \| CPU ms\nw7 \| 7\nw6 \| 6\nw5 \| 5\nw4 \| 4\nw3 \| 3$/m); + assert.doesNotMatch(top.text, /^w2 \|/m); + assert.match(top.text, /^do_requests by script \(2 of 2\) \| requests\nsetkernel-ops \| 40\nsk-status \| 2$/m); + assert.doesNotMatch(top.text, /s3cret/); + const all = await run({ allScripts: true }); + assert.match(all.text, /^workers_cpu_ms by script \(7 of 7\)/m); + assert.match(all.text, /^w1 \| 1$/m); +}); + +test("baseline: any Can't read line fails it, and the token is never printed", async () => { + const out = []; + const env = { ...ENV, CF_ANALYTICS_TOKEN: "s3cret-token-value" }; + const types = fakeSchema([["durableObjectsPeriodicGroups", "sum.activeTime"]]); + const code = await baseline(env, { now: NOW, fetchImpl: withSchema(types, { durableObjectsPeriodicGroups: new Error("unknown field activeTime") }), log: (l) => out.push(l) }); + assert.equal(code, 1); + const text = out.join("\n"); + assert.match(text, /^Can't read durableObjectsPeriodicGroups\.sum\.activeTime: not in the schema/m); + assert.match(text, /^Can't read do_duration_gbs: durableObjectsPeriodicGroups: unknown field/m); + assert.doesNotMatch(text, /s3cret/); +}); + +test("baseline: an introspection failure is a Can't read line, not a pass", async () => { + const out = []; + const refused = async () => new Response("no", { status: 403 }); + assert.equal(await baseline(ENV, { now: NOW, fetchImpl: refused, log: (l) => out.push(l) }), 1); + assert.match(out.join("\n"), /^Can't read the schema: introspection: HTTP 403/m); +}); + +test("baseline: no token or account, exit 2 before any request", async () => { + let called = false; + const code = await baseline({ BILLING_DAY: "13" }, { fetchImpl: async () => { called = true; }, log: () => {} }); + assert.equal(code, 2); + assert.equal(called, false); +}); + +test("baseline: one transient dataset exits 0, the metric is '?', and the line says so", async () => { + const out = []; + const { retry } = noSleep(); + const code = await baseline(ENV, { + now: NOW, retry, log: (l) => out.push(l), + fetchImpl: withSchema(fakeSchema(), ({ ...rowsFor("kv_reads", 42), workflowsAdaptiveGroups: TRY_AGAIN })), + }); + assert.equal(code, 0); + const text = out.join("\n"); + assert.match(text, /^Unknown workflows_steps: .*try again later/m); + assert.match(text, /^workflows_steps \| \? \|/m); + assert.match(text, /^kv_reads \| 42 \|/m); + assert.doesNotMatch(text, /Can't read/); +}); + +test("baseline: a schema error alongside a transient one still exits 1", async () => { + const out = []; + const code = await baseline(ENV, { + now: NOW, retry: noSleep().retry, log: (l) => out.push(l), + fetchImpl: withSchema(fakeSchema(), ({ workflowsAdaptiveGroups: TRY_AGAIN, d1AnalyticsAdaptiveGroups: new Error("unknown field rowsRead") })), + }); + assert.equal(code, 1); + const text = out.join("\n"); + assert.match(text, /^Can't read d1_rows_read: /m); + assert.match(text, /^Unknown workflows_steps: /m); +}); + +test("baseline: a transient AI Gateway read exits 0 and is not called a 'Can't read'", async () => { + const out = []; + const code = await baseline(ENV, { + now: NOW, retry: noSleep().retry, log: (l) => out.push(l), + fetchImpl: withSchema(fakeSchema(), ({ aiGatewayRequestsAdaptiveGroups: new Error("unable to execute query, please try again later") })), + }); + assert.equal(code, 0); + assert.doesNotMatch(out.join("\n"), /Can't read/); + assert.match(out.join("\n"), /AI Gateway traffic is unknown this run/); +}); diff --git a/workers/meter/test/fakes.js b/workers/meter/test/fakes.js new file mode 100644 index 0000000..6f7b0d6 --- /dev/null +++ b/workers/meter/test/fakes.js @@ -0,0 +1,76 @@ +// Test doubles: a KV namespace in memory that counts writes, and a GraphQL endpoint that answers +// each dataset with the rows a test sets. + +import { ENDPOINT } from "../src/usage.js"; + +export function memoryKV(initial = {}) { + const data = new Map(Object.entries(initial)); + const kv = { + data, + puts: 0, + deletes: 0, + async get(key, opts) { + const v = data.get(key); + if (v == null) return null; + const type = typeof opts === "string" ? opts : opts?.type; + return type === "json" ? JSON.parse(v) : v; + }, + async put(key, value) { kv.puts++; data.set(key, value); }, + async delete(key) { kv.deletes++; data.delete(key); }, + }; + return kv; +} + +// A dataset that fails its first `times` requests with `failure` (an Error, answered as a GraphQL +// error, or an HTTP status number) and then answers with `rows`. Pass it as a fakeGraphQL value. +export const flaky = (times, failure, rows = []) => (attempt) => (attempt <= times ? failure : rows); + +// No waiting between retries in tests; records what it was asked to wait. +export const noSleep = () => { const waits = []; const sleep = async (ms) => { waits.push(ms); }; return { waits, retry: { sleep } }; }; + +// rows: { datasetName: [row, ...] | Error | (attempt) => ... }. Unlisted datasets answer with no rows. +export function fakeGraphQL(rows = {}, { webhook = [] } = {}) { + const calls = []; + const impl = async (url, init) => { + if (url !== ENDPOINT) { + webhook.push({ url, init, body: JSON.parse(init.body) }); + return new Response("ok", { status: webhook.status ?? 200 }); + } + const body = JSON.parse(init.body); + const dataset = body.query.match(/rows: (\w+)\(/)[1]; + calls.push({ dataset, body, auth: init.headers.authorization }); + let r = rows[dataset] ?? []; + if (typeof r === "function") r = r(calls.filter((c) => c.dataset === dataset).length); + if (typeof r === "number") return new Response("down", { status: r }); + if (r instanceof Error) return Response.json({ data: null, errors: [{ message: r.message }] }); + return Response.json({ data: { viewer: { accounts: [{ rows: r }] } }, errors: null }); + }; + impl.calls = calls; + return impl; +} + +// Rows that put one metric at `used`, in the shape its dataset returns. +export function rowsFor(metric, used) { + const table = { + workers_requests: ["workersInvocationsAdaptive", { sum: { requests: used, cpuTimeUs: 0 }, dimensions: { scriptName: "w" } }], + workers_cpu_ms: ["workersInvocationsAdaptive", { sum: { requests: 0, cpuTimeUs: used * 1000 }, dimensions: { scriptName: "w" } }], + do_requests: ["durableObjectsInvocationsAdaptiveGroups", { sum: { requests: used }, dimensions: { scriptName: "w" } }], + do_duration_gbs: ["durableObjectsPeriodicGroups", { sum: { activeTime: (used / 0.125) * 1e6 } }], + workflows_steps: ["workflowsAdaptiveGroups", { count: used }], + queues_ops: ["queueMessageOperationsAdaptiveGroups", { sum: { billableOperations: used } }], + d1_rows_read: ["d1AnalyticsAdaptiveGroups", { sum: { rowsRead: used, rowsWritten: 0 } }], + d1_rows_written: ["d1AnalyticsAdaptiveGroups", { sum: { rowsRead: 0, rowsWritten: used } }], + r2_storage_bytes: ["r2StorageAdaptiveGroups", { max: { payloadSize: used, metadataSize: 0 }, dimensions: { bucketName: "b" } }], + r2_class_a: ["r2OperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "PutObject" } }], + r2_class_b: ["r2OperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "GetObject" } }], + kv_reads: ["kvOperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "read" } }], + kv_writes: ["kvOperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "write" } }], + kv_deletes: ["kvOperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "delete" } }], + kv_lists: ["kvOperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "list" } }], + ai_neurons: ["aiInferenceAdaptiveGroups", { sum: { totalNeurons: used } }], + }; + const [dataset, row] = table[metric]; + return { [dataset]: [row] }; +} + +export const ENV = { CF_ANALYTICS_TOKEN: "test-token", CF_ACCOUNT_ID: "acct", BILLING_DAY: "1" }; diff --git a/workers/meter/test/flags.test.js b/workers/meter/test/flags.test.js new file mode 100644 index 0000000..6aa2147 --- /dev/null +++ b/workers/meter/test/flags.test.js @@ -0,0 +1,54 @@ +// What the Workers reading the guard see: readGuard and refusal. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { readGuard, refusal, KEYS, DEGRADED } from "../src/flags.js"; +import { memoryKV } from "./fakes.js"; + +const flag = (until = null) => JSON.stringify({ since: "2026-10-15T10:07:00.000Z", metrics: ["ai_neurons"], until }); + +test("no flags: carry on", async () => { + const g = await readGuard(memoryKV()); + assert.deepEqual([g.alert, g.degrade, g.refuse], [false, false, false]); + assert.equal(refusal(g), null); +}); + +test("refuse implies degrade and alert, even if only the refuse key is there", async () => { + const g = await readGuard(memoryKV({ [KEYS.refuse]: flag() })); + assert.deepEqual([g.alert, g.degrade, g.refuse], [true, true, true]); +}); + +test("refuse: non-essential requests get a 503 with Retry-After; essential ones carry on", async () => { + const until = new Date(Date.now() + 3600e3).toISOString(); + const g = await readGuard(memoryKV({ [KEYS.alert]: flag(until), [KEYS.degrade]: flag(until), [KEYS.refuse]: flag(until) })); + const res = refusal(g); + assert.equal(res.status, 503); + const retry = Number(res.headers.get("retry-after")); + assert.ok(retry > 3500 && retry <= 3600, String(retry)); + assert.equal(refusal(g, { essential: true }), null); +}); + +test("degrade alone refuses nothing", async () => { + const g = await readGuard(memoryKV({ [KEYS.alert]: flag(), [KEYS.degrade]: flag() })); + assert.equal(g.degrade, true); + assert.equal(refusal(g), null); +}); + +test("KV unreadable: degrade (start nothing new) but keep serving", async () => { + const broken = { get: async () => { throw new Error("KV down"); } }; + const g = await readGuard(broken); + assert.deepEqual([g.alert, g.degrade, g.refuse], [true, true, false]); + assert.match(g.error, /KV down/); +}); + +test("readGuard asks KV to cache each flag at the edge", async () => { + const seen = []; + await readGuard({ get: async (k, o) => { seen.push([k, o]); return null; } }); + assert.deepEqual(seen.map(([k]) => k), [KEYS.alert, KEYS.degrade, KEYS.refuse]); + for (const [, o] of seen) assert.deepEqual(o, { type: "json", cacheTtl: 60 }); +}); + +test("the degraded settings are the spec's", () => { + assert.deepEqual(DEGRADED, { heartbeatSeconds: 60, retentionDays: 7, bundlesPaused: true, newLeases: false, workersAi: false }); + assert.ok(Object.isFrozen(DEGRADED)); +}); diff --git a/workers/meter/test/meter.test.js b/workers/meter/test/meter.test.js new file mode 100644 index 0000000..ce70d14 --- /dev/null +++ b/workers/meter/test/meter.test.js @@ -0,0 +1,202 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import allowances from "../allowances.json" with { type: "json" }; +import { evaluate, levelOf, activeDrill, message, signature } from "../src/meter.js"; +import { windowFor, elapsed, billingDay } from "../src/windows.js"; +import { SOURCES } from "../src/usage.js"; + +const NOW = new Date("2026-10-15T12:00:00Z"); +const usage = (values, errors = {}) => ({ values, errors, gateways: null }); +const flagsOn = (r) => ["alert", "degrade", "refuse"].filter((f) => r.flags[f]); + +test("allowances.json pins every metric with a positive allowance, a window and a pricing URL", () => { + for (const [id, m] of Object.entries(allowances.metrics)) { + assert.ok(m.allowance > 0, id); + assert.ok(["month", "day", "gauge"].includes(m.window), id); + assert.match(m.source, /^https:\/\/developers\.cloudflare\.com\/.+pricing/, id); + } + assert.deepEqual(allowances.thresholds, { alert: 0.5, degrade: 0.8, refuse: 0.95 }); + assert.equal(allowances.metrics.ai_neurons.allowance, 10000); + assert.equal(allowances.metrics.ai_neurons.window, "day"); +}); + +test("every metric is read by exactly one query, over the window allowances.json gives it", () => { + const seen = {}; + for (const s of SOURCES) for (const m of s.metrics) { + assert.equal(seen[m], undefined, `${m} read twice`); + seen[m] = s.window; + } + for (const [id, m] of Object.entries(allowances.metrics)) assert.equal(seen[id], m.window, id); + assert.deepEqual(Object.keys(seen).sort(), Object.keys(allowances.metrics).sort()); +}); + +test("levelOf: the boundaries are inclusive", () => { + const th = allowances.thresholds; + assert.equal(levelOf(0.4999, th), "ok"); + assert.equal(levelOf(0.5, th), "alert"); + assert.equal(levelOf(0.7999, th), "alert"); + assert.equal(levelOf(0.8, th), "degrade"); + assert.equal(levelOf(0.9499, th), "degrade"); + assert.equal(levelOf(0.95, th), "refuse"); + assert.equal(levelOf(3, th), "refuse"); + assert.equal(levelOf(null, th), "unknown"); +}); + +// The acceptance test: synthetic usage past each threshold, on every metric, flips the right flags. +for (const [id, m] of Object.entries(allowances.metrics)) { + test(`${id}: synthetic usage past 50/80/95% flips alert, degrade, refuse`, () => { + const cases = [[0.49, []], [0.51, ["alert"]], [0.81, ["alert", "degrade"]], [0.96, ["alert", "degrade", "refuse"]]]; + for (const [share, want] of cases) { + const r = evaluate({ allowances, usage: usage({ [id]: m.allowance * share }), now: NOW }); + assert.deepEqual(flagsOn(r), want, `${id} at ${share}`); + for (const f of want) assert.deepEqual(r.flags[f].metrics, [id]); + } + }); +} + +test("the worst metric decides; each flag lists the metrics holding it", () => { + const r = evaluate({ + allowances, + usage: usage({ queues_ops: 600000, ai_neurons: 9600, kv_writes: 850000 }), + now: NOW, + }); + assert.deepEqual(r.flags.alert.metrics, ["queues_ops", "kv_writes", "ai_neurons"]); + assert.deepEqual(r.flags.degrade.metrics, ["kv_writes", "ai_neurons"]); + assert.deepEqual(r.flags.refuse.metrics, ["ai_neurons"]); + // refuse lasts until the neurons reset at 00:00 UTC; degrade until the month resets too + assert.equal(r.flags.refuse.until, "2026-10-16T00:00:00.000Z"); + assert.equal(r.flags.degrade.until, "2026-11-01T00:00:00.000Z"); +}); + +test("a flag held by storage has no reset time", () => { + const r = evaluate({ allowances, usage: usage({ r2_storage_bytes: 9e9 }), now: NOW }); + assert.equal(r.flags.degrade.until, null); +}); + +test("a flag keeps its since time while it stays on", () => { + const first = evaluate({ allowances, usage: usage({ ai_neurons: 8500 }), now: NOW }); + const later = new Date("2026-10-15T13:00:00Z"); + const second = evaluate({ allowances, usage: usage({ ai_neurons: 8700 }), now: later, prev: first }); + assert.equal(second.flags.degrade.since, NOW.toISOString()); +}); + +test("the window reset clears the flags", () => { + const before = evaluate({ allowances, usage: usage({ ai_neurons: 9900 }), now: NOW }); + const after = evaluate({ allowances, usage: usage({ ai_neurons: 10 }), now: new Date("2026-10-16T00:07:00Z"), prev: before }); + assert.deepEqual(flagsOn(after), []); +}); + +test("an unreadable metric keeps its last value in the same window, so a flag can't drop", () => { + const first = evaluate({ allowances, usage: usage({ queues_ops: 900000 }), now: NOW }); + const failed = evaluate({ + allowances, + usage: usage({}, { queues_ops: "queueMessageOperationsAdaptiveGroups: HTTP 500" }), + now: new Date("2026-10-15T13:00:00Z"), + prev: first, + }); + assert.deepEqual(flagsOn(failed), ["alert", "degrade"]); + assert.equal(failed.metrics.queues_ops.stale, true); + assert.deepEqual(failed.errors, ["queues_ops"]); +}); + +test("an unreadable metric from a previous window is unknown, not zero", () => { + const first = evaluate({ allowances, usage: usage({ queues_ops: 900000 }), now: NOW }); + const failed = evaluate({ + allowances, + usage: usage({}, { queues_ops: "boom" }), + now: new Date("2026-11-01T01:00:00Z"), + prev: first, + }); + assert.equal(failed.metrics.queues_ops.used, null); + assert.equal(failed.metrics.queues_ops.level, "unknown"); + assert.deepEqual(failed.errors, ["queues_ops"]); +}); + +test("pro-rating: projected use at the end of the window, once 10% of it has passed", () => { + // 15 Oct 12:00 is 14.5 days into a 31-day month + const r = evaluate({ allowances, usage: usage({ queues_ops: 480000 }), now: NOW }); + assert.equal(r.metrics.queues_ops.level, "ok"); + assert.ok(Math.abs(r.metrics.queues_ops.projected - 480000 / (14.5 / 31)) < 1); + assert.deepEqual(r.overPace, ["queues_ops"]); + // two hours into the day, Workers AI's window is under 10% gone: no projection yet + const early = evaluate({ allowances, usage: usage({ ai_neurons: 2000 }), now: new Date("2026-10-15T02:00:00Z") }); + assert.equal(early.metrics.ai_neurons.projected, null); + assert.deepEqual(early.overPace, []); +}); + +test("a drill lowers one allowance until it expires and can never raise one", () => { + const drill = { metric: "ai_neurons", allowance: 100, expiresAt: "2026-10-15T14:00:00Z" }; + const r = evaluate({ allowances, usage: usage({ ai_neurons: 85 }), now: NOW, drill }); + assert.deepEqual(flagsOn(r), ["alert", "degrade"]); + assert.equal(r.metrics.ai_neurons.allowance, 100); + assert.equal(r.drill.metric, "ai_neurons"); + assert.match(message(r).text, /^\[DRILL\] /); + + assert.equal(activeDrill({ ...drill, expiresAt: "2026-10-15T11:59:59Z" }, allowances, NOW), null); + assert.equal(activeDrill({ ...drill, allowance: 1e9 }, allowances, NOW), null); + assert.equal(activeDrill({ ...drill, allowance: 10000 }, allowances, NOW), null); + assert.equal(activeDrill({ ...drill, allowance: -1 }, allowances, NOW), null); + assert.equal(activeDrill({ ...drill, metric: "nope" }, allowances, NOW), null); + assert.equal(activeDrill("garbage", allowances, NOW), null); +}); + +test("message: the highest flag leads, degrade and refuse mention Raghav, the baseline lists everything", () => { + const values = Object.fromEntries(Object.keys(allowances.metrics).map((id) => [id, 0])); + const quiet = evaluate({ allowances, usage: { ...usage({ ...values, queues_ops: 12000 }), gateways: { all: ["a", "b"], workersAi: [] } }, now: NOW }); + const base = message(quiet, { full: true }); + assert.match(base.text, /every metric is under 50%/); + assert.match(base.text, /Queues operations: 12,000 of 1,000,000 operations this month \(1%\)/); + assert.match(base.text, /none send requests to Workers AI/); + assert.equal(base.mention, false); + assert.equal(message(quiet).text.split("\n").length, 1); + + const hot = evaluate({ allowances, usage: { ...usage({ ai_neurons: 9700 }), gateways: { all: ["g"], workersAi: ["g"] } }, now: NOW }); + const m = message(hot); + assert.match(m.text, /^๐Ÿ”ด \*\*sk-meter: REFUSE\*\*/); + assert.match(m.text, /Workers AI neurons: 9,700 of 10,000 neurons a day \(97%\), resets 2026-10-16 00:00 UTC/); + assert.match(m.text, /AI Gateways sending requests to Workers AI: g/); + assert.equal(m.mention, true); +}); + +test("signature changes with the flags, errors, pace and Workers AI gateways, not with usage inside a level", () => { + const a = evaluate({ allowances, usage: usage({ kv_writes: 510000 }), now: NOW }); + const b = evaluate({ allowances, usage: usage({ kv_writes: 520000 }), now: NOW, prev: a }); + const c = evaluate({ allowances, usage: usage({ kv_writes: 810000 }), now: NOW, prev: b }); + assert.equal(signature(a), signature(b)); + assert.notEqual(signature(b), signature(c)); + const g = evaluate({ allowances, usage: { ...usage({ kv_writes: 510000 }), gateways: { workersAi: ["x"] } }, now: NOW }); + assert.notEqual(signature(a), signature(g)); +}); + +test("the top script is named for a metric projected at 50%, and stays out of the signature", () => { + // NOW is 14.5 days into a 31-day month, so 10M CPU ms used projects to ~21M of 30M: 71%. + const scripts = (top) => ({ workers_cpu_ms: { [top]: 6e6, other: 4e6 }, do_requests: { a: 1 } }); + const hot = (top) => evaluate({ allowances, usage: { ...usage({ workers_cpu_ms: 1e7, do_requests: 1 }), scripts: scripts(top) }, now: NOW }); + const a = hot("alpha"); + assert.deepEqual(a.topScripts, { workers_cpu_ms: { script: "alpha", used: 6e6, share: 0.6 } }); + assert.match(message(a).text, /^Top script for Workers CPU time: `alpha`, 6,000,000 CPU ms \(60% of the account's\)\.$/m); + assert.equal(signature(a), signature(hot("zeta"))); + assert.equal(hot("zeta").topScripts.workers_cpu_ms.script, "zeta"); + // below 50% projected, or no per-script rows this run: nothing named + const low = evaluate({ allowances, usage: { ...usage({ workers_cpu_ms: 1e6 }), scripts: scripts("alpha") }, now: NOW }); + assert.deepEqual(low.topScripts, {}); + assert.doesNotMatch(message(low).text, /Top script/); + assert.deepEqual(evaluate({ allowances, usage: usage({ workers_cpu_ms: 1e7 }), now: NOW }).topScripts, {}); + // a script name can't carry markdown or a mention into the chat message + const odd = evaluate({ allowances, usage: { ...usage({ workers_cpu_ms: 1e7 }), scripts: { workers_cpu_ms: { "`@here`**x": 5 } } }, now: NOW }); + assert.match(message(odd).text, /Top script for Workers CPU time: `\?\?here\?\?\?x`/); +}); + +test("windows: billing day, the daily UTC reset, elapsed share", () => { + assert.deepEqual(windowFor("month", NOW, 1), { start: new Date("2026-10-01T00:00:00Z"), end: new Date("2026-11-01T00:00:00Z") }); + assert.deepEqual(windowFor("month", NOW, 20), { start: new Date("2026-09-20T00:00:00Z"), end: new Date("2026-10-20T00:00:00Z") }); + assert.deepEqual(windowFor("month", new Date("2026-01-05T00:00:00Z"), 10), { start: new Date("2025-12-10T00:00:00Z"), end: new Date("2026-01-10T00:00:00Z") }); + assert.deepEqual(windowFor("day", NOW), { start: new Date("2026-10-15T00:00:00Z"), end: new Date("2026-10-16T00:00:00Z") }); + assert.equal(elapsed(windowFor("day", NOW), NOW), 0.5); + assert.throws(() => billingDay(undefined), /not set/); + assert.throws(() => billingDay(""), /not set/); + assert.equal(billingDay("13"), 13); + assert.equal(billingDay("28"), 28); + assert.throws(() => billingDay("29")); + assert.throws(() => billingDay("x")); +}); diff --git a/workers/meter/test/notify.test.js b/workers/meter/test/notify.test.js new file mode 100644 index 0000000..a33dd24 --- /dev/null +++ b/workers/meter/test/notify.test.js @@ -0,0 +1,69 @@ +// The alert webhook: the headers the /meter receiver behind Cloudflare Access needs. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { notify } from "../src/notify.js"; + +const URL = "https://alerts.example.test/meter"; +const ALERT = { text: "x", mention: false }; + +function capture(status = 200) { + const seen = []; + const impl = async (url, init) => { + seen.push({ url, init }); + return new Response(null, { status, headers: status >= 300 && status < 400 ? { location: "https://login.example.test/" } : {} }); + }; + impl.seen = seen; + return impl; +} + +test("sends the Access service token and the bearer, and doesn't follow redirects", async () => { + const f = capture(); + const env = { ALERT_WEBHOOK_URL: URL, ALERT_WEBHOOK_TOKEN: "t", ALERT_ACCESS_CLIENT_ID: "id", ALERT_ACCESS_CLIENT_SECRET: "s" }; + assert.equal(await notify(env, ALERT, f), true); + const { headers, redirect } = f.seen[0].init; + assert.equal(headers["cf-access-client-id"], "id"); + assert.equal(headers["cf-access-client-secret"], "s"); + assert.equal(headers.authorization, "Bearer t"); + assert.equal(redirect, "manual"); +}); + +test("no Access headers unless both halves of the service token are set", async () => { + const f = capture(); + await notify({ ALERT_WEBHOOK_URL: URL, ALERT_ACCESS_CLIENT_ID: "id" }, ALERT, f); + assert.equal(f.seen[0].init.headers["cf-access-client-id"], undefined); + assert.equal(f.seen[0].init.headers["cf-access-client-secret"], undefined); +}); + +test("an Access redirect to its login page is a failure, so the alert is retried", async () => { + await assert.rejects(notify({ ALERT_WEBHOOK_URL: URL }, ALERT, capture(302)), /HTTP 302/); +}); + +test("the receiver's contract: POST { source, text, mention } as JSON, and its 204 counts as delivered", async () => { + const f = capture(204); + const env = { ALERT_WEBHOOK_URL: "https://alerts.setkernel.net/meter", ALERT_WEBHOOK_TOKEN: "t" }; + assert.equal(await notify(env, { text: "line 1\nline 2", mention: true }, f), true); + const { url, init } = f.seen[0]; + assert.equal(url, "https://alerts.setkernel.net/meter"); + assert.equal(init.method, "POST"); + assert.equal(init.headers["content-type"], "application/json"); + assert.deepEqual(JSON.parse(init.body), { source: "sk-meter", text: "line 1\nline 2", mention: true }); +}); + +test("an http:// URL is refused before anything is sent, so no secret leaves in the clear", async () => { + const f = capture(); + const env = { ALERT_WEBHOOK_URL: "http://alerts.setkernel.net/meter", ALERT_WEBHOOK_TOKEN: "t", ALERT_ACCESS_CLIENT_ID: "id", ALERT_ACCESS_CLIENT_SECRET: "s" }; + await assert.rejects(notify(env, ALERT, f), /must be https/); + assert.equal(f.seen.length, 0); +}); + +test("a receiver error is a failure, so the alert is retried", async () => { + await assert.rejects(notify({ ALERT_WEBHOOK_URL: URL }, ALERT, capture(401)), /HTTP 401/); +}); + +test("the source is sk-meter unless the alert names its own (another sender)", async () => { + const f = capture(); + await notify({ ALERT_WEBHOOK_URL: URL }, ALERT, f); + await notify({ ALERT_WEBHOOK_URL: URL }, { ...ALERT, source: "other-sender" }, f); + assert.deepEqual(f.seen.map((s) => JSON.parse(s.init.body).source), ["sk-meter", "other-sender"]); +}); diff --git a/workers/meter/test/retry.test.js b/workers/meter/test/retry.test.js new file mode 100644 index 0000000..cd55374 --- /dev/null +++ b/workers/meter/test/retry.test.js @@ -0,0 +1,108 @@ +// A transient GraphQL failure on one dataset: retried, then unknown for this run (flags kept, run +// not failed); a schema error is still an error. Retry waits are recorded, never slept. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { run } from "../src/index.js"; +import { evaluate, message, signature } from "../src/meter.js"; +import allowances from "../allowances.json" with { type: "json" }; +import { readUsage, queryRows, RETRY_DELAYS_MS, SOURCES } from "../src/usage.js"; +import { memoryKV, fakeGraphQL, flaky, noSleep, rowsFor, ENV } from "./fakes.js"; +import { KEYS } from "../src/flags.js"; + +const NOW = new Date("2026-10-15T12:00:00Z"); +const TRY_AGAIN = new Error("unable to execute query, please try again later"); +const WORKFLOWS = SOURCES.find((s) => s.dataset === "workflowsAdaptiveGroups"); +const WINDOW = { start: new Date("2026-10-01T00:00:00Z"), end: NOW }; + +test("the retry waits are 2 s, 5 s and 10 s", () => { + assert.deepEqual(RETRY_DELAYS_MS, [2000, 5000, 10000]); +}); + +test("a dataset that fails twice with 'try again' and then answers is read, after two waits", async () => { + const { waits, retry } = noSleep(); + const gql = fakeGraphQL({ workflowsAdaptiveGroups: flaky(2, TRY_AGAIN, [{ count: 7 }]) }); + const usage = await readUsage(ENV, NOW, 1, gql, retry); + assert.equal(usage.values.workflows_steps, 7); + assert.deepEqual(usage.unknown, {}); + assert.deepEqual(usage.errors, {}); + assert.deepEqual(waits, [2000, 5000]); + assert.equal(gql.calls.filter((c) => c.dataset === "workflowsAdaptiveGroups").length, 3); +}); + +test("HTTP 5xx, 429 and a network error are transient too", async () => { + for (const failure of [502, 429]) { + const { retry } = noSleep(); + const usage = await readUsage(ENV, NOW, 1, fakeGraphQL({ workflowsAdaptiveGroups: flaky(1, failure, [{ count: 3 }]) }), retry); + assert.equal(usage.values.workflows_steps, 3, `HTTP ${failure}`); + } + const { retry } = noSleep(); + let n = 0; + const gql = fakeGraphQL({ workflowsAdaptiveGroups: [{ count: 4 }] }); + const dropped = async (url, init) => { + if (JSON.parse(init.body).query.includes("workflowsAdaptiveGroups") && n++ === 0) throw new TypeError("fetch failed"); + return gql(url, init); + }; + assert.equal((await readUsage(ENV, NOW, 1, dropped, retry)).values.workflows_steps, 4); +}); + +test("a dataset that always says 'try again' is unknown after four requests, and nothing else is affected", async () => { + const { waits, retry } = noSleep(); + const gql = fakeGraphQL({ ...rowsFor("kv_reads", 5), workflowsAdaptiveGroups: TRY_AGAIN }); + const usage = await readUsage(ENV, NOW, 1, gql, retry); + assert.deepEqual(Object.keys(usage.unknown), ["workflows_steps"]); + assert.match(usage.unknown.workflows_steps, /try again later/); + assert.deepEqual(usage.errors, {}); + assert.equal(usage.values.kv_reads, 5); + assert.deepEqual(waits, [2000, 5000, 10000]); + assert.equal(gql.calls.filter((c) => c.dataset === "workflowsAdaptiveGroups").length, 4); +}); + +test("a schema error is not retried and stays an error", async () => { + const { waits, retry } = noSleep(); + const gql = fakeGraphQL({ workflowsAdaptiveGroups: new Error('unknown field "count"') }); + const usage = await readUsage(ENV, NOW, 1, gql, retry); + assert.deepEqual(Object.keys(usage.errors), ["workflows_steps"]); + assert.deepEqual(usage.unknown, {}); + assert.deepEqual(waits, []); + assert.equal(gql.calls.filter((c) => c.dataset === "workflowsAdaptiveGroups").length, 1); +}); + +test("a 403 is not retried", async () => { + const { waits, retry } = noSleep(); + await assert.rejects(queryRows(ENV, WORKFLOWS, WINDOW, async () => new Response("no", { status: 403 }), retry), /HTTP 403/); + assert.deepEqual(waits, []); +}); + +test("an empty workflows dataset reads as 0", async () => { + const usage = await readUsage(ENV, NOW, 1, fakeGraphQL({ workflowsAdaptiveGroups: [] }), noSleep().retry); + assert.equal(usage.values.workflows_steps, 0); + assert.deepEqual(usage.errors, {}); + assert.deepEqual(usage.unknown, {}); +}); + +test("cron: an unknown metric keeps its flag and last value, is listed in the report, and a blip sends no new alert", async () => { + const kv = memoryKV(); + const env = { ...ENV, GUARD: kv }; + const { retry } = noSleep(); + const at = (h) => new Date(Date.UTC(2026, 9, 15, h, 7)); + await run(env, at(1), { fetchImpl: fakeGraphQL({ ...rowsFor("workflows_steps", 300000) }), retry }); + assert.ok(kv.data.has(KEYS.alert)); + const r = await run(env, at(2), { fetchImpl: fakeGraphQL({ workflowsAdaptiveGroups: TRY_AGAIN }), retry }); + assert.ok(kv.data.has(KEYS.alert)); + assert.deepEqual(r.unknown, ["workflows_steps"]); + assert.deepEqual(r.errors, []); + assert.equal(r.metrics.workflows_steps.used, 300000); + assert.equal(r.metrics.workflows_steps.stale, true); + assert.match(message(r).text, /Unknown this run .*Workflows steps.*flags keep their last known values/s); + assert.doesNotMatch(message(r).text, /Can't read/); +}); + +test("signature: an unknown metric with a last known value is quiet, one never read is not", () => { + const usage = { values: {}, errors: {}, unknown: { workflows_steps: "try again" }, gateways: null }; + const fresh = evaluate({ allowances, usage, now: NOW }); + assert.deepEqual(JSON.parse(signature(fresh)).unknown, ["workflows_steps"]); + const prev = evaluate({ allowances, usage: { values: { workflows_steps: 10 }, errors: {}, gateways: null }, now: NOW }); + const known = evaluate({ allowances, usage, now: NOW, prev }); + assert.deepEqual(JSON.parse(signature(known)).unknown, []); +}); diff --git a/workers/meter/test/run.test.js b/workers/meter/test/run.test.js new file mode 100644 index 0000000..e08d184 --- /dev/null +++ b/workers/meter/test/run.test.js @@ -0,0 +1,266 @@ +// The hourly run end to end: fake GraphQL in, KV flags and the Buzz webhook out. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { run } from "../src/index.js"; +import worker from "../src/index.js"; +import { KEYS, REPORT_KEY, DRILL_KEY, readGuard } from "../src/flags.js"; +import { SOURCES, GATEWAYS, buildQuery, readUsage, NO_SCRIPT, R2_CLASS_B } from "../src/usage.js"; +import { memoryKV, fakeGraphQL, rowsFor, ENV } from "./fakes.js"; + +const HOUR = (h) => new Date(Date.UTC(2026, 9, 15, h, 7)); +const WEBHOOK = { ALERT_WEBHOOK_URL: "https://alerts.example.test/meter", ALERT_WEBHOOK_TOKEN: "hook-token" }; + +test("one query per dataset, the account and token on each, nothing else sent", async () => { + const kv = memoryKV(); + const gql = fakeGraphQL(); + await run({ ...ENV, GUARD: kv }, HOUR(12), { fetchImpl: gql }); + assert.equal(gql.calls.length, SOURCES.length + 1); + for (const c of gql.calls) { + assert.equal(c.auth, "Bearer test-token"); + assert.equal(c.body.variables.account, "acct"); + assert.equal(c.body.variables.end, HOUR(12).toISOString()); + } + const by = Object.fromEntries(gql.calls.map((c) => [c.dataset, c.body.variables.start])); + assert.equal(by.queueMessageOperationsAdaptiveGroups, "2026-10-01T00:00:00.000Z"); + assert.equal(by.aiInferenceAdaptiveGroups, "2026-10-15T00:00:00.000Z"); + assert.equal(by.r2StorageAdaptiveGroups, "2026-10-14T12:07:00.000Z"); + assert.match(buildQuery(SOURCES.find((s) => s.metrics.includes("workflows_steps"))), /eventType: "STEP_START"/); + assert.match(buildQuery(GATEWAYS), /aiGatewayRequestsAdaptiveGroups/); +}); + +test("rows are summed and classified: R2 Class A/B, KV by action, DO duration in GB-s", async () => { + const kv = memoryKV(); + const gql = fakeGraphQL({ + r2OperationsAdaptiveGroups: [ + { sum: { requests: 10 }, dimensions: { actionType: "PutObject" } }, + { sum: { requests: 5 }, dimensions: { actionType: "SomethingNew" } }, + { sum: { requests: 7 }, dimensions: { actionType: "GetObject" } }, + { sum: { requests: 1 }, dimensions: { actionType: "HeadObject" } }, + ], + kvOperationsAdaptiveGroups: [ + { sum: { requests: 100 }, dimensions: { actionType: "read" } }, + { sum: { requests: 3 }, dimensions: { actionType: "write" } }, + { sum: { requests: 2 }, dimensions: { actionType: "mystery" } }, + { sum: { requests: 4 }, dimensions: { actionType: "list" } }, + { sum: { requests: 1 }, dimensions: { actionType: "delete" } }, + ], + durableObjectsPeriodicGroups: [{ sum: { activeTime: 8e6 } }, { sum: { activeTime: 8e6 } }], + r2StorageAdaptiveGroups: [ + { max: { payloadSize: 1000, metadataSize: 10 }, dimensions: { bucketName: "a" } }, + { max: { payloadSize: 500, metadataSize: 0 }, dimensions: { bucketName: "b" } }, + ], + workersInvocationsAdaptive: [{ sum: { requests: 40, cpuTimeUs: 2500 } }], + }); + const r = await run({ ...ENV, GUARD: kv }, HOUR(12), { fetchImpl: gql }); + const used = (id) => r.metrics[id].used; + assert.equal(used("r2_class_a"), 15); // unknown actions count as Class A + assert.equal(used("r2_class_b"), 8); + assert.equal(used("kv_reads"), 100); + assert.equal(used("kv_writes"), 5); // unknown actions count as writes + assert.equal(used("kv_lists"), 4); + assert.equal(used("kv_deletes"), 1); + assert.equal(used("do_duration_gbs"), 2); // 16 s x 0.125 GB + assert.equal(used("r2_storage_bytes"), 1510); + assert.equal(used("workers_requests"), 40); + assert.equal(used("workers_cpu_ms"), 2.5); + assert.ok(R2_CLASS_B.has("GetObject") && !R2_CLASS_B.has("PutObject")); +}); + +test("synthetic usage past each threshold sets exactly the right KV flags, and they clear", async () => { + const kv = memoryKV(); + const env = { ...ENV, GUARD: kv }; + const steps = [ + [4000, []], + [5100, ["alert"]], + [8100, ["alert", "degrade"]], + [9600, ["alert", "degrade", "refuse"]], + ]; + let h = 1; + for (const [neurons, want] of steps) { + await run(env, HOUR(h++), { fetchImpl: fakeGraphQL(rowsFor("ai_neurons", neurons)) }); + const keys = Object.entries(KEYS).filter(([, k]) => kv.data.has(k)).map(([f]) => f); + assert.deepEqual(keys, want, `${neurons} neurons`); + const g = await readGuard(kv); + assert.deepEqual([g.alert, g.degrade, g.refuse], ["alert", "degrade", "refuse"].map((f) => want.includes(f))); + } + const refuse = JSON.parse(kv.data.get(KEYS.refuse)); + assert.deepEqual(refuse.metrics, ["ai_neurons"]); + assert.equal(refuse.until, "2026-10-16T00:00:00.000Z"); + // the next UTC day: the window has reset + await run(env, new Date("2026-10-16T00:07:00Z"), { fetchImpl: fakeGraphQL(rowsFor("ai_neurons", 20)) }); + assert.deepEqual(Object.values(KEYS).filter((k) => kv.data.has(k)), []); +}); + +test("KV writes: one report a run, flags only when they change", async () => { + const kv = memoryKV(); + const env = { ...ENV, GUARD: kv }; + await run(env, HOUR(1), { fetchImpl: fakeGraphQL(rowsFor("queues_ops", 850000)) }); + const afterFirst = kv.puts; + assert.equal(afterFirst, 3); // report, alert, degrade + for (let h = 2; h <= 6; h++) await run(env, HOUR(h), { fetchImpl: fakeGraphQL(rowsFor("queues_ops", 850000 + h)) }); + assert.equal(kv.puts - afterFirst, 5); // the report only + // first run with no report clears leftover flags; later quiet runs delete nothing + assert.equal(kv.deletes, 1); // refuse, which was never set +}); + +test("first run with leftover flags and no report clears them", async () => { + const kv = memoryKV({ [KEYS.refuse]: JSON.stringify({ since: "x", metrics: ["ai_neurons"], until: null }) }); + await run({ ...ENV, GUARD: kv }, HOUR(1), { fetchImpl: fakeGraphQL() }); + assert.equal(kv.data.has(KEYS.refuse), false); +}); + +test("a failing query keeps the flag it was holding and reports the error", async () => { + const kv = memoryKV(); + const env = { ...ENV, GUARD: kv }; + await run(env, HOUR(1), { fetchImpl: fakeGraphQL(rowsFor("d1_rows_written", 41e6)) }); + assert.ok(kv.data.has(KEYS.degrade)); + const r = await run(env, HOUR(2), { + fetchImpl: fakeGraphQL({ d1AnalyticsAdaptiveGroups: new Error("unknown field rowsWritten") }), + }); + assert.ok(kv.data.has(KEYS.degrade)); + assert.deepEqual(r.errors, ["d1_rows_read", "d1_rows_written"]); + assert.equal(r.metrics.d1_rows_written.stale, true); + assert.match(r.metrics.d1_rows_written.error, /unknown field rowsWritten/); +}); + +test("per-script rows come from the same two queries: summed into the totals, kept per script", async () => { + const gql = fakeGraphQL({ + workersInvocationsAdaptive: [ + { sum: { requests: 10, cpuTimeUs: 3000 }, dimensions: { scriptName: "setkernel-ops" } }, + { sum: { requests: 5, cpuTimeUs: 1000 }, dimensions: { scriptName: "sk-meter" } }, + { sum: { requests: 1, cpuTimeUs: 500 }, dimensions: { scriptName: null } }, + ], + durableObjectsInvocationsAdaptiveGroups: [ + { sum: { requests: 7 }, dimensions: { scriptName: "setkernel-ops" } }, + { sum: { requests: 2 }, dimensions: { scriptName: "sk-status" } }, + ], + }); + const u = await readUsage(ENV, HOUR(12), 1, gql); + assert.equal(u.values.workers_cpu_ms, 4.5); + assert.equal(u.values.do_requests, 9); + assert.deepEqual(u.scripts.workers_cpu_ms, { "setkernel-ops": 3, "sk-meter": 1, [NO_SCRIPT]: 0.5 }); + assert.deepEqual(u.scripts.do_requests, { "setkernel-ops": 7, "sk-status": 2 }); + assert.equal(u.scripts.workers_requests["sk-meter"], 5); + for (const c of gql.calls.filter((c) => /workersInvocations|durableObjectsInvocations/.test(c.dataset))) { + assert.match(c.body.query, /dimensions \{ scriptName \}/); + } + assert.equal(gql.calls.length, SOURCES.length + 1); +}); + +test("a failing per-script dataset leaves its metrics out of scripts, and nothing else", async () => { + const gql = fakeGraphQL({ durableObjectsInvocationsAdaptiveGroups: new Error("unknown field scriptName") }); + const u = await readUsage(ENV, HOUR(12), 1, gql); + assert.match(u.errors.do_requests, /unknown field scriptName/); + assert.equal(u.scripts.do_requests, undefined); + assert.ok(u.scripts.workers_cpu_ms); +}); + +test("the token never appears in the report or an error", async () => { + const kv = memoryKV(); + const failing = async () => new Response("denied", { status: 403 }); + await run({ ...ENV, CF_ANALYTICS_TOKEN: "s3cret-token-value", GUARD: kv }, HOUR(1), { fetchImpl: failing }); + const stored = [...kv.data.values()].join("\n"); + assert.doesNotMatch(stored, /s3cret-token-value/); + assert.match(stored, /HTTP 403/); +}); + +test("the token never appears in the report when per-script rows are read", async () => { + const kv = memoryKV(); + const gql = fakeGraphQL({ workersInvocationsAdaptive: [{ sum: { requests: 1, cpuTimeUs: 2e13 }, dimensions: { scriptName: "setkernel-ops" } }] }); + await run({ ...ENV, CF_ANALYTICS_TOKEN: "s3cret-token-value", GUARD: kv }, HOUR(12), { fetchImpl: gql }); + const stored = [...kv.data.values()].join("\n"); + assert.match(stored, /"topScripts":\{"workers_cpu_ms":\{"script":"setkernel-ops"/); + assert.doesNotMatch(stored, /s3cret-token-value/); +}); + +test("alerts: the baseline on the first run, then only on change, retried until delivered", async () => { + const kv = memoryKV(); + const hook = []; + const env = { ...ENV, ...WEBHOOK, GUARD: kv }; + const gw = { aiGatewayRequestsAdaptiveGroups: [{ count: 3, dimensions: { gateway: "brief", provider: "workers-ai" } }, { count: 9, dimensions: { gateway: "chat", provider: "openai" } }] }; + await run(env, HOUR(1), { fetchImpl: fakeGraphQL({ ...gw, ...rowsFor("queues_ops", 1000) }, { webhook: hook }) }); + assert.equal(hook.length, 1); + assert.equal(hook[0].url, WEBHOOK.ALERT_WEBHOOK_URL); + assert.equal(hook[0].init.headers.authorization, "Bearer hook-token"); + assert.match(hook[0].body.text, /Queues operations: 1,000 of 1,000,000/); + assert.match(hook[0].body.text, /AI Gateways sending requests to Workers AI: brief/); + + await run(env, HOUR(2), { fetchImpl: fakeGraphQL({ ...gw, ...rowsFor("queues_ops", 2000) }, { webhook: hook }) }); + assert.equal(hook.length, 1, "nothing changed, nothing sent"); + + hook.status = 502; + const failed = await run(env, HOUR(3), { fetchImpl: fakeGraphQL({ ...gw, ...rowsFor("queues_ops", 810000) }, { webhook: hook }) }); + assert.equal(hook.length, 2); + assert.match(failed.notifyError, /HTTP 502/); + assert.ok(kv.data.has(KEYS.degrade), "the flag is set even when the alert fails"); + + hook.status = 200; + await run(env, HOUR(4), { fetchImpl: fakeGraphQL({ ...gw, ...rowsFor("queues_ops", 820000) }, { webhook: hook }) }); + assert.equal(hook.length, 3, "retried"); + assert.match(hook[2].body.text, /DEGRADE/); + assert.equal(hook[2].body.mention, true); + + await run(env, HOUR(5), { fetchImpl: fakeGraphQL({ ...gw, ...rowsFor("queues_ops", 830000) }, { webhook: hook }) }); + assert.equal(hook.length, 3, "delivered once"); +}); + +test("no webhook configured: the flags still act and the alert is logged", async () => { + const kv = memoryKV(); + const r = await run({ ...ENV, GUARD: kv }, HOUR(1), { fetchImpl: fakeGraphQL(rowsFor("ai_neurons", 9000)) }); + assert.ok(kv.data.has(KEYS.degrade)); + assert.equal(r.notified, null); +}); + +test("an http:// webhook is refused so the token isn't sent in the clear", async () => { + const kv = memoryKV(); + const hook = []; + const r = await run({ ...ENV, ALERT_WEBHOOK_URL: "http://alerts.example.test/", ALERT_WEBHOOK_TOKEN: "t", GUARD: kv }, HOUR(1), { + fetchImpl: fakeGraphQL({}, { webhook: hook }), + }); + assert.equal(hook.length, 0); + assert.match(r.notifyError, /must be https/); +}); + +test("the drill: a lowered allowance drives one metric past 80% and degrade is observed, then it expires", async () => { + const kv = memoryKV(); + const env = { ...ENV, GUARD: kv }; + // 500 real neurons used today: 5% of the 10,000 allowance + const usage = rowsFor("ai_neurons", 500); + await run(env, HOUR(9), { fetchImpl: fakeGraphQL(usage) }); + assert.equal((await readGuard(kv)).degrade, false); + + kv.data.set(DRILL_KEY, JSON.stringify({ metric: "ai_neurons", allowance: 600, expiresAt: "2026-10-15T12:00:00Z" })); + const r = await run(env, HOUR(10), { fetchImpl: fakeGraphQL(usage) }); + assert.equal(r.metrics.ai_neurons.level, "degrade"); // 500 of 600 = 83% + const g = await readGuard(kv); + assert.equal(g.degrade, true); + assert.equal(g.refuse, false); + + await run(env, HOUR(12), { fetchImpl: fakeGraphQL(usage) }); + assert.equal((await readGuard(kv)).degrade, false, "the drill expired at 12:00"); +}); + +test("the scheduled handler runs at the cron's time and stores the report", async () => { + const kv = memoryKV(); + const realFetch = globalThis.fetch; + globalThis.fetch = fakeGraphQL(rowsFor("kv_lists", 10)); + try { + await worker.scheduled({ scheduledTime: HOUR(3).getTime(), cron: "7 * * * *" }, { ...ENV, GUARD: kv }); + } finally { + globalThis.fetch = realFetch; + } + const report = JSON.parse(kv.data.get(REPORT_KEY)); + assert.equal(report.at, HOUR(3).toISOString()); + assert.equal(report.metrics.kv_lists.used, 10); +}); + +test("no BILLING_DAY (the sync isn't set up yet): the run fails before touching KV or the API", async () => { + const kv = memoryKV(); + const f = fakeGraphQL(); + const { BILLING_DAY, ...env } = ENV; + assert.equal(BILLING_DAY, "1"); + await assert.rejects(run({ ...env, GUARD: kv }, HOUR(1), { fetchImpl: f }), /BILLING_DAY is not set/); + assert.equal(f.calls.length, 0); + assert.equal(kv.puts, 0); +}); diff --git a/workers/meter/tools/baseline.mjs b/workers/meter/tools/baseline.mjs new file mode 100644 index 0000000..b2f231c --- /dev/null +++ b/workers/meter/tools/baseline.mjs @@ -0,0 +1,78 @@ +#!/usr/bin/env node +// The baseline: what the account already uses, measured the way the hourly run measures it, with +// nothing written anywhere. Also the schema check before a deploy: every dataset, field and filter +// key is looked up by introspection first, and anything the API doesn't have, or a query it +// rejects, is a "Can't read" line. Any "Can't read" line fails the run (exit 1). A dataset the API +// keeps answering "try again later" (after retries) is an "Unknown" line instead: that metric shows +// as "?" and the run still exits 0, because waiting fixes it and no name was wrong. +// +// CF_ACCOUNT_ID=... CF_ANALYTICS_TOKEN=... BILLING_DAY=13 node tools/baseline.mjs [--scripts] +// +// It also prints Workers CPU-ms and Durable Object requests by script (hq#243): the top +// TOP_SCRIPTS of each, or every script with --scripts. +// +// The token needs Account Analytics: Read and nothing else. Pass it from Infisical in the +// environment; it is never printed. CI: the `baseline` workflow (workflow_dispatch). + +import { pathToFileURL } from "node:url"; +import allowances from "../allowances.json" with { type: "json" }; +import { readUsage } from "../src/usage.js"; +import { evaluate, message } from "../src/meter.js"; +import { billingDay } from "../src/windows.js"; +import { checkSchema, introspector } from "./schema.mjs"; + +export const CANT_READ = "Can't read"; +export const UNKNOWN = "Unknown"; +export const TOP_SCRIPTS = 5; +// The metrics read per script that the baseline lists, in this order. +const BY_SCRIPT = ["workers_cpu_ms", "do_requests"]; + +export async function baseline(env, { now = new Date(), fetchImpl = fetch, log = console.log, retry, allScripts = false } = {}) { + if (!env.CF_ANALYTICS_TOKEN || !env.CF_ACCOUNT_ID) { + log("baseline: set CF_ACCOUNT_ID and CF_ANALYTICS_TOKEN (Account Analytics: Read)"); + return 2; + } + const day = billingDay(env.BILLING_DAY); + const lines = []; + let failed = false; + + let problems; + try { + problems = await checkSchema(introspector(env, fetchImpl)); + } catch (e) { + problems = [{ where: "the schema", missing: "", has: [], error: e.message }]; + } + for (const p of problems) { + failed = true; + lines.push(p.error + ? `${CANT_READ} the schema: ${p.error}` + : `${CANT_READ} ${p.where}.${p.missing}: not in the schema; there: ${p.has.join(" ") || "nothing"}`); + } + + const usage = await readUsage(env, now, day, fetchImpl, retry); + const report = evaluate({ allowances, usage, now, day }); + lines.push(message(report, { full: true }).text, "", `billing day ${day}`, "metric | used | allowance | share | projected at reset"); + for (const [id, m] of Object.entries(report.metrics)) { + const p = m.projected == null ? "" : `${Math.round((m.projected / m.allowance) * 100)}%`; + lines.push(`${id} | ${m.used ?? "?"} | ${m.allowance} | ${m.share == null ? "?" : (m.share * 100).toFixed(2) + "%"} | ${p}`); + } + for (const id of BY_SCRIPT) { + const by = usage.scripts?.[id]; + if (!by) continue; + const rows = Object.entries(by).sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1)); + const shown = allScripts ? rows : rows.slice(0, TOP_SCRIPTS); + lines.push("", `${id} by script (${shown.length} of ${rows.length}) | ${report.metrics[id].unit}`); + for (const [name, v] of shown) lines.push(`${name} | ${Math.round(v * 1000) / 1000}`); + } + for (const id of report.errors) lines.push(`${CANT_READ} ${id}: ${report.metrics[id].error}`); + for (const id of report.unknown) lines.push(`${UNKNOWN} ${id}: ${report.metrics[id].unknown}; the API said to try again later, so it is not a schema problem`); + failed ||= report.errors.length > 0 || Boolean(report.gateways?.error && !report.gateways.transient); + + for (const l of lines) log(l); + return failed ? 1 : 0; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + const { CF_ANALYTICS_TOKEN, CF_ACCOUNT_ID, BILLING_DAY } = process.env; + process.exit(await baseline({ CF_ANALYTICS_TOKEN, CF_ACCOUNT_ID, BILLING_DAY }, { allScripts: process.argv.includes("--scripts") })); +} diff --git a/workers/meter/tools/drill.mjs b/workers/meter/tools/drill.mjs new file mode 100644 index 0000000..6077b6c --- /dev/null +++ b/workers/meter/tools/drill.mjs @@ -0,0 +1,37 @@ +#!/usr/bin/env node +// The offline drill (`make drill`): the live drill in README.md, step for step, against an +// in-memory KV and a fake analytics API. Nothing leaves this machine. Exit 1 if any step doesn't +// see what the live drill must see. + +import { run } from "../src/index.js"; +import { KEYS, DRILL_KEY, readGuard } from "../src/flags.js"; +import { memoryKV, fakeGraphQL, rowsFor } from "../test/fakes.js"; + +const at = (h) => new Date(Date.UTC(2026, 9, 15, h, 7)); +const kv = memoryKV(); +const webhook = []; +const env = { CF_ANALYTICS_TOKEN: "drill", CF_ACCOUNT_ID: "drill", BILLING_DAY: "13", ALERT_WEBHOOK_URL: "https://alerts.example.test/meter", GUARD: kv }; +// 1,200 queue operations used this cycle: 0.12% of the 1,000,000 allowance +const fetchImpl = fakeGraphQL(rowsFor("queues_ops", 1200), { webhook }); + +let failed = 0; +const step = (ok, what) => { console.log(`${ok ? "ok " : "FAIL"} ${what}`); if (!ok) failed++; }; + +let r = await run(env, at(9), { fetchImpl }); +step(!(await readGuard(kv)).degrade, `1. baseline run: queues_ops at ${r.metrics.queues_ops.used}, guard:degrade absent`); + +const allowance = Math.round(r.metrics.queues_ops.used / 0.85); +kv.data.set(DRILL_KEY, JSON.stringify({ metric: "queues_ops", allowance, expiresAt: at(12).toISOString() })); +step(true, `2. put ${DRILL_KEY}: queues_ops allowance ${allowance} (used รท 0.85) until ${at(12).toISOString()}`); + +r = await run(env, at(10), { fetchImpl }); +const g = await readGuard(kv); +step(g.degrade && !g.refuse && kv.data.has(KEYS.degrade), `3. next run: guard:degrade present (${Math.round(r.metrics.queues_ops.share * 100)}%), guard:refuse absent`); +step(webhook.at(-1)?.body.text.startsWith("[DRILL]") && webhook.at(-1)?.body.mention === true, "4. the alert says [DRILL] and mentions Raghav"); + +kv.data.delete(DRILL_KEY); +r = await run(env, at(11), { fetchImpl }); +step(!kv.data.has(KEYS.degrade) && !kv.data.has(KEYS.alert) && r.drill === null, `5. delete ${DRILL_KEY}, next run: every flag cleared`); + +console.log(failed ? `drill: ${failed} step(s) failed` : "drill: passed"); +process.exit(failed ? 1 : 0); diff --git a/workers/meter/tools/schema.mjs b/workers/meter/tools/schema.mjs new file mode 100644 index 0000000..82a14ef --- /dev/null +++ b/workers/meter/tools/schema.mjs @@ -0,0 +1,97 @@ +// The schema check: every dataset, field and filter key src/usage.js asks for, looked up in the +// live GraphQL schema by introspection, so a wrong name is found by name before the first deploy +// rather than as a query error. Read-only; the token is never part of a message. + +import { ENDPOINT, SOURCES, GATEWAYS } from "../src/usage.js"; + +// "sum { a b } dimensions { c } count" -> [["sum", "a"], ["sum", "b"], ["dimensions", "c"], ["count"]] +export function selectionPaths(fields) { + const tokens = fields.match(/[{}]|\w+/g) ?? []; + const out = []; + const stack = []; + for (let i = 0; i < tokens.length; i++) { + const t = tokens[i]; + if (t === "{") continue; + if (t === "}") { stack.pop(); continue; } + if (tokens[i + 1] === "{") stack.push(t); + else out.push([...stack, t]); + } + return out; +} + +// The filter keys a query uses: the window bounds plus the source's own (eventType, ...). +export function filterKeys(source) { + const own = [...(source.filter ?? "").matchAll(/(\w+)\s*:/g)].map((m) => m[1]); + return ["datetime_geq", "datetime_leq", ...own]; +} + +// LIST / NON_NULL wrappers down to the named type. +const named = (t) => { while (t?.ofType) t = t.ofType; return t?.name ?? null; }; + +const TYPE_REF = "kind name ofType { kind name ofType { kind name ofType { kind name } } }"; +const TYPE_QUERY = `query T($name: String!) { __type(name: $name) { + name fields { name type { ${TYPE_REF} } args { name type { ${TYPE_REF} } } } + inputFields { name type { ${TYPE_REF} } } +} }`; + +export function introspector(env, fetchImpl = fetch) { + const cache = new Map(); + async function ask(query, variables) { + const res = await fetchImpl(ENDPOINT, { + method: "POST", + headers: { "content-type": "application/json", authorization: `Bearer ${env.CF_ANALYTICS_TOKEN}` }, + body: JSON.stringify({ query, variables }), + }); + if (!res.ok) throw new Error(`introspection: HTTP ${res.status}`); + const body = await res.json(); + if (body.errors?.length) throw new Error(`introspection: ${String(body.errors[0].message).slice(0, 200)}`); + return body.data; + } + return { + async root() { + return (await ask("{ __schema { queryType { name } } }")).__schema.queryType.name; + }, + async type(name) { + if (!cache.has(name)) cache.set(name, ask(TYPE_QUERY, { name }).then((d) => d.__type)); + return cache.get(name); + }, + }; +} + +// Problems as { where, missing, has }: `where` is the dataset (or the account type) and the path +// that couldn't be found, `has` the names that type does offer, for the fix. +export async function checkSchema(schema, sources = [...SOURCES, GATEWAYS]) { + const problems = []; + const field = async (typeName, name) => (await schema.type(typeName))?.fields?.find((f) => f.name === name) ?? null; + const names = async (typeName, key = "fields") => ((await schema.type(typeName))?.[key] ?? []).map((f) => f.name).sort(); + + const viewer = await field(await schema.root(), "viewer"); + const accounts = viewer && (await field(named(viewer.type), "accounts")); + if (!accounts) return [{ where: "viewer.accounts", missing: "accounts", has: [] }]; + const account = named(accounts.type); + + for (const s of sources) { + const ds = await field(account, s.dataset); + if (!ds) { + problems.push({ where: s.dataset, missing: s.dataset, has: (await names(account)).filter((n) => /Groups$|Adaptive$/.test(n)) }); + continue; + } + const filterType = named(ds.args?.find((a) => a.name === "filter")?.type); + const offered = filterType ? await names(filterType, "inputFields") : []; + for (const key of filterKeys(s)) { + if (!offered.includes(key)) problems.push({ where: `${s.dataset}(filter)`, missing: key, has: offered }); + } + for (const path of selectionPaths(s.fields)) { + let typeName = named(ds.type); + for (const [i, step] of path.entries()) { + const f = await field(typeName, step); + if (!f) { + problems.push({ where: [s.dataset, ...path.slice(0, i)].join("."), missing: step, has: await names(typeName) }); + break; + } + typeName = named(f.type); + } + } + } + return problems; +} diff --git a/workers/meter/wrangler.toml b/workers/meter/wrangler.toml new file mode 100644 index 0000000..7322970 --- /dev/null +++ b/workers/meter/wrangler.toml @@ -0,0 +1,33 @@ +# sk-meter: the Cloudflare billing guard. One hourly cron, no routes, no public URL. +name = "sk-meter" +main = "src/index.js" +compatibility_date = "2026-10-01" +workers_dev = false +preview_urls = false +# Every var and secret reaches the Worker through the Infisical -> Cloudflare Workers secret sync +# (project sk-ctl-workers, env prod), never through this file or CI. keep_vars stops a deploy +# from deleting what the sync set; there is deliberately no [vars] block. +keep_vars = true + +[triggers] +# 7 past the hour, clear of the top-of-hour rush +crons = ["7 * * * *"] + +[observability] +enabled = true + +# The guard flags (guard:alert, guard:degrade, guard:refuse) and the last report. Every Worker that +# reads the guard binds this same namespace as GUARD. It is the `sk-meter` namespace, owned by tofu in +# setkernel/setkernel-cloudflare (output sk_meter_kv_namespace_id): bind it, never create it. +[[kv_namespaces]] +binding = "GUARD" +id = "f79ca12aeb9347f9851f5dfd53313f39" + +# Set by the sync from Infisical sk-ctl-workers/prod (never in this file): +# BILLING_DAY the day of the month the billing cycle starts (13); required +# CF_ACCOUNT_ID the account to meter +# CF_ANALYTICS_TOKEN Account Analytics: Read, on this account only +# ALERT_WEBHOOK_URL https://alerts.setkernel.net/meter, forwards { source, text, mention } to Buzz +# ALERT_WEBHOOK_TOKEN bearer token that endpoint checks +# ALERT_ACCESS_CLIENT_ID the sk-meter-alerts Access service token in front of that +# ALERT_ACCESS_CLIENT_SECRET endpoint (hq observability/README.md, "Meter alerts") From 553630100ed2e9cbad72252459ba98c6c5381445 Mon Sep 17 00:00:00 2001 From: Raghav Gupta Date: Mon, 9 Nov 2026 09:14:17 -0400 Subject: [PATCH 2/3] feat(status): sk-status's code moves here as workers/status, not deployed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The /status read model (Board Durable Object, views, Access check, phone page) moves from the retiring control repository at c1e3253, plan v6 F3, with its tests. It has never been deployed and its only writer is being retired, so nothing deploys it here; its README lists the first deploy's owner steps if the page is wanted. Refs setkernel/hq#1449 โ€” Claude Lead Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 8 +- package.json | 5 +- workers/status/README.md | 70 +++++++ .../public/status/fonts/OFL-Manrope.txt | 93 +++++++++ .../fonts/manrope-latin-400-normal.woff2 | Bin 0 -> 14108 bytes .../fonts/manrope-latin-600-normal.woff2 | Bin 0 -> 14172 bytes workers/status/src/access.js | 85 ++++++++ workers/status/src/board.js | 93 +++++++++ workers/status/src/html.js | 50 +++++ workers/status/src/index.js | 111 +++++++++++ workers/status/src/model.js | 129 +++++++++++++ workers/status/test/board.test.js | 122 ++++++++++++ workers/status/test/cloudflare-workers.js | 14 ++ workers/status/test/fakes.js | 81 ++++++++ workers/status/test/model.test.js | 91 +++++++++ workers/status/test/setup.js | 11 ++ workers/status/test/worker.test.js | 182 ++++++++++++++++++ workers/status/wrangler.toml | 47 +++++ 18 files changed, 1189 insertions(+), 3 deletions(-) create mode 100644 workers/status/README.md create mode 100644 workers/status/public/status/fonts/OFL-Manrope.txt create mode 100644 workers/status/public/status/fonts/manrope-latin-400-normal.woff2 create mode 100644 workers/status/public/status/fonts/manrope-latin-600-normal.woff2 create mode 100644 workers/status/src/access.js create mode 100644 workers/status/src/board.js create mode 100644 workers/status/src/html.js create mode 100644 workers/status/src/index.js create mode 100644 workers/status/src/model.js create mode 100644 workers/status/test/board.test.js create mode 100644 workers/status/test/cloudflare-workers.js create mode 100644 workers/status/test/fakes.js create mode 100644 workers/status/test/model.test.js create mode 100644 workers/status/test/setup.js create mode 100644 workers/status/test/worker.test.js create mode 100644 workers/status/wrangler.toml diff --git a/AGENTS.md b/AGENTS.md index fe6edf2..f33b2d4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,6 +10,8 @@ The public page at https://live.setkernel.com: a Cloudflare Worker with static a Durable Object. sk-obs01 pushes filtered, delayed events to it; visitors watch over a WebSocket. It also holds `workers/meter`, sk-meter, the Cloudflare billing guard (an hourly cron, no route; `workers/meter/README.md`), moved here on 2026-11-09 (setkernel/hq#1449). +`workers/status` is sk-status's code (the `/status` read model on ctl.setkernel.net), moved the same +day; it has never been deployed and nothing here deploys it (`workers/status/README.md`). Public repo: everything here is visible to anyone. ## Commands @@ -24,8 +26,9 @@ Public repo: everything here is visible to anyone. # and repo names against GitHub's public list, live) # worker: npm run test:worker (vitest in workerd: the real Worker and Hub, see test/worker/) # meter: npm run test:meter (node --test: sk-meter's guard, usage queries, alert, baseline) +# status: npm run test:status (node --test: sk-status's board, views, Access check, routes) # lint: npm run check (node --check on every script) -# validate: npm run validate (check + test + test:worker + test:meter + a wrangler deploy --dry-run +# validate: npm run validate (check + test + test:worker + test:meter + test:status + a wrangler deploy --dry-run # of each Worker; what CI runs) ``` @@ -100,6 +103,9 @@ checks that its version serves all traffic and rolls it back in the job if not. - `workers/meter/`: sk-meter, its own Worker (`wrangler.toml`, not `wrangler.jsonc`): `src/` the hourly run and the guard flags, `allowances.json`, `tools/` the baseline and the offline drill, `test/` its `node --test` suites. Its settings come from the Infisical -> Workers sync (`keep_vars = true`). +- `workers/status/`: sk-status (`wrangler.toml`): `src/` the routes, the `Board` Durable Object and the + views, `public/status/fonts/`, `test/` (its `setup.js` resolves `cloudflare:workers` to a stand-in). + Not deployed; its writer, the control Worker, is being retired. - `test/`: `node --test` suites for the pure modules and the page (`day.test.mjs` for Day). - `test/worker/`: vitest suites run in workerd by `@cloudflare/vitest-plugin` (`vitest.config.mjs` reads `wrangler.jsonc`). They fake Access's certs with a throwaway key and drive every route diff --git a/package.json b/package.json index cdcc6a4..b7a69da 100644 --- a/package.json +++ b/package.json @@ -8,8 +8,9 @@ "test": "node --test test/*.test.mjs", "test:worker": "vitest run", "test:meter": "node --test workers/meter/test/*.test.js", - "check": "for f in src/*.js public/*.js public/lib/*.js public/usage/*.js scripts/*.mjs workers/meter/src/*.js workers/meter/tools/*.mjs; do node --check \"$f\" || exit 1; done", - "validate": "npm run check && npm test && npm run test:worker && npm run test:meter && wrangler deploy --dry-run --outdir .wrangler/dry-run && wrangler deploy --dry-run --config workers/meter/wrangler.toml --outdir .wrangler/dry-run-meter" + "test:status": "node --import ./workers/status/test/setup.js --test workers/status/test/*.test.js", + "check": "for f in src/*.js public/*.js public/lib/*.js public/usage/*.js scripts/*.mjs workers/meter/src/*.js workers/meter/tools/*.mjs workers/status/src/*.js; do node --check \"$f\" || exit 1; done", + "validate": "npm run check && npm test && npm run test:worker && npm run test:meter && npm run test:status && wrangler deploy --dry-run --outdir .wrangler/dry-run && wrangler deploy --dry-run --config workers/meter/wrangler.toml --outdir .wrangler/dry-run-meter && wrangler deploy --dry-run --config workers/status/wrangler.toml --outdir .wrangler/dry-run-status" }, "dependencies": { "@noble/secp256k1": "3.2.0" diff --git a/workers/status/README.md b/workers/status/README.md new file mode 100644 index 0000000..f53c85c --- /dev/null +++ b/workers/status/README.md @@ -0,0 +1,70 @@ +# sk-status: the read model of the org + +A page for Raghav, the internal wall and the showcase's publisher: sk-meter's guard flags and +the last 24 hours of merges and failures, on `ctl.setkernel.net/status` behind Access. + +It moved here from a private repository on 2026-11-09 (setkernel/hq#1449, plan v6 F3). Its +history up to then is that repository's `status/` at commit `c1e3253`. + +**It has never been deployed, and nothing deploys it from here.** Its only writer was the +control Worker, which publishes over a service binding (the `Feed` entrypoint) and is +being retired in hq#1449. Without it the board stays empty and the page shows only the guard. +Deploying it is Raghav's call ("The first deploy", below); until then this is code and tests +only, checked by `npm run validate`. + +## How it works +- `src/board.js` `Board`: one SQLite Durable Object (`BOARD`, name `org`). Each writer publishes + its own view here after it changes, and `/status` reads this one object. + - An item applies only with a higher `rev` than the board holds. Reaching `merged` or `failed` + adds one row to `events`; `closed` and `new` take an item off the board without one. + - Each publish prunes ended items and events older than 24 hours. No alarms, no WebSockets. +- `src/model.js`: checks a published batch against the state list (`STATES`) and builds the two + views. One bad entry is rejected with its index and reason; the rest of the batch applies. +- `src/index.js`: the routes, the Access check, the guard, and the `Feed` entrypoint. +- `src/access.js`: who may read (`STATUS_CLIENTS` scopes) and the Access token check. +- `src/html.js`: the phone page, no script. Manrope is self-hosted under `public/status/fonts/` + with its licence; the last fallback is `sans-serif`. +- It imports one module from outside this folder: `../../meter/src/flags.js` (the guard). + +## Routes +All on `ctl.setkernel.net`, behind an Access app for `ctl.setkernel.net/status` and +`ctl.setkernel.net/status/*` (not in setkernel-cloudflare tofu yet). The Worker routes are the +same two patterns. + +| Route | Who | What | +|---|---|---| +| `GET /status` | a person | HTML when the browser asks for it, JSON otherwise | +| `GET /status/internal.json` | a person, or an `internal` service token | `{ generated_at, snapshot_at, guard, last_24h }` | +| `GET /status/public.json` | anyone let in, including a `public` token | `{ generated_at, last_24h: { merged, failed } }`, built field by field | +| `/status/fonts/*` | (static assets) | the two font files and `OFL-Manrope.txt` | + +The Worker verifies `Cf-Access-Jwt-Assertion` itself (RS256, setkernel's team, audience +`STATUS_ACCESS_AUD`). A token with an `email` is a person and sees everything. A service token +gets in only if `STATUS_CLIENTS` lists its client id as `internal` or `public`; a `public` token +gets a 403 on anything but `/status/public.json`. With `STATUS_ACCESS_AUD` unset, every request +is a 401. + +## The guard +`/status` reads sk-meter's flags at most once a minute per isolate. Under `guard:degrade` +snapshots are reused longer (person 60 s, service token 600 s; normally 15 s and 120 s). Under +`guard:refuse`, `/status/public.json` answers 503 with `retry-after`; people still see the page. + +## What it costs +Estimated with two pollers at one a minute: about 176,000 Workers requests (1.8% of the +included 10M) and up to 190,000 Durable Object requests (19% of 1M at worst) a month +(https://developers.cloudflare.com/workers/platform/pricing/, +https://developers.cloudflare.com/durable-objects/platform/pricing/). + +## Commands +```sh +npm run test:status # node --test, with cloudflare:workers resolved to test/cloudflare-workers.js +``` + +## The first deploy (Raghav, only if the page is wanted) +1. setkernel-cloudflare: the Access app for the two paths (allow Raghav; two service tokens for + the wall and the publisher), `setkernel.net` in `deploy_tokens["setkernel-live"].zones`, and + `workers_admin` for one deploy (Workers Editor can't create a Worker). +2. Infisical `sk-ctl-workers/prod`, folder `/sk-status`: `STATUS_ACCESS_AUD` and `STATUS_CLIENTS` + from tofu outputs, and a Workers sync from that folder (never `/`) to the Worker `sk-status`. +3. A deploy job here, like `meter` in `.github/workflows/deploy.yml`. +4. Remove `workers_admin` again. diff --git a/workers/status/public/status/fonts/OFL-Manrope.txt b/workers/status/public/status/fonts/OFL-Manrope.txt new file mode 100644 index 0000000..462280f --- /dev/null +++ b/workers/status/public/status/fonts/OFL-Manrope.txt @@ -0,0 +1,93 @@ +Copyright 2019 The Manrope Project Authors (https://github.com/sharanda/manrope) + +This Font Software is licensed under the SIL Open Font License, Version 1.1. +This license is copied below, and is also available with a FAQ at: +http://scripts.sil.org/OFL + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/workers/status/public/status/fonts/manrope-latin-400-normal.woff2 b/workers/status/public/status/fonts/manrope-latin-400-normal.woff2 new file mode 100644 index 0000000000000000000000000000000000000000..7eaaa9586760977088706a11f8ffb6eda257be56 GIT binary patch literal 14108 zcmV+%H{-~6Pew8T0RR9105==}4*&oF0F0af05-b-0RR9100000000000000000000 z0000RvS=KW3I800A}vBm;3SOP11aI_U)ghC6g-1|*3@`YSf%J)y847Y-1JoXxxEj3Dss88B-=$eEcQA2LSYqx zNrVk)<&n{^`4?ysx`z zKG6CatzaxC%mEF1X8ec!`D=9xX~`XG6-vR${9t-G_nwLV^XvMvblcC6q-Z7H5uOIh z?GxeczBH|xs0xiqW&*$OUjIC><<2&{i;=~fS2Lysg`GyBs|y3*$7}A;-*<_noHNXpLgMfj z-BcWj$xGCkx?YC0olF(Ds{H>p^=s!@nZMFUPtg7q`W!7nX?_32Dp2jOx5VntLq3b9I2*n^AI2>OxNR;jh9{&GO`JMo!TR#R1 zRx0vO_RH_!dHiEeDAqNBO1w4TokCS8^qV}Id$tDXAFO2JL?my%@uGoryZb1iNDvW` z5JXDez3r>JY=x_;M>7Qu&Ox?moR{@C$kxnN8*dW<4B{*hvFGRU z6ZCg^%s~HSrUEn)up-vE{yP@nf$z;o_@%@n|A7D==86O(K8(w_aJJ7dMl?mn%4w4f z$$j-3kj9aeolSC!DR&))tnxdWFne|~&}f8V5aO+lV7q?rOlKSaO z@*q<9w$dnz8e43tBmp6JkTe4*aDWb|se1wFLDLuq*|BYjdX8f5_8S6B!3PahZMi=d zjKf|ta{Sf&BtY79graXt-4s*kJOdjJ6TId_uwc#hX%uW-Fn+{N+f(+{`pi6W2dqOX+dppZjNUD#wa<7MLp(_|kg+W>qu`Z-Mj&+cq8*b(OQ?jegZo^0C^z?R6ku0+ zjqE5qj{FJx!%NNFU|Sk%=je;8t#9eD99A-lZCe#*i3_%iPgNCPs7$@`yLe`{`-CY< zE1c00%KMkKZX`Zj2E|7PkAc3l-5z(XWo0iLsBkA7TzP@DT$7_-b;d*BLy}<7m;pm3j8d@io)8%rq(B`RGi1CRZxFP8Ad0=>1+hejiiN9G*xVh@PJ?lGZ=|o zme?VlaSY^OAO{0E7|6ju4%+BQd;?fIlfOU?BY{(wi3zVWul3j1{(Bzn?%r6uv?rqT z{i!W$E1>f?dl^gGgs5-WPEWZq$PQ2UlTYSEnK;L1ak7UoxIs?beQ=t)#Tp-}ySTG= zqFQV^%FgczNIwWh{J?q+9vVZ=nJbT|OF+nPg6 zFR30jM?)i1Oj}=>PF1&#J{{>^II6be?^2vKoe->0VZud3q(oim)DDWVZac6iXBfiV zDcP`P$DRX6PF#~mgdhJCP9B=zz{r?z8xt+wa9{|nrz`{#2Zf7=U(0f(GEOec8X77B z3xUKz;o{-%%Dl>FYCs^ciXdXrWVRwzC8-a3LS!L75>p4rCLxB<$N~TWfMpokv7WB( zUO=krxGW9bCT>@ggoSygY8Lv_Y6_*o2qX>)7Z1Od0r%4xAz#a8OWiqhsiXd1Kc-EC zlxLp+0t(Oo73cvi&<6$p1A+U2K~s)OwGygQ>EFGaGstZ$!i(j&&G>=1<6=m<8xLZK zXVq98A>H&w=4TDCqEaq`3D%N~{dbfe)3)jvmWSew+3jS%VnafxNUqAaPK*c=!jh90+SjmRN)e6D~res1%b=@e(8j+9D@LnxxB+DGLib zl~I{uT2!i5b@lY53kjOEXwxxG>W;XTs*DGPxU{)}*%b#U4Hg24gTlqb?+g?n7r{!X zFySIZib^p#7AsCdBtDWPONpF1awSi`0)>jG%kUJ1xl~qpotI{Jlp|!@|EgtK?ssSy zVcJc<+Do;5U2USGy?<_k-T%qHh#6x5LO}olXgUxe zApjTDjRA|7{NXJW_s6Y%rl0=T|H{H?6S2#8uOrhNR33c3i1%V5e68BF>(HsoBahv6 z`Tz4SXw;-xiyplm3QsB+)vs<({~tj^0WngjcW)2B`w)cEU%3*hRi|Eq;(ga#SACw) z5ht8-TK(=Nmkosa%o|SQ&Tsn`E^WhCxfJ}kN7r^?yY<3&fnf! zck?gvCf(qxyr$<*KRMUV_8C6XlXnOQb`-}DJ!`f0gpmqQ=8qSyIBkoCN)b@65W==S zFqD=wiSp7$+x$8W888I0A6*yc$EClIxG@s$SMGv_*m>8^acfkf3d}4vV3B zR;sR^z$6@u!gCp!ZK;(K8WY`Vdtjt?>S)_R>scuIiV-$J#tb4^HkT{IhL_GngkAKL zB96#gLxN$Ji5YLM?4_SmV5lWD7=aSxa5Tac1DA)z5Dc-07+W$&n@+QD%OM2tl-W|f zhd}@vog!AiE_YppUm=%>>WaD}_P|hq%kVR-8(o+mhSOqu+YD5k+TJjI;`s7Kj-29( zQ5=&|Xv1~FyV9k_!CU~EZ*^R1KCe7VoE|z`lOT4x+iec>@B)xpKW&Q=D=aIH|dx?Ns`=1f&)EgT+l}3GiWryD6ycu@H`R^ zk87WelC2+>Gucv0l3{G5dorX~NIx`q4l_h~()~Do3j0opF)Az(zPsodLW~O#(sWoc zl8gmY0@GlbrovP-uvAkJQe{FYsfwM~?YSSGYFm^NKxQo8#!wbp6RBXiu2mvfmvk7R zTBWy`gT(xDhdLbXsxlS2+Ju(z*T31mx&`Y91*0eftKyS?Gf#=%CE^qPfABIc! zlKGx%Z%v&3db&eBqgh|P0sr@c0cQ^F+ku`3)D5cxW*{NcFnF8n6!@M$URnb{K>#HP zv?c?WgwAC?Z_hu^E>I&I`;u;_xv)>2tKy8loyW<`>=dFK1!o!y_J!gHg?&&JkKc-Cn z@J{5&+Yb>iV8jH?oCQlV3^^+bO3qxlap%GFM?l94)~rU0Kkk&CkV@i$KloW$Wv~pH zU;f$-HQWdz6&r1Wi6$vASz~EKlV5&y!fWrn@gWQmg-j3^X6W(T&C)>T3E-zgc80#c z!GY6ZV!N4o9dgiNr%*5w4+j^YAt6(Jn2{0@lQ3q@o()@$?0n(Mmltn7g18tZ94l0$ zFcG4}O8FtywGYxye*?tJR#15nHS3Cu(=xb;B*U-Gre0HyjWX0A&S; z55f3p0Stu^4`FgcP!O!O(foG{q1Iy*+jKe4UmXQRSQxN_(IKp)olvg% zpc~z^KGR_$LJs1KS>xHtJ+3M;R*8`a(rjHjBi-BJa%{9I=%_s~E_SsE%Mqm6!M0L9 zT;gD1XS!2^2sKd+SdbA+wf9VzLqyURCJAQonR^&(A9_c{yA{p+7ss3Q|%ewKg`uLmZ{flUE;1;|i0 zj`Mm6TAs`k(jEk-m@MZ?RH(f~El~R51PRSHwW^jG9#)bOki<}zqGf*SketuoAHc1b zTo?d!+7TysAgK4?f+Fw5veYAQ-k#&OSpm! z)2fNTo>vIV5fFH-`{u^`Mp#`q8ms}=6Snng=$!DrobVktLK9{KfQz(8YirG5@1(-N zczegg=kK9$#bDJgE)!dlR&*7tMN+J$j84BGRxcjQ^-C1#`hN`{Ce?$ z$ep}3IF-+3^FM)lT;U)6y$u`}{!!ch0kq>`6Wpa<$wv@xC3(O)UU=^<$}WvhTpQDH zfQ+y_npM~o)4cm64Y)Q%B$h01szvtPraUAHabh+MK9wNe{=kN1mV-l58PXwiM+imi z`1V~lF8xng0`Kjb%D8Ec!rIe>4%;zg+Ab(Acjm(pot~UD{6f3go*qqigm@stY|NmF zY~1Ax+5 zlj&J`dS^hd(JV{PBy>R*1%sAQbR;B-0lP%ik6#piP3doZ%!412$UvuTcLd4kzCq0; zG_RRS=%krPY=>qMnRkrH{Ve^{|tMYD;aqL-MKBI>*AV>Btuf(``T zd==#uIZal~#|TSbHa(fMlC0M^wVv5e`0);N=z?}A$%^1>DyhVYrQu5KP^A^Y7f$aL zp(~UsDnc`;Tk`;%{YVhf$Fo`*2hVr-fYY=>;GIV}IMn2&rbHpjE$SOONc(uDv3nL~ zVHv}pNYmWD>Vm+M?0)6Zq6Q()VwP4yx+GET1&WR^R!PQNV=VK=y-9JnlqL%8rB+Bs z_)w~hG*yXyJc7CCyn>3KIyYdpB$WxN`Ei*i^tkf23HFl7PCZL0%mDBd-3ZgL%~tu$ zogH>#guQ}V6IV<$P*tJBd|2AgOH3A4hf6|O73ZzLa zd}4W7e24lH1;S#fly3So7Oi;+Qxj2zIERt}VlCNtYjcly;)N&65W1{@Ynt|CC!akQ zFI6l(JKk76_>Hy~kU%cGC0LC2yH-;H0GCKRiht6sqm8bKVr&Sd2o8(uw?Q^ zFx>+YwcGGI*k7y4eA6|DQdE$yB5OH;(Beh7Z)J-prP)gB9K=|#cA&C`=2+Qc!9W6w zb%ntq_>@KiL8{!S;Xs$^u~@OH0^&VT$&d+QQ&C(99Xu*7T(@p+Wp6R6cm$1$SVelr z>R9ilw7t$r<6N#5H!)!s()1dBat?g1rzXc^WL8N!CSG938VnI{Xx6)cWwzU8*Xu;K zMYIWX%Gmw}mX07}q5KPxV`CxU+k_MV}9B)Km{_K zHE-1!r|guB(d5%3Qb@#T%TDW!!|{9n1Mr}Kmsy!>gmoZn-z=Ee_-=yOSZ%Bmg7MRz z4&X|elI7UspsgIdfJxV96?`w%dAo@D)HdO$U&78c3mV_{vH!&N;WMXl`+=z4N7~OU zO)W(k2k79OWi!MEy4Y}5%9lhi=N1@1#y&#tI&P{1stEaRcPtEYcJJy~Xv4(~bkd=W zh|G&`4;;b(3K_OGWJ4J6E1#530`Iy2k?c2Wz>rG|9fJX3Lh`>yNQD@#?Q5wfKyY2u z?il-Z^PIVy+|lQmd`#iaqCTU{!n|Sk%Waz;=bl#Qc=C0hl99xIqZodF-9TOV3e`Oi z9}W5A2R=e%dut2Hl+W5%%tMCg%C9)+56C8d-MYu=_$xl{ zIFnmyi}|V*F+N1HLs7CjFbm%fBR$LUK_+q^u+tDJ7l>>kooi`8FYPTK3rIEqUuZwQ zciQ!ZTIIsxnt~d&7B%g{AsemX^C4Ove9Sq%6flDVu^2mu({7Xuy?PKVGdr>a6BH!t z?+r?12l^Ox$3?r@xozItJ-nX+Ylu{ZNPaP(*|M(;;!1q*a*NXSO3IA6`AGW19?*C< zE~nXhnx#jFKYuA`zbfIR&Cs=0C-LNE&GH2^nKM#BrR1|l7@5Rwpy`R|HA}DBq1fEk*GpR|%G~Ql zbUJzwS*t{Vd|#tvB_N}k!!?_vDpZpxmDLZ$mvf7wj(XajEH!0K&Ovg(_TU~2yHxBh zwZ>QIZies`3AUWu9A3viJmIhpD>6%VAO9cG<^21FZr4BISM%@BC*C)Ou)V zO+`o;vZ*iume5=j1B+=4hlN8M9N?Ntd||%LgHO3r9$0&%SB$-lD{3m+godl4d5$(ZE`9TKQ?b%O7YRNpvw))r!@;bbQRW4+ zEgUAOD^t9BRiUU*f29Cz!@mvoD&JzU=D9_c+dK!oR@d=w+pBZ7&t}$KZkwXDM&QH* z9HK^o!k=y(h!;h~2$HyMHG%7c%qVWa%i1Rd2 zg4%~o^C2#WOn$s9@H=*lfP?%gy|GBi*s9IiJx#@=+IV<8@!w@#iT`apSY>RIq2g){ z3_B$y63tG0t4)Re|Gg?#l&k+2^Psrpe%#fibKesEe&lnVre`|+-OD}82Ej;(q{9p3 zL$kzf+y~0xO&ZoS|Hil5w-H>k9(yFJ);k0Z2x@8HBdcsM9D z0D76M-s9nUFw$0<{ZMTXC3)eZq2w6Tm}1n` zUUVJf?rKgq^2$d<_@TXBoSk<3%22T5it@4Vvs1tQ(I?n&mC<*W9!)w4mQLredR+j#!s zc-Ioq7IdKIz>~%PIjUg>{$-CIH?8S2b8LM%HkmsYlU_-NxniON&<@DFyl8`hPy)fb zAgQ8e@L6fgf74G?Bpq-ymxT6iaV6Fg0^35Z{; z8~H_K(zE}Xlx^h~_~!qL4-UR>UTJ8-IqQmPD!z+COQ@>A;*9Wy z-`JcnNz{r|8m}FiEEr_j@Y~&;IvkGRSIb74K z*XT5IM8#=p8n3j7|05&nZAKn+75=|=Kn(ezV`%L5Fvmc7yCYEfQvZ{sqQ3lui_!}V zzt82&aG}|?fv#z?`lYz=^_rrsY!{rwK%m4>b`I2BFV+^gBoUFRHC&!#>)X|CMqF`z zoYcNddn!berd9!)uElY4r|CBQRu6Ngq;5N2Fn_hvLHd&_)g_|YQ`B95IJi+yq9Xx9 zrtf-xj?(0GF1Xn<*t^pwgiKC}Ab0z#tbwYoSFGnv$QAX8ke4be5EuJJMWS8-;zCU| z;!(c>A` z@v}SLW=}4d-j<_uycH-#HEMyU~y?5-?;oJKz zerBRwa#AsX#EiXYt@(OjqZ;z8b#&r!w^(20bbp}o?d7|(tH&PSzIN=jPe_U-0QQwQ;oF$)O_U+L=u;SSaVI5u0Z0`3>hzH7mt1H--_DJ+4LF(pAbwtEMYXEG~(d{r#V>()P!FZxq_*nhYN6RqFJ$-eJ?wi#9E<`P$M*#3y-O|p7iw__x*%UJ+ zamLDGM&6-|-z|Yh^m8Ckrlkjc1JRD$QkgrBLc-}GIw!QZE*-Ps$x5`KJ zQ0F8Uw|)nX&y+e!+T377bl~F7Xr$r+dY8*v?G9b^S!k4Wot3DH=I*bc^MDe`y!HNwL9SkWG%mQ8_LLes zCX(6IsrSF1TgX53yk$U51lj^+DqXfAIP=~D7m?Z@Evv1%YxJNZ{kkYiU}JUW%^}OK zJ~Pvl#XL!+f=l_6~e^mJG~>ARQzfLjcMb@i55$Hc2D71*D< z@5L!G92{dt&BX>@u_2O665wVLMUUC*O1B>fwR1XCuH%J;mM*jh2HcGD{)oK`z#OW- zoA5_Xo&6QR!;{laj+&~f%Q0@h9W^7TuEMkKx=04=GkjfUxKlQ8KLRMEr$f4do^qm| zVPXSi^c)kr^)bPudQX{pJ4|3SV?S!V9y1*|z0P0LS^JHob>up5et0WbY0kR0dDJZS zYLp4}66&+*MH63H+S(s!Vk%Z!)u?Q}a zjN$-~$3tGjckX*x9iGo+9BuV@_nI9s)=^4{tXNb4@@=` zNh733E#N`YAuV`BS4Pj`W}7S7CX|!8>Tac($W~iLktEUcvD&` zedq%e{9x~_Khgo~us7BRF-+qPe$j>chT2=QAoF}T@o`a^VEIgHRn^&4!E%`>|ES0V zAq$kb{Y^1Yg>dvbaGgP)}`#ZN>MfTT1m4M~hu zQb0b_uruOt+e*XmOSA?Gp%7XFrGb)mxqPM~tkW1tC;^$0L~6+V@N=yUo*@t2GfWsC zxWF>DpW24Uccefw@Le5*)Z|+DwzJn59*1tAQBg8kGEkDv zjPu>ZHEv?Nufnf6{L~rH$juq6w#>@4)Qje$BMAi}Y;Yir77!Rl@)Kt{TkJi{S^d_i z`FC`lZ#YqgLCHx(VpAI)oQF``2>AAnx*U`={JVQqQ0*lM6L_JJQX(Oqe8Wsjjfm#M ze0{!AfH@dPR9Sp-C6PjslkAietr6lR@t~}%3~+;(pKPf_b|A|r#IA|U<0-k=Oe%q# zLBZiNDP%$x6>~0u6rVtQpE zs8H+}8IkMYXlq0FA5_MK_fO4?`r@+ILc`)-xz-_x@UAi=as(zOVO)hAGu(7(BNh*?RoULz;bO2KRhxgCMGgBH!L=S7Z#Bd8+|AlA;EjadWxdJ$c<& zRZI0#qDVjSdUxSEIxz&tbmpOft$bp@OqH@g@9l(Jg%{}RJ@w#sH8O4EVV4anZkR_g z1mGAP8bsp6t|4{qGG`;Os6Fb`xnZ=1uma_g$=X}n1!ug%tiRI=fOBEuIt6nnnjbPN4xPfj^`gw>H;QCfAU+8tx zqXB*90LmrKpk*%WB~+BAdfi*Ctw<&3z0Q+LENt6&ayc;F0vbC@n*IjdRev`D<1OWp zEdduiDmPNsYs(vATw6kKslS9Kae0yw1S!Z*!KrUaGbw929=^68_L>8O*I&`PDJjg) z`*#>l9f6r3NH3SZN#CHwIpjGA*5bf{j(Ls(@N>>!2r_uk6H1%ZCPR=pRrUwJOfXZb z%1&VUcXt7|43^!K{ot}#7n%l3E+-~p(YpsY5Wo2ZNgw2};DMERBy`JWBco@4o>5Uk zKU3imk~zT0?6unP*^EQM12j+nSm&T*Z5gscZusi!C%@SBw zEPE$_*u~sXA~Do6KWkwzJuj`3kT>QmlzX~K%cM+RRh?ivKoPdtt9YXT>!jqv%=b27 zSC=21G+^_7FNA7okyyFSlI9jrx@<~J{x98R$^P$ zF%3CyCZ>)|YD#sPUmKVmg-l8&Arkr(wXu+#lYro^J7Gef?1(g9j{`Q8|5g~0oSJfR z1h~k)jmR_~4?kz3`a6W+)RIdCVUrE-V|OnN#d6)}ce08B=#g#uS?!V9(Ez!Fg~vHzEQmq7mE)BzW*?-#d*_^<;$`4iY$UmypeuKGccI~*RsdVSGohRR{Wn6L z@dbg&cUKGrGp8-ej@}?a)S5L(jiLHy}`V)uBJ)GSkfa-7}iu$)A=D<+kSJO1r#n2`mz)A}=3o7H)IF@rnSr^SvnpqA=| zDMC_o`jUNL?JSKZd5vAPObs-#V=BKm=m*xsaJnAh%~-wB3jA@Q5O;jEvOw=mjQ*d1 zI;$Yk^MME)16fBR>za)7`pgUx)2L-`A&~;lSW+ZBK1KoMTZwI!64G)gS%s3dAf|>d z&!T~Y3dKjadooOmtk)7Ui`aR&-G`A-52F~CUlc7eJTCTWW1dyvJ3KxuFC|;lxE`Cz z+dzxkwA1@6bWkLQ$Q1_Xy223wp~RSWPnt=Q&02h>n8nSK&-w#@-=HC#!mC%sI>W;N z4zhY^WS<}41F$PPr2`%m8xZX>fC6a8KAoVLuE_M@bdx{6BIuu{I-NOwX)DgaF*F@& zZfP!r!L3BUea`R>!xUd4-mJE##S#(Mxz@U+%TY375fYUVu;NL5!oppAPDYyikH z0$~_?4Jnr+*EnqCs$8~~%;u~n0{{RUYPD98ED?8(&6?wJscDYI<$;H!%Af7B!xgu$ zMpG%e?w-gV>IMxvB|7wHwaqDe$M3DH0jzuTZkW7w}u#s1JPcveriXb?%U^?hyC-b%Kim0VSxHawQ74FbkAD12d^Jx*Gb!uAGPYF+Py0{lmDbt&6m9MNy#cn8-%b;og zV!+OVznRk(s;swY0pLDt1ONXbdAqzr-YM^rcguU^y$$>7a8SjU4mYmt*KgU=UjU)_ z&f-+nU%WD{(&GF$s0mbi?vS+AjZ2T^L zwjlnBKDyNE-QUaH#i2}_KyM=A+3hR7I&OS)Zcpv9BZKny(OC`ENQSyHPu9AM6~skL zOY?Y<$}4eCyV)5|7a=!ne(kh*)&Jan8W2xi88s+x-9f6{bPr{IhM8EOMm6QU-*c77 zS*{WJzPpDm8gC(b6W16R9jGT9kgv=p-$c`#&^Ij0xAfPiBeQqv{CkxkwX2i*2hOE) z7-af5eoB(MoU4mx-*ac~#BC%l8&~O+PULMqPk2{oo4-1+y3A~Ak|&vFVJBhKoUpn3 zr{2PM>-}{(ehPNTt|_ZCein6#>N0`1f6*lT?^}-GUIKjm zSC;|c+xNcX-I@;bvY)F!3j+kWyC^zv+CW;fFJ%7;jtSeUbzgHkyw~maww<;&faT%2 zI{>{Uu_St%f1it&!?V8T%{=0nt$uj=nAbR!=~&|Tofky(aZ`Ppwpx5CS1fAm(Yuod{N@9< z^Z%W#9KBYoxlPyYhP{dS_N4T%#_2$JDOGPCj0{IY-*xw>MfDPGvkx>~qk0L^9f`dD z@E^qgtdh+%xqH^$x8A&weO8l|WCdAD%E?Vr2jgfeX-gf=*inx*s3tISdw5C5FIUNE z8ts%>T)$d;uj!+rcjOpz9Hu^-FNQQt(bFt$Xa6-(#6Fx~EvY9_muZe=J8us5deMyf zL1kEUUS>#{nxDawn3)<0^x)j-dZ{ED^e)W%m_5{S0~Gbv^&1^F$4TWEYO4?3h2`~n zX8v_okEWLN9%>A@cQSiS?oyqgIQ7*_>>T@S);Y?+?gr6EKT4&TZyYJUwf6D3S+eIR zA1tU9*H>vT_ev_v)wu1xw&|SgF7n|(+gG_~s%YQpsTOjcMC?AIZNDlT8Mc1g2IQj8 zg8SFM&DE$nQsbC;IG0i<1NF?^&XDk3G~ka32*d=LjAWu#kkwGBL1v!Tt9$@2MCjIn z*p@{LW*od0LW24hk}quwhs4Adinj=EJey(|MW8UCMd(_C?}6{5&y%g_0KZTuluwTs zOJUE24q1T`^SsCl8qS;4$4Y>LmMLE$g+d(h$`rvfTU|NnjX)1iu{>YC5Cun};iS{;Tc2^vC}GdEGb8!MJkU9Qj{_gB(SNaCcDa7wCKxHHMM zSfX^a;_2gLxwwkB1rZi+X`UQbvRIZlYitPuva;P`cdl=sgZ z-$%(Gg;ubv>g3D?o2%bA;`mOx4o*CH@^Vd=ZWr&Nd-?rv`Mc$^;dU&yW?g{4_})G(f;UprWKVNWT_mn76nug=8c5h9*G6dx+ zRjDwk+6JP(4M0ZGjEY7=s#eYIcbxe%p0ioWDHX(ldJUxpn{5g8P52SveS5*X>i*f5 zl9d56L`KLMnIO~pWR@^G^DMGVlsHM!WHHIJN|ANSY_iQR`y6sq?TfeG(Px4ocK~Sz zrR^@LWQ1ZSyqt2*C07Ov<;oW(qP%m@BhS3@W?DXuIwq2JMJZ^eP+aVZuk5xCQbJF{?Ks9X`NhR-IUU;o3YpFWWwJ$NVI3r zqwM}&bpDr(oYZMcVZ#|EEu_^&536RggEf{?F+YzpHStS|9TzEz>K5gg@H4@@m+b90 z-7@5oTYfU-ktKV|TFP_D&d!~Z@}5Q#QBtNnvSbh2_!n>Zt=hR0yEK{3anMq~_8F>L z#)lS|aKk17WXgxeU($uBYpLd$SaDC#D5imB_bw<^L^ezpgm%h@XV4-w*t&K8<-n_y zXII7;aKj2=54;KQH+M6iF?BPUX=I@msiApxVUlp#2ru9o?qteFR+?8$CJ7j>)vRx) z9{aw6;@2c3!WK;{7C~uLv-!e$w7=)?f#cM1I4lV3WA@Sp8*A%v|DqyOJvVkdm|9O^ aJXej=OBL5mo$WN1dRPew8T0RR9105@C!4*&oF0F0yn05<#p0RR9100000000000000000000 z0000RvS=KW3I41_z2)3r5jAQ-#TYOm zN)W-Egw6z`0Glv*VuE7z)93g3e7^WTp0`%Cmri7Rw2hV&(nw>+kjFfnAG6a0Mc;K%pmaL1H@-Yqplwj-~LX z*1z^!3jj4^T2R<&6uP=F0Din?#M}l%LV^~clC@<>akuMe6tNdf{Ju4{VWC^+ke%Di z?z*+&1M?G{imdJ?9U zzzXnBRW*_Dz)H2_>qP(0RI{DI-kukt1DDdh2gxbnoN}pDI_26$ng0G+?Ck6i;9>zu z2n-Mu84ZF8fTV)JEfnIV7?$D%&=GVBsY45PlHc><|+`%Me?VmIY_z^WEmf)B{BT=_dCbvhrESD5P z@!X_!71FFl^FzSJ$x!E>iQInDQk}XZOL<>Z zQI%X(4OgQiabb=VC0lv1=Y%3ON1*+A%2Oa{jDe?WoF2eAOfZ;gfqr3T9u3lzb#fcu z>)dFqtV5m0)=0eEu8*XV`>hQCKnuOvJ-trP=tdG*df+V=hEkjc>Bh>Av$XB{$8)E&mwb z)!jMDiT+++_+QY(ZyAe zd6MT9*WoRY3~C%J2n`}cP0{0sPz(x~Y9KC^tQ33* zfvU??Gg)daTean=hP-Nc`HG>H5K>gy#c@Q+DXEAPh*dIBtp?hwVN9E0>4>9<&UH$3 zNp)RTJqxO9QS~gTu4UD8MRi?OJ=av%^;}PZX2#nZd+Qx@AE4$ws)=6}<&$dqoEz!W zNS?q_4w^g^Lt$GLij=5QHfE~FHDMGJ13@ZQ@T{7Q;%11fpja!0ZDFFJ*(0}5vYAd& zL3EQu3@v~>P|pQ}1XDyqEs)5FWm+$DX=CGGxr}bjti?vOD%x_PYVw?*o*<$m8lsGO zKyIl|GrSlHUW^1UMuHb3!HbdL9H&WAQFJ2J-+?#`1MEK>SLHf)&0IV8%f8sVyVB{x z&Zeak<0}q_AzrxIo3t@YOOtFIOfkb83oNn18oL(9FgM)y5pt?@0pwUos|neD#pX&7I4FPy0~Q%%kwd<0 ziEG!Du543@B!eQ*V89}SEON+?8cL-UPMz(E>YR~5vy1Ixi2K1Yb1cnCP zk|4=Q0BNL?0T&+pLI4@}*7Nr}i1{&ZF++b)Z9N37E6>w$GtyHHk`xLw7_i77iyZQ0 z3vfkeHH2n5fm>Gcd@CdWS3l^9L9}K}0Rm(|4n&{;%7Aj90;mM8098%S+cBG&j+)`S zuI3rB&Ij6O%eIdES#c&}ILRY>rZXHSHi=kCClS_p0>xRxuF$|zl#Vi?d@pKjfYF~9 zXKcsEm$8xY!3RTJVT*rduVbj9Qm@Py~A3%cviwv^JAwO=IPGRYtEg#^E zAO27XAh0DU&%w~38UbUfuY-=nb$)N}|7_i77iyZPJouhvBKtL=AILx2MOXe)0t^>)u4VQTk|2}Oy#m;gJE9K#NG;+dFQKMzoVz9f(b1yIZ^T6WPgi^fKnp>sd0Ty}7LRwZ6Wz6g5w zg+&F>mMP5OK|_8{Rt8k%7xD5z6iA_D3wIk%VOfC?;);hcV2QxPs9hdJPCy2%pac;R zv|vO*WJPA9BJBsGmro-j`{T@?$un~OjJDH+e7dAVfv|2wvS;^K;?G9WCGU!>uDR}p zn_hV75rn);E?cl@$+8WPJumD{pj_PXH~%kz)B-}8pjh$e|LI4eiGG(IP3N3fd@*H|lt8t%k}kc9DxvO30fWPG5=>6Q3CR*BiH;wo{jK zXh^|@@CnC3ZJG!VgkfkbOD}NH0%{=sBOB3@wAw!#+;0dD&AwYJm8gL8BBL*1v{VrK z03$eUhIn33B%5meD=7LHX_B|(gsS#n*n&|9J>#H}y5{H(4nol;^3vcXWtu>jftjfi zyx!eQ2n~&mftJX)MPKSX5!2$#js@(9Ahk{pBwIw1qWREHsCcMasgWdq?s6SuV}8Re z!XXt5QjNyk_=E;DXf83}hQU{}QCqT&TZ+U@g_RozwFOZ^GQn3|yY#Fjl`veTuG&P|)k()67fSEMa7y#q6R;dV&w;JDD+Fq0fi%aM?WFPeSNak6xe>_r|G_M51W z1k5*xvf)Ku?@Ql~xK@WAA5jn$&R%lk*(W4HSzeUi>;>1JJuyUOaZz#|mt1-Ae)TK8 zETQP!D7WE2f9$0MI5)~kCt?1L1-C!*vYXG`pke-{1v8&{(e-Bz6Q(XuuLVg5U3Qfk z9fX6UDBMU0?|%t~9L|$Bkio&`!-ip3%VL5G)0Hg72b%ncHgBOQk?|@kUMuk@jaDHO zbfE3%%icf>(vOY6sDYCA>t{%t8U@u*5>-KA5kq0BgvOKzA%iK;B`eX?izmyvz4H7IlBsm_1NLYxv^OZBie z9yo$N!H7U4SP;AjL4*^7M3ce6p&<>T)_?!hA|UZ1DO1@G)FkK<3>WVCWiSIA0Hh>Y zu=MTexkO&*02Y2N^e#*-WbJCU|L0F_=Qls+W+1=>u+v>YxNUj~!qxP6bzdF#%$!et z76zVw4%0vDtC^IaHvc`!><{0Gp8Nd39cv zoVDyi%C?I9=Mt)p_iYgAO}I1`388ELrlDXi}q2g8^N7^rLEd=e!A#!7ZnEIW#nV)Xgg`Z^$%`0XJ|W4ydwCjZ zWha6K-h?I-Ud<+mV1{cZgB6K*=;Jl`2>ukcK!4yeQxiPNA2GDdv6u zgOE6D!4?r(41%;2Pv>c^U|V+K*p0V?z%!cyw#nwGkwg}5l8A`R?R6G#O6#9js1pZw zm01EpydIc|1D{3E0%9u#o!3jy@?@TH>^}%j@saK>Q=$GUwE%lFK|-@lt*T`P!$vbk zL~NxTE$NIgJDixK&$e%?n8-=lOF8`EGeK3 zw5bW~G9V)H!pC7D@|S7DM^gP-UJgNp)IQ_~@Y;fqC0}rP*!IDuP6@uPY};@+p%EbR z_`-O~hbtKP%aVA}ioo(T@bvTK>zlES?hOi6&fN&*5z-`vCxiC$Yz)IPkmuK-FI?? zOli@HEsT}jaSh$sA5Ux^{Jo1YJ0|GCJ-ZmMKPY$FFoBB@X&)pkd&wW4EYCOjw4}oV zl}XY{?-pow|UM6uuCN1bjHa^LZNo3Qm2E4B$s)3nZg^{S5qaT<@u3y{5~?b_c8W<=~*VzVh~ z7X@dgak6Z-POKw)Me}m44VH8@#6QtyseSC6j*Ym3oy5gB60wb$Rt`y!%Wfd+_$I2$ zd1s7g#ki|jbXUT)l{gza90skp)U@Z4?vhmib3}qPBJA$DCj**sy|D$e>k**xSR+qqCOUR7fkCZ1QV4Dr ze_Evw|Bosg@EUH4V70r=L3jdxz)UM<5lxNm5CXkoYza2E+@av|B`zA^Pa#I)D*$$X zlW@Fms1kbXVp`Gns@R5ok322=@6lX-S7El)JJ)m1eA5b{%>3xWzks>{;w-rM@Z1F; zF1qdv(;+t$aFHojB$ddZ-ko>p+;m6v;1SwhKmc>qEWv7z?p8@v7I?{kI3#m0;&Td+ zi_lA08={!ftWf|bS?jXM2T?XoO0eg-WF(dO)?xZZ^0lOd+>T{o3Zwly4GZol3B3Iy zaVYp7dGEi+-aiu0)`j20qxfSTBlze9k3z?X+9+iq1CyRlSSpO0WEng(y}$r-YLf>x zr1B!Mk8XWzUbg>~XqeG&nmrZ*CH&@&jn{^5i(q~l6F1obv5ST4z;dY7ghNdgZcR6q z0y#3KmBXwB^VqfsP1+K9RbXOsLZ=O>qBpLy0EoD2m28&}wqwo5N^Y$HZ?;57wC>SexdD0PPf34c4I21 zq2~*nf)gGYiP%fl5 zvso*dmvZ=TGD(`DPx03~K;8v$;0?S}Wla zWL?AF4HC6>x4M1e#{Dy=;_?B+>qpwp%n}6QDZ{uVmq<*!Xcx3YnjNdA@;$~(xm6ut zVv$g}i*7NuhPc6n0L%`<@p|aLcpjaiM=MEWCzV3e6_PG#ca}2YK9d{cVHn^5G%bJquSA`2}$Zdadg-1k7jC>e!iGQp`Y%2;JP- zTr?Ha{_rdW^*4UJ*?|58CVnOE-2uwD5dZJM+Y9A4d8Mp6B$vf4I0YXbn4Po`k-@MhNdA zh;C$E6;`+;F&7R;6ZP-7!nINiGmUs)!nXy(CKa2@Smo>MVTSJ;5~1@hk8b17NY~VU zO%}=a|9^BV`}YftpFhzx@bAz5@~;TVLTbYRi_B6HQ5ZQaBBYUNb47vVz;++954?jD zC{j)$ljaKT$aYy9vW=^okUUp_MyCTQZ%4Eb&Bb%4~;TA$v5> zA!hPBZPItMLq9)yn@3wY1vKW+*dmilJN>WQm4zKafR&6cE)7R4Mx0$*gdV_3+&j+{ zrOa?EMEkf#YMmxS3(fWXxeM8yTb9Hj#cmWnEr6Vz>qBo{O!MM6xF`6>6y>F+!7Kcl z<`>(%uEYaN^NG~H8CCZ9mFMmj9T*O1E|9Y!w?M=nO5T~682 zV0k=t6+4%HEGpkAi0yHd?Umt~p&5;n6ojnF8_JI`9iD0CEDHGIiCt|OQM#>^1#9@h z-}%Qb^0bb;-xpzjeDSBlNB3XunAiTk5#yriBGVEV1xK(glm)$PBRS1&*8j=UCH?nKE2eC;HTfQR*iVpXpzEhT#YQG7B zUT}{c(u(e|U#3`&`k~lixbyyAtKJooa-c9Z#8KebipFyM?%cgY0iH(uVKVnjvP!oq zN!o>}Id|ZhQseYYmwwc9^>5r+!lXArTa@AK;^+ai$m+i8TgctZ%b7qvBw>!fJ|oTn6`2U-!Xx)P$T%h z-f+u-bB|njy|2%#+;0ijo@cm}4(JvgV~7?Gq-otgUZEkWjXWz$ zGDN+8Y=}f!Tq06@eS$+XZS3X4@O2L5(af%#g80eN!etNm{n69K{>7({;3|F}s%!A? z{k?afnd94?mD1oy4Qe1|rTBCwH{b2sH~0^#3!HrTq$;=1V;oxhWc)g*^5v;`p7VF_ z3?&`axY8kQbrOiMHY~t-QrOCLzunXex>;N+;jF&7W+w7@D zBclEB(<@6bj~DMwU~fubdsdgkvGqf6CycKcaFxTYk-EaF_SoJCgQQE|mR-*#<_7O` zopOg2%Yw$v{OKQ9y&^8zpIGn5%s8%!sW7AJa#eQgG-*2?4z$j*wXvqfT7&h%+GKCa zr1t$7KF3lyzGlyv36tNs_1L<8N=`EqbwY8-8^^5WoR|D1mP8_Pl zJR8_Lv9?QKAb)y^fgC8myfwR?>vgDj|E6mL37xSPu@@-j>7DI|Y2Ir4c-vBp%{AiD zF%W2~q`s*@CDVdR_X*9436LINIMSaGR9-}Hwro&0)zLaBpX1hC;>U{bW8&k_dl|*o zx1C}L58m!KuYW0%VZz_x$m=aX(;t2uoHIGe|g=qHG0G))KjIP_2^1bwVd(fF6{UE2$4^8$TcJ7GgH-H zUpH@?AJ}?&eD{u1JunE{PEJg2KcN^sCC(uiRb)dhe)rd&wtx2?X+eML-8uDkRH&zT zag~A0bsg*5v14)W=#ER9xqS(i>>Cunf&X^me@>t$z^-Mj2|Bu$VkjI-4YxL&+zMvj z>2#aKtW__2{S|{dhyIpojpZ07Y4v27r(>-<4%UUy>C`q|>VR~!6zKntA9ribwq;z% zAO~o z1H;R*av%ZhxmCFn1SCoTrphXsxISEWpL{mo87bi(IFy<{eSY;Y+)f2pGfUQmld!$K zyp_)rw-@sKa|Kg}(h5s=r^g&BgtGuMOh(=QNX} zN^*74$)xt`Z@PZt_jl1nkkOr$*Hu-jJKS>QE(a&yN!DoQ(b?Lm_y3*6b*j4Zvmm1j zJ>*%LK^OnHZM}R#j*Q#_vq6zf@Ji7WQcS?gaPiMW20QK%G@}R7^SOnQG;40PeVs{{ z={P&6Fp`@!I3HPGbCFrmrUP7aQ&CjN6I3VuD1`l6NOytD$$C zQrb;LhAOFgxVo}-zIqFspX#Z%ZD%UU7&dxI~HsaMhwJsCCuGE5`ao0_-K4(2hgQ9t@J}}Et3Sw>*_l<7N86&pkTx;fHAkPk ztH`Pu-4^m4_1~R8;?6w~gt$r16?p#Y<*Va>Z+oTG5?QKxSw;1Z3gCuJy0m^M3FU=9 z^gBIl@|Q*(t`6`|Z;?@Z@uK#J+URl`bH;#V^FneS6$uk z4zYfVYwCw~Ruq#y%C}8$7vXv5NV6srD-tIdJ`y z*#^6wf{d;;L!R?Ixn|*SgYzBoY^ztq3ATtKwmq#2jrgJG4GqtSz6^cAZHJ-%1Z-Mn zH8Ge?QA#WVlB`n)ODXE>9@=MBCf}>ZR!vxNHH6&n zsk{BSyuaf?Rn5Ilo=ay|88hhRbl&S_Fw)PPV|8pc5?u8#Y0&@`i3xOLS-=igum@K`Nf?v5k&Wvjvd|+ZMUl{@3yyf zK782-RrsE#ElqESukUF3^saeSxExE58SQJS@}niS$1pl0<$8Ttr&4l$E7XYZwGB-k zK5J;b5>X!>ULO$=R<9vB4L>zz`1_Ip4~!?*8<)Au#>4B7fbU%^uDaSdJfV=O*h< z8{4i%G=v>zg*T9T(ty&BKt6uc3wx6nbzD!PLJZ2FQ6UCp6v1NgoEltf{7C&4h#@tk zt?=+^LmQL^C0wbeVc$rgKQWTn%Vuv%)6ZmY8Dev3)y`itxjX4*Rvv5@`)JbsRkqM8 zAg-Ai-WnSl)7i(0X`^v7LV~$m8jZ^h4$k1g4NUp~J92ZD3O{mWAUZv*KZ-SysglVY z>1SuOveNT8YKf^u?8tBb-6e2}fne}Knz>!dA7}xd0aQ10)=cT9AqgooQn;OUn8ILU zWJ^ydC$60l(H8G-5=RX+u#{6;Z3U5xU|eXJ(P1VR=jBR+Bo2+9m_pY&pchc+ucLgf z&AeUaACveR2DO0|7uD9qfbRxlL#;%?Zld7pZ+*YTx5ke_yKI)E+JM_;`i>LmPRrm1 zIodPA?Ht_b)}CT({fFU%9b&6>U4{wc)iRf#e8prW zt~GA#I^tG!{85Zp*{de`0?GL$xCQz9p*&O3(e!gA*W z{6gdpSu&WW6y zvwg_^{+AtFY3b1cs6g_R@j1=s5!on#e_j#co)QW3a%Bd4GQ;6mE~6@#E@m(UbU!1c zJeX#IfSQ8YyDbG<&b*^|9(@+C_;r4EpT{5WdrBFeT54W~7RVc}bX}?nlBslC5cA7# zvef(=Rq3)&MU#PNcvZNOJFH2;yOT9&GMZrAD!HSJl_KCyNWL1-K-JRM46qUDZ8M-l zRNA(e9WO$ixj=ydr&4CGRWu)yjRH9Pz+lW%F_w@v{jh!8yc%a>3FhfkXU_ShfiF4m2;8o7|yV za0#)euyBEG^14vI++5lC5fJC-+>fi$_0@~3x~rEf9vA?f10W}r%)bIz(yMu}J@Eak zN#HYCzN7D6GcmqH*)}r_R1l>J(mC;%KG7|9GRFl3|B(#_V+ID?25NQd zM^Ka#JW^mp%G#A8*lK*5bcW6-ZpF~0B`paWz0QryX{1|iRyxZ{s9y zzk@(A`-@l48;Jig1MJd=)ep+r)Q_tlv9Gs|e9RkJ(r#ZN;Oj4sbaw1-6wu=mGmN6L zY&nW`;xmr~Lp{ht-oTte^IdagMnYLzKr}=kHLmnjpFlx!0YQ#&!fp47EiFw89}9s@ zGP>t@VS)|onCIgx``Np^(~c3O!b7}9GwK*z>*oK3j3ZNmAJ;p!KsTx+wBSG_=?uwh z@SAi>>J=5LK0Xk-=;|Aj;Qh8R=iqT}kzh+={2tYvo)4?*Xs!y@F6E6&6d2OQa=2sHk~DX8($gvS(n~7Y2*Qlz*bp z7_@RR7yFalVB8Cqn{fmO z6rdJ}Wf7v7jQ@b`D>5_G&T;-GjNfWknL($J7tsWX0(9 z4HJyT#qY?@OWwXMmK)IHmfl?7hRS4mGre8Av;{~2YHP-wJx$AAyq@)hVM*9SqzT;^ z`^(tSDzlc>WwedFjbSP!rjkO^qPU4MNzCNLw3s+f3h?ojt2A18zUw9wJIZ0yHcx!s zKRBQ+zN-FGp^aINigVIlmsz%J{NtOBs_9B`INOjevP+z=yLrIxW|3#w^R?70m0m7tdu;o}_U(QclkQ))aO8e5uPTrfHp;cJFwjYQeGgt|EG zX>HHcFMq#<#+4)$mGvJBO3ykLD(+R{-?KR66CQ8sLW;!yuySE|beP5K2#v<0t)=kO zTMv7Hn;;;zfvb0XT(J?u{Dp;%*FF7W$ zhile2larkF+8WLp0J)ml^_0Cv-~+%0LW@n#$%xLbh?6CExGiLVt#TaC@i2^HYE>FDQL z`NQ&uYd>$gu^gVX6n^6*_h--Mo}XNp__PWV$Dbcjb*xKkRZAI%RvSWolWD(+$;X3@ zjE#H2%KcduY5bn&R^-8S&yOgN4a)B4W6usCyiq?tbr!)sGw3f4-T0s?Gq6{2FO;A0 zC;|T%vn|nAaOb5dqssNQO5>@?N+S&x0AjCs?M9PR+S+>XZCfQ?os!u($KyMz>={u%}x(ly}AARg@u_Pl+f)uIXC?eaQClXU}f8a zvbz?hs?V1UoR*~YOMLVNR?3oG*$%EI{{NOhq`WmKtGOA#NW{j&ZOETZRynCRnA)P|m#>w`A# zpE)yR5)hwzY@;y9+-xUa;c4#JXg!zP{92Fhv!)G>S*X=%BQmjR-Ky&#WwzDbBT@@t z9PZi|X07Bg?w^VEKN7u;(E3<-_a+Y|)Fisbx?-YAnduHg2@i-^HQGr}}U;>^TS!OIQ8d$)>c@=T8nTdkZlA=Uxi{v(KY;{hYs1 za-8!IPzD7Mz;7%{R<^BV{|k99W_ZdQoley*%8`{V>h`a&7N zV_LJ?rF_m}c;kpiz-0orYOvOZH9XYD3#5`G(FHAM?ch~k=9_p-6Xv{y>7QT66I ze@-ROCu#l`Yx$a4!NhC8#@Z~8vlB?x5ntwwKYnoiL%dTm+5LQumlrn`SQ%n1T%pC$ z;waHw7+pitpc5CQCzqoWJ8j6?rF)uf1yU$&bw`o7xl0uWXG^B^{`K=Qy3wK*5!ZVw zhZOrg&!T&3NooK%FWz6pURhXkf^E%bR)lVUP0G>USW|kc`P2P=DDDkkI2f0h@c{B( zmcsFY~KDc#>_8zcZ8!N!qpEiNb!5F>$x>S)PsqVo_$*c>6M1VbBAA z(a7-K8N=vJu6zB_mnQl0@Gl7jgbs<6C6a-dJWq|)7>Ra?@&Kla?6d7@B$>q|Ff-1q_bO-8tTk1vy#SqG(t_{uqHp`Fv(Z zp|Ge}KnNC#gc*#o{@Fy4xR;}2DbI2|6IpdAwR~L`3A6a3ku(PNfyd+9%dt4mg_HqS z$eX)5xRLZ;KJFb&XNDej9@u48Sw5e-umQ&>&^Wc^^hzOab7zv1Cs~Y(V88Uviph0e z(Re^vd%dXcX@XU&i&Ae{Ylf#V4-6 z$?na&>z-`)dGg}T$49<`Hg8<6LIos((G&^s0bOw;x=Jo_a7Pb_l&dH-l>PR*4`~!J zwo27%RN1Z85z#)25i3rocnRt>XjJdFZisLn-?>#4)5e#hF^UIO>?da*bQuEm(Bi z2`8Ph&wl5ehoulaLXKy=;td^q;u}9L&XNuOT8$Kiir;*38i5I8mJUG%OhNz>0CGqm zLPdg*U@!%F=zuophA!xVW=Ml~2n`8=KA=J`tifYghllV4q00ntg@ST{e2znY5idW6 z9gfaS%fw1~TE3hpP2}*~v$v2fE6gFelAOW=-4u_aOCVMDKf6&T%9JT86x`^diXtg@ z1z96IvKN+RE|pV`=_t(+vkT~=V^_n)hmOLWLZn{hOd^v))k$?$T~t@qO?6j2RPq6y z(t~mpi4^Dws&fof57k5UpdJiV@?ks`{=eSv-?Lq=DiHJXGYT$K%TfvTtJ!kjsd=&7 zIPV(C%y+8%YrF93wrr=UN?{||V;t^YONY8)<}c8}JGYc9twugqR_8N7YRPUEHB zVm>Bpkc6VL4Xy(0q+MrNrddU%EwaZtn19k9Yj=;clbcRU-IR`mavb?m^R0C=gz*&> mWySI#zT?4-H41Sxg1hyLh0 c.charCodeAt(0)); +} +const b64urlJson = (s) => JSON.parse(new TextDecoder().decode(b64urlBytes(s))); + +// Access's signing keys, cached for an hour (they rotate rarely). An unknown kid refetches, but +// not more than once every minRefetchMs, so junk tokens can't turn every request into a fetch. +export function jwksFetcher(url = CERTS, ttlMs = 3600_000, fetchFn = (u) => fetch(u), minRefetchMs = 60_000) { + let keys = null; + let at = 0; + return async (kid) => { + const age = Date.now() - at; + const unknown = kid && keys && !keys.some((k) => k.kid === kid); + if (!keys || age >= ttlMs || (unknown && age >= minRefetchMs)) { + const r = await fetchFn(url); + if (!r.ok) throw new Error(`certs: HTTP ${r.status}`); + keys = (await r.json()).keys ?? []; + at = Date.now(); + } + return keys; + }; +} + +// The claims of an unexpired RS256 token from setkernel's Access, for this app (aud), signed by +// one of Access's current keys; null for anything else. +export async function verifyAccessJwt(token, { aud, getKeys, issuer = ISSUER, now = Date.now() }) { + try { + const parts = (token || "").split("."); + if (parts.length !== 3 || !aud) return null; + const header = b64urlJson(parts[0]); + const claims = b64urlJson(parts[1]); + if (header.alg !== "RS256" || claims.iss !== issuer) return null; + const auds = Array.isArray(claims.aud) ? claims.aud : [claims.aud]; + if (!auds.includes(aud)) return null; + if (!Number.isFinite(claims.exp) || claims.exp * 1000 <= now) return null; + if (Number.isFinite(claims.nbf) && claims.nbf * 1000 > now + 60_000) return null; + const jwk = (await getKeys(header.kid)).find((k) => k.kid === header.kid); + if (!jwk) return null; + const key = await crypto.subtle.importKey("jwk", jwk, { name: "RSASSA-PKCS1-v1_5", hash: "SHA-256" }, false, ["verify"]); + const ok = await crypto.subtle.verify("RSASSA-PKCS1-v1_5", key, b64urlBytes(parts[2]), new TextEncoder().encode(`${parts[0]}.${parts[1]}`)); + return ok ? claims : null; + } catch { + return null; + } +} + + +export const SCOPES = ["internal", "public"]; + +// STATUS_CLIENTS: { "": "internal" | "public" }. The wall's token gets +// internal; the showcase publisher's gets public, so it can only ever read the sanitized view. +// Answers a Map; an unset or malformed variable is an empty one (no service token is let in). +export function clientScopes(raw) { + try { + const parsed = typeof raw === "string" ? JSON.parse(raw) : null; + if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) return new Map(); + const entries = Object.entries(parsed); + if (entries.some(([id, scope]) => !id || !SCOPES.includes(scope))) return new Map(); + return new Map(entries); + } catch { + return new Map(); + } +} + +// Verified claims -> who is asking: { who: "person", email } for someone who signed in through +// Access, { who: "service", scope } for a listed service token, or null. A service token's +// claims carry common_name (its client id) and no email. +export function viewer(claims, scopes) { + if (!claims) return null; + if (typeof claims.common_name === "string" && claims.common_name) { + const scope = scopes.get(claims.common_name); + return scope ? { who: "service", scope } : null; + } + return typeof claims.email === "string" && claims.email ? { who: "person", email: claims.email } : null; +} diff --git a/workers/status/src/board.js b/workers/status/src/board.js new file mode 100644 index 0000000..33e506e --- /dev/null +++ b/workers/status/src/board.js @@ -0,0 +1,93 @@ +// Board: the one object that holds the read model. The control Worker's objects are one per +// item and can't be listed, so each publishes its view here after it changes (README.md, +// "Feeding it"); /status reads this one object instead of hundreds. Seats are no longer kept: +// a deployed object's old `seats` table stays, unread. +// +// SQLite storage, no alarms and no WebSockets, so it is idle between requests and bills no +// duration then. Old rows are pruned inside each publish, not by a timer. + +import { DurableObject } from "cloudflare:workers"; +import { DAY_MS, ENDED, LOGGED, checkBatch } from "./model.js"; + +export const BOARD_NAME = "org"; + +const SCHEMA = [ + `CREATE TABLE IF NOT EXISTS items ( + ref TEXT PRIMARY KEY, repo TEXT NOT NULL, number INTEGER NOT NULL, kind TEXT NOT NULL, + state TEXT NOT NULL, rev INTEGER NOT NULL, seat TEXT, pr INTEGER, session_url TEXT, + updated_at TEXT NOT NULL)`, + // one row each time an item reaches merged or failed: the "last 24 h" list + `CREATE TABLE IF NOT EXISTS events ( + seq INTEGER PRIMARY KEY AUTOINCREMENT, at TEXT NOT NULL, ref TEXT NOT NULL, repo TEXT NOT NULL, + number INTEGER NOT NULL, kind TEXT NOT NULL, state TEXT NOT NULL, seat TEXT, pr INTEGER)`, + "CREATE INDEX IF NOT EXISTS events_at ON events (at)", +]; + +export class Board extends DurableObject { + constructor(ctx, env) { + super(ctx, env); + this.sql = ctx.storage.sql; + for (const s of SCHEMA) this.sql.exec(s); + this.now = () => new Date(); + } + + // { items } (model.js checkBatch) -> { ok, applied, stale, rejected }. An item applies only + // with a higher rev than the board holds, so a late or repeated publish changes nothing. Two + // exceptions, both writes the control Worker makes without raising rev: a fired dispatch after the move to + // dispatched, so at the same rev a session link the board doesn't hold yet still applies; and + // another PR taking over (WorkItem.github()), so at the same rev a newer updated_at applies + // too. Neither takes a held link away. + async publish(batch) { + const checked = checkBatch(batch); + if (checked.error) return { ok: false, status: 400, error: checked.error }; + const now = this.now(); + const cutoff = new Date(now.getTime() - DAY_MS).toISOString(); + let applied = 0; + let stale = 0; + this.ctx.storage.transactionSync(() => { + for (const r of checked.items) { + const held = this.sql.exec("SELECT rev, state, session_url, updated_at FROM items WHERE ref = ?", r.ref).toArray()[0]; + const same = held && held.rev === r.rev; + const link = same && held.session_url == null && r.session_url != null; + const newer = same && r.updated_at > held.updated_at; + if (held && held.rev >= r.rev && !link && !newer) { + stale++; + continue; + } + const session = same ? r.session_url ?? held.session_url : r.session_url; + this.sql.exec( + `INSERT INTO items (ref, repo, number, kind, state, rev, seat, pr, session_url, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT (ref) DO UPDATE SET kind = excluded.kind, state = excluded.state, rev = excluded.rev, + seat = excluded.seat, pr = excluded.pr, session_url = excluded.session_url, updated_at = excluded.updated_at`, + r.ref, r.repo, r.number, r.kind, r.state, r.rev, r.seat, r.pr, session, r.updated_at, + ); + if (LOGGED.includes(r.state) && held?.state !== r.state) { + this.sql.exec( + "INSERT INTO events (at, ref, repo, number, kind, state, seat, pr) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", + r.updated_at, r.ref, r.repo, r.number, r.kind, r.state, r.seat, r.pr, + ); + } + applied++; + } + // the page shows a day of merges and failures; ended items and events older than that go + this.sql.exec("DELETE FROM events WHERE at < ?", cutoff); + this.sql.exec(`DELETE FROM items WHERE state IN (${ENDED.map(() => "?").join(", ")}) AND updated_at < ?`, ...ENDED, cutoff); + }); + return { ok: true, applied, stale, rejected: checked.rejected }; + } + + // Everything the board holds, in one call: the open items and a day of events, and + // when they were read (the Worker reuses a snapshot for up to FRESH seconds). + async snapshot() { + const now = this.now(); + const cutoff = new Date(now.getTime() - DAY_MS).toISOString(); + const items = this.sql.exec( + `SELECT ref, repo, number, kind, state, rev, seat, pr, session_url, updated_at FROM items + WHERE state NOT IN (${ENDED.map(() => "?").join(", ")}) ORDER BY ref`, + ...ENDED, + ).toArray(); + const events = this.sql.exec("SELECT at, ref, repo, number, kind, state, seat, pr FROM events WHERE at >= ? ORDER BY seq", cutoff).toArray(); + return { ok: true, at: now.toISOString(), items, events }; + } +} diff --git a/workers/status/src/html.js b/workers/status/src/html.js new file mode 100644 index 0000000..1c7fbbf --- /dev/null +++ b/workers/status/src/html.js @@ -0,0 +1,50 @@ +// The phone view of the internal read model: one small page, no script, nothing loaded from +// anywhere but this hostname. Manrope (Google Fonts, OFL) is self-hosted as a static asset at +// /status/fonts/ (public/status/fonts, with its licence); sans-serif is only the last fallback. + +const ESC = { "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }; +export const esc = (v) => String(v ?? "").replace(/[&<>"']/g, (c) => ESC[c]); + +// "4 min", "3 h", "2 d" since (or until) iso, from now +export function ago(iso, now) { + const s = Math.abs(now.getTime() - Date.parse(iso)) / 1000; + if (!Number.isFinite(s)) return "?"; + if (s < 90) return `${Math.round(s)} s`; + if (s < 5400) return `${Math.round(s / 60)} min`; + if (s < 172800) return `${Math.round(s / 3600)} h`; + return `${Math.round(s / 86400)} d`; +} + +const link = (href, text) => (href ? `${esc(text)}` : esc(text)); + +const STYLE = ` +@font-face{font-family:Manrope;src:url(/status/fonts/manrope-latin-400-normal.woff2) format("woff2");font-weight:400;font-display:swap} +@font-face{font-family:Manrope;src:url(/status/fonts/manrope-latin-600-normal.woff2) format("woff2");font-weight:600;font-display:swap} +:root{color-scheme:light dark;--fg:#1b1f24;--muted:#5b6470;--bg:#fbfbfa;--line:#d9dde3;--warn:#8a4b00;--bad:#a1101a;--ok:#1d6b3a} +@media (prefers-color-scheme:dark){:root{--fg:#e8eaed;--muted:#a3abb5;--bg:#14171a;--line:#2e343b;--warn:#f0b35a;--bad:#ff8a8f;--ok:#7fd49a}} +*{box-sizing:border-box}body{margin:0 auto;max-width:40rem;padding:1rem;font:400 1rem/1.45 Manrope,sans-serif;color:var(--fg);background:var(--bg)} +h1{font-size:1.25rem;margin:0 0 .25rem}h2{font-size:1rem;font-weight:600;margin:1.5rem 0 .5rem;border-bottom:1px solid var(--line);padding-bottom:.25rem} +a{color:inherit}ul{list-style:none;margin:0;padding:0}li{padding:.35rem 0;border-bottom:1px solid var(--line)} +.m{color:var(--muted);font-size:.875rem}.g{padding:.5rem .75rem;border:1px solid currentColor;border-radius:.25rem;margin:.75rem 0} +.alert{color:var(--warn)}.degrade,.refuse,.bad{color:var(--bad)}.ok{color:var(--ok)}.n{font-weight:600}`; + +function guardBlock(g, now) { + if (g.level === "ok") return `

Guard: ok

`; + const what = g.unreadable ? "flags unreadable, treated as degrade" : g.metrics.join(", ") || "no metric named"; + const until = g.until ? `, until ${esc(g.until.slice(0, 16).replace("T", " "))} UTC (${ago(g.until, now)})` : ""; + return `

Guard: ${esc(g.level)} (${esc(what)}${until})

`; +} + +// internal: model.js internalView() +export function renderHtml(v) { + const now = new Date(v.generated_at); + const events = v.last_24h.events.length + ? v.last_24h.events.map((e) => `
  • ${esc(e.state)} ${link(e.pr_url ?? e.url, e.ref)} ${ago(e.at, now)} ago${e.seat ? ` ยท ${esc(e.seat)}` : ""}
  • `).join("") + : `
  • None.
  • `; + return `` + + `setkernel status` + + `

    setkernel status

    ${esc(v.generated_at.slice(0, 19).replace("T", " "))} UTC ยท JSON

    ` + + guardBlock(v.guard, now) + + `

    Last 24 h: ${v.last_24h.merged} merged, ${v.last_24h.failed} failed

      ${events}
    ` + + ``; +} diff --git a/workers/status/src/index.js b/workers/status/src/index.js new file mode 100644 index 0000000..8f4bc0a --- /dev/null +++ b/workers/status/src/index.js @@ -0,0 +1,111 @@ +// sk-status: the read model of the org (hq#208), for Raghav, the wall and Polaris. +// +// GET /status the internal view: HTML for a browser, JSON otherwise +// GET /status/internal.json the internal view as JSON (the wall, hq#127; Polaris) +// GET /status/public.json the sanitized view (the showcase's publisher, hq#213) +// /status/fonts/* static assets: served by Cloudflare before this code runs +// +// The routes are ctl.setkernel.net/status and /status/*, behind their own Access app (README.md, +// "Routes"), and every request's Access token is checked again here. A person sees everything; +// a service token sees what STATUS_CLIENTS gives it. The control Worker writes through the Feed entrypoint, over a service +// binding, so there is no write route at all. + +import { WorkerEntrypoint } from "cloudflare:workers"; +import { readGuard, refusal } from "../../meter/src/flags.js"; +import { clientScopes, jwksFetcher, verifyAccessJwt, viewer } from "./access.js"; +import { BOARD_NAME } from "./board.js"; +import { renderHtml } from "./html.js"; +import { internalView, publicView } from "./model.js"; + +export { Board } from "./board.js"; + +// How long one isolate reuses a board snapshot, in seconds. The Cache API isn't available +// behind Access, so this is per isolate. Pollers get the longer wait (twice their one-a-minute +// interval, so at most every other poll reads the object); under guard:degrade every wait goes up. +export const FRESH = { person: 15, service: 120 }; +export const DEGRADED_FRESH = { person: 60, service: 600 }; +export const GUARD_SECONDS = 60; + +const fail = (status, error) => Response.json({ ok: false, error }, { status, headers: { "cache-control": "no-store" } }); + +const board = (env) => env.BOARD.get(env.BOARD.idFromName(BOARD_NAME)); + +const HTML_HEADERS = { + "content-type": "text/html; charset=utf-8", + "content-security-policy": "default-src 'none'; style-src 'unsafe-inline'; font-src 'self'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'", + "x-content-type-options": "nosniff", + "referrer-policy": "no-referrer", +}; + +// One handler with its own caches; tests make their own with a fake clock and fake keys. +export function createHandler({ getKeys = jwksFetcher(), now = () => new Date() } = {}) { + let guardMemo = null; + let snapMemo = null; + + async function guardOf(env) { + const t = now().getTime(); + if (!guardMemo || t - guardMemo.at >= GUARD_SECONDS * 1000) guardMemo = { at: t, value: await readGuard(env.GUARD) }; + return guardMemo.value; + } + + async function snapshotOf(env, maxAgeS) { + const t = now().getTime(); + if (!snapMemo || t - snapMemo.at >= maxAgeS * 1000) { + const snap = await board(env).snapshot(); + if (!snap?.ok) throw new Error("board snapshot failed"); + snapMemo = { at: t, value: snap }; + } + return snapMemo.value; + } + + return async function handle(request, env) { + const { pathname } = new URL(request.url); + const views = { "/status": "page", "/status/internal.json": "internal", "/status/public.json": "public" }; + const view = views[pathname]; + if (!view) return fail(404, "no such route"); + if (request.method !== "GET" && request.method !== "HEAD") return fail(405, "GET only"); + + // fails closed: with no audience configured, nothing is served + const claims = await verifyAccessJwt(request.headers.get("cf-access-jwt-assertion"), { aud: env.STATUS_ACCESS_AUD, getKeys, now: now().getTime() }); + const who = viewer(claims, clientScopes(env.STATUS_CLIENTS)); + if (!who) return fail(401, "unauthorized"); + if (who.who === "service" && who.scope === "public" && view !== "public") return fail(403, "this token reads /status/public.json only"); + + const guard = await guardOf(env); + // the public feed is the one that exists for polling from outside: it stops first + if (view === "public") { + const paused = refusal(guard); + if (paused) return paused; + } + const wait = (guard.degrade ? DEGRADED_FRESH : FRESH)[who.who]; + const internal = internalView(await snapshotOf(env, wait), guard, now()); + // /status answers HTML or JSON by Accept, so a cache must key on it + const headers = { "cache-control": `private, max-age=${wait}`, vary: "accept" }; + if (view === "public") return Response.json(publicView(internal), { headers }); + const wantsHtml = view === "page" && who.who === "person" && /text\/html/.test(request.headers.get("accept") ?? ""); + if (wantsHtml) return new Response(renderHtml(internal), { headers: { ...headers, ...HTML_HEADERS } }); + return Response.json(internal, { headers }); + }; +} + + +// The control Worker's service binding: [[services]] binding = "STATUS", service = "sk-status", entrypoint = "Feed". +// publish({ items: [WorkItem.get() results] }) -> { ok, applied, stale, rejected }. Items feed only +// the "last 24 h" list now; a batch's `seats` is ignored. +export class Feed extends WorkerEntrypoint { + async publish(batch) { + return board(this.env).publish(batch); + } +} + +// A 500 that logs the error and tells the caller nothing more. +export const guarded = (handle) => async (request, env) => { + try { + return await handle(request, env); + } catch (e) { + console.error(`sk-status: ${request.method} ${new URL(request.url).pathname}: ${e?.stack ?? e}`); + return fail(500, "internal error"); + } +}; + +export default { fetch: guarded(createHandler()) }; diff --git a/workers/status/src/model.js b/workers/status/src/model.js new file mode 100644 index 0000000..de1e247 --- /dev/null +++ b/workers/status/src/model.js @@ -0,0 +1,129 @@ +// The read model (hq#208): what the control Worker publishes to the board, checked, and the views +// built from the board's rows. Pure functions; board.js stores, index.js serves. +// +// The control Worker publishes what its objects already answer (the control Worker's data model): +// items: WorkItem.get() results, { item: { ref, kind, state, rev, seat, pr, updated_at }, dispatch } +// (dispatch is the control Worker's last routine fire, with session_url; it may be absent) +// Anything else is dropped with a reason, so one bad entry never blocks the rest of a batch. +// The queue, running work and seats are no longer shown (hq#1330): a batch's `seats` +// is ignored, and items only feed the "last 24 h" list. + +// The control Worker's work item states, so an item is never rejected for a state the control Worker can reach. Kept in +// step by hand with STATES in the control Worker's model.js; status has no import from the control Worker. +export const STATES = ["new", "queued", "dispatched", "running", "pr_open", "reviewing", "merged", "failed", "closed"]; +// off the board: finished, closed without a merge, or back to new (out of the agent flow, +// the control Worker). These rows are pruned after a day; they would otherwise stay forever. +export const ENDED = ["merged", "failed", "closed", "new"]; +// the ends the "last 24 h" list counts: one event each time an item reaches one +export const LOGGED = ["merged", "failed"]; +export const DAY_MS = 24 * 3600_000; +export const BATCH_MAX = { items: 100 }; +export const EVENTS_SHOWN = 50; + +const OWNER = /^[a-z0-9](?:[a-z0-9-]{0,38})$/; +const REPO = /^[a-z0-9._-]{1,100}$/; +const LOGIN = /^[a-z0-9](?:[a-z0-9-]{0,38})$/; +const REF = /^([^/#]+)\/([^/#]+)#([1-9][0-9]{0,9})$/; + +const iso = (v) => { + if (typeof v !== "string" && typeof v !== "number") return null; + const t = new Date(v).getTime(); + return Number.isFinite(t) ? new Date(t).toISOString() : null; +}; +const login = (v) => (typeof v === "string" && LOGIN.test(v.toLowerCase()) ? v.toLowerCase() : null); +const posInt = (v) => (Number.isInteger(v) && v > 0 && v < 1e10 ? v : null); + +// "owner/repo#N" -> { repo, number }, or null. Lower-cased, as the control Worker keys its objects. +export function parseRef(ref) { + const m = typeof ref === "string" ? REF.exec(ref.toLowerCase()) : null; + if (!m || !OWNER.test(m[1]) || !REPO.test(m[2]) || m[2] === "." || m[2] === "..") return null; + return { ref: `${m[1]}/${m[2]}#${m[3]}`, repo: `${m[1]}/${m[2]}`, number: Number(m[3]) }; +} + +// Only a claude.ai session link is ever shown or linked (the control Worker keeps the same rule). +export const sessionUrl = (v) => (typeof v === "string" && v.length <= 200 && /^https:\/\/claude\.ai\/[A-Za-z0-9/_?=&.-]*$/.test(v) ? v : null); + +// One WorkItem.get() result -> a board row, or { error }. +export function itemRow(entry) { + const item = entry?.item ?? null; + if (!item || typeof item !== "object") return { error: "no item" }; + const name = parseRef(item.ref); + if (!name) return { error: "ref must be owner/repo#N" }; + if (!STATES.includes(item.state)) return { error: `state must be one of ${STATES.join(", ")}` }; + const rev = posInt(item.rev); + if (!rev) return { error: "rev must be a positive integer" }; + const updated = iso(item.updated_at); + if (!updated) return { error: "updated_at must be a time" }; + const seat = item.seat == null ? null : login(item.seat); + if (item.seat != null && !seat) return { error: "seat must be a GitHub login" }; + // the session link belongs to the dispatch that put the item on its current seat + const d = entry.dispatch; + const session = d && d.outcome === "fired" && (d.seat == null || login(d.seat) === seat) ? sessionUrl(d.session_url) : null; + return { + row: { + ref: name.ref, repo: name.repo, number: name.number, + kind: item.kind === "pr" ? "pr" : "issue", + state: item.state, rev, seat, pr: posInt(item.pr), session_url: session, updated_at: updated, + }, + }; +} + +// { items } -> { items: rows, rejected: [{ kind, index, error }] } +export function checkBatch(batch) { + if (!batch || typeof batch !== "object" || Array.isArray(batch)) return { error: "batch must be { items }" }; + const items = batch.items ?? []; + if (!Array.isArray(items)) return { error: "items must be an array" }; + if (items.length > BATCH_MAX.items) return { error: `at most ${BATCH_MAX.items} items a batch` }; + const out = { items: [], rejected: [] }; + items.forEach((e, index) => { + const r = itemRow(e); + if (r.error) out.rejected.push({ kind: "item", index, error: r.error }); + else out.items.push(r.row); + }); + return out; +} + +const github = (repo, kind, n) => `https://github.com/${repo}/${kind === "pr" ? "pull" : "issues"}/${n}`; +const byTime = (a, b) => (a < b ? -1 : a > b ? 1 : 0); + +// The guard as the page shows it: the levels, and the metrics and reset time of the highest one. +export function guardView(guard) { + const level = guard.refuse ? "refuse" : guard.degrade ? "degrade" : guard.alert ? "alert" : "ok"; + const d = guard.detail?.[level]; + return { + level, alert: guard.alert, degrade: guard.degrade, refuse: guard.refuse, + metrics: Array.isArray(d?.metrics) ? d.metrics.filter((m) => typeof m === "string") : [], + until: iso(d?.until), + ...(guard.error ? { unreadable: true } : {}), + }; +} + +// The internal view: for Raghav, the wall (hq#127) and Polaris. snapshot is Board.snapshot(). +export function internalView(snapshot, guard, now) { + const since = now.getTime() - DAY_MS; + const events = snapshot.events.filter((e) => Date.parse(e.at) > since).sort((a, b) => byTime(b.at, a.at)); + return { + generated_at: now.toISOString(), + // the board's data as of this time: a reused snapshot can be up to FRESH seconds older + snapshot_at: iso(snapshot.at), + guard: guardView(guard), + last_24h: { + merged: events.filter((e) => e.state === "merged").length, + failed: events.filter((e) => e.state === "failed").length, + events: events.slice(0, EVENTS_SHOWN).map((e) => ({ + at: e.at, state: e.state, ref: e.ref, seat: e.seat, pr: e.pr, + url: github(e.repo, e.kind, e.number), pr_url: e.pr ? github(e.repo, "pr", e.pr) : null, + })), + }, + }; +} + +// The sanitized view for the public showcase's publisher (hq#213). Built field by field from the +// internal view, never by deleting from it: counts only. No repo, ref, URL, login, session or guard +// detail can reach it, because none is copied. +export function publicView(internal) { + return { + generated_at: internal.generated_at, + last_24h: { merged: internal.last_24h.merged, failed: internal.last_24h.failed }, + }; +} diff --git a/workers/status/test/board.test.js b/workers/status/test/board.test.js new file mode 100644 index 0000000..834f6dc --- /dev/null +++ b/workers/status/test/board.test.js @@ -0,0 +1,122 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { Board } from "../src/board.js"; +import { fakeState } from "./fakes.js"; + +const T0 = Date.parse("2026-10-16T12:00:00Z"); + +function board(state = fakeState(), at = T0) { + const b = new Board(state, {}); + b.clock = at; + b.now = () => new Date(b.clock); + return b; +} + +const item = (ref, state, rev, extra = {}) => ({ item: { ref, kind: "issue", state, rev, seat: null, pr: null, updated_at: new Date(T0).toISOString(), ...extra } }); +const seat = (login, seen, utilization = 0.5) => ({ seat: { login, active: true, updated_at: seen, windows: [{ name: "five_hour", utilization, resets_at: "2026-10-16T15:00:00Z", observed_at: seen }] } }); + +test("publish then snapshot: open items, and seats are neither written nor read", async () => { + const b = board(); + const r = await b.publish({ items: [item("a/b#1", "queued", 1), item("a/b#2", "running", 3, { seat: "vega-setkernel" })], seats: [seat("vega-setkernel", "2026-10-16T11:00:00Z")] }); + assert.deepEqual(r, { ok: true, applied: 2, stale: 0, rejected: [] }); + const s = await b.snapshot(); + assert.deepEqual(s.items.map((i) => [i.ref, i.state, i.seat]), [["a/b#1", "queued", null], ["a/b#2", "running", "vega-setkernel"]]); + assert.equal(s.seats, undefined); + assert.deepEqual(b.sql.exec("SELECT name FROM sqlite_master WHERE name = 'seats'").toArray(), []); + assert.deepEqual(s.events, []); +}); + +test("the rev fence: an older or repeated item view changes nothing", async () => { + const b = board(); + await b.publish({ items: [item("a/b#1", "running", 3)] }); + const r = await b.publish({ items: [item("a/b#1", "queued", 1), item("a/b#1", "running", 3)] }); + assert.equal(r.applied, 0); + assert.equal(r.stale, 2); + assert.equal((await b.snapshot()).items[0].state, "running"); +}); + +test("the session link written after the move to dispatched applies at the same rev, and stays", async () => { + const b = board(); + const moved = item("a/b#1", "dispatched", 2, { seat: "vega-setkernel" }); + const fired = { ...moved, dispatch: { at: new Date(T0).toISOString(), seat: "vega-setkernel", rev: 2, outcome: "fired", session_url: "https://claude.ai/code/session_01abc" } }; + assert.equal((await b.publish({ items: [moved] })).applied, 1); + assert.deepEqual(await b.publish({ items: [fired] }), { ok: true, applied: 1, stale: 0, rejected: [] }); + assert.equal((await b.snapshot()).items[0].session_url, "https://claude.ai/code/session_01abc"); + const r = await b.publish({ items: [moved, fired] }); // late or repeated: no change either way + assert.equal(r.stale, 2); + assert.equal((await b.snapshot()).items[0].session_url, "https://claude.ai/code/session_01abc"); + assert.deepEqual((await b.snapshot()).events, []); +}); + +test("another PR taking over at the same rev applies when newer, and a late replay doesn't undo it", async () => { + const b = board(); + const at = (m) => new Date(T0 + m * 60_000).toISOString(); + const fired = { ...item("a/b#1", "pr_open", 4, { seat: "vega-setkernel", pr: 10, updated_at: at(0) }), dispatch: { at: at(0), seat: "vega-setkernel", rev: 4, outcome: "fired", session_url: "https://claude.ai/code/session_01abc" } }; + const first = item("a/b#1", "pr_open", 4, { seat: "vega-setkernel", pr: 10, updated_at: at(0) }); + const took = item("a/b#1", "pr_open", 4, { seat: "vega-setkernel", pr: 11, updated_at: at(30) }); + assert.equal((await b.publish({ items: [fired] })).applied, 1); + assert.deepEqual(await b.publish({ items: [took] }), { ok: true, applied: 1, stale: 0, rejected: [] }); + let s = await b.snapshot(); + assert.deepEqual(s.items.map((i) => [i.ref, i.rev, i.pr, i.session_url]), [["a/b#1", 4, 11, "https://claude.ai/code/session_01abc"]]); + const r = await b.publish({ items: [first, fired, took] }); // late or repeated: no change + assert.equal(r.stale, 3); + s = await b.snapshot(); + assert.deepEqual(s.items.map((i) => [i.pr, i.updated_at]), [[11, at(30)]]); + assert.deepEqual(s.events, []); +}); + +test("reaching merged or failed records one event; a retry records the next failure too", async () => { + const b = board(); + await b.publish({ items: [item("a/b#1", "pr_open", 4, { pr: 9, seat: "vega-setkernel" })] }); + await b.publish({ items: [item("a/b#1", "merged", 5, { pr: 9, seat: "vega-setkernel" })] }); + await b.publish({ items: [item("a/b#1", "merged", 5, { pr: 9 })] }); // the same view again + await b.publish({ items: [item("c/d#2", "failed", 2), item("c/d#2", "queued", 3), item("c/d#2", "failed", 4)] }); + const s = await b.snapshot(); + assert.deepEqual(s.events.map((e) => [e.ref, e.state, e.pr]), [["a/b#1", "merged", 9], ["c/d#2", "failed", null], ["c/d#2", "failed", null]]); + assert.deepEqual(s.items, [], "ended items aren't open"); +}); + +test("a day later, ended items and their events are pruned on the next publish", async () => { + const b = board(); + await b.publish({ items: [item("a/b#1", "merged", 2), item("a/b#2", "queued", 1)] }); + b.clock = T0 + 25 * 3600_000; + assert.equal((await b.snapshot()).events.length, 0, "the snapshot hides events past a day"); + await b.publish({ items: [item("a/b#3", "queued", 1, { updated_at: new Date(b.clock).toISOString() })] }); + const rows = b.sql.exec("SELECT ref FROM items ORDER BY ref").toArray().map((r) => r.ref); + assert.deepEqual(rows, ["a/b#2", "a/b#3"], "an open item stays however old"); + assert.equal(b.sql.exec("SELECT count(*) AS n FROM events").toArray()[0].n, 0); +}); + +test("closed and new leave the board without an event, and are pruned a day later", async () => { + const b = board(); + await b.publish({ items: [item("a/b#1", "queued", 1), item("a/b#2", "running", 2, { seat: "vega-setkernel" })] }); + await b.publish({ items: [item("a/b#1", "closed", 2), item("a/b#2", "new", 3)] }); + const s = await b.snapshot(); + assert.deepEqual(s.items, [], "neither is queued or running any more"); + assert.deepEqual(s.events, [], "only merged and failed are counted"); + b.clock = T0 + 25 * 3600_000; + await b.publish({ items: [] }); + assert.equal(b.sql.exec("SELECT count(*) AS n FROM items").toArray()[0].n, 0); +}); + +test("a bad entry is rejected with its reason and the rest of the batch applies", async () => { + const b = board(); + const r = await b.publish({ items: [item("bad ref", "queued", 1), item("a/b#1", "queued", 1)] }); + assert.equal(r.applied, 1); + assert.deepEqual(r.rejected, [{ kind: "item", index: 0, error: "ref must be owner/repo#N" }]); + assert.deepEqual(await b.publish("nope"), { ok: false, status: 400, error: "batch must be { items }" }); +}); + +test("a failed write rolls the whole batch back", async () => { + const state = fakeState(); + const b = board(state); + state.db.exec("CREATE TRIGGER boom BEFORE INSERT ON events BEGIN SELECT RAISE(ABORT, 'disk full'); END"); + await assert.rejects(b.publish({ items: [item("a/b#1", "queued", 1), item("a/b#2", "merged", 2)] }), /disk full/); + assert.deepEqual((await b.snapshot()).items, []); +}); + +test("state survives the object being rebuilt on the same storage", async () => { + const state = fakeState(); + await board(state).publish({ items: [item("a/b#1", "queued", 1)] }); + assert.equal((await board(state).snapshot()).items.length, 1); +}); diff --git a/workers/status/test/cloudflare-workers.js b/workers/status/test/cloudflare-workers.js new file mode 100644 index 0000000..bbfe1ba --- /dev/null +++ b/workers/status/test/cloudflare-workers.js @@ -0,0 +1,14 @@ +// The part of `cloudflare:workers` sk-status uses: both base classes keep ctx and env. +export class DurableObject { + constructor(ctx, env) { + this.ctx = ctx; + this.env = env; + } +} + +export class WorkerEntrypoint { + constructor(ctx, env) { + this.ctx = ctx; + this.env = env; + } +} diff --git a/workers/status/test/fakes.js b/workers/status/test/fakes.js new file mode 100644 index 0000000..722a9b8 --- /dev/null +++ b/workers/status/test/fakes.js @@ -0,0 +1,81 @@ +// Test doubles: a namespace that hands out one object per name as the runtime does, and Access +// tokens signed by a key the test owns. fakeState is Durable Object storage on a real SQLite; +// memoryKV is sk-meter's. +import { DatabaseSync } from "node:sqlite"; + +export { memoryKV } from "../../meter/test/fakes.js"; + +// ctx.storage.sql.exec(query, ...bindings) returns a cursor; transactionSync rolls back on throw. +export function fakeState() { + const db = new DatabaseSync(":memory:"); + const sql = { + exec(query, ...bindings) { + const st = db.prepare(query); + const rows = st.columns().length ? st.all(...bindings) : (st.run(...bindings), []); + return { toArray: () => rows, one: () => rows[0] }; + }, + }; + // One alarm, as the runtime keeps it: setAlarm replaces it, and storage.alarm is the time it is + // set for (epoch ms) or null. A test runs it with object.alarm(). + const storage = { + sql, + alarm: null, + async setAlarm(at) { storage.alarm = typeof at === "number" ? at : at.getTime(); }, + async getAlarm() { return storage.alarm; }, + async deleteAlarm() { storage.alarm = null; }, + transactionSync(fn) { + db.exec("BEGIN"); + try { + const r = fn(); + db.exec("COMMIT"); + return r; + } catch (e) { + db.exec("ROLLBACK"); + throw e; + } + }, + }; + // ctx.waitUntil: the promises it was given, for a test to await (waits) + const waits = []; + return { storage, db, waits, waitUntil: (p) => { waits.push(p); } }; +} + +// init(object) runs once on each new object, e.g. to point a Board's clock at the test's. +export function fakeNamespace(Class, env = {}, init = () => {}) { + const objects = new Map(); + const ns = { + objects, + gets: 0, + idFromName: (name) => ({ name, toString: () => name }), + get(id) { + ns.gets++; + if (!objects.has(id.name)) { + const o = new Class(fakeState(), env); + init(o); + objects.set(id.name, o); + } + return objects.get(id.name); + }, + }; + return ns; +} + +const b64url = (bytes) => Buffer.from(bytes).toString("base64url"); + +// An RS256 signer standing in for setkernel's Access: sign(claims) -> a token, getKeys -> its JWKS. +export async function fakeAccess({ kid = "k1" } = {}) { + const pair = await crypto.subtle.generateKey( + { name: "RSASSA-PKCS1-v1_5", modulusLength: 2048, publicExponent: new Uint8Array([1, 0, 1]), hash: "SHA-256" }, + true, ["sign", "verify"], + ); + const jwk = { ...(await crypto.subtle.exportKey("jwk", pair.publicKey)), kid }; + return { + getKeys: async () => [jwk], + async sign(claims, header = { alg: "RS256", kid }) { + const enc = (o) => b64url(new TextEncoder().encode(JSON.stringify(o))); + const body = `${enc(header)}.${enc(claims)}`; + const sig = await crypto.subtle.sign("RSASSA-PKCS1-v1_5", pair.privateKey, new TextEncoder().encode(body)); + return `${body}.${b64url(new Uint8Array(sig))}`; + }, + }; +} diff --git a/workers/status/test/model.test.js b/workers/status/test/model.test.js new file mode 100644 index 0000000..2c9c340 --- /dev/null +++ b/workers/status/test/model.test.js @@ -0,0 +1,91 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { ENDED, LOGGED, STATES, checkBatch, internalView, itemRow, parseRef, publicView, sessionUrl } from "../src/model.js"; +import { esc } from "../src/html.js"; + +const NOW = new Date("2026-10-16T12:00:00Z"); +const OK = { alert: false, degrade: false, refuse: false, detail: {} }; +const item = (ref, state, rev = 1, extra = {}, dispatch) => ({ item: { ref, kind: "issue", state, rev, seat: null, pr: null, updated_at: "2026-10-16T11:00:00Z", ...extra }, dispatch }); + +test("parseRef lower-cases and rejects what isn't owner/repo#N", () => { + assert.deepEqual(parseRef("SetKernel/SK-demo#14"), { ref: "setkernel/sk-demo#14", repo: "setkernel/sk-demo", number: 14 }); + for (const bad of ["setkernel/sk-demo", "setkernel#1", "a/b/c#1", "a/b#0", "a/..#1", "-a/b#1", "a/b c#1", 7, null]) assert.equal(parseRef(bad), null, String(bad)); +}); + +test("itemRow keeps a fired dispatch's claude.ai link only for the item's own seat", () => { + const fired = { outcome: "fired", seat: "vega-setkernel", session_url: "https://claude.ai/code/session_01ab" }; + assert.equal(itemRow(item("a/b#1", "running", 3, { seat: "Vega-Setkernel" }, fired)).row.session_url, fired.session_url); + assert.equal(itemRow(item("a/b#1", "running", 3, { seat: "lyra-setkernel" }, fired)).row.session_url, null); + assert.equal(itemRow(item("a/b#1", "running", 3, { seat: "vega-setkernel" }, { ...fired, outcome: "failed" })).row.session_url, null); + for (const url of ["javascript:alert(1)", "https://claude.ai.evil.example/x", "http://claude.ai/x", "https://claude.ai/\">"]) { + assert.equal(sessionUrl(url), null, url); + } +}); + +test("itemRow rejects bad fields with a reason", () => { + assert.match(itemRow(item("nope", "queued")).error, /ref/); + assert.match(itemRow(item("a/b#1", "lost")).error, /state/); + assert.match(itemRow(item("a/b#1", "queued", 0)).error, /rev/); + assert.match(itemRow(item("a/b#1", "queued", 1, { seat: "not a login" })).error, /seat/); + assert.match(itemRow(item("a/b#1", "queued", 1, { updated_at: "soon" })).error, /updated_at/); + assert.match(itemRow({}).error, /no item/); +}); + +test("the ends the board logs and prunes are states the control Worker can reach", () => { + for (const state of ENDED) assert.ok(STATES.includes(state), state); + for (const state of LOGGED) assert.ok(ENDED.includes(state), state); +}); + +test("checkBatch drops bad entries with their index, keeps the rest and ignores seats", () => { + const out = checkBatch({ items: [item("a/b#1", "queued"), item("bad", "queued")], seats: [{ seat: { login: "x", active: true, updated_at: NOW.toISOString() } }] }); + assert.equal(out.items.length, 1); + assert.equal(out.seats, undefined); + assert.deepEqual(out.rejected, [{ kind: "item", index: 1, error: "ref must be owner/repo#N" }]); + assert.match(checkBatch([]).error, /batch/); + assert.match(checkBatch({ items: Array(101).fill(item("a/b#1", "queued")) }).error, /at most/); +}); + +const SNAPSHOT = { + items: [ + { ref: "acme-client/secret-app#3", repo: "acme-client/secret-app", number: 3, kind: "issue", state: "queued", rev: 1, seat: null, pr: null, session_url: null, updated_at: "2026-10-16T09:00:00Z" }, + { ref: "setkernel/sk-demo#14", repo: "setkernel/sk-demo", number: 14, kind: "issue", state: "queued", rev: 1, seat: null, pr: null, session_url: null, updated_at: "2026-10-16T10:00:00Z" }, + { ref: "setkernel/sk-demo#12", repo: "setkernel/sk-demo", number: 12, kind: "issue", state: "queued", rev: 1, seat: null, pr: null, session_url: null, updated_at: "2026-10-16T08:00:00Z" }, + { ref: "acme-client/secret-app#9", repo: "acme-client/secret-app", number: 9, kind: "issue", state: "pr_open", rev: 4, seat: "vega-setkernel", pr: 10, session_url: "https://claude.ai/code/session_01ab", updated_at: "2026-10-16T11:00:00Z" }, + { ref: "setkernel/hq#1", repo: "setkernel/hq", number: 1, kind: "issue", state: "running", rev: 3, seat: "realraghavgupta", pr: null, session_url: null, updated_at: "2026-10-16T10:30:00Z" }, + ], + events: [ + { at: "2026-10-16T11:30:00Z", ref: "acme-client/secret-app#7", repo: "acme-client/secret-app", number: 7, kind: "issue", state: "merged", seat: "vega-setkernel", pr: 8 }, + { at: "2026-10-16T07:00:00Z", ref: "setkernel/hq#2", repo: "setkernel/hq", number: 2, kind: "issue", state: "failed", seat: "lyra-setkernel", pr: null }, + { at: "2026-10-15T11:00:00Z", ref: "setkernel/hq#3", repo: "setkernel/hq", number: 3, kind: "issue", state: "merged", seat: null, pr: null }, + ], +}; + +test("internalView: the guard and a day of events, with no queue, running or seats", () => { + const v = internalView(SNAPSHOT, OK, NOW); + assert.deepEqual(Object.keys(v), ["generated_at", "snapshot_at", "guard", "last_24h"]); + assert.equal(v.last_24h.events[0].pr_url, "https://github.com/acme-client/secret-app/pull/8"); + assert.equal(v.last_24h.merged, 1); + assert.equal(v.last_24h.failed, 1); + assert.deepEqual(v.last_24h.events.map((e) => e.ref), ["acme-client/secret-app#7", "setkernel/hq#2"]); + assert.equal(v.guard.level, "ok"); +}); + +test("internalView names the guard's highest level, its metrics and reset", () => { + const g = { alert: true, degrade: true, refuse: false, detail: { degrade: { since: "x", metrics: ["workers_cpu_ms"], until: "2026-11-13T00:00:00Z" } } }; + assert.deepEqual(internalView(SNAPSHOT, g, NOW).guard, { level: "degrade", alert: true, degrade: true, refuse: false, metrics: ["workers_cpu_ms"], until: "2026-11-13T00:00:00.000Z" }); + assert.equal(internalView(SNAPSHOT, { alert: true, degrade: true, refuse: false, error: "kv down" }, NOW).guard.unreadable, true); +}); + +test("publicView: counts only; no repo, ref, URL, login or usage", () => { + const v = publicView(internalView(SNAPSHOT, OK, NOW)); + assert.deepEqual(v, { generated_at: NOW.toISOString(), last_24h: { merged: 1, failed: 1 } }); + const text = JSON.stringify(v).toLowerCase(); + for (const leak of ["acme", "secret-app", "sk-demo", "setkernel/", "http", "claude.ai", "raghav", "-setkernel", "five_hour", "headroom", "guard"]) { + assert.ok(!text.includes(leak), `public view leaks ${leak}`); + } +}); + +test("esc escapes everything HTML gives meaning to", () => { + assert.equal(esc(`&`), "<a href="x" onclick='y'>&</a>"); + assert.equal(esc(null), ""); +}); diff --git a/workers/status/test/setup.js b/workers/status/test/setup.js new file mode 100644 index 0000000..73acad3 --- /dev/null +++ b/workers/status/test/setup.js @@ -0,0 +1,11 @@ +// Loaded before the tests (package.json: node --import): `cloudflare:workers` exists only on the +// Workers runtime, so the tests resolve it to a stand-in with the same base classes. +import { registerHooks } from "node:module"; + +const STAND_IN = new URL("./cloudflare-workers.js", import.meta.url).href; + +registerHooks({ + resolve(specifier, context, next) { + return specifier === "cloudflare:workers" ? { url: STAND_IN, shortCircuit: true } : next(specifier, context); + }, +}); diff --git a/workers/status/test/worker.test.js b/workers/status/test/worker.test.js new file mode 100644 index 0000000..cb9cd33 --- /dev/null +++ b/workers/status/test/worker.test.js @@ -0,0 +1,182 @@ +// End to end in one process: the control Worker's views go in through the Feed entrypoint, land in the real +// Board class on a real SQLite, and come out of /status as HTML and JSON behind the Access check. +import { test } from "node:test"; +import assert from "node:assert/strict"; +import worker, { Board, Feed, createHandler, guarded } from "../src/index.js"; +import { fakeAccess, fakeNamespace, memoryKV } from "./fakes.js"; + +const AUD = "aud-sk-status"; +const T0 = Date.parse("2026-10-16T12:00:00Z"); +const WALL = "wall-client-id.access"; +const PUBLISHER = "publisher-client-id.access"; + +async function setup({ guard = {}, env: extra = {} } = {}) { + const access = await fakeAccess(); + const kv = memoryKV(Object.fromEntries(Object.entries(guard).map(([k, v]) => [`guard:${k}`, JSON.stringify(v)]))); + const env = { GUARD: kv, STATUS_ACCESS_AUD: AUD, STATUS_CLIENTS: JSON.stringify({ [WALL]: "internal", [PUBLISHER]: "public" }), ...extra }; + const clock = { t: T0 }; + // the board prunes and filters by its own clock, so it runs on the test's, not the real date + env.BOARD = fakeNamespace(Board, env, (b) => { b.now = () => new Date(clock.t); }); + const handle = createHandler({ getKeys: access.getKeys, now: () => new Date(clock.t) }); + const base = { iss: "https://setkernel.cloudflareaccess.com", aud: [AUD], exp: T0 / 1000 + 3600 }; + const tokens = { + raghav: await access.sign({ ...base, email: "raghav@example.com", sub: "u1" }), + wall: await access.sign({ ...base, common_name: WALL, sub: "" }), + publisher: await access.sign({ ...base, common_name: PUBLISHER, sub: "" }), + stranger: await access.sign({ ...base, common_name: "someone-else.access", sub: "" }), + otherApp: await access.sign({ ...base, aud: ["another-app"], email: "raghav@example.com" }), + expired: await access.sign({ ...base, exp: T0 / 1000 - 1, email: "raghav@example.com" }), + }; + const get = (path, token, headers = {}) => handle(new Request(`https://ctl.setkernel.net${path}`, { headers: { ...(token ? { "cf-access-jwt-assertion": token } : {}), ...headers } }), env); + return { env, kv, clock, tokens, get, feed: new Feed({}, env) }; +} + +// What the control Worker's WorkItem.get() answers +const item = (ref, state, rev, extra = {}, dispatch = null) => ({ item: { ref, kind: "issue", state, rev, seat: null, pr: null, created_at: "2026-10-16T08:00:00Z", updated_at: "2026-10-16T11:00:00Z", ...extra }, dispatch }); +const fired = { at: "2026-10-16T10:59:00Z", seat: "vega-setkernel", rev: 2, outcome: "fired", status: 200, session_id: "session_01ab", session_url: "https://claude.ai/code/session_01ab", error: null }; + +async function fill(feed) { + const r = await feed.publish({ + items: [ + item("acme-client/secret-app#3", "queued", 1), + item("setkernel/sk-demo#14", "queued", 1), + item("setkernel/sk-demo#12", "running", 3, { seat: "vega-setkernel" }, fired), + item("setkernel/hq#7", "merged", 6, { pr: 8, seat: "lyra-setkernel" }), + ], + }); + assert.equal(r.applied, 4); +} + +test("GET /status renders the board for Raghav's phone, from the Durable Object's state", async () => { + const { feed, get, tokens } = await setup(); + await fill(feed); + const res = await get("/status", tokens.raghav, { accept: "text/html,application/xhtml+xml" }); + assert.equal(res.status, 200); + assert.match(res.headers.get("content-type"), /text\/html/); + assert.match(res.headers.get("content-security-policy"), /default-src 'none'/); + const html = await res.text(); + assert.doesNotMatch(html, /Running|Queue|Seats|sk-demo#12|acme-client|session_01ab/, "no running, queue or seats parts"); + assert.match(html, /Last 24 h: 1 merged, 0 failed/); + assert.match(html, /setkernel\/hq#7<\/a>/); + assert.match(html, /Guard: ok/); + assert.match(html, /url\(\/status\/fonts\/manrope-latin-400-normal.woff2\)/); + assert.doesNotMatch(html, / { + const { feed, get, tokens } = await setup(); + await fill(feed); + const res = await get("/status/internal.json", tokens.wall); + assert.equal(res.status, 200); + assert.equal(res.headers.get("cache-control"), "private, max-age=120"); + const v = await res.json(); + assert.deepEqual(Object.keys(v), ["generated_at", "snapshot_at", "guard", "last_24h"], "no queue, running or seats"); + assert.deepEqual(v.last_24h.events.map((e) => [e.ref, e.state, e.pr_url]), [["setkernel/hq#7", "merged", "https://github.com/setkernel/hq/pull/8"]]); + // a browser asking /status with a service token still gets JSON + assert.match((await get("/status", tokens.wall, { accept: "text/html" })).headers.get("content-type"), /json/); +}); + +test("the publisher's token reads the sanitized view and nothing else", async () => { + const { feed, get, tokens } = await setup(); + await fill(feed); + const res = await get("/status/public.json", tokens.publisher); + assert.equal(res.status, 200); + const body = await res.text(); + assert.deepEqual(JSON.parse(body), { generated_at: new Date(T0).toISOString(), last_24h: { merged: 1, failed: 0 } }); + for (const leak of ["acme", "secret-app", "sk-demo", "http", "-setkernel", "session"]) assert.ok(!body.includes(leak), leak); + assert.equal((await get("/status/internal.json", tokens.publisher)).status, 403); + assert.equal((await get("/status", tokens.publisher)).status, 403); + assert.equal((await get("/status/public.json", tokens.raghav)).status, 200); +}); + +test("no valid Access token for this app, no status", async () => { + const { get, tokens, env } = await setup(); + for (const [why, token] of [["none", null], ["garbage", "a.b.c"], ["another app", tokens.otherApp], ["expired", tokens.expired], ["unlisted service token", tokens.stranger]]) { + assert.equal((await get("/status/internal.json", token)).status, 401, why); + } + assert.equal(env.BOARD.gets, 0, "the board is never read before auth"); +}); + +test("fails closed: with no audience configured, even a good token gets a 401", async () => { + const { get, tokens } = await setup({ env: { STATUS_ACCESS_AUD: undefined } }); + assert.equal((await get("/status/internal.json", tokens.raghav)).status, 401); +}); + +test("a malformed STATUS_CLIENTS lets no service token in, and people still get in", async () => { + const { get, tokens } = await setup({ env: { STATUS_CLIENTS: JSON.stringify({ [WALL]: "admin" }) } }); + assert.equal((await get("/status/internal.json", tokens.wall)).status, 401); + assert.equal((await get("/status/internal.json", tokens.raghav)).status, 200); +}); + +test("guard:refuse stops the public feed first; Raghav and the wall still see the guard", async () => { + const until = new Date(T0 + 3 * 86400_000).toISOString(); + const { get, tokens } = await setup({ guard: { refuse: { since: "x", metrics: ["workers_cpu_ms"], until } } }); + const pub = await get("/status/public.json", tokens.publisher); + assert.equal(pub.status, 503); + assert.ok(Number(pub.headers.get("retry-after")) > 0); + const internal = await get("/status/internal.json", tokens.wall); + assert.equal(internal.status, 200); + assert.equal(internal.headers.get("cache-control"), "private, max-age=600"); + assert.deepEqual((await internal.json()).guard, { level: "refuse", alert: true, degrade: true, refuse: true, metrics: ["workers_cpu_ms"], until }); + const html = await (await get("/status", tokens.raghav, { accept: "text/html" })).text(); + assert.match(html, /Guard: refuse<\/span> \(workers_cpu_ms, until/); +}); + +test("pollers reuse a snapshot for 120 s (600 s under degrade), people for 15 s", async () => { + const { get, tokens, env, clock, kv } = await setup(); + await get("/status/internal.json", tokens.wall); + await get("/status/internal.json", tokens.wall); + assert.equal(env.BOARD.gets, 1); + clock.t += 119_000; + const reused = await get("/status/internal.json", tokens.wall); + assert.equal(env.BOARD.gets, 1); + assert.equal(reused.headers.get("vary"), "accept"); + const body = await reused.json(); + assert.equal(body.generated_at, new Date(clock.t).toISOString()); + assert.equal(body.snapshot_at, new Date(T0).toISOString(), "the data's age shows"); + await get("/status/internal.json", tokens.raghav); // older than a person's 15 s + assert.equal(env.BOARD.gets, 2); + await kv.put("guard:degrade", JSON.stringify({ since: "x", metrics: ["do_requests"], until: null })); + clock.t += 121_000; // the guard is re-read after 60 s: degrade; the snapshot is 121 s old + await get("/status/internal.json", tokens.wall); + assert.equal(env.BOARD.gets, 2, "under degrade the wall waits 600 s, not 120"); + clock.t += 480_000; + await get("/status/internal.json", tokens.wall); + assert.equal(env.BOARD.gets, 3); +}); + +test("unknown paths are 404, writes are 405, and an error hides its detail", async () => { + const { get, tokens, env } = await setup(); + assert.equal((await get("/status/../work-items", tokens.raghav)).status, 404); + assert.equal((await get("/statusx", tokens.raghav)).status, 404); + const post = await createHandler()(new Request("https://ctl.setkernel.net/status", { method: "POST" }), env); + assert.equal(post.status, 405); + const broken = { ...env, BOARD: { idFromName: () => ({}), get: () => ({ snapshot: async () => { throw new Error("secret detail"); } }) } }; + const logged = []; + const quiet = console.error; + console.error = (m) => logged.push(m); + try { + const access = await fakeAccess(); + const handle = guarded(createHandler({ getKeys: access.getKeys, now: () => new Date(T0) })); + const token = await access.sign({ iss: "https://setkernel.cloudflareaccess.com", aud: AUD, exp: T0 / 1000 + 60, email: "raghav@example.com" }); + const res = await handle(new Request("https://ctl.setkernel.net/status/internal.json", { headers: { "cf-access-jwt-assertion": token } }), broken); + assert.equal(res.status, 500); + assert.doesNotMatch(await res.text(), /secret detail/); + assert.match(logged.join("\n"), /secret detail/); + } finally { + console.error = quiet; + } +}); + +test("the default export is a fetch handler that fails closed without an audience", async () => { + const res = await worker.fetch(new Request("https://ctl.setkernel.net/status/internal.json", { headers: { "cf-access-jwt-assertion": "a.b.c" } }), { STATUS_ACCESS_AUD: "" }); + assert.equal(res.status, 401); +}); + +test("Feed.publish answers what the board did, so the control Worker can log a rejected view", async () => { + const { feed } = await setup(); + const r = await feed.publish({ items: [item("setkernel/sk-demo#1", "queued", 1), item("bad", "queued", 1)] }); + assert.equal(r.ok, true); + assert.equal(r.applied, 1); + assert.equal(r.rejected[0].error, "ref must be owner/repo#N"); +}); diff --git a/workers/status/wrangler.toml b/workers/status/wrangler.toml new file mode 100644 index 0000000..2659c99 --- /dev/null +++ b/workers/status/wrangler.toml @@ -0,0 +1,47 @@ +# sk-status: the read model of the org, as one Durable Object (hq#208). Routes on +# ctl.setkernel.net/status only, behind its own Access app; no workers.dev or preview URL. The control Worker +# writes to it over a service binding (the Feed entrypoint), never over HTTP (README.md). +name = "sk-status" +main = "src/index.js" +compatibility_date = "2026-10-01" +workers_dev = false +preview_urls = false +# Every var reaches the Worker through the Infisical -> Cloudflare Workers secret sync (project +# sk-ctl-workers, env prod, folder /sk-status only: / holds sk-meter's and the control Worker's credentials), +# never through this file or CI (decision 0003). +keep_vars = true + +# /status and everything under /status/, nothing else on the hostname. The DNS record and the +# Access app are tofu in setkernel/setkernel-cloudflare; the routes need them first (README.md). +routes = [ + { pattern = "ctl.setkernel.net/status", zone_name = "setkernel.net" }, + { pattern = "ctl.setkernel.net/status/*", zone_name = "setkernel.net" }, +] + +# The page's two font files and their licence. Static assets are served before any Worker code +# and aren't billed as Worker requests. +[assets] +directory = "public" + +[observability] +enabled = true + +# sk-meter's guard flags. The `sk-meter` namespace, owned by tofu in setkernel/setkernel-cloudflare +# (output sk_meter_kv_namespace_id): bind it, never create it. sk-status only reads it. +[[kv_namespaces]] +binding = "GUARD" +id = "f79ca12aeb9347f9851f5dfd53313f39" + +[[durable_objects.bindings]] +name = "BOARD" +class_name = "Board" + +# Migration tags are permanent once deployed: never edit or reuse "v1"; a change is a new tag. +[[migrations]] +tag = "v1" +new_sqlite_classes = ["Board"] + +# Set by the sync from Infisical sk-ctl-workers/prod, folder /sk-status (never in this file): +# STATUS_ACCESS_AUD the audience tag of the Access app `sk-status`. Unset, every request is a 401. +# STATUS_CLIENTS JSON { "": "internal" | "public" }: the wall's +# token and the showcase publisher's. Unset, only people get in. From ff1841fa4dd69f6c890f75d81e4a3c94599592c8 Mon Sep 17 00:00:00 2001 From: Raghav Gupta Date: Mon, 9 Nov 2026 09:27:58 -0400 Subject: [PATCH 3/3] fix(deploy): a failed sk-meter rollback reaches the issue, with owner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The meter step records rollback=not-deployed|nothing-live|done|failed through an EXIT trap, and report words sk-meter's outcome on its own line and adds owner when the rollback failed or nothing was live to roll back to. Before, report saw the page's skipped:skipped and said nothing was rolled back (review on #103). Also: AGENTS.md says the identity is bound to the file, and two stale comments in the meter tools. Refs setkernel/hq#1449 โ€” Claude Lead Co-Authored-By: Claude Opus 5.5 --- .github/workflows/deploy.yml | 44 +++++++++++++++++++++++++++++--- AGENTS.md | 6 +++-- workers/meter/tools/baseline.mjs | 2 +- workers/meter/tools/drill.mjs | 2 +- 4 files changed, 46 insertions(+), 8 deletions(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 81ab650..27e00c5 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -209,10 +209,17 @@ jobs: # sk-meter (workers/meter/README.md, "Deploy"): one hourly cron and no route, so nothing can be asked over # HTTP. The job records the version serving all traffic, deploys, and passes only if this run's version # ("deploy ") now serves all of it. Otherwise it rolls back to the recorded version and fails. + # That is a check of the deployment, not a smoke: a version whose cron throws still passes it, and + # sk-watch's failed-cron-event alert covers that. Its `rollback` output tells `report` what happened: + # not-deployed (the deploy failed, nothing changed), nothing-live (no version to roll back to), + # done (rolled back, and the old version serves all traffic again) or failed (the rollback failed + # or didn't take); empty when the job passed. meter: needs: check runs-on: setkernel-linux timeout-minutes: 10 + outputs: + rollback: ${{ steps.meter.outputs.rollback }} permissions: contents: read id-token: write # OIDC login to Infisical @@ -240,7 +247,11 @@ jobs: # Only CLOUDFLARE_API_TOKEN reaches wrangler: env -i drops the OIDC request variables and the # page's two Worker secrets, which sk-meter never gets. Its settings come from the Infisical sync. - name: Deploy sk-meter, check it, roll back if it isn't live + id: meter run: | + # Whatever ends this script, the trap records how far it got (bash -e exits on any failed command). + state=not-deployed + trap 'echo "rollback=${state}" >> "$GITHUB_OUTPUT"' EXIT wrangler() { env -i PATH="$PATH" HOME="$HOME" CI=true WRANGLER_SEND_METRICS=false \ CLOUDFLARE_API_TOKEN="$CLOUDFLARE_API_TOKEN" \ @@ -249,24 +260,32 @@ jobs: live() { wrangler deployments status --json | jq -r '[.versions[]? | select(.percentage == 100) | .version_id][0] // empty'; } before=$(live) || before= wrangler deploy --message "deploy ${GITHUB_SHA}" + state=failed # deployed: from here a failure leaves sk-meter on an unknown version now=$(live) || now= message=$(wrangler versions view "$now" --json | jq -r '.annotations["workers/message"] // empty') || message= if [ "$message" = "deploy ${GITHUB_SHA}" ]; then echo "sk-meter serves ${now} (deploy ${GITHUB_SHA}) to all traffic" + state= exit 0 fi echo "::error::sk-meter's live version ${now:-(none)} is '${message}', not 'deploy ${GITHUB_SHA}'" if [ -z "$before" ]; then echo "::error::no version was live before, so there is nothing to roll back to" + state=nothing-live exit 1 fi wrangler rollback "$before" --message "rollback from ${GITHUB_SHA::7}" --yes - [ "$(live)" = "$before" ] && echo "::error::rolled back to ${before}, which serves all traffic again" || + if [ "$(live)" = "$before" ]; then + echo "::error::rolled back to ${before}, which serves all traffic again" + state=done + else echo "::error::rolled back to ${before}, but it doesn't serve all traffic" + fi exit 1 # Any failure above, the checks included, opens one issue for the agents, and for the owner too when - # a failed smoke could not be rolled back or failed again after it. The runner image has curl and jq + # a failed smoke could not be rolled back or failed again after it, or sk-meter's rollback failed or + # had nothing to roll back to. The runner image has curl and jq # but no gh. report: needs: [check, deploy, smoke, rollback, smoke-again, meter] @@ -286,6 +305,7 @@ jobs: ROLLBACK: ${{ needs.rollback.result }} SMOKE_AGAIN: ${{ needs.smoke-again.result }} METER: ${{ needs.meter.result }} + METER_ROLLBACK: ${{ needs.meter.outputs.rollback }} VERSION: ${{ needs.deploy.outputs.previous-version }} PREVIOUS: ${{ needs.deploy.outputs.previous-sha }} run: | @@ -304,11 +324,27 @@ jobs: more='["owner"]' rolled="The smoke failed and the rollback failed (the job log says why), so ${GITHUB_SHA::7} may still be live. main is unchanged." ;; *) - rolled="Nothing was rolled back: no smoke failed after a deploy." ;; + rolled="The page: nothing was rolled back, because no smoke failed after its deploy." ;; + esac + # sk-meter's own outcome (the meter job). Its rollback failing, or nothing to roll back to, is the owner's. + case "$METER_ROLLBACK" in + not-deployed) + meter_text="sk-meter: the deploy failed before anything changed, so the version live before still serves." ;; + nothing-live) + more='["owner"]' + meter_text="sk-meter: this run's version was not serving all traffic, and no version was live before, so nothing was rolled back. sk-meter may be running no version at all." ;; + done) + meter_text="sk-meter: this run's version was not serving all traffic, so the meter job rolled back to the version live before, which serves all traffic again." ;; + failed) + more='["owner"]' + meter_text="sk-meter: this run's version was not serving all traffic and the rollback failed or did not take, so sk-meter may be on neither version. Read wrangler deployments status --config workers/meter/wrangler.toml." ;; + *) + meter_text="" ;; esac - body="The deploy of ${GITHUB_SHA} failed. Jobs: check ${CHECK}, deploy ${DEPLOY}, smoke ${SMOKE}, rollback ${ROLLBACK}, smoke-again ${SMOKE_AGAIN}, meter ${METER} (sk-meter; its log says whether it rolled back). + body="The deploy of ${GITHUB_SHA} failed. Jobs: check ${CHECK}, deploy ${DEPLOY}, smoke ${SMOKE}, rollback ${ROLLBACK}, smoke-again ${SMOKE_AGAIN}, meter ${METER} (sk-meter). ${rolled} + ${meter_text} Run: ${RUN_URL} diff --git a/AGENTS.md b/AGENTS.md index fe6edf2..a9e0e01 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,11 +38,13 @@ if the rollback or its smoke fails). Where it differs from the template, the hea validate`, the identity and folder, `VERSION` and the Worker secrets in the deploy's one version). It reads `CLOUDFLARE_API_TOKEN`, `GITHUB_WEBHOOK_SECRET` and `BUZZ_PRIVATE_KEY` from Infisical (project `setkernel-live`, prod, `/`) through the OIDC identity `setkernel-live-deploy`, which accepts only that -workflow on `main`, in the `deploy` and `rollback` jobs only. The token itself is +workflow file on `main` (its binding is the file, not a job); only the `deploy`, `rollback` and +`meter` jobs ask for it. The token itself is `setkernel-live-deploy` in setkernel-cloudflare `stacks/services/api_tokens.tf`. Never deploy from a laptop. The same workflow's `meter` job deploys sk-meter (`workers/meter/wrangler.toml`) with that token, -checks that its version serves all traffic and rolls it back in the job if not. +checks that its version serves all traffic and rolls it back in the job if not. A failed rollback, +or nothing live to roll back to, gives the issue `owner` too. ## Layout - `src/worker.js`: routes (`/ws`, `/api/state`, `/api/replay`, `/ingest`, `/ingest/usage`, `/ingest/limits`, `/usage/data`, `/usage/`, `/github/webhook`, `/github/queue`) and the `Hub` and `Usage` Durable Objects. diff --git a/workers/meter/tools/baseline.mjs b/workers/meter/tools/baseline.mjs index b2f231c..78baec3 100644 --- a/workers/meter/tools/baseline.mjs +++ b/workers/meter/tools/baseline.mjs @@ -12,7 +12,7 @@ // TOP_SCRIPTS of each, or every script with --scripts. // // The token needs Account Analytics: Read and nothing else. Pass it from Infisical in the -// environment; it is never printed. CI: the `baseline` workflow (workflow_dispatch). +// environment; it is never printed. Run it by hand (README.md, "The baseline"). import { pathToFileURL } from "node:url"; import allowances from "../allowances.json" with { type: "json" }; diff --git a/workers/meter/tools/drill.mjs b/workers/meter/tools/drill.mjs index 6077b6c..6a8d508 100644 --- a/workers/meter/tools/drill.mjs +++ b/workers/meter/tools/drill.mjs @@ -1,5 +1,5 @@ #!/usr/bin/env node -// The offline drill (`make drill`): the live drill in README.md, step for step, against an +// The offline drill (`node workers/meter/tools/drill.mjs`): the live drill in README.md, step for step, against an // in-memory KV and a fake analytics API. Nothing leaves this machine. Exit 1 if any step doesn't // see what the live drill must see.