From ac8e3e54f0cbc31e59fd41721dda9325469131ad Mon Sep 17 00:00:00 2001 From: Raghav Gupta Date: Mon, 9 Nov 2026 09:12:16 -0400 Subject: [PATCH 1/3] feat(meter): sk-meter moves here as workers/meter, deployed by deploy.yml's meter job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The billing guard's code (src, allowances, tests, baseline and offline drill) moves from the retiring control repository at c1e3253, plan v6 F3. Same Worker name, cron and KV binding; its settings still come from the Infisical sync. deploy.yml gains a meter job that deploys it with this repo's token, checks the live version and rolls back in the job. Refs setkernel/hq#1449 โ€” Claude Lead Co-Authored-By: Claude Opus 5.5 --- .github/workflows/deploy.yml | 70 +++++++- AGENTS.md | 11 +- package.json | 5 +- workers/meter/README.md | 104 +++++++++++ workers/meter/allowances.json | 119 +++++++++++++ workers/meter/src/flags.js | 49 +++++ workers/meter/src/index.js | 57 ++++++ workers/meter/src/meter.js | 176 ++++++++++++++++++ workers/meter/src/notify.js | 33 ++++ workers/meter/src/usage.js | 243 +++++++++++++++++++++++++ workers/meter/src/windows.js | 35 ++++ workers/meter/test/baseline.test.js | 167 +++++++++++++++++ workers/meter/test/fakes.js | 76 ++++++++ workers/meter/test/flags.test.js | 54 ++++++ workers/meter/test/meter.test.js | 202 +++++++++++++++++++++ workers/meter/test/notify.test.js | 69 ++++++++ workers/meter/test/retry.test.js | 108 +++++++++++ workers/meter/test/run.test.js | 266 ++++++++++++++++++++++++++++ workers/meter/tools/baseline.mjs | 78 ++++++++ workers/meter/tools/drill.mjs | 37 ++++ workers/meter/tools/schema.mjs | 97 ++++++++++ workers/meter/wrangler.toml | 33 ++++ 22 files changed, 2083 insertions(+), 6 deletions(-) create mode 100644 workers/meter/README.md create mode 100644 workers/meter/allowances.json create mode 100644 workers/meter/src/flags.js create mode 100644 workers/meter/src/index.js create mode 100644 workers/meter/src/meter.js create mode 100644 workers/meter/src/notify.js create mode 100644 workers/meter/src/usage.js create mode 100644 workers/meter/src/windows.js create mode 100644 workers/meter/test/baseline.test.js create mode 100644 workers/meter/test/fakes.js create mode 100644 workers/meter/test/flags.test.js create mode 100644 workers/meter/test/meter.test.js create mode 100644 workers/meter/test/notify.test.js create mode 100644 workers/meter/test/retry.test.js create mode 100644 workers/meter/test/run.test.js create mode 100644 workers/meter/tools/baseline.mjs create mode 100644 workers/meter/tools/drill.mjs create mode 100644 workers/meter/tools/schema.mjs create mode 100644 workers/meter/wrangler.toml diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index f3b2f04..81ab650 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -7,7 +7,7 @@ name: Deploy # smokes that, and a failed run opens an agent-ready issue with the run's URL. # - One concurrency group, never cancelled: a newer push replaces the pending run, so the newest main # always deploys and no deploy stops half way. -# - Only `deploy` and `rollback` can get the deploy token (id-token: write), and they run no app code: +# - Only `deploy`, `rollback` and `meter` can get the deploy token (id-token: write), and they run no app code: # install scripts are off and wrangler runs without the OIDC request variables. The checks and the # smoke run in jobs that have no token; only `report` can write issues. # - Rollback: `deploy` records the live version, and the commit its "deploy " message names, before @@ -22,6 +22,10 @@ name: Deploy # the two Worker secrets from Infisical into the same version, so each deploy is one version. # - The smoke may wait 10 minutes for the Durable Object to run the new code, so its jobs get 15, and # scripts/smoke.mjs has no dependencies, so they install nothing. +# - The `meter` job deploys the second Worker, sk-meter (workers/meter, setkernel/hq#1449), with the same +# token after the same `check`. It has no route, so it checks the live deployment itself and rolls back +# in the same job; like `deploy` and `rollback` it installs no scripts and runs nothing but wrangler. +# It doesn't wait for the page's deploy, and the page's smoke doesn't wait for it. on: push: branches: [main] @@ -202,11 +206,70 @@ jobs: fi node scripts/smoke.mjs --expect "$EXPECT" + # sk-meter (workers/meter/README.md, "Deploy"): one hourly cron and no route, so nothing can be asked over + # HTTP. The job records the version serving all traffic, deploys, and passes only if this run's version + # ("deploy ") now serves all of it. Otherwise it rolls back to the recorded version and fails. + meter: + needs: check + runs-on: setkernel-linux + timeout-minutes: 10 + permissions: + contents: read + id-token: write # OIDC login to Infisical + env: + WRANGLER_SEND_METRICS: "false" + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - run: npm ci --ignore-scripts --no-audit --no-fund + - name: Fetch the deploy token from Infisical + uses: Infisical/secrets-action@d2e351f16c6ca20d17c85e6c992e04bdeb64e87d # v1.0.18 + with: + method: oidc + identity-id: 37b06681-c931-4bad-b3cf-4e76bfb633dc # setkernel-live-deploy + oidc-audience: https://github.com/setkernel + domain: https://secrets-api.setkernel.net + project-slug: ${{ github.event.repository.name }} + env-slug: prod + secret-path: / + export-type: env + extra-headers: | + CF-Access-Client-Id: ${{ secrets.CF_ACCESS_CLIENT_ID }} + CF-Access-Client-Secret: ${{ secrets.CF_ACCESS_CLIENT_SECRET }} + # Only CLOUDFLARE_API_TOKEN reaches wrangler: env -i drops the OIDC request variables and the + # page's two Worker secrets, which sk-meter never gets. Its settings come from the Infisical sync. + - name: Deploy sk-meter, check it, roll back if it isn't live + run: | + wrangler() { + env -i PATH="$PATH" HOME="$HOME" CI=true WRANGLER_SEND_METRICS=false \ + CLOUDFLARE_API_TOKEN="$CLOUDFLARE_API_TOKEN" \ + npx --no-install wrangler "$@" --config workers/meter/wrangler.toml + } + live() { wrangler deployments status --json | jq -r '[.versions[]? | select(.percentage == 100) | .version_id][0] // empty'; } + before=$(live) || before= + wrangler deploy --message "deploy ${GITHUB_SHA}" + now=$(live) || now= + message=$(wrangler versions view "$now" --json | jq -r '.annotations["workers/message"] // empty') || message= + if [ "$message" = "deploy ${GITHUB_SHA}" ]; then + echo "sk-meter serves ${now} (deploy ${GITHUB_SHA}) to all traffic" + exit 0 + fi + echo "::error::sk-meter's live version ${now:-(none)} is '${message}', not 'deploy ${GITHUB_SHA}'" + if [ -z "$before" ]; then + echo "::error::no version was live before, so there is nothing to roll back to" + exit 1 + fi + wrangler rollback "$before" --message "rollback from ${GITHUB_SHA::7}" --yes + [ "$(live)" = "$before" ] && echo "::error::rolled back to ${before}, which serves all traffic again" || + echo "::error::rolled back to ${before}, but it doesn't serve all traffic" + exit 1 + # Any failure above, the checks included, opens one issue for the agents, and for the owner too when # a failed smoke could not be rolled back or failed again after it. The runner image has curl and jq # but no gh. report: - needs: [check, deploy, smoke, rollback, smoke-again] + needs: [check, deploy, smoke, rollback, smoke-again, meter] if: failure() runs-on: setkernel-linux timeout-minutes: 5 @@ -222,6 +285,7 @@ jobs: SMOKE: ${{ needs.smoke.result }} ROLLBACK: ${{ needs.rollback.result }} SMOKE_AGAIN: ${{ needs.smoke-again.result }} + METER: ${{ needs.meter.result }} VERSION: ${{ needs.deploy.outputs.previous-version }} PREVIOUS: ${{ needs.deploy.outputs.previous-sha }} run: | @@ -242,7 +306,7 @@ jobs: *) rolled="Nothing was rolled back: no smoke failed after a deploy." ;; esac - body="The deploy of ${GITHUB_SHA} failed. Jobs: check ${CHECK}, deploy ${DEPLOY}, smoke ${SMOKE}, rollback ${ROLLBACK}, smoke-again ${SMOKE_AGAIN}. + body="The deploy of ${GITHUB_SHA} failed. Jobs: check ${CHECK}, deploy ${DEPLOY}, smoke ${SMOKE}, rollback ${ROLLBACK}, smoke-again ${SMOKE_AGAIN}, meter ${METER} (sk-meter; its log says whether it rolled back). ${rolled} diff --git a/AGENTS.md b/AGENTS.md index acb897a..fe6edf2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,6 +8,8 @@ secrets, branches and commits, the end-of-run report) come from setkernel's cent ## What this repo is The public page at https://live.setkernel.com: a Cloudflare Worker with static assets and one Durable Object. sk-obs01 pushes filtered, delayed events to it; visitors watch over a WebSocket. +It also holds `workers/meter`, sk-meter, the Cloudflare billing guard (an hourly cron, no route; +`workers/meter/README.md`), moved here on 2026-11-09 (setkernel/hq#1449). Public repo: everything here is visible to anyone. ## Commands @@ -21,8 +23,10 @@ Public repo: everything here is visible to anyone. # its poster (the artifact, the pull request, the shots branch, the comment), # and repo names against GitHub's public list, live) # worker: npm run test:worker (vitest in workerd: the real Worker and Hub, see test/worker/) +# meter: npm run test:meter (node --test: sk-meter's guard, usage queries, alert, baseline) # lint: npm run check (node --check on every script) -# validate: npm run validate (check + test + test:worker + wrangler deploy --dry-run; what CI runs) +# validate: npm run validate (check + test + test:worker + test:meter + a wrangler deploy --dry-run +# of each Worker; what CI runs) ``` ## Deploy @@ -37,6 +41,8 @@ reads `CLOUDFLARE_API_TOKEN`, `GITHUB_WEBHOOK_SECRET` and `BUZZ_PRIVATE_KEY` fro workflow on `main`, in the `deploy` and `rollback` jobs only. The token itself is `setkernel-live-deploy` in setkernel-cloudflare `stacks/services/api_tokens.tf`. Never deploy from a laptop. +The same workflow's `meter` job deploys sk-meter (`workers/meter/wrangler.toml`) with that token, +checks that its version serves all traffic and rolls it back in the job if not. ## Layout - `src/worker.js`: routes (`/ws`, `/api/state`, `/api/replay`, `/ingest`, `/ingest/usage`, `/ingest/limits`, `/usage/data`, `/usage/`, `/github/webhook`, `/github/queue`) and the `Hub` and `Usage` Durable Objects. @@ -91,6 +97,9 @@ a laptop. put text from the artifact in the comment, and never write any ref but `visual-diff-shots`. The poster reads only the files main's `SHOTS` (in `scripts/visual-diff.mjs`) names, so a pull request that changes `SHOTS` gets no comment ("unexpected file in the artifact") until it merges. +- `workers/meter/`: sk-meter, its own Worker (`wrangler.toml`, not `wrangler.jsonc`): `src/` the hourly + run and the guard flags, `allowances.json`, `tools/` the baseline and the offline drill, `test/` its + `node --test` suites. Its settings come from the Infisical -> Workers sync (`keep_vars = true`). - `test/`: `node --test` suites for the pure modules and the page (`day.test.mjs` for Day). - `test/worker/`: vitest suites run in workerd by `@cloudflare/vitest-plugin` (`vitest.config.mjs` reads `wrangler.jsonc`). They fake Access's certs with a throwaway key and drive every route diff --git a/package.json b/package.json index 926179d..cdcc6a4 100644 --- a/package.json +++ b/package.json @@ -7,8 +7,9 @@ "scripts": { "test": "node --test test/*.test.mjs", "test:worker": "vitest run", - "check": "for f in src/*.js public/*.js public/lib/*.js public/usage/*.js scripts/*.mjs; do node --check \"$f\" || exit 1; done", - "validate": "npm run check && npm test && npm run test:worker && wrangler deploy --dry-run --outdir .wrangler/dry-run" + "test:meter": "node --test workers/meter/test/*.test.js", + "check": "for f in src/*.js public/*.js public/lib/*.js public/usage/*.js scripts/*.mjs workers/meter/src/*.js workers/meter/tools/*.mjs; do node --check \"$f\" || exit 1; done", + "validate": "npm run check && npm test && npm run test:worker && npm run test:meter && wrangler deploy --dry-run --outdir .wrangler/dry-run && wrangler deploy --dry-run --config workers/meter/wrangler.toml --outdir .wrangler/dry-run-meter" }, "dependencies": { "@noble/secp256k1": "3.2.0" diff --git a/workers/meter/README.md b/workers/meter/README.md new file mode 100644 index 0000000..4eb666a --- /dev/null +++ b/workers/meter/README.md @@ -0,0 +1,104 @@ +# sk-meter: the Cloudflare billing guard + +Cloudflare's budget alerts email a day late and don't cap usage, and overage on Workers Paid +bills automatically. sk-meter keeps setkernel inside the $5 plan. Every hour (cron `7 * * * *`) +it reads the account's usage through the GraphQL Analytics API, totals each metric over its +window, compares it with the plan's included allowance and sets three KV flags. + +| Share of any allowance | KV key | What happens | +|---|---|---| +| 50% | `guard:alert` | an alert to Raghav | +| 80% | `guard:degrade` | readers run with `DEGRADED` (`src/flags.js`) | +| 95% | `guard:refuse` | readers refuse non-essential requests until the window resets | + +Cloudflare's own $10 budget alert stays on as the backstop. + +It moved here from a private repository on 2026-11-09 (setkernel/hq#1449, plan v6 F3). Its +history up to then is that repository's `meter/` at commit `c1e3253`. Not moved, and retired +with that repository: the Workers Logs `readings` tool and the `baseline` and live `drill` +workflows (each read Infisical through an identity bound to a workflow there). + +## Files +- `allowances.json`: the included allowance for each metric, its window (`month` from the + billing day, `day` from 00:00 UTC for Workers AI, `gauge` for storage), and the pricing page + it came from. Change it only from those pages. +- `src/usage.js`: the GraphQL queries, one per dataset. +- `src/meter.js`: shares, levels, flags, the pro-rated projection, the alert text. Pure. +- `src/flags.js`: the contract for every reader: `readGuard(env.GUARD)`, + `refusal(guard, { essential })` and `DEGRADED`. +- `src/index.js`: the hourly run. `src/notify.js`: the alert webhook. +- `tools/baseline.mjs`: a read-only report of current usage, also the schema check. +- `tools/drill.mjs`: the offline drill (below). + +## How it decides +- Each metric's total is recomputed from the start of its window every hour, so a missed run + loses nothing. +- A flag is on while any metric is at or past its threshold, and lists those metrics and when + the last of them resets (`until`; `null` when storage holds it, because only deleting data + brings that down). +- A query that fails keeps the metric's last value from the same window, so an outage of the + analytics API never lowers a flag; the failure is in the alert. +- Pro-rated: once 10% of a window has passed, a metric on pace to pass its allowance before the + reset is named in the alert. Flags act on actual use, not the projection. +- A reader that can't reach KV treats it as degrade, not refuse. +- The Workers and Durable Object queries also group by `scriptName` in the same query, so a + metric projected to reach 50% names its top script in the alert (`topScripts` in the report). +- An alert goes out when the flags, errors, on-pace list or Workers AI gateways change, and is + retried each hour until delivered. + +## What it costs +Its own use of the plan (https://developers.cloudflare.com/workers/platform/pricing/), a month: + +| Metric | sk-meter's use | Included | Share | +|---|---|---|---| +| Workers requests (cron runs) | 720 | 10,000,000 | 0.01% | +| Workers CPU time | estimate: a few ms a run, well under 36,000 ms | 30,000,000 ms | <0.2% | +| KV reads | 1,440 (report + drill each run) | 10,000,000 | 0.01% | +| KV writes | 720 (report) + a few on flag changes | 1,000,000 | <0.1% | +| GraphQL Analytics subrequests | ~11 a run; up to ~44 when every dataset needs its 3 retries | not billed | | + +## Commands +```sh +npm run test:meter # the tests (node --test); npm run validate runs them too +node workers/meter/tools/drill.mjs # the offline drill: in-memory KV, fake analytics API, nothing leaves the machine +CF_ACCOUNT_ID=โ€ฆ CF_ANALYTICS_TOKEN=โ€ฆ BILLING_DAY=13 node workers/meter/tools/baseline.mjs # live, read-only +``` + +## Deploy +Only from CI: the `meter` job in `.github/workflows/deploy.yml`, on every push to `main` and by +hand, after the same `check` job as the page. It reads the repo's deploy token (Infisical +project `setkernel-live`, identity `setkernel-live-deploy`: Workers Editor on the account, which +deploys an existing Worker and binds an existing KV namespace). sk-meter has no route, so the +job checks the live deployment itself: the version serving all traffic must be this run's +(`deploy `). If not, it rolls back to the version that was live before, and the job fails, +so `report` opens an issue. Nothing sets vars or secrets on deploy (`keep_vars = true`). + +### The Worker's settings +All in Infisical `sk-ctl-workers/prod`, reaching the Worker through the Infisical -> Cloudflare +Workers secret sync (target Worker `sk-meter`). The sync is by Worker name, so the move changed +nothing there. + +| Key | Value | +|---|---| +| `BILLING_DAY` | `13` (cycle starts on the 13th; required, 1-28) | +| `CF_ACCOUNT_ID` | the account to meter | +| `CF_ANALYTICS_TOKEN` | `sk-meter-analytics-read`: Account Analytics Read, nothing else (setkernel-cloudflare tofu) | +| `ALERT_WEBHOOK_URL` | `https://alerts.setkernel.net/meter` (https only; anything else is refused) | +| `ALERT_WEBHOOK_TOKEN` | bearer the receiver checks | +| `ALERT_ACCESS_CLIENT_ID`, `ALERT_ACCESS_CLIENT_SECRET` | the `sk-meter-alerts` Access service token | + +The KV namespace `sk-meter` (`f79ca12aeb9347f9851f5dfd53313f39`) is tofu-owned in +setkernel-cloudflare: this repo binds it and never creates or deletes it. + +## The baseline +`tools/baseline.mjs` introspects the GraphQL schema for every dataset, field and filter key in +`src/usage.js`, then reads the current usage the way the hourly run does, and prints both. Any +"Can't read" line fails it. A dataset still answering "try again later" after three retries is +an "Unknown" line: its metric shows as `?` and the run passes. It writes nothing and prints no +secret. + +## The drill +Proves the degrade end to end without extra spend: `meter:drill` in the namespace lowers one +metric's allowance below what is already used, until its `expiresAt`. A drill can only lower an +allowance. `tools/drill.mjs` runs the steps offline and fails if any step doesn't see what it +should. diff --git a/workers/meter/allowances.json b/workers/meter/allowances.json new file mode 100644 index 0000000..9864ad8 --- /dev/null +++ b/workers/meter/allowances.json @@ -0,0 +1,119 @@ +{ + "plan": "Workers Paid, $5 a month; no spend above it (setkernel/hq docs/decisions/0007-model-v4c.html)", + "checked": "2026-10-15", + "thresholds": { "alert": 0.5, "degrade": 0.8, "refuse": 0.95 }, + "metrics": { + "workers_requests": { + "label": "Workers requests", + "allowance": 10000000, + "unit": "requests", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "workers_cpu_ms": { + "label": "Workers CPU time", + "allowance": 30000000, + "unit": "CPU ms", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "do_requests": { + "label": "Durable Objects requests", + "allowance": 1000000, + "unit": "requests", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "do_duration_gbs": { + "label": "Durable Objects duration", + "allowance": 400000, + "unit": "GB-s", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "workflows_steps": { + "label": "Workflows steps", + "allowance": 500000, + "unit": "steps", + "window": "month", + "source": "https://developers.cloudflare.com/workflows/reference/pricing/" + }, + "queues_ops": { + "label": "Queues operations", + "allowance": 1000000, + "unit": "operations", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "d1_rows_read": { + "label": "D1 rows read", + "allowance": 25000000000, + "unit": "rows", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "d1_rows_written": { + "label": "D1 rows written", + "allowance": 50000000, + "unit": "rows", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "r2_storage_bytes": { + "label": "R2 storage", + "allowance": 10000000000, + "unit": "bytes (10 GB-month, counted as 10^9-byte GB so the guard errs low)", + "window": "gauge", + "source": "https://developers.cloudflare.com/r2/pricing/" + }, + "r2_class_a": { + "label": "R2 Class A operations", + "allowance": 1000000, + "unit": "requests", + "window": "month", + "source": "https://developers.cloudflare.com/r2/pricing/" + }, + "r2_class_b": { + "label": "R2 Class B operations", + "allowance": 10000000, + "unit": "requests", + "window": "month", + "source": "https://developers.cloudflare.com/r2/pricing/" + }, + "kv_reads": { + "label": "KV reads", + "allowance": 10000000, + "unit": "operations", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "kv_writes": { + "label": "KV writes", + "allowance": 1000000, + "unit": "operations", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "kv_deletes": { + "label": "KV deletes", + "allowance": 1000000, + "unit": "operations", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "kv_lists": { + "label": "KV lists", + "allowance": 1000000, + "unit": "operations", + "window": "month", + "source": "https://developers.cloudflare.com/workers/platform/pricing/" + }, + "ai_neurons": { + "label": "Workers AI neurons", + "allowance": 10000, + "unit": "neurons", + "window": "day", + "source": "https://developers.cloudflare.com/workers-ai/platform/pricing/" + } + } +} diff --git a/workers/meter/src/flags.js b/workers/meter/src/flags.js new file mode 100644 index 0000000..26d3332 --- /dev/null +++ b/workers/meter/src/flags.js @@ -0,0 +1,49 @@ +// The guard's contract with every Worker that reads it. sk-meter writes these KV keys in the GUARD +// namespace; every other component binds the same namespace read-only in spirit and imports this +// module. A key is present while its threshold is passed and deleted when usage falls back (the +// window resets). Each value is JSON: { since, metrics: [ids], until: ISO time or null }. +// +// guard:alert 50% of any allowance: Buzz alert to Raghav (sk-meter sends it) +// guard:degrade 80%: run with DEGRADED below +// guard:refuse 95%: refuse non-essential requests until the window resets +// +// The static status page is served as static assets, which run before any Worker code, so it +// keeps serving whatever these flags say. + +export const KEYS = { alert: "guard:alert", degrade: "guard:degrade", refuse: "guard:refuse" }; +export const REPORT_KEY = "meter:report"; +export const DRILL_KEY = "meter:drill"; + +// What a component does while guard:degrade is on. +export const DEGRADED = Object.freeze({ + heartbeatSeconds: 60, + retentionDays: 7, + bundlesPaused: true, + newLeases: false, + workersAi: false, +}); + +// Read the three flags. KV caches each for a minute at the edge, so a busy Worker costs a few +// reads a minute, not one per request. If KV can't be read the guard can't be confirmed, so the +// answer is degrade (stop starting new spend) but not refuse (keep serving). +export async function readGuard(kv, { cacheTtl = 60 } = {}) { + try { + const [alert, degrade, refuse] = await Promise.all( + [KEYS.alert, KEYS.degrade, KEYS.refuse].map((k) => kv.get(k, { type: "json", cacheTtl })), + ); + // degrade is implied by refuse, alert by both: a reader never sees refuse without degrade. + return { alert: Boolean(alert || degrade || refuse), degrade: Boolean(degrade || refuse), refuse: Boolean(refuse), detail: { alert, degrade, refuse } }; + } catch (e) { + return { alert: true, degrade: true, refuse: false, error: String(e?.message ?? e) }; + } +} + +// For a Worker's fetch handler: a 503 to send instead of doing the work, or null to carry on. +// Essential requests (the status page, health checks, the guard's own readers) always carry on. +export function refusal(guard, { essential = false } = {}) { + if (!guard.refuse || essential) return null; + const until = guard.detail?.refuse?.until; + const headers = { "content-type": "text/plain; charset=utf-8" }; + if (until) headers["retry-after"] = String(Math.max(60, Math.ceil((Date.parse(until) - Date.now()) / 1000))); + return new Response("Paused: setkernel is holding Cloudflare usage inside its plan until the usage window resets.\n", { status: 503, headers }); +} diff --git a/workers/meter/src/index.js b/workers/meter/src/index.js new file mode 100644 index 0000000..4c0ca06 --- /dev/null +++ b/workers/meter/src/index.js @@ -0,0 +1,57 @@ +// sk-meter: Cloudflare's budget alerts arrive a day late and stop nothing, and overage on Workers +// Paid bills automatically. Every hour this reads the account's usage, compares it with the +// plan's included allowances (allowances.json) and sets the guard flags (flags.js) that the +// Workers reading them obey. +// +// Its own cost: 720 cron runs a month, ~11 GraphQL subrequests each (up to ~44 when every dataset +// needs all its retries), two KV reads, one KV write for the report and a write only when a flag +// changes. + +import allowances from "../allowances.json" with { type: "json" }; +import { evaluate, message, signature, FLAGS } from "./meter.js"; +import { KEYS, REPORT_KEY, DRILL_KEY } from "./flags.js"; +import { notify } from "./notify.js"; +import { readUsage } from "./usage.js"; +import { billingDay } from "./windows.js"; + +export async function run(env, now, { fetchImpl = fetch, table = allowances, retry } = {}) { + const kv = env.GUARD; + const day = billingDay(env.BILLING_DAY); + const [prev, drill] = await Promise.all([kv.get(REPORT_KEY, "json"), kv.get(DRILL_KEY, "json")]); + const usage = await readUsage(env, now, day, fetchImpl, retry); + const report = evaluate({ allowances: table, usage, now, day, prev, drill }); + + // Flags first: they are what stops spend. Write only on change; on the first run (no previous + // report) clear any flag left over from before. + for (const name of FLAGS) { + const was = prev ? prev.flags?.[name] ?? null : undefined; + const is = report.flags[name]; + if (is && JSON.stringify(is) !== JSON.stringify(was)) await kv.put(KEYS[name], JSON.stringify(is)); + else if (!is && was !== null) await kv.delete(KEYS[name]); + } + + // Alert when the state changes, and retry each hour until one is delivered. The first run has + // nothing to compare with, so it always reports: that is the baseline. + const sig = signature(report); + report.notified = prev?.notified ?? null; + if (sig !== report.notified) { + try { + if (await notify(env, message(report, { full: !prev }), fetchImpl)) report.notified = sig; + } catch (e) { + report.notifyError = e.message; + console.error(`sk-meter: alert failed: ${e.message}`); + } + } + + await kv.put(REPORT_KEY, JSON.stringify(report)); + const on = FLAGS.filter((f) => report.flags[f]); + console.log(`sk-meter: flags ${on.join(",") || "none"}; errors ${report.errors.join(",") || "none"}; unknown ${report.unknown.join(",") || "none"}`); + return report; +} + +export default { + // Awaited, so a run that throws shows as a failed cron event (sk-watch watches for those). + async scheduled(controller, env) { + await run(env, new Date(controller.scheduledTime)); + }, +}; diff --git a/workers/meter/src/meter.js b/workers/meter/src/meter.js new file mode 100644 index 0000000..715cd36 --- /dev/null +++ b/workers/meter/src/meter.js @@ -0,0 +1,176 @@ +// Turns usage into the guard's state: a share of the allowance per metric, the three flags, and +// the alert to send. Pure: no I/O, so every threshold is tested with synthetic usage. + +import { windowFor, elapsed } from "./windows.js"; + +export const LEVELS = ["ok", "alert", "degrade", "refuse"]; +export const FLAGS = ["alert", "degrade", "refuse"]; + +// Projection is noisy early in a window; it only speaks after this share has passed. +const PACE_AFTER = 0.1; + +export function levelOf(share, thresholds) { + if (share == null) return "unknown"; + if (share >= thresholds.refuse) return "refuse"; + if (share >= thresholds.degrade) return "degrade"; + if (share >= thresholds.alert) return "alert"; + return "ok"; +} + +// A drill lowers one metric's allowance until it expires, so real usage crosses a threshold with +// no extra spend. It can only make the guard stricter: a drill that would raise an allowance, or +// names an unknown metric, or has expired, is ignored. +export function activeDrill(drill, allowances, now) { + if (!drill || typeof drill !== "object") return null; + const pinned = allowances.metrics[drill.metric]?.allowance; + const a = Number(drill.allowance); + const until = Date.parse(drill.expiresAt); + if (!pinned || !(a > 0) || a >= pinned || !(until > now.getTime())) return null; + return { metric: drill.metric, allowance: a, expiresAt: new Date(until).toISOString() }; +} + +// The script using most of a metric that is projected to reach the alert level (50%) of its +// allowance, for any metric the run read per script. Ties go to the name that sorts first. +function topScripts(metrics, scripts, th) { + const out = {}; + for (const [id, by] of Object.entries(scripts ?? {})) { + const m = metrics[id]; + if (!m || m.projected == null || m.projected / m.allowance < th.alert) continue; + const rows = Object.entries(by).sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1)); + const sum = rows.reduce((t, [, v]) => t + v, 0); + if (!rows.length || !(sum > 0)) continue; + out[id] = { script: rows[0][0], used: rows[0][1], share: rows[0][1] / sum }; + } + return out; +} + +export function evaluate({ allowances, usage, now, day = 1, prev = null, drill = null }) { + const th = allowances.thresholds; + const d = activeDrill(drill, allowances, now); + const metrics = {}; + for (const [id, spec] of Object.entries(allowances.metrics)) { + const w = windowFor(spec.window, now, day); + const windowStart = w.start.toISOString(); + const allowance = d?.metric === id ? d.allowance : spec.allowance; + let used = usage.values[id] ?? null; + let stale = false; + const error = usage.errors[id]; + const unknown = usage.unknown?.[id]; + if (used == null) { + // Unreadable: keep the last value from the same window, so a failing query can never + // lower a flag. A gauge's window moves every hour, so match on the window kind instead. + const last = prev?.metrics?.[id]; + const sameWindow = last && (spec.window === "gauge" || last.windowStart === windowStart); + if (sameWindow && last.used != null) { used = last.used; stale = true; } + } + const share = used == null ? null : used / allowance; + const f = spec.window === "gauge" ? null : elapsed(w, now); + const projected = used == null || f == null || f < PACE_AFTER ? null : used / f; + metrics[id] = { + label: spec.label, unit: spec.unit, window: spec.window, used, allowance, share, + level: levelOf(share, th), + projected, + windowStart, + resetsAt: spec.window === "gauge" ? null : w.end.toISOString(), + ...(stale && { stale }), + ...(error && { error }), + ...(unknown && { unknown }), + ...(d?.metric === id && { drill: d.expiresAt }), + }; + } + + const flags = {}; + for (const name of FLAGS) { + const rank = LEVELS.indexOf(name); + const hit = Object.entries(metrics).filter(([, m]) => LEVELS.indexOf(m.level) >= rank); + if (!hit.length) { flags[name] = null; continue; } + const resets = hit.map(([, m]) => m.resetsAt); + flags[name] = { + since: prev?.flags?.[name]?.since ?? now.toISOString(), + metrics: hit.map(([id]) => id), + // When the last metric holding the flag resets; null while storage holds it (deleting + // objects is the only way down). + until: resets.includes(null) ? null : resets.sort().at(-1), + }; + } + + const errors = Object.keys(metrics).filter((id) => metrics[id].error).sort(); + // Still failing with a "try again" after the retries: not an error, but nothing is known this run. + const unknown = Object.keys(metrics).filter((id) => metrics[id].unknown).sort(); + const overPace = Object.keys(metrics) + .filter((id) => metrics[id].projected != null && metrics[id].projected >= metrics[id].allowance) + .sort(); + return { + at: now.toISOString(), + metrics, + flags, + errors, + unknown, + overPace, + drill: d, + gateways: usage.gateways ?? null, + topScripts: topScripts(metrics, usage.scripts, th), + }; +} + +// What the alert is about. A new alert goes out when this changes, and keeps being retried each +// hour until one is delivered. +export function signature(report) { + return JSON.stringify({ + flags: FLAGS.filter((f) => report.flags[f]).map((f) => [f, report.flags[f].metrics]), + errors: report.errors, + // A blip with a last known value changes nothing and sends nothing; a metric that has never + // been read is unguarded, so that stays in the alert. + unknown: (report.unknown ?? []).filter((id) => report.metrics[id].used == null), + overPace: report.overPace, + drill: report.drill?.metric ?? null, + workersAi: report.gateways?.workersAi ?? [], + // Not report.topScripts: the top script moves as usage does, and would re-send the alert. + }); +} + +const n = (x) => Math.round(x).toLocaleString("en-US"); +const pct = (x) => `${Math.round(x * 100)}%`; +const when = (iso) => iso.replace("T", " ").replace(/:00\.000Z$/, " UTC"); + +// A script name is the account owner's, but it goes into a chat message: keep it to name characters. +const scriptName = (name) => `\`${String(name).replace(/[^\w.\-/() ]/g, "?").slice(0, 64)}\``; + +function line(m) { + const per = m.window === "day" ? " a day" : m.window === "month" ? " this month" : ""; + const reset = m.resetsAt ? `, resets ${when(m.resetsAt)}` : ", stays until data is deleted"; + return `- ${m.label}: ${n(m.used)} of ${n(m.allowance)} ${m.unit.split(" (")[0]}${per} (${pct(m.share)})${reset}${m.stale ? " [last known]" : ""}`; +} + +// The alert text. `full` lists every metric, for the baseline report on the first run. +export function message(report, { full = false } = {}) { + const top = [...FLAGS].reverse().find((f) => report.flags[f]); + const head = { + refuse: "๐Ÿ”ด **sk-meter: REFUSE** (95%): non-essential requests are refused until the window resets.", + degrade: "๐ŸŸ  **sk-meter: DEGRADE** (80%): heartbeats 60 s, retention 7 days, bundles paused, no new leases, Workers AI off.", + alert: "๐ŸŸก **sk-meter: 50%** of an allowance is used.", + }[top] ?? "โœ… **sk-meter**: every metric is under 50% of its allowance."; + const lines = [(report.drill ? "[DRILL] " : "") + head]; + for (const m of Object.values(report.metrics)) { + if (m.used != null && (full || m.level !== "ok")) lines.push(line(m)); + } + for (const [id, t] of Object.entries(report.topScripts ?? {})) { + lines.push(`Top script for ${report.metrics[id].label}: ${scriptName(t.script)}, ${n(t.used)} ${report.metrics[id].unit.split(" (")[0]} (${pct(t.share)} of the account's).`); + } + const gw = report.gateways; + if (gw?.workersAi?.length) lines.push(`AI Gateways sending requests to Workers AI: ${gw.workersAi.join(", ")}.`); + else if (full && gw?.all) lines.push(`AI Gateways with traffic this month: ${gw.all.length}; none send requests to Workers AI.`); + else if (full && gw?.transient) lines.push(`AI Gateway traffic is unknown this run (${gw.error}).`); + else if (full && gw?.error) lines.push(`Can't read AI Gateway traffic (${gw.error}).`); + if (report.overPace.length) { + lines.push(`On pace to pass the allowance before it resets: ${report.overPace.map((id) => report.metrics[id].label).join(", ")}.`); + } + if (report.errors.length) { + lines.push(`Can't read ${report.errors.map((id) => report.metrics[id].label).join(", ")} (${report.metrics[report.errors[0]].error}); flags keep their last known values.`); + } + if (report.unknown?.length) { + lines.push(`Unknown this run (the API said to try again later): ${report.unknown.map((id) => report.metrics[id].label).join(", ")} (${report.metrics[report.unknown[0]].unknown}); flags keep their last known values.`); + } + if (report.drill) lines.push(`Drill: ${report.drill.metric} allowance lowered to ${n(report.drill.allowance)} until ${when(report.drill.expiresAt)}.`); + return { text: lines.join("\n"), mention: Boolean(report.flags.degrade || report.flags.refuse) }; +} diff --git a/workers/meter/src/notify.js b/workers/meter/src/notify.js new file mode 100644 index 0000000..d8f09c8 --- /dev/null +++ b/workers/meter/src/notify.js @@ -0,0 +1,33 @@ +// Alert delivery. A Worker can't run the buzz CLI or hold a Buzz identity the way hq's +// observability bridge does, so it POSTs { source, text, mention } to ALERT_WEBHOOK_URL (a Worker +// secret) with ALERT_WEBHOOK_TOKEN as a bearer token, and the receiver sends it to Buzz. The +// receiver is the buzz-alerts bridge's /meter route on sk-obs01 (hq#222, observability/README.md), +// behind Cloudflare Access: ALERT_ACCESS_CLIENT_ID and ALERT_ACCESS_CLIENT_SECRET are the +// sk-meter-alerts service token. A caller may name its own `source`. Without a webhook the alert is logged and the KV flags still act; +// the guard never depends on the alert arriving. + +export async function notify(env, alert, fetchImpl = fetch) { + const url = env.ALERT_WEBHOOK_URL; + if (!url) { + console.warn(`sk-meter: no ALERT_WEBHOOK_URL, alert not delivered: ${alert.text}`); + return false; + } + // Never send the bearer token over plain HTTP. + if (!url.startsWith("https://")) throw new Error("ALERT_WEBHOOK_URL must be https"); + const headers = { "content-type": "application/json" }; + if (env.ALERT_WEBHOOK_TOKEN) headers.authorization = `Bearer ${env.ALERT_WEBHOOK_TOKEN}`; + if (env.ALERT_ACCESS_CLIENT_ID && env.ALERT_ACCESS_CLIENT_SECRET) { + headers["cf-access-client-id"] = env.ALERT_ACCESS_CLIENT_ID; + headers["cf-access-client-secret"] = env.ALERT_ACCESS_CLIENT_SECRET; + } + const res = await fetchImpl(url, { + method: "POST", + headers, + // Access answers a refused token with a redirect to its login page; following it could end in + // a 200 and count the alert as delivered. A redirect is a failure, retried next hour. + redirect: "manual", + body: JSON.stringify({ source: alert.source ?? "sk-meter", text: alert.text, mention: alert.mention }), + }); + if (!res.ok) throw new Error(`alert webhook answered HTTP ${res.status}`); + return true; +} diff --git a/workers/meter/src/usage.js b/workers/meter/src/usage.js new file mode 100644 index 0000000..90ebb4c --- /dev/null +++ b/workers/meter/src/usage.js @@ -0,0 +1,243 @@ +// Account usage from Cloudflare's GraphQL Analytics API. One query per dataset, so a field the API +// rejects costs that dataset's metrics only (they are reported as errors and keep their last +// value), never the whole run. A server-side "try again" (HTTP 5xx or 429, a network error, or a +// GraphQL "try again later") is retried, then reported as unknown for this run; only a rejection +// that retrying can't fix (a missing field, a 403) is an error. Each query covers the metric's whole window, so a missed hour is +// caught up on the next run and nothing drifts. +// +// The two datasets behind the metrics hq#243 can't place (Workers CPU, Durable Object requests) are +// also grouped by script, in the same query, so the totals don't change: no `datetime` dimension, so +// a query returns one row per script. `perScript` maps a metric to what one row contributes to it. +// +// Dataset and field names come from the product docs' analytics pages (2026-10-15). Names marked +// "unconfirmed" were not on those pages; `node tools/baseline.mjs` checks every name here +// against the live schema before the first deploy. + +import { windowFor } from "./windows.js"; + +export const ENDPOINT = "https://api.cloudflare.com/client/v4/graphql"; + +// R2 operation classes, from https://developers.cloudflare.com/r2/pricing/. Anything not Class B +// counts as Class A, the smaller allowance, so an action we don't know about errs high. +export const R2_CLASS_B = new Set([ + "HeadBucket", "HeadObject", "GetObject", "UsageSummary", "GetBucketEncryption", + "GetBucketLocation", "GetBucketCors", "GetBucketLifecycleConfiguration", +]); + +const total = (rows, pick) => rows.reduce((n, r) => n + (Number(pick(r)) || 0), 0); + +export const SOURCES = [ + { + // unconfirmed: sum.cpuTimeUs (the tutorial shows only quantiles of CPU time) and + // dimensions.scriptName on this dataset + dataset: "workersInvocationsAdaptive", window: "month", + fields: "sum { requests cpuTimeUs } dimensions { scriptName }", + metrics: ["workers_requests", "workers_cpu_ms"], + perScript: { + workers_requests: (r) => r.sum?.requests, + workers_cpu_ms: (r) => (Number(r.sum?.cpuTimeUs) || 0) / 1000, + }, + read: (rows) => ({ + workers_requests: total(rows, (r) => r.sum?.requests), + workers_cpu_ms: total(rows, (r) => r.sum?.cpuTimeUs) / 1000, + }), + }, + { + // unconfirmed: dimensions.scriptName on this dataset + dataset: "durableObjectsInvocationsAdaptiveGroups", window: "month", + fields: "sum { requests } dimensions { scriptName }", + metrics: ["do_requests"], + perScript: { do_requests: (r) => r.sum?.requests }, + read: (rows) => ({ do_requests: total(rows, (r) => r.sum?.requests) }), + }, + { + // unconfirmed: sum.activeTime in microseconds. Duration bills 128 MB per active object, so + // GB-s = seconds active x 0.125. + dataset: "durableObjectsPeriodicGroups", window: "month", + fields: "sum { activeTime }", + metrics: ["do_duration_gbs"], + read: (rows) => ({ do_duration_gbs: (total(rows, (r) => r.sum?.activeTime) / 1e6) * 0.125 }), + }, + { + // A billed step is a step run; retries (ATTEMPT_*) and rollbacks don't count. + dataset: "workflowsAdaptiveGroups", window: "month", + filter: 'eventType: "STEP_START"', + fields: "count", + metrics: ["workflows_steps"], + read: (rows) => ({ workflows_steps: total(rows, (r) => r.count) }), + }, + { + dataset: "queueMessageOperationsAdaptiveGroups", window: "month", + fields: "sum { billableOperations }", + metrics: ["queues_ops"], + read: (rows) => ({ queues_ops: total(rows, (r) => r.sum?.billableOperations) }), + }, + { + dataset: "d1AnalyticsAdaptiveGroups", window: "month", + fields: "sum { rowsRead rowsWritten }", + metrics: ["d1_rows_read", "d1_rows_written"], + read: (rows) => ({ + d1_rows_read: total(rows, (r) => r.sum?.rowsRead), + d1_rows_written: total(rows, (r) => r.sum?.rowsWritten), + }), + }, + { + dataset: "r2OperationsAdaptiveGroups", window: "month", + fields: "sum { requests } dimensions { actionType }", + metrics: ["r2_class_a", "r2_class_b"], + read: (rows) => ({ + r2_class_a: total(rows.filter((r) => !R2_CLASS_B.has(r.dimensions?.actionType)), (r) => r.sum?.requests), + r2_class_b: total(rows.filter((r) => R2_CLASS_B.has(r.dimensions?.actionType)), (r) => r.sum?.requests), + }), + }, + { + // The largest size each bucket reached in the last day, summed over buckets. + dataset: "r2StorageAdaptiveGroups", window: "gauge", + fields: "max { payloadSize metadataSize } dimensions { bucketName }", + metrics: ["r2_storage_bytes"], + read: (rows) => ({ + r2_storage_bytes: total(rows, (r) => (Number(r.max?.payloadSize) || 0) + (Number(r.max?.metadataSize) || 0)), + }), + }, + { + // actionType is read, write, delete or list; anything else counts as a write (smallest allowance). + dataset: "kvOperationsAdaptiveGroups", window: "month", + fields: "sum { requests } dimensions { actionType }", + metrics: ["kv_reads", "kv_writes", "kv_deletes", "kv_lists"], + read: (rows) => { + const out = { kv_reads: 0, kv_writes: 0, kv_deletes: 0, kv_lists: 0 }; + const key = { read: "kv_reads", delete: "kv_deletes", list: "kv_lists" }; + for (const r of rows) out[key[r.dimensions?.actionType] ?? "kv_writes"] += Number(r.sum?.requests) || 0; + return out; + }, + }, + { + // unconfirmed: dataset aiInferenceAdaptiveGroups and sum.totalNeurons + dataset: "aiInferenceAdaptiveGroups", window: "day", + fields: "sum { totalNeurons }", + metrics: ["ai_neurons"], + read: (rows) => ({ ai_neurons: total(rows, (r) => r.sum?.totalNeurons) }), + }, +]; + +// Not a metric: which AI Gateways send requests to Workers AI. Any that do spend neurons, which +// the rule allows only inside the free 10,000 a day. +export const GATEWAYS = { + dataset: "aiGatewayRequestsAdaptiveGroups", window: "month", + fields: "count dimensions { gateway provider }", + read: (rows) => { + const all = new Set(); + const workersAi = new Set(); + for (const r of rows) { + const g = r.dimensions?.gateway; + if (!g) continue; + all.add(g); + if (String(r.dimensions?.provider).toLowerCase() === "workers-ai") workersAi.add(g); + } + return { all: [...all].sort(), workersAi: [...workersAi].sort() }; + }, +}; + +// Rows of a source's per-script metrics: { metric: { script: amount } }. A row with no script name +// (the API's null) is kept under NO_SCRIPT so the per-script amounts still add up to the total. +export const NO_SCRIPT = "(no script name)"; +export function scriptsOf(source, rows) { + const out = {}; + for (const [metric, pick] of Object.entries(source.perScript ?? {})) { + const by = {}; + for (const r of rows) { + const name = r.dimensions?.scriptName || NO_SCRIPT; + by[name] = (by[name] ?? 0) + (Number(pick(r)) || 0); + } + out[metric] = by; + } + return out; +} + +export function buildQuery(source) { + const extra = source.filter ? `, ${source.filter}` : ""; + return `query Usage($account: string!, $start: Time!, $end: Time!) { + viewer { accounts(filter: { accountTag: $account }) { + rows: ${source.dataset}(limit: 10000, filter: { datetime_geq: $start, datetime_leq: $end${extra} }) { ${source.fields} } + } } +}`; +} + +// A failure that asking again may fix. Everything else thrown here is a real error. +export class TransientError extends Error {} + +// Waits after the 1st, 2nd and 3rd failure: up to four requests for one dataset, then it is unknown. +export const RETRY_DELAYS_MS = [2000, 5000, 10000]; +const TRY_AGAIN = /try again/i; +const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +async function queryOnce(env, source, window, fetchImpl) { + let res; + try { + res = await fetchImpl(ENDPOINT, { + method: "POST", + headers: { "content-type": "application/json", authorization: `Bearer ${env.CF_ANALYTICS_TOKEN}` }, + body: JSON.stringify({ + query: buildQuery(source), + variables: { account: env.CF_ACCOUNT_ID, start: window.start.toISOString(), end: window.end.toISOString() }, + }), + }); + } catch (e) { + throw new TransientError(`${source.dataset}: ${String(e.message).slice(0, 200)}`); + } + if (!res.ok) { + const Kind = res.status >= 500 || res.status === 429 ? TransientError : Error; + throw new Kind(`${source.dataset}: HTTP ${res.status}`); + } + const body = await res.json(); + if (body.errors?.length) { + const message = String(body.errors[0].message).slice(0, 200); + throw new (TRY_AGAIN.test(message) ? TransientError : Error)(`${source.dataset}: ${message}`); + } + const rows = body.data?.viewer?.accounts?.[0]?.rows; + if (!Array.isArray(rows)) throw new Error(`${source.dataset}: no data in the response`); + if (rows.length >= 10000) throw new Error(`${source.dataset}: 10,000 rows, the total would be cut short`); + return rows; +} + +// Rows of one dataset over one window. Throws with a message that never contains the token: a +// TransientError once the retries are used up, a plain Error at once for anything retrying can't fix. +export async function queryRows(env, source, window, fetchImpl, { delays = RETRY_DELAYS_MS, sleep = wait } = {}) { + for (let attempt = 0; ; attempt++) { + try { + return await queryOnce(env, source, window, fetchImpl); + } catch (e) { + if (!(e instanceof TransientError) || attempt >= delays.length) throw e; + await sleep(delays[attempt]); + } + } +} + +// { values: { metric: number }, errors: { metric: message }, unknown: { metric: message }, +// gateways: { all, workersAi } | null, scripts: { metric: { script: number } } }. `scripts` has +// the metrics of a source with `perScript` that was read this run. `unknown` is a dataset still failing with a "try again" +// after the retries; `errors` is one that can't be fixed by waiting. +export async function readUsage(env, now, day, fetchImpl = fetch, retry = {}) { + const values = {}; + const errors = {}; + const unknown = {}; + const scripts = {}; + // Each window up to now; the API has no data from the future. + const until = (kind) => ({ start: windowFor(kind, now, day).start, end: now }); + await Promise.all(SOURCES.map(async (s) => { + try { + const rows = await queryRows(env, s, until(s.window), fetchImpl, retry); + Object.assign(values, s.read(rows)); + Object.assign(scripts, scriptsOf(s, rows)); + } catch (e) { + for (const m of s.metrics) (e instanceof TransientError ? unknown : errors)[m] = e.message; + } + })); + let gateways; + try { + gateways = GATEWAYS.read(await queryRows(env, GATEWAYS, until(GATEWAYS.window), fetchImpl, retry)); + } catch (e) { + gateways = { error: e.message, ...(e instanceof TransientError && { transient: true }) }; + } + return { values, errors, unknown, gateways, scripts }; +} diff --git a/workers/meter/src/windows.js b/workers/meter/src/windows.js new file mode 100644 index 0000000..09a36de --- /dev/null +++ b/workers/meter/src/windows.js @@ -0,0 +1,35 @@ +// Usage windows. Monthly allowances reset on the account's billing day (BILLING_DAY, 1-28, read +// it from the dashboard's billing page); Workers AI's free neurons reset daily at 00:00 UTC. +// A "gauge" metric (storage) has no reset: it is read over the last day, and what is stored now +// is what the month will bill. + +const DAY = 86400000; + +// Required: with no [vars] in wrangler.toml it arrives through the Infisical sync, and a +// silent default would measure the wrong month. +export function billingDay(value) { + if (value == null || value === "") throw new Error("BILLING_DAY is not set (Infisical sk-ctl-workers)"); + const n = Number(value); + if (!Number.isInteger(n) || n < 1 || n > 28) throw new Error(`BILLING_DAY must be 1-28, got ${value}`); + return n; +} + +// The window containing `now`: { start, end } as Dates, end exclusive. +export function windowFor(kind, now, day = 1) { + if (kind === "gauge") return { start: new Date(now.getTime() - DAY), end: now }; + if (kind === "day") { + const start = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate())); + return { start, end: new Date(start.getTime() + DAY) }; + } + if (kind !== "month") throw new Error(`unknown window ${kind}`); + const m = now.getUTCMonth() - (now.getUTCDate() < day ? 1 : 0); + const start = new Date(Date.UTC(now.getUTCFullYear(), m, day)); + const end = new Date(Date.UTC(start.getUTCFullYear(), start.getUTCMonth() + 1, day)); + return { start, end }; +} + +// Share of the window that has passed, 0..1. +export function elapsed(window, now) { + const f = (now - window.start) / (window.end - window.start); + return Math.min(1, Math.max(0, f)); +} diff --git a/workers/meter/test/baseline.test.js b/workers/meter/test/baseline.test.js new file mode 100644 index 0000000..a790493 --- /dev/null +++ b/workers/meter/test/baseline.test.js @@ -0,0 +1,167 @@ +// The baseline and its schema check, against a fake GraphQL endpoint that also answers +// introspection with a schema built from the names src/usage.js asks for. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { SOURCES, GATEWAYS } from "../src/usage.js"; +import { selectionPaths, filterKeys, checkSchema, introspector } from "../tools/schema.mjs"; +import { baseline } from "../tools/baseline.mjs"; +import { fakeGraphQL, rowsFor, noSleep, ENV } from "./fakes.js"; + +const NOW = new Date("2026-10-15T12:00:00Z"); +const TRY_AGAIN = new Error("unable to execute query, please try again later"); +const ref = (name) => ({ kind: "OBJECT", name, ofType: null }); +const list = (name) => ({ kind: "LIST", name: null, ofType: { kind: "NON_NULL", name: null, ofType: ref(name) } }); + +// Every name usage.js uses, minus `drop` (["dataset", "path.to.field"] or ["dataset"]). +function fakeSchema(drop = []) { + const types = { Query: { fields: [{ name: "viewer", type: ref("viewer") }] }, viewer: { fields: [{ name: "accounts", type: list("account") }] }, account: { fields: [] } }; + const add = (type, name, t) => { types[type] ??= { fields: [] }; if (!types[type].fields.some((f) => f.name === name)) types[type].fields.push({ name, type: t }); }; + for (const s of [...SOURCES, GATEWAYS]) { + if (drop.some(([d, p]) => d === s.dataset && !p)) continue; + const row = `${s.dataset}Row`; + types.account.fields.push({ name: s.dataset, type: list(row), args: [{ name: "filter", type: ref(`${s.dataset}Filter`) }] }); + types[`${s.dataset}Filter`] = { inputFields: filterKeys(s).map((name) => ({ name, type: ref("Time") })) }; + types[row] ??= { fields: [] }; + for (const path of selectionPaths(s.fields)) { + const dropped = drop.some(([d, p]) => d === s.dataset && p === path.join(".")); + let type = row; + for (const [i, step] of path.entries()) { + const last = i === path.length - 1; + const next = last ? "Leaf" : `${type}_${step}`; + if (last && dropped) types[type] ??= { fields: [] }; + else add(type, step, ref(next)); + type = next; + } + } + } + return types; +} + +function withSchema(types, rows = {}) { + const graphql = fakeGraphQL(rows); + return async (url, init) => { + const { query, variables } = JSON.parse(init.body); + if (query.includes("__schema")) return Response.json({ data: { __schema: { queryType: { name: "Query" } } } }); + if (query.includes("__type")) return Response.json({ data: { __type: types[variables.name] ? { name: variables.name, ...types[variables.name] } : null } }); + return graphql(url, init); + }; +} + +test("selectionPaths: nested selections become field paths", () => { + assert.deepEqual(selectionPaths("max { payloadSize metadataSize } dimensions { bucketName }"), + [["max", "payloadSize"], ["max", "metadataSize"], ["dimensions", "bucketName"]]); + assert.deepEqual(selectionPaths("count dimensions { gateway provider }"), [["count"], ["dimensions", "gateway"], ["dimensions", "provider"]]); + assert.deepEqual(filterKeys(SOURCES.find((s) => s.filter)), ["datetime_geq", "datetime_leq", "eventType"]); +}); + +test("schema check: every name present, no problems", async () => { + assert.deepEqual(await checkSchema(introspector(ENV, withSchema(fakeSchema())), [...SOURCES, GATEWAYS]), []); +}); + +test("schema check names a missing field, a missing dataset and a missing filter key, with what is there", async () => { + const types = fakeSchema([["workersInvocationsAdaptive", "sum.cpuTimeUs"], ["aiInferenceAdaptiveGroups"]]); + types.workflowsAdaptiveGroupsFilter.inputFields = types.workflowsAdaptiveGroupsFilter.inputFields.filter((f) => f.name !== "eventType"); + const problems = await checkSchema(introspector(ENV, withSchema(types))); + assert.deepEqual(problems.map((p) => `${p.where}.${p.missing}`).sort(), [ + "aiInferenceAdaptiveGroups.aiInferenceAdaptiveGroups", + "workersInvocationsAdaptive.sum.cpuTimeUs", + "workflowsAdaptiveGroups(filter).eventType", + ]); + assert.deepEqual(problems.find((p) => p.missing === "cpuTimeUs").has, ["requests"]); +}); + +test("baseline: exit 0 and every metric listed when everything reads", async () => { + const out = []; + const code = await baseline({ ...ENV, BILLING_DAY: "13" }, { now: NOW, fetchImpl: withSchema(fakeSchema(), rowsFor("kv_reads", 42)), log: (l) => out.push(l) }); + assert.equal(code, 0); + const text = out.join("\n"); + assert.match(text, /billing day 13/); + assert.match(text, /^kv_reads \| 42 \| 10000000 \|/m); + assert.doesNotMatch(text, /Can't read/); +}); + +test("baseline: CPU-ms and DO requests by script, the top five unless --scripts asks for all", async () => { + const rows = { + workersInvocationsAdaptive: Array.from({ length: 7 }, (_, i) => ({ sum: { requests: 1, cpuTimeUs: (i + 1) * 1000 }, dimensions: { scriptName: `w${i + 1}` } })), + durableObjectsInvocationsAdaptiveGroups: [{ sum: { requests: 40 }, dimensions: { scriptName: "setkernel-ops" } }, { sum: { requests: 2 }, dimensions: { scriptName: "sk-status" } }], + }; + const run = async (opts) => { + const out = []; + const code = await baseline({ ...ENV, CF_ANALYTICS_TOKEN: "s3cret-token-value" }, { now: NOW, fetchImpl: withSchema(fakeSchema(), rows), log: (l) => out.push(l), ...opts }); + return { code, text: out.join("\n") }; + }; + const top = await run(); + assert.equal(top.code, 0); + assert.match(top.text, /^workers_cpu_ms by script \(5 of 7\) \| CPU ms\nw7 \| 7\nw6 \| 6\nw5 \| 5\nw4 \| 4\nw3 \| 3$/m); + assert.doesNotMatch(top.text, /^w2 \|/m); + assert.match(top.text, /^do_requests by script \(2 of 2\) \| requests\nsetkernel-ops \| 40\nsk-status \| 2$/m); + assert.doesNotMatch(top.text, /s3cret/); + const all = await run({ allScripts: true }); + assert.match(all.text, /^workers_cpu_ms by script \(7 of 7\)/m); + assert.match(all.text, /^w1 \| 1$/m); +}); + +test("baseline: any Can't read line fails it, and the token is never printed", async () => { + const out = []; + const env = { ...ENV, CF_ANALYTICS_TOKEN: "s3cret-token-value" }; + const types = fakeSchema([["durableObjectsPeriodicGroups", "sum.activeTime"]]); + const code = await baseline(env, { now: NOW, fetchImpl: withSchema(types, { durableObjectsPeriodicGroups: new Error("unknown field activeTime") }), log: (l) => out.push(l) }); + assert.equal(code, 1); + const text = out.join("\n"); + assert.match(text, /^Can't read durableObjectsPeriodicGroups\.sum\.activeTime: not in the schema/m); + assert.match(text, /^Can't read do_duration_gbs: durableObjectsPeriodicGroups: unknown field/m); + assert.doesNotMatch(text, /s3cret/); +}); + +test("baseline: an introspection failure is a Can't read line, not a pass", async () => { + const out = []; + const refused = async () => new Response("no", { status: 403 }); + assert.equal(await baseline(ENV, { now: NOW, fetchImpl: refused, log: (l) => out.push(l) }), 1); + assert.match(out.join("\n"), /^Can't read the schema: introspection: HTTP 403/m); +}); + +test("baseline: no token or account, exit 2 before any request", async () => { + let called = false; + const code = await baseline({ BILLING_DAY: "13" }, { fetchImpl: async () => { called = true; }, log: () => {} }); + assert.equal(code, 2); + assert.equal(called, false); +}); + +test("baseline: one transient dataset exits 0, the metric is '?', and the line says so", async () => { + const out = []; + const { retry } = noSleep(); + const code = await baseline(ENV, { + now: NOW, retry, log: (l) => out.push(l), + fetchImpl: withSchema(fakeSchema(), ({ ...rowsFor("kv_reads", 42), workflowsAdaptiveGroups: TRY_AGAIN })), + }); + assert.equal(code, 0); + const text = out.join("\n"); + assert.match(text, /^Unknown workflows_steps: .*try again later/m); + assert.match(text, /^workflows_steps \| \? \|/m); + assert.match(text, /^kv_reads \| 42 \|/m); + assert.doesNotMatch(text, /Can't read/); +}); + +test("baseline: a schema error alongside a transient one still exits 1", async () => { + const out = []; + const code = await baseline(ENV, { + now: NOW, retry: noSleep().retry, log: (l) => out.push(l), + fetchImpl: withSchema(fakeSchema(), ({ workflowsAdaptiveGroups: TRY_AGAIN, d1AnalyticsAdaptiveGroups: new Error("unknown field rowsRead") })), + }); + assert.equal(code, 1); + const text = out.join("\n"); + assert.match(text, /^Can't read d1_rows_read: /m); + assert.match(text, /^Unknown workflows_steps: /m); +}); + +test("baseline: a transient AI Gateway read exits 0 and is not called a 'Can't read'", async () => { + const out = []; + const code = await baseline(ENV, { + now: NOW, retry: noSleep().retry, log: (l) => out.push(l), + fetchImpl: withSchema(fakeSchema(), ({ aiGatewayRequestsAdaptiveGroups: new Error("unable to execute query, please try again later") })), + }); + assert.equal(code, 0); + assert.doesNotMatch(out.join("\n"), /Can't read/); + assert.match(out.join("\n"), /AI Gateway traffic is unknown this run/); +}); diff --git a/workers/meter/test/fakes.js b/workers/meter/test/fakes.js new file mode 100644 index 0000000..6f7b0d6 --- /dev/null +++ b/workers/meter/test/fakes.js @@ -0,0 +1,76 @@ +// Test doubles: a KV namespace in memory that counts writes, and a GraphQL endpoint that answers +// each dataset with the rows a test sets. + +import { ENDPOINT } from "../src/usage.js"; + +export function memoryKV(initial = {}) { + const data = new Map(Object.entries(initial)); + const kv = { + data, + puts: 0, + deletes: 0, + async get(key, opts) { + const v = data.get(key); + if (v == null) return null; + const type = typeof opts === "string" ? opts : opts?.type; + return type === "json" ? JSON.parse(v) : v; + }, + async put(key, value) { kv.puts++; data.set(key, value); }, + async delete(key) { kv.deletes++; data.delete(key); }, + }; + return kv; +} + +// A dataset that fails its first `times` requests with `failure` (an Error, answered as a GraphQL +// error, or an HTTP status number) and then answers with `rows`. Pass it as a fakeGraphQL value. +export const flaky = (times, failure, rows = []) => (attempt) => (attempt <= times ? failure : rows); + +// No waiting between retries in tests; records what it was asked to wait. +export const noSleep = () => { const waits = []; const sleep = async (ms) => { waits.push(ms); }; return { waits, retry: { sleep } }; }; + +// rows: { datasetName: [row, ...] | Error | (attempt) => ... }. Unlisted datasets answer with no rows. +export function fakeGraphQL(rows = {}, { webhook = [] } = {}) { + const calls = []; + const impl = async (url, init) => { + if (url !== ENDPOINT) { + webhook.push({ url, init, body: JSON.parse(init.body) }); + return new Response("ok", { status: webhook.status ?? 200 }); + } + const body = JSON.parse(init.body); + const dataset = body.query.match(/rows: (\w+)\(/)[1]; + calls.push({ dataset, body, auth: init.headers.authorization }); + let r = rows[dataset] ?? []; + if (typeof r === "function") r = r(calls.filter((c) => c.dataset === dataset).length); + if (typeof r === "number") return new Response("down", { status: r }); + if (r instanceof Error) return Response.json({ data: null, errors: [{ message: r.message }] }); + return Response.json({ data: { viewer: { accounts: [{ rows: r }] } }, errors: null }); + }; + impl.calls = calls; + return impl; +} + +// Rows that put one metric at `used`, in the shape its dataset returns. +export function rowsFor(metric, used) { + const table = { + workers_requests: ["workersInvocationsAdaptive", { sum: { requests: used, cpuTimeUs: 0 }, dimensions: { scriptName: "w" } }], + workers_cpu_ms: ["workersInvocationsAdaptive", { sum: { requests: 0, cpuTimeUs: used * 1000 }, dimensions: { scriptName: "w" } }], + do_requests: ["durableObjectsInvocationsAdaptiveGroups", { sum: { requests: used }, dimensions: { scriptName: "w" } }], + do_duration_gbs: ["durableObjectsPeriodicGroups", { sum: { activeTime: (used / 0.125) * 1e6 } }], + workflows_steps: ["workflowsAdaptiveGroups", { count: used }], + queues_ops: ["queueMessageOperationsAdaptiveGroups", { sum: { billableOperations: used } }], + d1_rows_read: ["d1AnalyticsAdaptiveGroups", { sum: { rowsRead: used, rowsWritten: 0 } }], + d1_rows_written: ["d1AnalyticsAdaptiveGroups", { sum: { rowsRead: 0, rowsWritten: used } }], + r2_storage_bytes: ["r2StorageAdaptiveGroups", { max: { payloadSize: used, metadataSize: 0 }, dimensions: { bucketName: "b" } }], + r2_class_a: ["r2OperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "PutObject" } }], + r2_class_b: ["r2OperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "GetObject" } }], + kv_reads: ["kvOperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "read" } }], + kv_writes: ["kvOperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "write" } }], + kv_deletes: ["kvOperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "delete" } }], + kv_lists: ["kvOperationsAdaptiveGroups", { sum: { requests: used }, dimensions: { actionType: "list" } }], + ai_neurons: ["aiInferenceAdaptiveGroups", { sum: { totalNeurons: used } }], + }; + const [dataset, row] = table[metric]; + return { [dataset]: [row] }; +} + +export const ENV = { CF_ANALYTICS_TOKEN: "test-token", CF_ACCOUNT_ID: "acct", BILLING_DAY: "1" }; diff --git a/workers/meter/test/flags.test.js b/workers/meter/test/flags.test.js new file mode 100644 index 0000000..6aa2147 --- /dev/null +++ b/workers/meter/test/flags.test.js @@ -0,0 +1,54 @@ +// What the Workers reading the guard see: readGuard and refusal. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { readGuard, refusal, KEYS, DEGRADED } from "../src/flags.js"; +import { memoryKV } from "./fakes.js"; + +const flag = (until = null) => JSON.stringify({ since: "2026-10-15T10:07:00.000Z", metrics: ["ai_neurons"], until }); + +test("no flags: carry on", async () => { + const g = await readGuard(memoryKV()); + assert.deepEqual([g.alert, g.degrade, g.refuse], [false, false, false]); + assert.equal(refusal(g), null); +}); + +test("refuse implies degrade and alert, even if only the refuse key is there", async () => { + const g = await readGuard(memoryKV({ [KEYS.refuse]: flag() })); + assert.deepEqual([g.alert, g.degrade, g.refuse], [true, true, true]); +}); + +test("refuse: non-essential requests get a 503 with Retry-After; essential ones carry on", async () => { + const until = new Date(Date.now() + 3600e3).toISOString(); + const g = await readGuard(memoryKV({ [KEYS.alert]: flag(until), [KEYS.degrade]: flag(until), [KEYS.refuse]: flag(until) })); + const res = refusal(g); + assert.equal(res.status, 503); + const retry = Number(res.headers.get("retry-after")); + assert.ok(retry > 3500 && retry <= 3600, String(retry)); + assert.equal(refusal(g, { essential: true }), null); +}); + +test("degrade alone refuses nothing", async () => { + const g = await readGuard(memoryKV({ [KEYS.alert]: flag(), [KEYS.degrade]: flag() })); + assert.equal(g.degrade, true); + assert.equal(refusal(g), null); +}); + +test("KV unreadable: degrade (start nothing new) but keep serving", async () => { + const broken = { get: async () => { throw new Error("KV down"); } }; + const g = await readGuard(broken); + assert.deepEqual([g.alert, g.degrade, g.refuse], [true, true, false]); + assert.match(g.error, /KV down/); +}); + +test("readGuard asks KV to cache each flag at the edge", async () => { + const seen = []; + await readGuard({ get: async (k, o) => { seen.push([k, o]); return null; } }); + assert.deepEqual(seen.map(([k]) => k), [KEYS.alert, KEYS.degrade, KEYS.refuse]); + for (const [, o] of seen) assert.deepEqual(o, { type: "json", cacheTtl: 60 }); +}); + +test("the degraded settings are the spec's", () => { + assert.deepEqual(DEGRADED, { heartbeatSeconds: 60, retentionDays: 7, bundlesPaused: true, newLeases: false, workersAi: false }); + assert.ok(Object.isFrozen(DEGRADED)); +}); diff --git a/workers/meter/test/meter.test.js b/workers/meter/test/meter.test.js new file mode 100644 index 0000000..ce70d14 --- /dev/null +++ b/workers/meter/test/meter.test.js @@ -0,0 +1,202 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import allowances from "../allowances.json" with { type: "json" }; +import { evaluate, levelOf, activeDrill, message, signature } from "../src/meter.js"; +import { windowFor, elapsed, billingDay } from "../src/windows.js"; +import { SOURCES } from "../src/usage.js"; + +const NOW = new Date("2026-10-15T12:00:00Z"); +const usage = (values, errors = {}) => ({ values, errors, gateways: null }); +const flagsOn = (r) => ["alert", "degrade", "refuse"].filter((f) => r.flags[f]); + +test("allowances.json pins every metric with a positive allowance, a window and a pricing URL", () => { + for (const [id, m] of Object.entries(allowances.metrics)) { + assert.ok(m.allowance > 0, id); + assert.ok(["month", "day", "gauge"].includes(m.window), id); + assert.match(m.source, /^https:\/\/developers\.cloudflare\.com\/.+pricing/, id); + } + assert.deepEqual(allowances.thresholds, { alert: 0.5, degrade: 0.8, refuse: 0.95 }); + assert.equal(allowances.metrics.ai_neurons.allowance, 10000); + assert.equal(allowances.metrics.ai_neurons.window, "day"); +}); + +test("every metric is read by exactly one query, over the window allowances.json gives it", () => { + const seen = {}; + for (const s of SOURCES) for (const m of s.metrics) { + assert.equal(seen[m], undefined, `${m} read twice`); + seen[m] = s.window; + } + for (const [id, m] of Object.entries(allowances.metrics)) assert.equal(seen[id], m.window, id); + assert.deepEqual(Object.keys(seen).sort(), Object.keys(allowances.metrics).sort()); +}); + +test("levelOf: the boundaries are inclusive", () => { + const th = allowances.thresholds; + assert.equal(levelOf(0.4999, th), "ok"); + assert.equal(levelOf(0.5, th), "alert"); + assert.equal(levelOf(0.7999, th), "alert"); + assert.equal(levelOf(0.8, th), "degrade"); + assert.equal(levelOf(0.9499, th), "degrade"); + assert.equal(levelOf(0.95, th), "refuse"); + assert.equal(levelOf(3, th), "refuse"); + assert.equal(levelOf(null, th), "unknown"); +}); + +// The acceptance test: synthetic usage past each threshold, on every metric, flips the right flags. +for (const [id, m] of Object.entries(allowances.metrics)) { + test(`${id}: synthetic usage past 50/80/95% flips alert, degrade, refuse`, () => { + const cases = [[0.49, []], [0.51, ["alert"]], [0.81, ["alert", "degrade"]], [0.96, ["alert", "degrade", "refuse"]]]; + for (const [share, want] of cases) { + const r = evaluate({ allowances, usage: usage({ [id]: m.allowance * share }), now: NOW }); + assert.deepEqual(flagsOn(r), want, `${id} at ${share}`); + for (const f of want) assert.deepEqual(r.flags[f].metrics, [id]); + } + }); +} + +test("the worst metric decides; each flag lists the metrics holding it", () => { + const r = evaluate({ + allowances, + usage: usage({ queues_ops: 600000, ai_neurons: 9600, kv_writes: 850000 }), + now: NOW, + }); + assert.deepEqual(r.flags.alert.metrics, ["queues_ops", "kv_writes", "ai_neurons"]); + assert.deepEqual(r.flags.degrade.metrics, ["kv_writes", "ai_neurons"]); + assert.deepEqual(r.flags.refuse.metrics, ["ai_neurons"]); + // refuse lasts until the neurons reset at 00:00 UTC; degrade until the month resets too + assert.equal(r.flags.refuse.until, "2026-10-16T00:00:00.000Z"); + assert.equal(r.flags.degrade.until, "2026-11-01T00:00:00.000Z"); +}); + +test("a flag held by storage has no reset time", () => { + const r = evaluate({ allowances, usage: usage({ r2_storage_bytes: 9e9 }), now: NOW }); + assert.equal(r.flags.degrade.until, null); +}); + +test("a flag keeps its since time while it stays on", () => { + const first = evaluate({ allowances, usage: usage({ ai_neurons: 8500 }), now: NOW }); + const later = new Date("2026-10-15T13:00:00Z"); + const second = evaluate({ allowances, usage: usage({ ai_neurons: 8700 }), now: later, prev: first }); + assert.equal(second.flags.degrade.since, NOW.toISOString()); +}); + +test("the window reset clears the flags", () => { + const before = evaluate({ allowances, usage: usage({ ai_neurons: 9900 }), now: NOW }); + const after = evaluate({ allowances, usage: usage({ ai_neurons: 10 }), now: new Date("2026-10-16T00:07:00Z"), prev: before }); + assert.deepEqual(flagsOn(after), []); +}); + +test("an unreadable metric keeps its last value in the same window, so a flag can't drop", () => { + const first = evaluate({ allowances, usage: usage({ queues_ops: 900000 }), now: NOW }); + const failed = evaluate({ + allowances, + usage: usage({}, { queues_ops: "queueMessageOperationsAdaptiveGroups: HTTP 500" }), + now: new Date("2026-10-15T13:00:00Z"), + prev: first, + }); + assert.deepEqual(flagsOn(failed), ["alert", "degrade"]); + assert.equal(failed.metrics.queues_ops.stale, true); + assert.deepEqual(failed.errors, ["queues_ops"]); +}); + +test("an unreadable metric from a previous window is unknown, not zero", () => { + const first = evaluate({ allowances, usage: usage({ queues_ops: 900000 }), now: NOW }); + const failed = evaluate({ + allowances, + usage: usage({}, { queues_ops: "boom" }), + now: new Date("2026-11-01T01:00:00Z"), + prev: first, + }); + assert.equal(failed.metrics.queues_ops.used, null); + assert.equal(failed.metrics.queues_ops.level, "unknown"); + assert.deepEqual(failed.errors, ["queues_ops"]); +}); + +test("pro-rating: projected use at the end of the window, once 10% of it has passed", () => { + // 15 Oct 12:00 is 14.5 days into a 31-day month + const r = evaluate({ allowances, usage: usage({ queues_ops: 480000 }), now: NOW }); + assert.equal(r.metrics.queues_ops.level, "ok"); + assert.ok(Math.abs(r.metrics.queues_ops.projected - 480000 / (14.5 / 31)) < 1); + assert.deepEqual(r.overPace, ["queues_ops"]); + // two hours into the day, Workers AI's window is under 10% gone: no projection yet + const early = evaluate({ allowances, usage: usage({ ai_neurons: 2000 }), now: new Date("2026-10-15T02:00:00Z") }); + assert.equal(early.metrics.ai_neurons.projected, null); + assert.deepEqual(early.overPace, []); +}); + +test("a drill lowers one allowance until it expires and can never raise one", () => { + const drill = { metric: "ai_neurons", allowance: 100, expiresAt: "2026-10-15T14:00:00Z" }; + const r = evaluate({ allowances, usage: usage({ ai_neurons: 85 }), now: NOW, drill }); + assert.deepEqual(flagsOn(r), ["alert", "degrade"]); + assert.equal(r.metrics.ai_neurons.allowance, 100); + assert.equal(r.drill.metric, "ai_neurons"); + assert.match(message(r).text, /^\[DRILL\] /); + + assert.equal(activeDrill({ ...drill, expiresAt: "2026-10-15T11:59:59Z" }, allowances, NOW), null); + assert.equal(activeDrill({ ...drill, allowance: 1e9 }, allowances, NOW), null); + assert.equal(activeDrill({ ...drill, allowance: 10000 }, allowances, NOW), null); + assert.equal(activeDrill({ ...drill, allowance: -1 }, allowances, NOW), null); + assert.equal(activeDrill({ ...drill, metric: "nope" }, allowances, NOW), null); + assert.equal(activeDrill("garbage", allowances, NOW), null); +}); + +test("message: the highest flag leads, degrade and refuse mention Raghav, the baseline lists everything", () => { + const values = Object.fromEntries(Object.keys(allowances.metrics).map((id) => [id, 0])); + const quiet = evaluate({ allowances, usage: { ...usage({ ...values, queues_ops: 12000 }), gateways: { all: ["a", "b"], workersAi: [] } }, now: NOW }); + const base = message(quiet, { full: true }); + assert.match(base.text, /every metric is under 50%/); + assert.match(base.text, /Queues operations: 12,000 of 1,000,000 operations this month \(1%\)/); + assert.match(base.text, /none send requests to Workers AI/); + assert.equal(base.mention, false); + assert.equal(message(quiet).text.split("\n").length, 1); + + const hot = evaluate({ allowances, usage: { ...usage({ ai_neurons: 9700 }), gateways: { all: ["g"], workersAi: ["g"] } }, now: NOW }); + const m = message(hot); + assert.match(m.text, /^๐Ÿ”ด \*\*sk-meter: REFUSE\*\*/); + assert.match(m.text, /Workers AI neurons: 9,700 of 10,000 neurons a day \(97%\), resets 2026-10-16 00:00 UTC/); + assert.match(m.text, /AI Gateways sending requests to Workers AI: g/); + assert.equal(m.mention, true); +}); + +test("signature changes with the flags, errors, pace and Workers AI gateways, not with usage inside a level", () => { + const a = evaluate({ allowances, usage: usage({ kv_writes: 510000 }), now: NOW }); + const b = evaluate({ allowances, usage: usage({ kv_writes: 520000 }), now: NOW, prev: a }); + const c = evaluate({ allowances, usage: usage({ kv_writes: 810000 }), now: NOW, prev: b }); + assert.equal(signature(a), signature(b)); + assert.notEqual(signature(b), signature(c)); + const g = evaluate({ allowances, usage: { ...usage({ kv_writes: 510000 }), gateways: { workersAi: ["x"] } }, now: NOW }); + assert.notEqual(signature(a), signature(g)); +}); + +test("the top script is named for a metric projected at 50%, and stays out of the signature", () => { + // NOW is 14.5 days into a 31-day month, so 10M CPU ms used projects to ~21M of 30M: 71%. + const scripts = (top) => ({ workers_cpu_ms: { [top]: 6e6, other: 4e6 }, do_requests: { a: 1 } }); + const hot = (top) => evaluate({ allowances, usage: { ...usage({ workers_cpu_ms: 1e7, do_requests: 1 }), scripts: scripts(top) }, now: NOW }); + const a = hot("alpha"); + assert.deepEqual(a.topScripts, { workers_cpu_ms: { script: "alpha", used: 6e6, share: 0.6 } }); + assert.match(message(a).text, /^Top script for Workers CPU time: `alpha`, 6,000,000 CPU ms \(60% of the account's\)\.$/m); + assert.equal(signature(a), signature(hot("zeta"))); + assert.equal(hot("zeta").topScripts.workers_cpu_ms.script, "zeta"); + // below 50% projected, or no per-script rows this run: nothing named + const low = evaluate({ allowances, usage: { ...usage({ workers_cpu_ms: 1e6 }), scripts: scripts("alpha") }, now: NOW }); + assert.deepEqual(low.topScripts, {}); + assert.doesNotMatch(message(low).text, /Top script/); + assert.deepEqual(evaluate({ allowances, usage: usage({ workers_cpu_ms: 1e7 }), now: NOW }).topScripts, {}); + // a script name can't carry markdown or a mention into the chat message + const odd = evaluate({ allowances, usage: { ...usage({ workers_cpu_ms: 1e7 }), scripts: { workers_cpu_ms: { "`@here`**x": 5 } } }, now: NOW }); + assert.match(message(odd).text, /Top script for Workers CPU time: `\?\?here\?\?\?x`/); +}); + +test("windows: billing day, the daily UTC reset, elapsed share", () => { + assert.deepEqual(windowFor("month", NOW, 1), { start: new Date("2026-10-01T00:00:00Z"), end: new Date("2026-11-01T00:00:00Z") }); + assert.deepEqual(windowFor("month", NOW, 20), { start: new Date("2026-09-20T00:00:00Z"), end: new Date("2026-10-20T00:00:00Z") }); + assert.deepEqual(windowFor("month", new Date("2026-01-05T00:00:00Z"), 10), { start: new Date("2025-12-10T00:00:00Z"), end: new Date("2026-01-10T00:00:00Z") }); + assert.deepEqual(windowFor("day", NOW), { start: new Date("2026-10-15T00:00:00Z"), end: new Date("2026-10-16T00:00:00Z") }); + assert.equal(elapsed(windowFor("day", NOW), NOW), 0.5); + assert.throws(() => billingDay(undefined), /not set/); + assert.throws(() => billingDay(""), /not set/); + assert.equal(billingDay("13"), 13); + assert.equal(billingDay("28"), 28); + assert.throws(() => billingDay("29")); + assert.throws(() => billingDay("x")); +}); diff --git a/workers/meter/test/notify.test.js b/workers/meter/test/notify.test.js new file mode 100644 index 0000000..a33dd24 --- /dev/null +++ b/workers/meter/test/notify.test.js @@ -0,0 +1,69 @@ +// The alert webhook: the headers the /meter receiver behind Cloudflare Access needs. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { notify } from "../src/notify.js"; + +const URL = "https://alerts.example.test/meter"; +const ALERT = { text: "x", mention: false }; + +function capture(status = 200) { + const seen = []; + const impl = async (url, init) => { + seen.push({ url, init }); + return new Response(null, { status, headers: status >= 300 && status < 400 ? { location: "https://login.example.test/" } : {} }); + }; + impl.seen = seen; + return impl; +} + +test("sends the Access service token and the bearer, and doesn't follow redirects", async () => { + const f = capture(); + const env = { ALERT_WEBHOOK_URL: URL, ALERT_WEBHOOK_TOKEN: "t", ALERT_ACCESS_CLIENT_ID: "id", ALERT_ACCESS_CLIENT_SECRET: "s" }; + assert.equal(await notify(env, ALERT, f), true); + const { headers, redirect } = f.seen[0].init; + assert.equal(headers["cf-access-client-id"], "id"); + assert.equal(headers["cf-access-client-secret"], "s"); + assert.equal(headers.authorization, "Bearer t"); + assert.equal(redirect, "manual"); +}); + +test("no Access headers unless both halves of the service token are set", async () => { + const f = capture(); + await notify({ ALERT_WEBHOOK_URL: URL, ALERT_ACCESS_CLIENT_ID: "id" }, ALERT, f); + assert.equal(f.seen[0].init.headers["cf-access-client-id"], undefined); + assert.equal(f.seen[0].init.headers["cf-access-client-secret"], undefined); +}); + +test("an Access redirect to its login page is a failure, so the alert is retried", async () => { + await assert.rejects(notify({ ALERT_WEBHOOK_URL: URL }, ALERT, capture(302)), /HTTP 302/); +}); + +test("the receiver's contract: POST { source, text, mention } as JSON, and its 204 counts as delivered", async () => { + const f = capture(204); + const env = { ALERT_WEBHOOK_URL: "https://alerts.setkernel.net/meter", ALERT_WEBHOOK_TOKEN: "t" }; + assert.equal(await notify(env, { text: "line 1\nline 2", mention: true }, f), true); + const { url, init } = f.seen[0]; + assert.equal(url, "https://alerts.setkernel.net/meter"); + assert.equal(init.method, "POST"); + assert.equal(init.headers["content-type"], "application/json"); + assert.deepEqual(JSON.parse(init.body), { source: "sk-meter", text: "line 1\nline 2", mention: true }); +}); + +test("an http:// URL is refused before anything is sent, so no secret leaves in the clear", async () => { + const f = capture(); + const env = { ALERT_WEBHOOK_URL: "http://alerts.setkernel.net/meter", ALERT_WEBHOOK_TOKEN: "t", ALERT_ACCESS_CLIENT_ID: "id", ALERT_ACCESS_CLIENT_SECRET: "s" }; + await assert.rejects(notify(env, ALERT, f), /must be https/); + assert.equal(f.seen.length, 0); +}); + +test("a receiver error is a failure, so the alert is retried", async () => { + await assert.rejects(notify({ ALERT_WEBHOOK_URL: URL }, ALERT, capture(401)), /HTTP 401/); +}); + +test("the source is sk-meter unless the alert names its own (another sender)", async () => { + const f = capture(); + await notify({ ALERT_WEBHOOK_URL: URL }, ALERT, f); + await notify({ ALERT_WEBHOOK_URL: URL }, { ...ALERT, source: "other-sender" }, f); + assert.deepEqual(f.seen.map((s) => JSON.parse(s.init.body).source), ["sk-meter", "other-sender"]); +}); diff --git a/workers/meter/test/retry.test.js b/workers/meter/test/retry.test.js new file mode 100644 index 0000000..cd55374 --- /dev/null +++ b/workers/meter/test/retry.test.js @@ -0,0 +1,108 @@ +// A transient GraphQL failure on one dataset: retried, then unknown for this run (flags kept, run +// not failed); a schema error is still an error. Retry waits are recorded, never slept. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { run } from "../src/index.js"; +import { evaluate, message, signature } from "../src/meter.js"; +import allowances from "../allowances.json" with { type: "json" }; +import { readUsage, queryRows, RETRY_DELAYS_MS, SOURCES } from "../src/usage.js"; +import { memoryKV, fakeGraphQL, flaky, noSleep, rowsFor, ENV } from "./fakes.js"; +import { KEYS } from "../src/flags.js"; + +const NOW = new Date("2026-10-15T12:00:00Z"); +const TRY_AGAIN = new Error("unable to execute query, please try again later"); +const WORKFLOWS = SOURCES.find((s) => s.dataset === "workflowsAdaptiveGroups"); +const WINDOW = { start: new Date("2026-10-01T00:00:00Z"), end: NOW }; + +test("the retry waits are 2 s, 5 s and 10 s", () => { + assert.deepEqual(RETRY_DELAYS_MS, [2000, 5000, 10000]); +}); + +test("a dataset that fails twice with 'try again' and then answers is read, after two waits", async () => { + const { waits, retry } = noSleep(); + const gql = fakeGraphQL({ workflowsAdaptiveGroups: flaky(2, TRY_AGAIN, [{ count: 7 }]) }); + const usage = await readUsage(ENV, NOW, 1, gql, retry); + assert.equal(usage.values.workflows_steps, 7); + assert.deepEqual(usage.unknown, {}); + assert.deepEqual(usage.errors, {}); + assert.deepEqual(waits, [2000, 5000]); + assert.equal(gql.calls.filter((c) => c.dataset === "workflowsAdaptiveGroups").length, 3); +}); + +test("HTTP 5xx, 429 and a network error are transient too", async () => { + for (const failure of [502, 429]) { + const { retry } = noSleep(); + const usage = await readUsage(ENV, NOW, 1, fakeGraphQL({ workflowsAdaptiveGroups: flaky(1, failure, [{ count: 3 }]) }), retry); + assert.equal(usage.values.workflows_steps, 3, `HTTP ${failure}`); + } + const { retry } = noSleep(); + let n = 0; + const gql = fakeGraphQL({ workflowsAdaptiveGroups: [{ count: 4 }] }); + const dropped = async (url, init) => { + if (JSON.parse(init.body).query.includes("workflowsAdaptiveGroups") && n++ === 0) throw new TypeError("fetch failed"); + return gql(url, init); + }; + assert.equal((await readUsage(ENV, NOW, 1, dropped, retry)).values.workflows_steps, 4); +}); + +test("a dataset that always says 'try again' is unknown after four requests, and nothing else is affected", async () => { + const { waits, retry } = noSleep(); + const gql = fakeGraphQL({ ...rowsFor("kv_reads", 5), workflowsAdaptiveGroups: TRY_AGAIN }); + const usage = await readUsage(ENV, NOW, 1, gql, retry); + assert.deepEqual(Object.keys(usage.unknown), ["workflows_steps"]); + assert.match(usage.unknown.workflows_steps, /try again later/); + assert.deepEqual(usage.errors, {}); + assert.equal(usage.values.kv_reads, 5); + assert.deepEqual(waits, [2000, 5000, 10000]); + assert.equal(gql.calls.filter((c) => c.dataset === "workflowsAdaptiveGroups").length, 4); +}); + +test("a schema error is not retried and stays an error", async () => { + const { waits, retry } = noSleep(); + const gql = fakeGraphQL({ workflowsAdaptiveGroups: new Error('unknown field "count"') }); + const usage = await readUsage(ENV, NOW, 1, gql, retry); + assert.deepEqual(Object.keys(usage.errors), ["workflows_steps"]); + assert.deepEqual(usage.unknown, {}); + assert.deepEqual(waits, []); + assert.equal(gql.calls.filter((c) => c.dataset === "workflowsAdaptiveGroups").length, 1); +}); + +test("a 403 is not retried", async () => { + const { waits, retry } = noSleep(); + await assert.rejects(queryRows(ENV, WORKFLOWS, WINDOW, async () => new Response("no", { status: 403 }), retry), /HTTP 403/); + assert.deepEqual(waits, []); +}); + +test("an empty workflows dataset reads as 0", async () => { + const usage = await readUsage(ENV, NOW, 1, fakeGraphQL({ workflowsAdaptiveGroups: [] }), noSleep().retry); + assert.equal(usage.values.workflows_steps, 0); + assert.deepEqual(usage.errors, {}); + assert.deepEqual(usage.unknown, {}); +}); + +test("cron: an unknown metric keeps its flag and last value, is listed in the report, and a blip sends no new alert", async () => { + const kv = memoryKV(); + const env = { ...ENV, GUARD: kv }; + const { retry } = noSleep(); + const at = (h) => new Date(Date.UTC(2026, 9, 15, h, 7)); + await run(env, at(1), { fetchImpl: fakeGraphQL({ ...rowsFor("workflows_steps", 300000) }), retry }); + assert.ok(kv.data.has(KEYS.alert)); + const r = await run(env, at(2), { fetchImpl: fakeGraphQL({ workflowsAdaptiveGroups: TRY_AGAIN }), retry }); + assert.ok(kv.data.has(KEYS.alert)); + assert.deepEqual(r.unknown, ["workflows_steps"]); + assert.deepEqual(r.errors, []); + assert.equal(r.metrics.workflows_steps.used, 300000); + assert.equal(r.metrics.workflows_steps.stale, true); + assert.match(message(r).text, /Unknown this run .*Workflows steps.*flags keep their last known values/s); + assert.doesNotMatch(message(r).text, /Can't read/); +}); + +test("signature: an unknown metric with a last known value is quiet, one never read is not", () => { + const usage = { values: {}, errors: {}, unknown: { workflows_steps: "try again" }, gateways: null }; + const fresh = evaluate({ allowances, usage, now: NOW }); + assert.deepEqual(JSON.parse(signature(fresh)).unknown, ["workflows_steps"]); + const prev = evaluate({ allowances, usage: { values: { workflows_steps: 10 }, errors: {}, gateways: null }, now: NOW }); + const known = evaluate({ allowances, usage, now: NOW, prev }); + assert.deepEqual(JSON.parse(signature(known)).unknown, []); +}); diff --git a/workers/meter/test/run.test.js b/workers/meter/test/run.test.js new file mode 100644 index 0000000..e08d184 --- /dev/null +++ b/workers/meter/test/run.test.js @@ -0,0 +1,266 @@ +// The hourly run end to end: fake GraphQL in, KV flags and the Buzz webhook out. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { run } from "../src/index.js"; +import worker from "../src/index.js"; +import { KEYS, REPORT_KEY, DRILL_KEY, readGuard } from "../src/flags.js"; +import { SOURCES, GATEWAYS, buildQuery, readUsage, NO_SCRIPT, R2_CLASS_B } from "../src/usage.js"; +import { memoryKV, fakeGraphQL, rowsFor, ENV } from "./fakes.js"; + +const HOUR = (h) => new Date(Date.UTC(2026, 9, 15, h, 7)); +const WEBHOOK = { ALERT_WEBHOOK_URL: "https://alerts.example.test/meter", ALERT_WEBHOOK_TOKEN: "hook-token" }; + +test("one query per dataset, the account and token on each, nothing else sent", async () => { + const kv = memoryKV(); + const gql = fakeGraphQL(); + await run({ ...ENV, GUARD: kv }, HOUR(12), { fetchImpl: gql }); + assert.equal(gql.calls.length, SOURCES.length + 1); + for (const c of gql.calls) { + assert.equal(c.auth, "Bearer test-token"); + assert.equal(c.body.variables.account, "acct"); + assert.equal(c.body.variables.end, HOUR(12).toISOString()); + } + const by = Object.fromEntries(gql.calls.map((c) => [c.dataset, c.body.variables.start])); + assert.equal(by.queueMessageOperationsAdaptiveGroups, "2026-10-01T00:00:00.000Z"); + assert.equal(by.aiInferenceAdaptiveGroups, "2026-10-15T00:00:00.000Z"); + assert.equal(by.r2StorageAdaptiveGroups, "2026-10-14T12:07:00.000Z"); + assert.match(buildQuery(SOURCES.find((s) => s.metrics.includes("workflows_steps"))), /eventType: "STEP_START"/); + assert.match(buildQuery(GATEWAYS), /aiGatewayRequestsAdaptiveGroups/); +}); + +test("rows are summed and classified: R2 Class A/B, KV by action, DO duration in GB-s", async () => { + const kv = memoryKV(); + const gql = fakeGraphQL({ + r2OperationsAdaptiveGroups: [ + { sum: { requests: 10 }, dimensions: { actionType: "PutObject" } }, + { sum: { requests: 5 }, dimensions: { actionType: "SomethingNew" } }, + { sum: { requests: 7 }, dimensions: { actionType: "GetObject" } }, + { sum: { requests: 1 }, dimensions: { actionType: "HeadObject" } }, + ], + kvOperationsAdaptiveGroups: [ + { sum: { requests: 100 }, dimensions: { actionType: "read" } }, + { sum: { requests: 3 }, dimensions: { actionType: "write" } }, + { sum: { requests: 2 }, dimensions: { actionType: "mystery" } }, + { sum: { requests: 4 }, dimensions: { actionType: "list" } }, + { sum: { requests: 1 }, dimensions: { actionType: "delete" } }, + ], + durableObjectsPeriodicGroups: [{ sum: { activeTime: 8e6 } }, { sum: { activeTime: 8e6 } }], + r2StorageAdaptiveGroups: [ + { max: { payloadSize: 1000, metadataSize: 10 }, dimensions: { bucketName: "a" } }, + { max: { payloadSize: 500, metadataSize: 0 }, dimensions: { bucketName: "b" } }, + ], + workersInvocationsAdaptive: [{ sum: { requests: 40, cpuTimeUs: 2500 } }], + }); + const r = await run({ ...ENV, GUARD: kv }, HOUR(12), { fetchImpl: gql }); + const used = (id) => r.metrics[id].used; + assert.equal(used("r2_class_a"), 15); // unknown actions count as Class A + assert.equal(used("r2_class_b"), 8); + assert.equal(used("kv_reads"), 100); + assert.equal(used("kv_writes"), 5); // unknown actions count as writes + assert.equal(used("kv_lists"), 4); + assert.equal(used("kv_deletes"), 1); + assert.equal(used("do_duration_gbs"), 2); // 16 s x 0.125 GB + assert.equal(used("r2_storage_bytes"), 1510); + assert.equal(used("workers_requests"), 40); + assert.equal(used("workers_cpu_ms"), 2.5); + assert.ok(R2_CLASS_B.has("GetObject") && !R2_CLASS_B.has("PutObject")); +}); + +test("synthetic usage past each threshold sets exactly the right KV flags, and they clear", async () => { + const kv = memoryKV(); + const env = { ...ENV, GUARD: kv }; + const steps = [ + [4000, []], + [5100, ["alert"]], + [8100, ["alert", "degrade"]], + [9600, ["alert", "degrade", "refuse"]], + ]; + let h = 1; + for (const [neurons, want] of steps) { + await run(env, HOUR(h++), { fetchImpl: fakeGraphQL(rowsFor("ai_neurons", neurons)) }); + const keys = Object.entries(KEYS).filter(([, k]) => kv.data.has(k)).map(([f]) => f); + assert.deepEqual(keys, want, `${neurons} neurons`); + const g = await readGuard(kv); + assert.deepEqual([g.alert, g.degrade, g.refuse], ["alert", "degrade", "refuse"].map((f) => want.includes(f))); + } + const refuse = JSON.parse(kv.data.get(KEYS.refuse)); + assert.deepEqual(refuse.metrics, ["ai_neurons"]); + assert.equal(refuse.until, "2026-10-16T00:00:00.000Z"); + // the next UTC day: the window has reset + await run(env, new Date("2026-10-16T00:07:00Z"), { fetchImpl: fakeGraphQL(rowsFor("ai_neurons", 20)) }); + assert.deepEqual(Object.values(KEYS).filter((k) => kv.data.has(k)), []); +}); + +test("KV writes: one report a run, flags only when they change", async () => { + const kv = memoryKV(); + const env = { ...ENV, GUARD: kv }; + await run(env, HOUR(1), { fetchImpl: fakeGraphQL(rowsFor("queues_ops", 850000)) }); + const afterFirst = kv.puts; + assert.equal(afterFirst, 3); // report, alert, degrade + for (let h = 2; h <= 6; h++) await run(env, HOUR(h), { fetchImpl: fakeGraphQL(rowsFor("queues_ops", 850000 + h)) }); + assert.equal(kv.puts - afterFirst, 5); // the report only + // first run with no report clears leftover flags; later quiet runs delete nothing + assert.equal(kv.deletes, 1); // refuse, which was never set +}); + +test("first run with leftover flags and no report clears them", async () => { + const kv = memoryKV({ [KEYS.refuse]: JSON.stringify({ since: "x", metrics: ["ai_neurons"], until: null }) }); + await run({ ...ENV, GUARD: kv }, HOUR(1), { fetchImpl: fakeGraphQL() }); + assert.equal(kv.data.has(KEYS.refuse), false); +}); + +test("a failing query keeps the flag it was holding and reports the error", async () => { + const kv = memoryKV(); + const env = { ...ENV, GUARD: kv }; + await run(env, HOUR(1), { fetchImpl: fakeGraphQL(rowsFor("d1_rows_written", 41e6)) }); + assert.ok(kv.data.has(KEYS.degrade)); + const r = await run(env, HOUR(2), { + fetchImpl: fakeGraphQL({ d1AnalyticsAdaptiveGroups: new Error("unknown field rowsWritten") }), + }); + assert.ok(kv.data.has(KEYS.degrade)); + assert.deepEqual(r.errors, ["d1_rows_read", "d1_rows_written"]); + assert.equal(r.metrics.d1_rows_written.stale, true); + assert.match(r.metrics.d1_rows_written.error, /unknown field rowsWritten/); +}); + +test("per-script rows come from the same two queries: summed into the totals, kept per script", async () => { + const gql = fakeGraphQL({ + workersInvocationsAdaptive: [ + { sum: { requests: 10, cpuTimeUs: 3000 }, dimensions: { scriptName: "setkernel-ops" } }, + { sum: { requests: 5, cpuTimeUs: 1000 }, dimensions: { scriptName: "sk-meter" } }, + { sum: { requests: 1, cpuTimeUs: 500 }, dimensions: { scriptName: null } }, + ], + durableObjectsInvocationsAdaptiveGroups: [ + { sum: { requests: 7 }, dimensions: { scriptName: "setkernel-ops" } }, + { sum: { requests: 2 }, dimensions: { scriptName: "sk-status" } }, + ], + }); + const u = await readUsage(ENV, HOUR(12), 1, gql); + assert.equal(u.values.workers_cpu_ms, 4.5); + assert.equal(u.values.do_requests, 9); + assert.deepEqual(u.scripts.workers_cpu_ms, { "setkernel-ops": 3, "sk-meter": 1, [NO_SCRIPT]: 0.5 }); + assert.deepEqual(u.scripts.do_requests, { "setkernel-ops": 7, "sk-status": 2 }); + assert.equal(u.scripts.workers_requests["sk-meter"], 5); + for (const c of gql.calls.filter((c) => /workersInvocations|durableObjectsInvocations/.test(c.dataset))) { + assert.match(c.body.query, /dimensions \{ scriptName \}/); + } + assert.equal(gql.calls.length, SOURCES.length + 1); +}); + +test("a failing per-script dataset leaves its metrics out of scripts, and nothing else", async () => { + const gql = fakeGraphQL({ durableObjectsInvocationsAdaptiveGroups: new Error("unknown field scriptName") }); + const u = await readUsage(ENV, HOUR(12), 1, gql); + assert.match(u.errors.do_requests, /unknown field scriptName/); + assert.equal(u.scripts.do_requests, undefined); + assert.ok(u.scripts.workers_cpu_ms); +}); + +test("the token never appears in the report or an error", async () => { + const kv = memoryKV(); + const failing = async () => new Response("denied", { status: 403 }); + await run({ ...ENV, CF_ANALYTICS_TOKEN: "s3cret-token-value", GUARD: kv }, HOUR(1), { fetchImpl: failing }); + const stored = [...kv.data.values()].join("\n"); + assert.doesNotMatch(stored, /s3cret-token-value/); + assert.match(stored, /HTTP 403/); +}); + +test("the token never appears in the report when per-script rows are read", async () => { + const kv = memoryKV(); + const gql = fakeGraphQL({ workersInvocationsAdaptive: [{ sum: { requests: 1, cpuTimeUs: 2e13 }, dimensions: { scriptName: "setkernel-ops" } }] }); + await run({ ...ENV, CF_ANALYTICS_TOKEN: "s3cret-token-value", GUARD: kv }, HOUR(12), { fetchImpl: gql }); + const stored = [...kv.data.values()].join("\n"); + assert.match(stored, /"topScripts":\{"workers_cpu_ms":\{"script":"setkernel-ops"/); + assert.doesNotMatch(stored, /s3cret-token-value/); +}); + +test("alerts: the baseline on the first run, then only on change, retried until delivered", async () => { + const kv = memoryKV(); + const hook = []; + const env = { ...ENV, ...WEBHOOK, GUARD: kv }; + const gw = { aiGatewayRequestsAdaptiveGroups: [{ count: 3, dimensions: { gateway: "brief", provider: "workers-ai" } }, { count: 9, dimensions: { gateway: "chat", provider: "openai" } }] }; + await run(env, HOUR(1), { fetchImpl: fakeGraphQL({ ...gw, ...rowsFor("queues_ops", 1000) }, { webhook: hook }) }); + assert.equal(hook.length, 1); + assert.equal(hook[0].url, WEBHOOK.ALERT_WEBHOOK_URL); + assert.equal(hook[0].init.headers.authorization, "Bearer hook-token"); + assert.match(hook[0].body.text, /Queues operations: 1,000 of 1,000,000/); + assert.match(hook[0].body.text, /AI Gateways sending requests to Workers AI: brief/); + + await run(env, HOUR(2), { fetchImpl: fakeGraphQL({ ...gw, ...rowsFor("queues_ops", 2000) }, { webhook: hook }) }); + assert.equal(hook.length, 1, "nothing changed, nothing sent"); + + hook.status = 502; + const failed = await run(env, HOUR(3), { fetchImpl: fakeGraphQL({ ...gw, ...rowsFor("queues_ops", 810000) }, { webhook: hook }) }); + assert.equal(hook.length, 2); + assert.match(failed.notifyError, /HTTP 502/); + assert.ok(kv.data.has(KEYS.degrade), "the flag is set even when the alert fails"); + + hook.status = 200; + await run(env, HOUR(4), { fetchImpl: fakeGraphQL({ ...gw, ...rowsFor("queues_ops", 820000) }, { webhook: hook }) }); + assert.equal(hook.length, 3, "retried"); + assert.match(hook[2].body.text, /DEGRADE/); + assert.equal(hook[2].body.mention, true); + + await run(env, HOUR(5), { fetchImpl: fakeGraphQL({ ...gw, ...rowsFor("queues_ops", 830000) }, { webhook: hook }) }); + assert.equal(hook.length, 3, "delivered once"); +}); + +test("no webhook configured: the flags still act and the alert is logged", async () => { + const kv = memoryKV(); + const r = await run({ ...ENV, GUARD: kv }, HOUR(1), { fetchImpl: fakeGraphQL(rowsFor("ai_neurons", 9000)) }); + assert.ok(kv.data.has(KEYS.degrade)); + assert.equal(r.notified, null); +}); + +test("an http:// webhook is refused so the token isn't sent in the clear", async () => { + const kv = memoryKV(); + const hook = []; + const r = await run({ ...ENV, ALERT_WEBHOOK_URL: "http://alerts.example.test/", ALERT_WEBHOOK_TOKEN: "t", GUARD: kv }, HOUR(1), { + fetchImpl: fakeGraphQL({}, { webhook: hook }), + }); + assert.equal(hook.length, 0); + assert.match(r.notifyError, /must be https/); +}); + +test("the drill: a lowered allowance drives one metric past 80% and degrade is observed, then it expires", async () => { + const kv = memoryKV(); + const env = { ...ENV, GUARD: kv }; + // 500 real neurons used today: 5% of the 10,000 allowance + const usage = rowsFor("ai_neurons", 500); + await run(env, HOUR(9), { fetchImpl: fakeGraphQL(usage) }); + assert.equal((await readGuard(kv)).degrade, false); + + kv.data.set(DRILL_KEY, JSON.stringify({ metric: "ai_neurons", allowance: 600, expiresAt: "2026-10-15T12:00:00Z" })); + const r = await run(env, HOUR(10), { fetchImpl: fakeGraphQL(usage) }); + assert.equal(r.metrics.ai_neurons.level, "degrade"); // 500 of 600 = 83% + const g = await readGuard(kv); + assert.equal(g.degrade, true); + assert.equal(g.refuse, false); + + await run(env, HOUR(12), { fetchImpl: fakeGraphQL(usage) }); + assert.equal((await readGuard(kv)).degrade, false, "the drill expired at 12:00"); +}); + +test("the scheduled handler runs at the cron's time and stores the report", async () => { + const kv = memoryKV(); + const realFetch = globalThis.fetch; + globalThis.fetch = fakeGraphQL(rowsFor("kv_lists", 10)); + try { + await worker.scheduled({ scheduledTime: HOUR(3).getTime(), cron: "7 * * * *" }, { ...ENV, GUARD: kv }); + } finally { + globalThis.fetch = realFetch; + } + const report = JSON.parse(kv.data.get(REPORT_KEY)); + assert.equal(report.at, HOUR(3).toISOString()); + assert.equal(report.metrics.kv_lists.used, 10); +}); + +test("no BILLING_DAY (the sync isn't set up yet): the run fails before touching KV or the API", async () => { + const kv = memoryKV(); + const f = fakeGraphQL(); + const { BILLING_DAY, ...env } = ENV; + assert.equal(BILLING_DAY, "1"); + await assert.rejects(run({ ...env, GUARD: kv }, HOUR(1), { fetchImpl: f }), /BILLING_DAY is not set/); + assert.equal(f.calls.length, 0); + assert.equal(kv.puts, 0); +}); diff --git a/workers/meter/tools/baseline.mjs b/workers/meter/tools/baseline.mjs new file mode 100644 index 0000000..b2f231c --- /dev/null +++ b/workers/meter/tools/baseline.mjs @@ -0,0 +1,78 @@ +#!/usr/bin/env node +// The baseline: what the account already uses, measured the way the hourly run measures it, with +// nothing written anywhere. Also the schema check before a deploy: every dataset, field and filter +// key is looked up by introspection first, and anything the API doesn't have, or a query it +// rejects, is a "Can't read" line. Any "Can't read" line fails the run (exit 1). A dataset the API +// keeps answering "try again later" (after retries) is an "Unknown" line instead: that metric shows +// as "?" and the run still exits 0, because waiting fixes it and no name was wrong. +// +// CF_ACCOUNT_ID=... CF_ANALYTICS_TOKEN=... BILLING_DAY=13 node tools/baseline.mjs [--scripts] +// +// It also prints Workers CPU-ms and Durable Object requests by script (hq#243): the top +// TOP_SCRIPTS of each, or every script with --scripts. +// +// The token needs Account Analytics: Read and nothing else. Pass it from Infisical in the +// environment; it is never printed. CI: the `baseline` workflow (workflow_dispatch). + +import { pathToFileURL } from "node:url"; +import allowances from "../allowances.json" with { type: "json" }; +import { readUsage } from "../src/usage.js"; +import { evaluate, message } from "../src/meter.js"; +import { billingDay } from "../src/windows.js"; +import { checkSchema, introspector } from "./schema.mjs"; + +export const CANT_READ = "Can't read"; +export const UNKNOWN = "Unknown"; +export const TOP_SCRIPTS = 5; +// The metrics read per script that the baseline lists, in this order. +const BY_SCRIPT = ["workers_cpu_ms", "do_requests"]; + +export async function baseline(env, { now = new Date(), fetchImpl = fetch, log = console.log, retry, allScripts = false } = {}) { + if (!env.CF_ANALYTICS_TOKEN || !env.CF_ACCOUNT_ID) { + log("baseline: set CF_ACCOUNT_ID and CF_ANALYTICS_TOKEN (Account Analytics: Read)"); + return 2; + } + const day = billingDay(env.BILLING_DAY); + const lines = []; + let failed = false; + + let problems; + try { + problems = await checkSchema(introspector(env, fetchImpl)); + } catch (e) { + problems = [{ where: "the schema", missing: "", has: [], error: e.message }]; + } + for (const p of problems) { + failed = true; + lines.push(p.error + ? `${CANT_READ} the schema: ${p.error}` + : `${CANT_READ} ${p.where}.${p.missing}: not in the schema; there: ${p.has.join(" ") || "nothing"}`); + } + + const usage = await readUsage(env, now, day, fetchImpl, retry); + const report = evaluate({ allowances, usage, now, day }); + lines.push(message(report, { full: true }).text, "", `billing day ${day}`, "metric | used | allowance | share | projected at reset"); + for (const [id, m] of Object.entries(report.metrics)) { + const p = m.projected == null ? "" : `${Math.round((m.projected / m.allowance) * 100)}%`; + lines.push(`${id} | ${m.used ?? "?"} | ${m.allowance} | ${m.share == null ? "?" : (m.share * 100).toFixed(2) + "%"} | ${p}`); + } + for (const id of BY_SCRIPT) { + const by = usage.scripts?.[id]; + if (!by) continue; + const rows = Object.entries(by).sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1)); + const shown = allScripts ? rows : rows.slice(0, TOP_SCRIPTS); + lines.push("", `${id} by script (${shown.length} of ${rows.length}) | ${report.metrics[id].unit}`); + for (const [name, v] of shown) lines.push(`${name} | ${Math.round(v * 1000) / 1000}`); + } + for (const id of report.errors) lines.push(`${CANT_READ} ${id}: ${report.metrics[id].error}`); + for (const id of report.unknown) lines.push(`${UNKNOWN} ${id}: ${report.metrics[id].unknown}; the API said to try again later, so it is not a schema problem`); + failed ||= report.errors.length > 0 || Boolean(report.gateways?.error && !report.gateways.transient); + + for (const l of lines) log(l); + return failed ? 1 : 0; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + const { CF_ANALYTICS_TOKEN, CF_ACCOUNT_ID, BILLING_DAY } = process.env; + process.exit(await baseline({ CF_ANALYTICS_TOKEN, CF_ACCOUNT_ID, BILLING_DAY }, { allScripts: process.argv.includes("--scripts") })); +} diff --git a/workers/meter/tools/drill.mjs b/workers/meter/tools/drill.mjs new file mode 100644 index 0000000..6077b6c --- /dev/null +++ b/workers/meter/tools/drill.mjs @@ -0,0 +1,37 @@ +#!/usr/bin/env node +// The offline drill (`make drill`): the live drill in README.md, step for step, against an +// in-memory KV and a fake analytics API. Nothing leaves this machine. Exit 1 if any step doesn't +// see what the live drill must see. + +import { run } from "../src/index.js"; +import { KEYS, DRILL_KEY, readGuard } from "../src/flags.js"; +import { memoryKV, fakeGraphQL, rowsFor } from "../test/fakes.js"; + +const at = (h) => new Date(Date.UTC(2026, 9, 15, h, 7)); +const kv = memoryKV(); +const webhook = []; +const env = { CF_ANALYTICS_TOKEN: "drill", CF_ACCOUNT_ID: "drill", BILLING_DAY: "13", ALERT_WEBHOOK_URL: "https://alerts.example.test/meter", GUARD: kv }; +// 1,200 queue operations used this cycle: 0.12% of the 1,000,000 allowance +const fetchImpl = fakeGraphQL(rowsFor("queues_ops", 1200), { webhook }); + +let failed = 0; +const step = (ok, what) => { console.log(`${ok ? "ok " : "FAIL"} ${what}`); if (!ok) failed++; }; + +let r = await run(env, at(9), { fetchImpl }); +step(!(await readGuard(kv)).degrade, `1. baseline run: queues_ops at ${r.metrics.queues_ops.used}, guard:degrade absent`); + +const allowance = Math.round(r.metrics.queues_ops.used / 0.85); +kv.data.set(DRILL_KEY, JSON.stringify({ metric: "queues_ops", allowance, expiresAt: at(12).toISOString() })); +step(true, `2. put ${DRILL_KEY}: queues_ops allowance ${allowance} (used รท 0.85) until ${at(12).toISOString()}`); + +r = await run(env, at(10), { fetchImpl }); +const g = await readGuard(kv); +step(g.degrade && !g.refuse && kv.data.has(KEYS.degrade), `3. next run: guard:degrade present (${Math.round(r.metrics.queues_ops.share * 100)}%), guard:refuse absent`); +step(webhook.at(-1)?.body.text.startsWith("[DRILL]") && webhook.at(-1)?.body.mention === true, "4. the alert says [DRILL] and mentions Raghav"); + +kv.data.delete(DRILL_KEY); +r = await run(env, at(11), { fetchImpl }); +step(!kv.data.has(KEYS.degrade) && !kv.data.has(KEYS.alert) && r.drill === null, `5. delete ${DRILL_KEY}, next run: every flag cleared`); + +console.log(failed ? `drill: ${failed} step(s) failed` : "drill: passed"); +process.exit(failed ? 1 : 0); diff --git a/workers/meter/tools/schema.mjs b/workers/meter/tools/schema.mjs new file mode 100644 index 0000000..82a14ef --- /dev/null +++ b/workers/meter/tools/schema.mjs @@ -0,0 +1,97 @@ +// The schema check: every dataset, field and filter key src/usage.js asks for, looked up in the +// live GraphQL schema by introspection, so a wrong name is found by name before the first deploy +// rather than as a query error. Read-only; the token is never part of a message. + +import { ENDPOINT, SOURCES, GATEWAYS } from "../src/usage.js"; + +// "sum { a b } dimensions { c } count" -> [["sum", "a"], ["sum", "b"], ["dimensions", "c"], ["count"]] +export function selectionPaths(fields) { + const tokens = fields.match(/[{}]|\w+/g) ?? []; + const out = []; + const stack = []; + for (let i = 0; i < tokens.length; i++) { + const t = tokens[i]; + if (t === "{") continue; + if (t === "}") { stack.pop(); continue; } + if (tokens[i + 1] === "{") stack.push(t); + else out.push([...stack, t]); + } + return out; +} + +// The filter keys a query uses: the window bounds plus the source's own (eventType, ...). +export function filterKeys(source) { + const own = [...(source.filter ?? "").matchAll(/(\w+)\s*:/g)].map((m) => m[1]); + return ["datetime_geq", "datetime_leq", ...own]; +} + +// LIST / NON_NULL wrappers down to the named type. +const named = (t) => { while (t?.ofType) t = t.ofType; return t?.name ?? null; }; + +const TYPE_REF = "kind name ofType { kind name ofType { kind name ofType { kind name } } }"; +const TYPE_QUERY = `query T($name: String!) { __type(name: $name) { + name fields { name type { ${TYPE_REF} } args { name type { ${TYPE_REF} } } } + inputFields { name type { ${TYPE_REF} } } +} }`; + +export function introspector(env, fetchImpl = fetch) { + const cache = new Map(); + async function ask(query, variables) { + const res = await fetchImpl(ENDPOINT, { + method: "POST", + headers: { "content-type": "application/json", authorization: `Bearer ${env.CF_ANALYTICS_TOKEN}` }, + body: JSON.stringify({ query, variables }), + }); + if (!res.ok) throw new Error(`introspection: HTTP ${res.status}`); + const body = await res.json(); + if (body.errors?.length) throw new Error(`introspection: ${String(body.errors[0].message).slice(0, 200)}`); + return body.data; + } + return { + async root() { + return (await ask("{ __schema { queryType { name } } }")).__schema.queryType.name; + }, + async type(name) { + if (!cache.has(name)) cache.set(name, ask(TYPE_QUERY, { name }).then((d) => d.__type)); + return cache.get(name); + }, + }; +} + +// Problems as { where, missing, has }: `where` is the dataset (or the account type) and the path +// that couldn't be found, `has` the names that type does offer, for the fix. +export async function checkSchema(schema, sources = [...SOURCES, GATEWAYS]) { + const problems = []; + const field = async (typeName, name) => (await schema.type(typeName))?.fields?.find((f) => f.name === name) ?? null; + const names = async (typeName, key = "fields") => ((await schema.type(typeName))?.[key] ?? []).map((f) => f.name).sort(); + + const viewer = await field(await schema.root(), "viewer"); + const accounts = viewer && (await field(named(viewer.type), "accounts")); + if (!accounts) return [{ where: "viewer.accounts", missing: "accounts", has: [] }]; + const account = named(accounts.type); + + for (const s of sources) { + const ds = await field(account, s.dataset); + if (!ds) { + problems.push({ where: s.dataset, missing: s.dataset, has: (await names(account)).filter((n) => /Groups$|Adaptive$/.test(n)) }); + continue; + } + const filterType = named(ds.args?.find((a) => a.name === "filter")?.type); + const offered = filterType ? await names(filterType, "inputFields") : []; + for (const key of filterKeys(s)) { + if (!offered.includes(key)) problems.push({ where: `${s.dataset}(filter)`, missing: key, has: offered }); + } + for (const path of selectionPaths(s.fields)) { + let typeName = named(ds.type); + for (const [i, step] of path.entries()) { + const f = await field(typeName, step); + if (!f) { + problems.push({ where: [s.dataset, ...path.slice(0, i)].join("."), missing: step, has: await names(typeName) }); + break; + } + typeName = named(f.type); + } + } + } + return problems; +} diff --git a/workers/meter/wrangler.toml b/workers/meter/wrangler.toml new file mode 100644 index 0000000..7322970 --- /dev/null +++ b/workers/meter/wrangler.toml @@ -0,0 +1,33 @@ +# sk-meter: the Cloudflare billing guard. One hourly cron, no routes, no public URL. +name = "sk-meter" +main = "src/index.js" +compatibility_date = "2026-10-01" +workers_dev = false +preview_urls = false +# Every var and secret reaches the Worker through the Infisical -> Cloudflare Workers secret sync +# (project sk-ctl-workers, env prod), never through this file or CI. keep_vars stops a deploy +# from deleting what the sync set; there is deliberately no [vars] block. +keep_vars = true + +[triggers] +# 7 past the hour, clear of the top-of-hour rush +crons = ["7 * * * *"] + +[observability] +enabled = true + +# The guard flags (guard:alert, guard:degrade, guard:refuse) and the last report. Every Worker that +# reads the guard binds this same namespace as GUARD. It is the `sk-meter` namespace, owned by tofu in +# setkernel/setkernel-cloudflare (output sk_meter_kv_namespace_id): bind it, never create it. +[[kv_namespaces]] +binding = "GUARD" +id = "f79ca12aeb9347f9851f5dfd53313f39" + +# Set by the sync from Infisical sk-ctl-workers/prod (never in this file): +# BILLING_DAY the day of the month the billing cycle starts (13); required +# CF_ACCOUNT_ID the account to meter +# CF_ANALYTICS_TOKEN Account Analytics: Read, on this account only +# ALERT_WEBHOOK_URL https://alerts.setkernel.net/meter, forwards { source, text, mention } to Buzz +# ALERT_WEBHOOK_TOKEN bearer token that endpoint checks +# ALERT_ACCESS_CLIENT_ID the sk-meter-alerts Access service token in front of that +# ALERT_ACCESS_CLIENT_SECRET endpoint (hq observability/README.md, "Meter alerts") From 7ffd866dbd27cc1e2dc58131606a1bdd97579223 Mon Sep 17 00:00:00 2001 From: Raghav Gupta Date: Mon, 9 Nov 2026 09:16:00 -0400 Subject: [PATCH 2/3] fix(meter): alerts carry { title, message } for the ops Worker's email relay MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The buzz-alerts bridge sk-meter posted to was removed in hq#1261, so its alerts have nowhere to go. The body now also carries title and message, which the ops Worker's POST /alert emails (plan v6 E3b). The old fields stay, so nothing breaks before the Worker's two settings change. Refs setkernel/hq#1449 โ€” Claude Lead Co-Authored-By: Claude Opus 5.5 --- workers/meter/README.md | 11 ++++++++--- workers/meter/src/notify.js | 25 +++++++++++++++++-------- workers/meter/test/notify.test.js | 14 +++++++++++--- workers/meter/wrangler.toml | 7 +++---- 4 files changed, 39 insertions(+), 18 deletions(-) diff --git a/workers/meter/README.md b/workers/meter/README.md index 4eb666a..d7fb015 100644 --- a/workers/meter/README.md +++ b/workers/meter/README.md @@ -83,9 +83,14 @@ nothing there. | `BILLING_DAY` | `13` (cycle starts on the 13th; required, 1-28) | | `CF_ACCOUNT_ID` | the account to meter | | `CF_ANALYTICS_TOKEN` | `sk-meter-analytics-read`: Account Analytics Read, nothing else (setkernel-cloudflare tofu) | -| `ALERT_WEBHOOK_URL` | `https://alerts.setkernel.net/meter` (https only; anything else is refused) | -| `ALERT_WEBHOOK_TOKEN` | bearer the receiver checks | -| `ALERT_ACCESS_CLIENT_ID`, `ALERT_ACCESS_CLIENT_SECRET` | the `sk-meter-alerts` Access service token | +| `ALERT_WEBHOOK_URL` | `https://setkernel-ops.setkernel.workers.dev/alert`, the ops Worker's email relay (https only; anything else is refused) | +| `ALERT_WEBHOOK_TOKEN` | the ops Worker's `OPS_ALERT_TOKEN` (Infisical `hq-workers/prod`, folder `/ops`), copied, never retyped | +| `ALERT_ACCESS_CLIENT_ID`, `ALERT_ACCESS_CLIENT_SECRET` | unused since the relay has no Access app; delete them with the `sk-meter-alerts` token | + +Until 2026-11-09 the URL was `https://alerts.setkernel.net/meter`, the buzz-alerts bridge on +sk-obs01, which hq#1261 removed. An alert that fails is retried every hour, so the alert that +was due arrives on the first run after the two values change. The body carries both shapes +(`src/notify.js`), so the change needs no deploy of its own. The KV namespace `sk-meter` (`f79ca12aeb9347f9851f5dfd53313f39`) is tofu-owned in setkernel-cloudflare: this repo binds it and never creates or deletes it. diff --git a/workers/meter/src/notify.js b/workers/meter/src/notify.js index d8f09c8..5aa33ae 100644 --- a/workers/meter/src/notify.js +++ b/workers/meter/src/notify.js @@ -1,10 +1,19 @@ -// Alert delivery. A Worker can't run the buzz CLI or hold a Buzz identity the way hq's -// observability bridge does, so it POSTs { source, text, mention } to ALERT_WEBHOOK_URL (a Worker -// secret) with ALERT_WEBHOOK_TOKEN as a bearer token, and the receiver sends it to Buzz. The -// receiver is the buzz-alerts bridge's /meter route on sk-obs01 (hq#222, observability/README.md), -// behind Cloudflare Access: ALERT_ACCESS_CLIENT_ID and ALERT_ACCESS_CLIENT_SECRET are the -// sk-meter-alerts service token. A caller may name its own `source`. Without a webhook the alert is logged and the KV flags still act; -// the guard never depends on the alert arriving. +// Alert delivery. sk-meter POSTs one JSON body to ALERT_WEBHOOK_URL (a Worker secret) with +// ALERT_WEBHOOK_TOKEN as a bearer token. The body carries two shapes at once, so either receiver reads it: +// { source, text, mention } the old buzz-alerts bridge's /meter route on sk-obs01 (removed in hq#1261) +// { title, message } the ops Worker's POST /alert (setkernel/hq workers/ops), which emails +// `title` as the subject and `message` as the body (plan v6 E3b: alerts are email) +// ALERT_ACCESS_CLIENT_ID and ALERT_ACCESS_CLIENT_SECRET, when both are set, are sent as a Cloudflare +// Access service token (the old route was behind Access; the ops Worker's is not). A caller may name its +// own `source`. Without a webhook the alert is logged and the KV flags still act; the guard never +// depends on the alert arriving. + +// The subject line is the source and the alert's first line, the body its whole text. +export function payload(alert) { + const source = alert.source ?? "sk-meter"; + const first = String(alert.text ?? "").split("\n").find((l) => l.trim()) ?? "alert"; + return { source, text: alert.text, mention: alert.mention, title: `${source}: ${first.trim()}`.slice(0, 200), message: alert.text }; +} export async function notify(env, alert, fetchImpl = fetch) { const url = env.ALERT_WEBHOOK_URL; @@ -26,7 +35,7 @@ export async function notify(env, alert, fetchImpl = fetch) { // Access answers a refused token with a redirect to its login page; following it could end in // a 200 and count the alert as delivered. A redirect is a failure, retried next hour. redirect: "manual", - body: JSON.stringify({ source: alert.source ?? "sk-meter", text: alert.text, mention: alert.mention }), + body: JSON.stringify(payload(alert)), }); if (!res.ok) throw new Error(`alert webhook answered HTTP ${res.status}`); return true; diff --git a/workers/meter/test/notify.test.js b/workers/meter/test/notify.test.js index a33dd24..f75c2e1 100644 --- a/workers/meter/test/notify.test.js +++ b/workers/meter/test/notify.test.js @@ -2,7 +2,7 @@ import { test } from "node:test"; import assert from "node:assert/strict"; -import { notify } from "../src/notify.js"; +import { notify, payload } from "../src/notify.js"; const URL = "https://alerts.example.test/meter"; const ALERT = { text: "x", mention: false }; @@ -39,7 +39,7 @@ test("an Access redirect to its login page is a failure, so the alert is retried await assert.rejects(notify({ ALERT_WEBHOOK_URL: URL }, ALERT, capture(302)), /HTTP 302/); }); -test("the receiver's contract: POST { source, text, mention } as JSON, and its 204 counts as delivered", async () => { +test("the receiver's contract: POST { source, text, mention, title, message } as JSON, and its 204 counts as delivered", async () => { const f = capture(204); const env = { ALERT_WEBHOOK_URL: "https://alerts.setkernel.net/meter", ALERT_WEBHOOK_TOKEN: "t" }; assert.equal(await notify(env, { text: "line 1\nline 2", mention: true }, f), true); @@ -47,7 +47,15 @@ test("the receiver's contract: POST { source, text, mention } as JSON, and its 2 assert.equal(url, "https://alerts.setkernel.net/meter"); assert.equal(init.method, "POST"); assert.equal(init.headers["content-type"], "application/json"); - assert.deepEqual(JSON.parse(init.body), { source: "sk-meter", text: "line 1\nline 2", mention: true }); + assert.deepEqual(JSON.parse(init.body), { source: "sk-meter", text: "line 1\nline 2", mention: true, title: "sk-meter: line 1", message: "line 1\nline 2" }); +}); + +test("the ops Worker's /alert reads title as the subject and message as the body", () => { + assert.deepEqual(payload({ text: "\n DEGRADE: queues_ops 85%\nmore", mention: true }), { + source: "sk-meter", text: "\n DEGRADE: queues_ops 85%\nmore", mention: true, title: "sk-meter: DEGRADE: queues_ops 85%", message: "\n DEGRADE: queues_ops 85%\nmore", + }); + assert.equal(payload({ text: "x".repeat(500) }).title.length, 200); + assert.equal(payload({ text: "" }).title, "sk-meter: alert"); }); test("an http:// URL is refused before anything is sent, so no secret leaves in the clear", async () => { diff --git a/workers/meter/wrangler.toml b/workers/meter/wrangler.toml index 7322970..31d2cd2 100644 --- a/workers/meter/wrangler.toml +++ b/workers/meter/wrangler.toml @@ -27,7 +27,6 @@ id = "f79ca12aeb9347f9851f5dfd53313f39" # BILLING_DAY the day of the month the billing cycle starts (13); required # CF_ACCOUNT_ID the account to meter # CF_ANALYTICS_TOKEN Account Analytics: Read, on this account only -# ALERT_WEBHOOK_URL https://alerts.setkernel.net/meter, forwards { source, text, mention } to Buzz -# ALERT_WEBHOOK_TOKEN bearer token that endpoint checks -# ALERT_ACCESS_CLIENT_ID the sk-meter-alerts Access service token in front of that -# ALERT_ACCESS_CLIENT_SECRET endpoint (hq observability/README.md, "Meter alerts") +# ALERT_WEBHOOK_URL the ops Worker's POST /alert, which emails { title, message } (README.md) +# ALERT_WEBHOOK_TOKEN its OPS_ALERT_TOKEN bearer +# ALERT_ACCESS_CLIENT_ID, ALERT_ACCESS_CLIENT_SECRET unused (the old route's Access token) From ff1841fa4dd69f6c890f75d81e4a3c94599592c8 Mon Sep 17 00:00:00 2001 From: Raghav Gupta Date: Mon, 9 Nov 2026 09:27:58 -0400 Subject: [PATCH 3/3] fix(deploy): a failed sk-meter rollback reaches the issue, with owner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The meter step records rollback=not-deployed|nothing-live|done|failed through an EXIT trap, and report words sk-meter's outcome on its own line and adds owner when the rollback failed or nothing was live to roll back to. Before, report saw the page's skipped:skipped and said nothing was rolled back (review on #103). Also: AGENTS.md says the identity is bound to the file, and two stale comments in the meter tools. Refs setkernel/hq#1449 โ€” Claude Lead Co-Authored-By: Claude Opus 5.5 --- .github/workflows/deploy.yml | 44 +++++++++++++++++++++++++++++--- AGENTS.md | 6 +++-- workers/meter/tools/baseline.mjs | 2 +- workers/meter/tools/drill.mjs | 2 +- 4 files changed, 46 insertions(+), 8 deletions(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 81ab650..27e00c5 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -209,10 +209,17 @@ jobs: # sk-meter (workers/meter/README.md, "Deploy"): one hourly cron and no route, so nothing can be asked over # HTTP. The job records the version serving all traffic, deploys, and passes only if this run's version # ("deploy ") now serves all of it. Otherwise it rolls back to the recorded version and fails. + # That is a check of the deployment, not a smoke: a version whose cron throws still passes it, and + # sk-watch's failed-cron-event alert covers that. Its `rollback` output tells `report` what happened: + # not-deployed (the deploy failed, nothing changed), nothing-live (no version to roll back to), + # done (rolled back, and the old version serves all traffic again) or failed (the rollback failed + # or didn't take); empty when the job passed. meter: needs: check runs-on: setkernel-linux timeout-minutes: 10 + outputs: + rollback: ${{ steps.meter.outputs.rollback }} permissions: contents: read id-token: write # OIDC login to Infisical @@ -240,7 +247,11 @@ jobs: # Only CLOUDFLARE_API_TOKEN reaches wrangler: env -i drops the OIDC request variables and the # page's two Worker secrets, which sk-meter never gets. Its settings come from the Infisical sync. - name: Deploy sk-meter, check it, roll back if it isn't live + id: meter run: | + # Whatever ends this script, the trap records how far it got (bash -e exits on any failed command). + state=not-deployed + trap 'echo "rollback=${state}" >> "$GITHUB_OUTPUT"' EXIT wrangler() { env -i PATH="$PATH" HOME="$HOME" CI=true WRANGLER_SEND_METRICS=false \ CLOUDFLARE_API_TOKEN="$CLOUDFLARE_API_TOKEN" \ @@ -249,24 +260,32 @@ jobs: live() { wrangler deployments status --json | jq -r '[.versions[]? | select(.percentage == 100) | .version_id][0] // empty'; } before=$(live) || before= wrangler deploy --message "deploy ${GITHUB_SHA}" + state=failed # deployed: from here a failure leaves sk-meter on an unknown version now=$(live) || now= message=$(wrangler versions view "$now" --json | jq -r '.annotations["workers/message"] // empty') || message= if [ "$message" = "deploy ${GITHUB_SHA}" ]; then echo "sk-meter serves ${now} (deploy ${GITHUB_SHA}) to all traffic" + state= exit 0 fi echo "::error::sk-meter's live version ${now:-(none)} is '${message}', not 'deploy ${GITHUB_SHA}'" if [ -z "$before" ]; then echo "::error::no version was live before, so there is nothing to roll back to" + state=nothing-live exit 1 fi wrangler rollback "$before" --message "rollback from ${GITHUB_SHA::7}" --yes - [ "$(live)" = "$before" ] && echo "::error::rolled back to ${before}, which serves all traffic again" || + if [ "$(live)" = "$before" ]; then + echo "::error::rolled back to ${before}, which serves all traffic again" + state=done + else echo "::error::rolled back to ${before}, but it doesn't serve all traffic" + fi exit 1 # Any failure above, the checks included, opens one issue for the agents, and for the owner too when - # a failed smoke could not be rolled back or failed again after it. The runner image has curl and jq + # a failed smoke could not be rolled back or failed again after it, or sk-meter's rollback failed or + # had nothing to roll back to. The runner image has curl and jq # but no gh. report: needs: [check, deploy, smoke, rollback, smoke-again, meter] @@ -286,6 +305,7 @@ jobs: ROLLBACK: ${{ needs.rollback.result }} SMOKE_AGAIN: ${{ needs.smoke-again.result }} METER: ${{ needs.meter.result }} + METER_ROLLBACK: ${{ needs.meter.outputs.rollback }} VERSION: ${{ needs.deploy.outputs.previous-version }} PREVIOUS: ${{ needs.deploy.outputs.previous-sha }} run: | @@ -304,11 +324,27 @@ jobs: more='["owner"]' rolled="The smoke failed and the rollback failed (the job log says why), so ${GITHUB_SHA::7} may still be live. main is unchanged." ;; *) - rolled="Nothing was rolled back: no smoke failed after a deploy." ;; + rolled="The page: nothing was rolled back, because no smoke failed after its deploy." ;; + esac + # sk-meter's own outcome (the meter job). Its rollback failing, or nothing to roll back to, is the owner's. + case "$METER_ROLLBACK" in + not-deployed) + meter_text="sk-meter: the deploy failed before anything changed, so the version live before still serves." ;; + nothing-live) + more='["owner"]' + meter_text="sk-meter: this run's version was not serving all traffic, and no version was live before, so nothing was rolled back. sk-meter may be running no version at all." ;; + done) + meter_text="sk-meter: this run's version was not serving all traffic, so the meter job rolled back to the version live before, which serves all traffic again." ;; + failed) + more='["owner"]' + meter_text="sk-meter: this run's version was not serving all traffic and the rollback failed or did not take, so sk-meter may be on neither version. Read wrangler deployments status --config workers/meter/wrangler.toml." ;; + *) + meter_text="" ;; esac - body="The deploy of ${GITHUB_SHA} failed. Jobs: check ${CHECK}, deploy ${DEPLOY}, smoke ${SMOKE}, rollback ${ROLLBACK}, smoke-again ${SMOKE_AGAIN}, meter ${METER} (sk-meter; its log says whether it rolled back). + body="The deploy of ${GITHUB_SHA} failed. Jobs: check ${CHECK}, deploy ${DEPLOY}, smoke ${SMOKE}, rollback ${ROLLBACK}, smoke-again ${SMOKE_AGAIN}, meter ${METER} (sk-meter). ${rolled} + ${meter_text} Run: ${RUN_URL} diff --git a/AGENTS.md b/AGENTS.md index fe6edf2..a9e0e01 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,11 +38,13 @@ if the rollback or its smoke fails). Where it differs from the template, the hea validate`, the identity and folder, `VERSION` and the Worker secrets in the deploy's one version). It reads `CLOUDFLARE_API_TOKEN`, `GITHUB_WEBHOOK_SECRET` and `BUZZ_PRIVATE_KEY` from Infisical (project `setkernel-live`, prod, `/`) through the OIDC identity `setkernel-live-deploy`, which accepts only that -workflow on `main`, in the `deploy` and `rollback` jobs only. The token itself is +workflow file on `main` (its binding is the file, not a job); only the `deploy`, `rollback` and +`meter` jobs ask for it. The token itself is `setkernel-live-deploy` in setkernel-cloudflare `stacks/services/api_tokens.tf`. Never deploy from a laptop. The same workflow's `meter` job deploys sk-meter (`workers/meter/wrangler.toml`) with that token, -checks that its version serves all traffic and rolls it back in the job if not. +checks that its version serves all traffic and rolls it back in the job if not. A failed rollback, +or nothing live to roll back to, gives the issue `owner` too. ## Layout - `src/worker.js`: routes (`/ws`, `/api/state`, `/api/replay`, `/ingest`, `/ingest/usage`, `/ingest/limits`, `/usage/data`, `/usage/`, `/github/webhook`, `/github/queue`) and the `Hub` and `Usage` Durable Objects. diff --git a/workers/meter/tools/baseline.mjs b/workers/meter/tools/baseline.mjs index b2f231c..78baec3 100644 --- a/workers/meter/tools/baseline.mjs +++ b/workers/meter/tools/baseline.mjs @@ -12,7 +12,7 @@ // TOP_SCRIPTS of each, or every script with --scripts. // // The token needs Account Analytics: Read and nothing else. Pass it from Infisical in the -// environment; it is never printed. CI: the `baseline` workflow (workflow_dispatch). +// environment; it is never printed. Run it by hand (README.md, "The baseline"). import { pathToFileURL } from "node:url"; import allowances from "../allowances.json" with { type: "json" }; diff --git a/workers/meter/tools/drill.mjs b/workers/meter/tools/drill.mjs index 6077b6c..6a8d508 100644 --- a/workers/meter/tools/drill.mjs +++ b/workers/meter/tools/drill.mjs @@ -1,5 +1,5 @@ #!/usr/bin/env node -// The offline drill (`make drill`): the live drill in README.md, step for step, against an +// The offline drill (`node workers/meter/tools/drill.mjs`): the live drill in README.md, step for step, against an // in-memory KV and a fake analytics API. Nothing leaves this machine. Exit 1 if any step doesn't // see what the live drill must see.