Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 105 additions & 5 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ name: Deploy
# smokes that, and a failed run opens an agent-ready issue with the run's URL.
# - One concurrency group, never cancelled: a newer push replaces the pending run, so the newest main
# always deploys and no deploy stops half way.
# - Only `deploy` and `rollback` can get the deploy token (id-token: write), and they run no app code:
# - Only `deploy`, `rollback` and `meter` can get the deploy token (id-token: write), and they run no app code:
# install scripts are off and wrangler runs without the OIDC request variables. The checks and the
# smoke run in jobs that have no token; only `report` can write issues.
# - Rollback: `deploy` records the live version, and the commit its "deploy <sha>" message names, before
Expand All @@ -22,6 +22,10 @@ name: Deploy
# the two Worker secrets from Infisical into the same version, so each deploy is one version.
# - The smoke may wait 10 minutes for the Durable Object to run the new code, so its jobs get 15, and
# scripts/smoke.mjs has no dependencies, so they install nothing.
# - The `meter` job deploys the second Worker, sk-meter (workers/meter, setkernel/hq#1449), with the same
# token after the same `check`. It has no route, so it checks the live deployment itself and rolls back
# in the same job; like `deploy` and `rollback` it installs no scripts and runs nothing but wrangler.
# It doesn't wait for the page's deploy, and the page's smoke doesn't wait for it.
on:
push:
branches: [main]
Expand Down Expand Up @@ -202,11 +206,89 @@ jobs:
fi
node scripts/smoke.mjs --expect "$EXPECT"

# sk-meter (workers/meter/README.md, "Deploy"): one hourly cron and no route, so nothing can be asked over
# HTTP. The job records the version serving all traffic, deploys, and passes only if this run's version
# ("deploy <sha>") now serves all of it. Otherwise it rolls back to the recorded version and fails.
# That is a check of the deployment, not a smoke: a version whose cron throws still passes it, and
# sk-watch's failed-cron-event alert covers that. Its `rollback` output tells `report` what happened:
# not-deployed (the deploy failed, nothing changed), nothing-live (no version to roll back to),
# done (rolled back, and the old version serves all traffic again) or failed (the rollback failed
# or didn't take); empty when the job passed.
meter:
needs: check
runs-on: setkernel-linux
timeout-minutes: 10
outputs:
rollback: ${{ steps.meter.outputs.rollback }}
permissions:
contents: read
id-token: write # OIDC login to Infisical
env:
WRANGLER_SEND_METRICS: "false"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- run: npm ci --ignore-scripts --no-audit --no-fund
- name: Fetch the deploy token from Infisical
uses: Infisical/secrets-action@d2e351f16c6ca20d17c85e6c992e04bdeb64e87d # v1.0.18
with:
method: oidc
identity-id: 37b06681-c931-4bad-b3cf-4e76bfb633dc # setkernel-live-deploy
oidc-audience: https://github.com/setkernel
domain: https://secrets-api.setkernel.net
project-slug: ${{ github.event.repository.name }}
env-slug: prod
secret-path: /
export-type: env
extra-headers: |
CF-Access-Client-Id: ${{ secrets.CF_ACCESS_CLIENT_ID }}
CF-Access-Client-Secret: ${{ secrets.CF_ACCESS_CLIENT_SECRET }}
# Only CLOUDFLARE_API_TOKEN reaches wrangler: env -i drops the OIDC request variables and the
# page's two Worker secrets, which sk-meter never gets. Its settings come from the Infisical sync.
- name: Deploy sk-meter, check it, roll back if it isn't live
id: meter
run: |
# Whatever ends this script, the trap records how far it got (bash -e exits on any failed command).
state=not-deployed
trap 'echo "rollback=${state}" >> "$GITHUB_OUTPUT"' EXIT
wrangler() {
env -i PATH="$PATH" HOME="$HOME" CI=true WRANGLER_SEND_METRICS=false \
CLOUDFLARE_API_TOKEN="$CLOUDFLARE_API_TOKEN" \
npx --no-install wrangler "$@" --config workers/meter/wrangler.toml
}
live() { wrangler deployments status --json | jq -r '[.versions[]? | select(.percentage == 100) | .version_id][0] // empty'; }
before=$(live) || before=
wrangler deploy --message "deploy ${GITHUB_SHA}"
state=failed # deployed: from here a failure leaves sk-meter on an unknown version
now=$(live) || now=
message=$(wrangler versions view "$now" --json | jq -r '.annotations["workers/message"] // empty') || message=
if [ "$message" = "deploy ${GITHUB_SHA}" ]; then
echo "sk-meter serves ${now} (deploy ${GITHUB_SHA}) to all traffic"
state=
exit 0
fi
echo "::error::sk-meter's live version ${now:-(none)} is '${message}', not 'deploy ${GITHUB_SHA}'"
if [ -z "$before" ]; then
echo "::error::no version was live before, so there is nothing to roll back to"
state=nothing-live
exit 1
fi
wrangler rollback "$before" --message "rollback from ${GITHUB_SHA::7}" --yes
if [ "$(live)" = "$before" ]; then
echo "::error::rolled back to ${before}, which serves all traffic again"
state=done
else
echo "::error::rolled back to ${before}, but it doesn't serve all traffic"
fi
exit 1

# Any failure above, the checks included, opens one issue for the agents, and for the owner too when
# a failed smoke could not be rolled back or failed again after it. The runner image has curl and jq
# a failed smoke could not be rolled back or failed again after it, or sk-meter's rollback failed or
# had nothing to roll back to. The runner image has curl and jq
# but no gh.
report:
needs: [check, deploy, smoke, rollback, smoke-again]
needs: [check, deploy, smoke, rollback, smoke-again, meter]
if: failure()
runs-on: setkernel-linux
timeout-minutes: 5
Expand All @@ -222,6 +304,8 @@ jobs:
SMOKE: ${{ needs.smoke.result }}
ROLLBACK: ${{ needs.rollback.result }}
SMOKE_AGAIN: ${{ needs.smoke-again.result }}
METER: ${{ needs.meter.result }}
METER_ROLLBACK: ${{ needs.meter.outputs.rollback }}
VERSION: ${{ needs.deploy.outputs.previous-version }}
PREVIOUS: ${{ needs.deploy.outputs.previous-sha }}
run: |
Expand All @@ -240,11 +324,27 @@ jobs:
more='["owner"]'
rolled="The smoke failed and the rollback failed (the job log says why), so ${GITHUB_SHA::7} may still be live. main is unchanged." ;;
*)
rolled="Nothing was rolled back: no smoke failed after a deploy." ;;
rolled="The page: nothing was rolled back, because no smoke failed after its deploy." ;;
esac
# sk-meter's own outcome (the meter job). Its rollback failing, or nothing to roll back to, is the owner's.
case "$METER_ROLLBACK" in
not-deployed)
meter_text="sk-meter: the deploy failed before anything changed, so the version live before still serves." ;;
nothing-live)
more='["owner"]'
meter_text="sk-meter: this run's version was not serving all traffic, and no version was live before, so nothing was rolled back. sk-meter may be running no version at all." ;;
done)
meter_text="sk-meter: this run's version was not serving all traffic, so the meter job rolled back to the version live before, which serves all traffic again." ;;
failed)
more='["owner"]'
meter_text="sk-meter: this run's version was not serving all traffic and the rollback failed or did not take, so sk-meter may be on neither version. Read wrangler deployments status --config workers/meter/wrangler.toml." ;;
*)
meter_text="" ;;
esac
body="The deploy of ${GITHUB_SHA} failed. Jobs: check ${CHECK}, deploy ${DEPLOY}, smoke ${SMOKE}, rollback ${ROLLBACK}, smoke-again ${SMOKE_AGAIN}.
body="The deploy of ${GITHUB_SHA} failed. Jobs: check ${CHECK}, deploy ${DEPLOY}, smoke ${SMOKE}, rollback ${ROLLBACK}, smoke-again ${SMOKE_AGAIN}, meter ${METER} (sk-meter).

${rolled}
${meter_text}

Run: ${RUN_URL}

Expand Down
15 changes: 13 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ secrets, branches and commits, the end-of-run report) come from setkernel's cent
## What this repo is
The public page at https://live.setkernel.com: a Cloudflare Worker with static assets and one
Durable Object. sk-obs01 pushes filtered, delayed events to it; visitors watch over a WebSocket.
It also holds `workers/meter`, sk-meter, the Cloudflare billing guard (an hourly cron, no route;
`workers/meter/README.md`), moved here on 2026-11-09 (setkernel/hq#1449).
Public repo: everything here is visible to anyone.

## Commands
Expand All @@ -21,8 +23,10 @@ Public repo: everything here is visible to anyone.
# its poster (the artifact, the pull request, the shots branch, the comment),
# and repo names against GitHub's public list, live)
# worker: npm run test:worker (vitest in workerd: the real Worker and Hub, see test/worker/)
# meter: npm run test:meter (node --test: sk-meter's guard, usage queries, alert, baseline)
# lint: npm run check (node --check on every script)
# validate: npm run validate (check + test + test:worker + wrangler deploy --dry-run; what CI runs)
# validate: npm run validate (check + test + test:worker + test:meter + a wrangler deploy --dry-run
# of each Worker; what CI runs)
```

## Deploy
Expand All @@ -34,9 +38,13 @@ if the rollback or its smoke fails). Where it differs from the template, the hea
validate`, the identity and folder, `VERSION` and the Worker secrets in the deploy's one version). It
reads `CLOUDFLARE_API_TOKEN`, `GITHUB_WEBHOOK_SECRET` and `BUZZ_PRIVATE_KEY` from Infisical (project
`setkernel-live`, prod, `/`) through the OIDC identity `setkernel-live-deploy`, which accepts only that
workflow on `main`, in the `deploy` and `rollback` jobs only. The token itself is
workflow file on `main` (its binding is the file, not a job); only the `deploy`, `rollback` and
`meter` jobs ask for it. The token itself is
`setkernel-live-deploy` in setkernel-cloudflare `stacks/services/api_tokens.tf`. Never deploy from
a laptop.
The same workflow's `meter` job deploys sk-meter (`workers/meter/wrangler.toml`) with that token,
checks that its version serves all traffic and rolls it back in the job if not. A failed rollback,
or nothing live to roll back to, gives the issue `owner` too.

## Layout
- `src/worker.js`: routes (`/ws`, `/api/state`, `/api/replay`, `/ingest`, `/ingest/usage`, `/ingest/limits`, `/usage/data`, `/usage/`, `/github/webhook`, `/github/queue`) and the `Hub` and `Usage` Durable Objects.
Expand Down Expand Up @@ -91,6 +99,9 @@ a laptop.
put text from the artifact in the comment, and never write any ref but `visual-diff-shots`.
The poster reads only the files main's `SHOTS` (in `scripts/visual-diff.mjs`) names, so a pull request
that changes `SHOTS` gets no comment ("unexpected file in the artifact") until it merges.
- `workers/meter/`: sk-meter, its own Worker (`wrangler.toml`, not `wrangler.jsonc`): `src/` the hourly
run and the guard flags, `allowances.json`, `tools/` the baseline and the offline drill, `test/` its
`node --test` suites. Its settings come from the Infisical -> Workers sync (`keep_vars = true`).
- `test/`: `node --test` suites for the pure modules and the page (`day.test.mjs` for Day).
- `test/worker/`: vitest suites run in workerd by `@cloudflare/vitest-plugin` (`vitest.config.mjs`
reads `wrangler.jsonc`). They fake Access's certs with a throwaway key and drive every route
Expand Down
5 changes: 3 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@
"scripts": {
"test": "node --test test/*.test.mjs",
"test:worker": "vitest run",
"check": "for f in src/*.js public/*.js public/lib/*.js public/usage/*.js scripts/*.mjs; do node --check \"$f\" || exit 1; done",
"validate": "npm run check && npm test && npm run test:worker && wrangler deploy --dry-run --outdir .wrangler/dry-run"
"test:meter": "node --test workers/meter/test/*.test.js",
"check": "for f in src/*.js public/*.js public/lib/*.js public/usage/*.js scripts/*.mjs workers/meter/src/*.js workers/meter/tools/*.mjs; do node --check \"$f\" || exit 1; done",
"validate": "npm run check && npm test && npm run test:worker && npm run test:meter && wrangler deploy --dry-run --outdir .wrangler/dry-run && wrangler deploy --dry-run --config workers/meter/wrangler.toml --outdir .wrangler/dry-run-meter"
},
"dependencies": {
"@noble/secp256k1": "3.2.0"
Expand Down
104 changes: 104 additions & 0 deletions workers/meter/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# sk-meter: the Cloudflare billing guard

Cloudflare's budget alerts email a day late and don't cap usage, and overage on Workers Paid
bills automatically. sk-meter keeps setkernel inside the $5 plan. Every hour (cron `7 * * * *`)
it reads the account's usage through the GraphQL Analytics API, totals each metric over its
window, compares it with the plan's included allowance and sets three KV flags.

| Share of any allowance | KV key | What happens |
|---|---|---|
| 50% | `guard:alert` | an alert to Raghav |
| 80% | `guard:degrade` | readers run with `DEGRADED` (`src/flags.js`) |
| 95% | `guard:refuse` | readers refuse non-essential requests until the window resets |

Cloudflare's own $10 budget alert stays on as the backstop.

It moved here from a private repository on 2026-11-09 (setkernel/hq#1449, plan v6 F3). Its
history up to then is that repository's `meter/` at commit `c1e3253`. Not moved, and retired
with that repository: the Workers Logs `readings` tool and the `baseline` and live `drill`
workflows (each read Infisical through an identity bound to a workflow there).

## Files
- `allowances.json`: the included allowance for each metric, its window (`month` from the
billing day, `day` from 00:00 UTC for Workers AI, `gauge` for storage), and the pricing page
it came from. Change it only from those pages.
- `src/usage.js`: the GraphQL queries, one per dataset.
- `src/meter.js`: shares, levels, flags, the pro-rated projection, the alert text. Pure.
- `src/flags.js`: the contract for every reader: `readGuard(env.GUARD)`,
`refusal(guard, { essential })` and `DEGRADED`.
- `src/index.js`: the hourly run. `src/notify.js`: the alert webhook.
- `tools/baseline.mjs`: a read-only report of current usage, also the schema check.
- `tools/drill.mjs`: the offline drill (below).

## How it decides
- Each metric's total is recomputed from the start of its window every hour, so a missed run
loses nothing.
- A flag is on while any metric is at or past its threshold, and lists those metrics and when
the last of them resets (`until`; `null` when storage holds it, because only deleting data
brings that down).
- A query that fails keeps the metric's last value from the same window, so an outage of the
analytics API never lowers a flag; the failure is in the alert.
- Pro-rated: once 10% of a window has passed, a metric on pace to pass its allowance before the
reset is named in the alert. Flags act on actual use, not the projection.
- A reader that can't reach KV treats it as degrade, not refuse.
- The Workers and Durable Object queries also group by `scriptName` in the same query, so a
metric projected to reach 50% names its top script in the alert (`topScripts` in the report).
- An alert goes out when the flags, errors, on-pace list or Workers AI gateways change, and is
retried each hour until delivered.

## What it costs
Its own use of the plan (https://developers.cloudflare.com/workers/platform/pricing/), a month:

| Metric | sk-meter's use | Included | Share |
|---|---|---|---|
| Workers requests (cron runs) | 720 | 10,000,000 | 0.01% |
| Workers CPU time | estimate: a few ms a run, well under 36,000 ms | 30,000,000 ms | <0.2% |
| KV reads | 1,440 (report + drill each run) | 10,000,000 | 0.01% |
| KV writes | 720 (report) + a few on flag changes | 1,000,000 | <0.1% |
| GraphQL Analytics subrequests | ~11 a run; up to ~44 when every dataset needs its 3 retries | not billed | |

## Commands
```sh
npm run test:meter # the tests (node --test); npm run validate runs them too
node workers/meter/tools/drill.mjs # the offline drill: in-memory KV, fake analytics API, nothing leaves the machine
CF_ACCOUNT_ID=… CF_ANALYTICS_TOKEN=… BILLING_DAY=13 node workers/meter/tools/baseline.mjs # live, read-only
```

## Deploy
Only from CI: the `meter` job in `.github/workflows/deploy.yml`, on every push to `main` and by
hand, after the same `check` job as the page. It reads the repo's deploy token (Infisical
project `setkernel-live`, identity `setkernel-live-deploy`: Workers Editor on the account, which
deploys an existing Worker and binds an existing KV namespace). sk-meter has no route, so the
job checks the live deployment itself: the version serving all traffic must be this run's
(`deploy <sha>`). If not, it rolls back to the version that was live before, and the job fails,
so `report` opens an issue. Nothing sets vars or secrets on deploy (`keep_vars = true`).

### The Worker's settings
All in Infisical `sk-ctl-workers/prod`, reaching the Worker through the Infisical -> Cloudflare
Workers secret sync (target Worker `sk-meter`). The sync is by Worker name, so the move changed
nothing there.

| Key | Value |
|---|---|
| `BILLING_DAY` | `13` (cycle starts on the 13th; required, 1-28) |
| `CF_ACCOUNT_ID` | the account to meter |
| `CF_ANALYTICS_TOKEN` | `sk-meter-analytics-read`: Account Analytics Read, nothing else (setkernel-cloudflare tofu) |
| `ALERT_WEBHOOK_URL` | `https://alerts.setkernel.net/meter` (https only; anything else is refused) |
| `ALERT_WEBHOOK_TOKEN` | bearer the receiver checks |
| `ALERT_ACCESS_CLIENT_ID`, `ALERT_ACCESS_CLIENT_SECRET` | the `sk-meter-alerts` Access service token |

The KV namespace `sk-meter` (`f79ca12aeb9347f9851f5dfd53313f39`) is tofu-owned in
setkernel-cloudflare: this repo binds it and never creates or deletes it.

## The baseline
`tools/baseline.mjs` introspects the GraphQL schema for every dataset, field and filter key in
`src/usage.js`, then reads the current usage the way the hourly run does, and prints both. Any
"Can't read" line fails it. A dataset still answering "try again later" after three retries is
an "Unknown" line: its metric shows as `?` and the run passes. It writes nothing and prints no
secret.

## The drill
Proves the degrade end to end without extra spend: `meter:drill` in the namespace lowers one
metric's allowance below what is already used, until its `expiresAt`. A drill can only lower an
allowance. `tools/drill.mjs` runs the steps offline and fails if any step doesn't see what it
should.
Loading
Loading