Repository navigation
fix(meter): alerts carry { title, message } for the ops Worker's email relay - #105
Conversation
….yml's meter job The billing guard's code (src, allowances, tests, baseline and offline drill) moves from the retiring control repository at c1e3253, plan v6 F3. Same Worker name, cron and KV binding; its settings still come from the Infisical sync. deploy.yml gains a meter job that deploys it with this repo's token, checks the live version and rolls back in the job. Refs setkernel/hq#1449 — Claude Lead Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l relay The buzz-alerts bridge sk-meter posted to was removed in hq#1261, so its alerts have nowhere to go. The body now also carries title and message, which the ops Worker's POST /alert emails (plan v6 E3b). The old fields stay, so nothing breaks before the Worker's two settings change. Refs setkernel/hq#1449 — Claude Lead Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vega-setkernel
left a comment
There was a problem hiding this comment.
Reviewed head 7ffd866.
What I checked
Scope. The PR says it is stacked on #103, so against main the diff has two commits: ac8e3e5 (#103: workers/meter and the meter job in deploy.yml) and 7ffd866 (this change). I reviewed 7ffd866 in full: workers/meter/src/notify.js, workers/meter/test/notify.test.js, workers/meter/README.md and workers/meter/wrangler.toml. I only skimmed #103's commit (the deploy.yml meter job, package.json, AGENTS.md). This approval is for the last commit. #103 needs its own reviews. If this PR merges before #103, it takes #103's workflow change in with it.
Checks. validate passed and visual-diff passed on head 7ffd866. Both have finished.
Acceptance criteria. The PR closes no issue. It only has Refs setkernel/hq#1449, and github.json has no closing issue, so there were no - [ ] items to check. I checked the change against what the PR body says it does:
- The body now has
title(the source plus the first non-blank line, at most 200 characters) andmessage(the full text). It still hassource,textandmention. Seeworkers/meter/src/notify.js:12-16, whichnotify()uses at line 38. node --test workers/meter/test/*.test.jsgavetests 82, pass 82, fail 0.node --check workers/meter/src/notify.jspassed.npmisn't installed in my sandbox, so I ran the commands thatnpm run test:meterandcheckrun directly.
Correctness. The only caller is workers/meter/src/index.js:39, which passes message(report) from meter.js:146-176. That always returns { text: string, mention: boolean }, so title and message are always set. Empty or blank text falls back to "sk-meter: alert", and the test covers it. Delivery, retries, the https-only check and redirect: "manual" are unchanged.
Org rules and security. I found no secrets in the diff. notify.test.js is a new file relative to main (it comes from #103's commit), and this repo has no test-diff guard workflow (.github/workflows: ci, deploy, visual-diff, visual-diff-comment). The bearer token still goes only over https.
Not checked: the ops Worker's POST /alert contract in setkernel/hq workers/ops, meaning whether it accepts { title, message } and ignores the extra fields. I have no access to that repo in this run.
Blockers
None.
Nits (non-blocking)
- The real subject repeats the name and keeps the Markdown.
payload({ text: "🔴 **sk-meter: REFUSE** (95%): …" }).titlegivessk-meter: 🔴 **sk-meter: REFUSE** (95%): …, becausemessage()already starts with**sk-meter: …**. An email subject doesn't render**. Consider removing*fromfirst, or not adding the prefix when the line already names the source. notify.js:28-31adds the Access headers whenever bothALERT_ACCESS_CLIENT_*are set. Until the secrets are deleted, as the README asks, thesk-meter-alertsAccess client secret goes to the ops Worker with every alert. That receiver is ours and the connection is https, but the steps on hq#1449 could delete the two secrets at the same moment the URL changes..slice(0, 200)counts UTF-16 code units, so it can split a surrogate pair, for example an emoji at character 199. This is cosmetic.
lyra-setkernel
left a comment
There was a problem hiding this comment.
Reviewed head 7ffd866.
No merge blockers found. Details below.
What I checked and how
- Checks:
validateandvisual-diffboth finished with success on this head. - Acceptance criteria: the PR closes no issue. It only refs setkernel/hq#1449 and #1247, so there was no
- [ ]list to check. I checked the change against what the PR body says it does:workers/meter/src/notify.js:payload()keeps{ source, text, mention }and addstitle(the source plus the first non-blank line, trimmed, at most 200 characters) andmessage(the whole text).notify()sendsJSON.stringify(payload(alert)). Nothing else innotify()changed: the https-only check, the bearer token,redirect: "manual"and treating non-2xx as a failure are all still there.- The only caller is
workers/meter/src/index.js:39. It gets{ text, mention }frommessage()(src/meter.js:175), andtextis always a string there, somessageis never missing from the body. node --test workers/meter/test/*.test.json this head gave 82 tests, 82 pass, 0 fail. That includes the newpayloadtest and the updated contract test.- The README and
wrangler.tomlcomments match the code: the new URL and token, and the two Access values marked unused.
- Stacking and scope: the base is
main, and main squash-merges (the last 10 commits onorigin/mainare squashes). So this head can only merge as a whole, which includes #103's commitac8e3e5: themeterjob in.github/workflows/deploy.yml,package.json,AGENTS.mdand all ofworkers/meter/. Because an approval of this head counts for all of it, I read that commit too:- The
meterjob usesid-token: writewithcontents: readonly. It runsnpm ci --ignore-scriptsand pins its actions by SHA. wrangler runs underenv -i, which passes onlyCLOUDFLARE_API_TOKEN, so the page's two Worker secrets and the OIDC request variables don't reach it. A failed deploy fails the step. A version that isn't live gets rolled back to the recorded one, and the job then fails.reportnowneedsthe job and names its result in the issue it opens. - I read
src/index.js,meter.js,usage.js,flags.jsandwindows.jsand found no blocker. I only skimmedtools/*.mjs.node --checkon them passes in CI'svalidate. - If this PR merges before #103, sk-meter's deploy job ships under this PR's R1 label. That is fine as long as #103 lands first, as the body says.
- The
- Secrets: I found none in the diff. The KV namespace id, the Infisical identity id and the
workers.devURL are identifiers, not credentials. - Prompt injection: the PR text and code contain nothing that tries to steer the review.
- Not checked:
- That the ops Worker's
POST /alertin setkernel/hq accepts this body (titleandmessage, plus the three extra fields) with this bearer token. I had no hq checkout in this run. - Whether hq still deploys a Worker named
sk-meter, which would compete with this repo'smeterjob. - The Infisical and setting changes, which are Raghav's steps on hq#1449.
- That the ops Worker's
Nits (non-blocking)
- The real email subject repeats the name and keeps the Markdown.
message()starts every alert with something like🔴 **sk-meter: REFUSE** (95%): ….payload()addssk-meter:in front, so a real REFUSE alert gets the subjectsk-meter: 🔴 **sk-meter: REFUSE** (95%): non-essential requests are refused until the window resets.(I ranpayload()on that text to confirm.) The new test usesDEGRADE: queues_ops 85%, whichmessage()never produces, so it doesn't show this. Consider removing**from the subject and leaving out the prefix when the first line already names the source. - The Access headers outlive the old route. If
ALERT_ACCESS_CLIENT_IDandALERT_ACCESS_CLIENT_SECRETare still set when the URL changes,notify()keeps sendingcf-access-client-secretto the ops Worker, which has no use for it. The README says to delete them. Putting that deletion in the same hq#1449 step as the URL change, or ignoring them when the URL isn't behind Access, would stop the secret going somewhere that doesn't need it. - Markdown in a plain email.
messagecarries Markdown (**…**, backticks around script names) that was written for Buzz. Whether that reads well depends on how the ops Worker renders the email body.
The meter step records rollback=not-deployed|nothing-live|done|failed through an EXIT trap, and report words sk-meter's outcome on its own line and adds owner when the rollback failed or nothing was live to roll back to. Before, report saw the page's skipped:skipped and said nothing was rolled back (review on #103). Also: AGENTS.md says the identity is bound to the file, and two stale comments in the meter tools. Refs setkernel/hq#1449 — Claude Lead Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3805bcd
vega-setkernel
left a comment
There was a problem hiding this comment.
Reviewed head 3805bcd.
What I checked
What changed since the last reviews. lyra-setkernel and I both reviewed 7ffd866, and both reviews have been dismissed. The head is now 3805bcd, a merge of ff1841f (#103's rollback-report fix) into this branch. git diff 7ffd866 3805bcd touches only .github/workflows/deploy.yml, AGENTS.md, workers/meter/tools/baseline.mjs and workers/meter/tools/drill.mjs. Nothing changed under workers/meter/src or workers/meter/test (git diff 7ffd866 3805bcd -- workers/meter/src workers/meter/test is empty). So this PR's own commit is the one I reviewed before, and in this round I read all of ff1841f.
Checks. validate and visual-diff both finished with success on this head.
Acceptance criteria. The PR closes no issue (closing_issues is empty; the body says only Refs setkernel/hq#1449), so there are no - [ ] items to check. I checked the change against what the body says it does. title and message are still built in workers/meter/src/notify.js:12-16, and the old fields are kept.
Tests. node --test workers/meter/test/*.test.js on this head: tests 82, pass 82, fail 0. deploy.yml parses as YAML (yaml.safe_load). actionlint isn't installed here, so I didn't run it.
Correctness of ff1841f (the new rollback output). I took the meter step's run: script out of deploy.yml and ran it under bash -e, the way the runner does. I swapped in a stub for npx wrangler and changed nothing else. Results:
| scenario | exit | GITHUB_OUTPUT |
|---|---|---|
| deploy OK, new version serves all traffic | 0 | rollback= |
wrangler deploy fails |
1 | rollback=not-deployed |
| nothing live before, new version not live | 1 | rollback=nothing-live |
| rolled back, old version serves all traffic | 1 | rollback=done |
wrangler rollback itself fails |
1 | rollback=failed |
| rollback runs but the old version doesn't come back | 1 | rollback=failed |
Every state matches the comment at deploy.yml:212-216. In report, METER_ROLLBACK comes in through env:, not through ${{ }} inside the script, so there's no injection path. Both nothing-live and failed set more='["owner"]', which matches the AGENTS.md line. When meter was skipped or passed, the output is empty, so meter_text is empty and the page's text is unchanged.
Scope and risk. Against main, the head still includes all of #103 (ac8e3e5 and now ff1841f), including .github/workflows/deploy.yml, a workflow path. The body's "R1" covers only the last commit. If #103 hasn't merged when this merges, the workflow change ships under this PR's label. Raghav opened this PR, and the body already says it's stacked, so I don't count this as a blocker. #103 still needs its own reviews.
Secrets. None in the diff. The Access client values are ${{ secrets.* }} references.
Prompt injection. Nothing in the PR text or code tries to steer the review.
Not checked. I had no setkernel/hq checkout, so I couldn't confirm that the ops Worker's POST /alert accepts this body and token. The Infisical setting changes on hq#1449 are Raghav's steps. I didn't run a live wrangler deploy or rollback; the table above uses a stub.
Blockers
None.
Nits (non-blocking)
- When
wrangler deployments statusfails before the deploy,beforeis empty, and the step reportsnothing-livewith "no version was live before" (in my stub run, a status failure gaverollback=nothing-live). The real cause is that the version couldn't be read. The issue still getsowner, so only the wording is wrong. - The
nothing-liveissue text ends "sk-meter may be running no version at all". Since the deploy just succeeded, some version is serving: just not the one taggeddeploy <sha>. "sk-meter may be on an unknown version" would be closer. - If the job hits
timeout-minutes: 10or is cancelled, the EXIT trap may not run. ThenMETER_ROLLBACKis empty, the issue has no sk-meter line, andmeter cancelledin the jobs list is the only hint. - My earlier nits on
7ffd866still apply, and neither changed. The email subject repeatssk-meter:and keeps the**(payload({ text: "🔴 **sk-meter: REFUSE** …" }).title). And the Access headers keep going to the ops Worker untilALERT_ACCESS_CLIENT_*are deleted.
lyra-setkernel
left a comment
There was a problem hiding this comment.
Reviewed head 3805bcd.
No merge blockers found. This head adds two commits to 7ffd866, which vega-setkernel and I reviewed earlier: ff1841f (#103's fix for the rollback report) and the merge commit 3805bcd. git diff 7ffd866 HEAD touches only .github/workflows/deploy.yml, AGENTS.md, workers/meter/tools/baseline.mjs and workers/meter/tools/drill.mjs. This PR's own change (workers/meter/src/notify.js, its test, the README and wrangler.toml) is the same as before. Main squash-merges, so approving this head approves #103's content too, and I reviewed that delta in full.
What I checked and how
- Checks:
validateandvisual-diffboth finished with success on this head. - Acceptance criteria: the PR closes no issue (
closing_issuesis empty; the body only refs setkernel/hq#1449 and #1247), so there was no- [ ]list to check. My earlier review covers the{ title, message }body against what the PR description says it does, and that code hasn't changed. - Tests:
node --test workers/meter/test/*.test.json this head gave 82 tests, 82 pass, 0 fail.node --checkpasses onnotify.js,tools/baseline.mjs,tools/drill.mjsandtools/schema.mjs. I pulled therun:scripts of themeterandreportjobs out with a YAML parser, andbash -npasses on both. I did not run the wholenpm run validate: npm and wrangler aren't installed in this sandbox. CI'svalidateran it and passed. - Correctness of the new
meterstep (deploy.yml, stepid: meter): the workflow sets noshell:, so the step runs underbash -e. I traced each way the script can end:wrangler deployfails.stateis stillnot-deployed, and the EXIT trap writesrollback=not-deployed.- After the deploy,
now=$(live) || now=andmessage=$(...) || message=are both guarded, so neither can end the script with the wrong state. - The run's version is live:
state=andexit 0write an emptyrollback=, soreportadds no sk-meter line. - Nothing was live before: the output is
nothing-live, andreportaddsowner. wrangler rollbackfails:-eexits withstate=failed, andreportaddsowner.- The rollback runs but the old version doesn't come back:
statestaysfailed. The$(live)inside theiftest can't end the script under-e. - The rollback takes: the output is
done, with noowner.
- Correctness of
report:METER_ROLLBACKcomes in throughenv:and is only matched in acase, so it is never pasted into the script.morestarts as'[]', and eithercaseonly ever sets it to'["owner"]', so the page's outcome and sk-meter's outcome can't fight over it. When the meter job never reached its step (it was skipped after a failedcheck, or an earlier step failed), the output is empty,meter_textis empty, and the jobs line still showsmeter <result>. The body still goes to the API throughjq --arg. - Scope: the new files are #103's, and the PR says it is stacked on #103.
AGENTS.mdnow says what the workflow does: the identity is bound to the workflow file, and themeterjob is the third job that asks for it. The two tools comments only drop stale references (make drill, abaselineworkflow). - Security (the head now changes a workflow): the job permissions are the same as before (
contents: read,id-token: write). The newrollbackoutput only ever holds one of four fixed words, or nothing, so no secret can leak through it.CLOUDFLARE_API_TOKENstill reaches wrangler only throughenv -i. Thereportjob'sissues: writeis unchanged. - Secrets: none in the delta.
- Prompt injection: I found no text in the PR, its comments or the code that tries to steer this review.
- Not checked: the ops Worker's
POST /alertcontract in setkernel/hq (I had no hq checkout), how wrangler behaves for real against Cloudflare (no network or token here), and the Infisical changes for hq#1449, which are Raghav's.
Blockers
None.
Nits (non-blocking)
- The
not-deployedwording may claim too much.wrangler deployuploads the script and then updates the cron triggers in a separate call. If only the trigger update fails, the command exits non-zero while the new code may already be live. The issue would then say "the version live before still serves" when it might not, and it would not addowner. I have not confirmed this against wrangler's source in this run. Something like "the deploy command failed; checkwrangler deployments status" would be safe whatever the cause. - The
failedtext names only one cause. It says "this run's version was not serving all traffic", but the state is alsofailedwhen the check after the deploy couldn't read the status at all (bothliveandversions viewfail, and the rollback then fails). "The check after the deploy did not see this run's version serving all traffic" covers both cases. - The risk class in the body is out of date. The body says R1, but this head now carries a change to
.github/workflows/deploy.yml(from #103), and the org rules put workflow changes in R3. Raghav opened the PR, so this changes no approval step. It only matters for how the PR is labelled. - My two nits from the earlier review still apply. The email subject repeats
sk-meter:and keeps the**Markdown, and the Access headers keep going to the new URL until the twoALERT_ACCESS_CLIENT_*secrets are deleted.
# Conflicts: # workers/meter/README.md # workers/meter/src/notify.js # workers/meter/test/notify.test.js # workers/meter/wrangler.toml
lyra-setkernel
left a comment
There was a problem hiding this comment.
Reviewed head aa73933.
No merge blockers found.
What changed since my last review
I approved 3805bcd. The head is now aa73933, which merges main into the branch after #103 was squash-merged as 4a2859d. git diff 3805bcd HEAD is empty, so the tree is exactly the one vega-setkernel and I reviewed last round. Against main, the diff is now only this PR's own change: workers/meter/src/notify.js, workers/meter/test/notify.test.js, workers/meter/README.md and workers/meter/wrangler.toml (git diff --stat origin/main...HEAD: 4 files, +39 −18). The workflow and #103 content that made my last nit about the risk class relevant is gone from this diff, so the body's R1 now fits.
What I checked and how
- Checks: all three finished on this head.
validatepassed,visual-diffpassed andgatefinishedneutral. None failed and none are still running. - Acceptance criteria: the PR closes no issue.
closing_issuesis empty, and the body only refs setkernel/hq#1449 and #1247, so there's no- [ ]list to check. I checked the change against what the body says it does:payload()innotify.jskeeps{ source, text, mention }and addstitle(the source plus the first non-blank line, trimmed, at most 200 characters) andmessage(the whole text).notify()sendsJSON.stringify(payload(alert)). Nothing else innotify()changed: the https-only check, the bearer token,redirect: "manual"and treating non-2xx as a failure are all still there.- The README and
wrangler.tomlcomments match the code.
- Tests:
node --test workers/meter/test/*.test.json this head gavetests 82, pass 82, fail 0.node --check workers/meter/src/notify.jspassed. - Callers: the only caller is
workers/meter/src/index.js:39, which passesmessage(report, …), and that always has a stringtext. - Org rules: the test edits are the contract test's name, the expected body and the new import, plus one new test. The repo has no test-diff guard workflow (
.github/workflows: ci, deploy, visual-diff, visual-diff-comment). I found no secrets in the diff. The README names the Infisical location ofOPS_ALERT_TOKEN, not its value. - Prompt injection: I found nothing in the PR text or code that tries to steer the review.
- Not checked:
- Whether the ops Worker's
POST /alertin setkernel/hq accepts this body and bearer token. I had no hq checkout in this run. - The Infisical setting changes on hq#1449, which are Raghav's steps.
- Whether the ops Worker's
Blockers
None.
Nits (non-blocking)
- The real email subject repeats the name and keeps the Markdown.
message()starts each alert with something like🔴 **sk-meter: REFUSE** …, so the subject becomessk-meter: 🔴 **sk-meter: REFUSE** …. The test'sDEGRADE: queues_ops 85%text doesn't show this. - The Access headers outlive the old route.
notify()keeps sendingcf-access-client-secretto the ops Worker until bothALERT_ACCESS_CLIENT_*values are deleted. The cleanest fix is to delete them in the same hq#1449 step that changes the URL.
vega-setkernel
left a comment
There was a problem hiding this comment.
Reviewed head aa73933.
What I checked
What changed since the last approvals. lyra-setkernel and I approved 3805bcd. The head is now aa73933, which merges main back in now that #103 has landed as 4a2859d. The merge base with origin/main is 4a2859d. Both git diff origin/main...HEAD and git diff origin/main HEAD now show only this PR's four files: workers/meter/README.md, workers/meter/src/notify.js, workers/meter/test/notify.test.js and workers/meter/wrangler.toml (39+/18−). git diff 3805bcd HEAD on those four paths is empty, so this PR's own change is byte-for-byte the one already reviewed. #103's workflow change is no longer part of this PR. That settles the stacking and R1-vs-R3 points in the earlier reviews.
Checks. On this head, validate and visual-diff finished with success, and gate finished with neutral. None failed and none are still running.
Acceptance criteria. The PR closes no issue: closing_issues is empty, and the body only says Refs setkernel/hq#1449. So there are no - [ ] items to check. I checked the change against what the body says it does:
payload()atworkers/meter/src/notify.js:12-16keeps{ source, text, mention }and addstitleandmessage.titleis the source plus the first non-blank line, trimmed, at most 200 characters.messageis the whole text.notify()sendsJSON.stringify(payload(alert))at line 38.- Nothing else in
notify()changed: the https-only check, the bearer token,redirect: "manual"and treating non-2xx as a failure are all still there. - The only caller is
workers/meter/src/index.js:39. It passesmessage(report, …), which always returns a stringtext.
Tests. On this head, node --test workers/meter/test/*.test.js gave tests 82, pass 82, fail 0, and node --check workers/meter/src/notify.js passed. npm isn't installed in this sandbox, so I ran those commands directly instead of npm run validate. CI's validate passed.
Scope. No issue names the files, so I compared against the PR body. It lists exactly these four files, and the diff touches nothing else.
Org rules. I found no secrets in the diff. The workers.dev URL and the Infisical path are locations, not credentials. One existing test assertion changed: the contract test now expects the two new fields. This repo has no test-diff guard: there is no test-diff or allow-test-changes in .github/, and the workflows are ci, deploy, visual-diff and visual-diff-comment. So no allow-test-changes: line is needed.
Prompt injection. Nothing in the PR text or code tries to steer the review.
Not checked. I had no setkernel/hq checkout, so I couldn't confirm that the ops Worker's POST /alert accepts this body and bearer token. The Infisical setting changes on hq#1449 are Raghav's steps, and I didn't check them either.
Blockers
None.
Nits (non-blocking, all from earlier rounds and unchanged)
- The email subject repeats the name and keeps the Markdown.
message()starts each alert with something like🔴 **sk-meter: REFUSE** …, so the subject comes out assk-meter: 🔴 **sk-meter: REFUSE** …. notify.jsstill sends the Access headers whenever bothALERT_ACCESS_CLIENT_*are set. If they aren't deleted when the URL changes, the Access client secret goes to the ops Worker, which doesn't need it..slice(0, 200)counts UTF-16 code units, so it can split an emoji at the cut. This is cosmetic.
Refs setkernel/hq#1449 (plan v6 F3, step 2) and setkernel/hq#1247 E3b ("Alerts are email only, through the ops Worker relay"). #103 is merged and main is merged in, so the diff is only this change.
Why
sk-meter's alerts still POST to
https://alerts.setkernel.net/meter. That is the buzz-alerts bridge on sk-obs01, which hq#1261 removed. So the billing guard's alerts have nowhere to go. It retries every hour, and its KV flags still act. The hq#1449 plan says email by default.What changes
src/notify.js: the body also carriestitle(the source and the alert's first line, at most 200 characters) andmessage(the whole text). That is the shape the ops Worker'sPOST /alertturns into one email. The old{ source, text, mention }fields stay, so the code works with either receiver and needs no coordination with the settings change.ALERT_WEBHOOK_URLandALERT_WEBHOOK_TOKEN.ALERT_ACCESS_CLIENT_ID/SECRETare now unused.What merging does
It deploys sk-meter with the new body (deploy.yml's
meterjob, live since #103). Alerts move to email only once Raghav changes the two Worker settings in Infisicalsk-ctl-workers: the URL becomes the ops Worker's/alert, and the token becomes a copy ofOPS_ALERT_TOKEN. The exact steps are on hq#1449.Checks
npm run validatelocally: passed.test:meterran 82 tests, 82 passing.Risk class
R1: the alert body of one Worker, compatible with both receivers.
— Claude Lead
🤖 Generated with Claude Code