diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 8c48f11..f3b2f04 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -1,48 +1,79 @@ name: Deploy +# The org's deploy-on-merge contract (setkernel/hq#801, setkernel-live#83), from setkernel/hq +# templates/repo/.github/workflows/deploy.yml at 6e8fdbd; setkernel/hq tools/check-deploy-contract.py +# checks it each week. Every push to main deploys main's head (docs-only pushes too), the smoke check +# runs against what was deployed, a failed smoke rolls back to the version that was live before and +# smokes that, and a failed run opens an agent-ready issue with the run's URL. +# - One concurrency group, never cancelled: a newer push replaces the pending run, so the newest main +# always deploys and no deploy stops half way. +# - Only `deploy` and `rollback` can get the deploy token (id-token: write), and they run no app code: +# install scripts are off and wrangler runs without the OIDC request variables. The checks and the +# smoke run in jobs that have no token; only `report` can write issues. +# - Rollback: `deploy` records the live version, and the commit its "deploy " message names, before +# it deploys. If the smoke fails, `rollback` puts that version back, and `smoke-again` smokes it with +# --expect that commit. If that fails too, or the rollback fails, or no version was live, nothing rolls +# back again and the issue gets `owner`. main is unchanged either way; the next push deploys again. +# Where this file differs from the template, for this repo: +# - `check` runs `npm run validate` (what CI runs) on .nvmrc's Node; the repo has no Makefile. +# - The identity and folder are this repo's own (setkernel-live-deploy, the project's root folder), not +# vars.DEPLOY_INFISICAL_IDENTITY and /deploy. The same identity serves `rollback`. +# - The deploy sets VERSION to the commit (scripts/smoke.mjs waits for /api/state's `version`) and puts +# the two Worker secrets from Infisical into the same version, so each deploy is one version. +# - The smoke may wait 10 minutes for the Durable Object to run the new code, so its jobs get 15, and +# scripts/smoke.mjs has no dependencies, so they install nothing. on: - # Deploy on merge (setkernel-live#30, hq#152): every push to main deploys, then a smoke check - # waits for the Durable Object to run the new commit and checks each route; a failed check rolls - # back to the previous version and opens an issue for the agents. Docs-only pushes don't deploy. push: branches: [main] - paths-ignore: ["**.md", "docs/**"] - workflow_dispatch: {} + workflow_dispatch: {} # re-deploys main's head + +permissions: {} concurrency: - group: deploy-production + group: deploy-${{ github.event.repository.name }} cancel-in-progress: false -permissions: - contents: read - jobs: - deploy: + # The same checks as ci.yml, on this exact commit, before anything is deployed. + check: + if: github.ref == 'refs/heads/main' runs-on: setkernel-linux - timeout-minutes: 25 # the smoke check may wait up to 10 minutes for the Durable Object + timeout-minutes: 20 permissions: contents: read - id-token: write # OIDC login to Infisical - issues: write # open an issue when the smoke check fails steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 with: node-version-file: .nvmrc cache: npm - - - run: npm ci - + - run: npm ci --no-audit --no-fund - name: Re-gate before deploy run: npm run validate - # The deploy token lives in Infisical (project setkernel-live, prod), not in GitHub - # (setkernel decision 0003 + amendments). Fetched last, so no npm script runs with it. The - # org-runner Access token gets past Cloudflare Access; the identity only accepts this - # workflow on main of this repo. + deploy: + needs: check + runs-on: setkernel-linux + timeout-minutes: 10 + permissions: + contents: read + id-token: write # OIDC login to Infisical + env: + WRANGLER_SEND_METRICS: "false" + outputs: + previous-version: ${{ steps.live.outputs.version }} + previous-sha: ${{ steps.live.outputs.sha }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + # wrangler from the lockfile; no install script runs in the job that can get the token + - run: npm ci --ignore-scripts --no-audit --no-fund + # The deploy token lives in Infisical (project setkernel-live, prod), not in GitHub (decision 0003 + # + amendments). The org-runner Access token gets past Cloudflare Access; the identity only accepts + # this workflow on main of this repo. - name: Fetch the deploy token from Infisical uses: Infisical/secrets-action@d2e351f16c6ca20d17c85e6c992e04bdeb64e87d # v1.0.18 with: @@ -50,54 +81,176 @@ jobs: identity-id: 37b06681-c931-4bad-b3cf-4e76bfb633dc # setkernel-live-deploy oidc-audience: https://github.com/setkernel domain: https://secrets-api.setkernel.net - project-slug: setkernel-live + project-slug: ${{ github.event.repository.name }} env-slug: prod secret-path: / export-type: env extra-headers: | CF-Access-Client-Id: ${{ secrets.CF_ACCESS_CLIENT_ID }} CF-Access-Client-Secret: ${{ secrets.CF_ACCESS_CLIENT_SECRET }} + # What a failed smoke rolls back to: the version serving all traffic, and the commit its message + # names. A Worker with no deployment yet has nothing to roll back to. + - name: Record the live version + id: live + run: | + unset ACTIONS_ID_TOKEN_REQUEST_URL ACTIONS_ID_TOKEN_REQUEST_TOKEN + if ! npx --no-install wrangler deployments status --json > "$RUNNER_TEMP/live.json"; then + echo "::warning::no live version read, so there is nothing to roll back to" + exit 0 + fi + version=$(jq -r '[.versions[]? | select(.percentage == 100) | .version_id][0] // empty' "$RUNNER_TEMP/live.json") + [ -n "$version" ] || exit 0 + message=$(npx --no-install wrangler versions view "$version" --json | + jq -r '.annotations["workers/message"] // empty') || message= + sha=$(printf '%s\n' "$message" | sed -n 's/^deploy \([0-9a-f]\{40\}\)$/\1/p') + echo "version=${version}" >> "$GITHUB_OUTPUT" + echo "sha=${sha}" >> "$GITHUB_OUTPUT" + # The account comes from the token; wrangler.jsonc carries no account id. VERSION = this commit, so + # /api/state shows which code the Durable Object runs. GITHUB_WEBHOOK_SECRET (/github/webhook) and + # BUZZ_PRIVATE_KEY (the Worker's Buzz identity, setkernel-live#9) come from the same Infisical + # project and go into this version as Worker secrets, read from stdin, never a file; the deploy + # fails if either is missing. + - name: Deploy + id: deploy + run: | + for name in GITHUB_WEBHOOK_SECRET BUZZ_PRIVATE_KEY; do + if [ -z "${!name}" ]; then + echo "::error::${name} is not in Infisical (project setkernel-live, prod, /)" + exit 1 + fi + done + jq -n '{GITHUB_WEBHOOK_SECRET: env.GITHUB_WEBHOOK_SECRET, BUZZ_PRIVATE_KEY: env.BUZZ_PRIVATE_KEY}' | + env -u ACTIONS_ID_TOKEN_REQUEST_URL -u ACTIONS_ID_TOKEN_REQUEST_TOKEN \ + npx --no-install wrangler deploy --message "deploy ${GITHUB_SHA}" \ + --var "VERSION:${GITHUB_SHA}" --secrets-file /dev/stdin - # Account is resolved from the token; wrangler.jsonc carries no account id. - # GITHUB_WEBHOOK_SECRET (same Infisical project) becomes a Worker secret for /github/webhook, - # and BUZZ_PRIVATE_KEY (the Worker's Buzz identity for the "updates stopped" alert, - # setkernel-live#9) another; the deploy fails if either is missing from Infisical. - - uses: cloudflare/wrangler-action@953926a2e2182532811c01a25e53647d93bf07c0 # v4.1.3 - env: - GITHUB_WEBHOOK_SECRET: ${{ env.GITHUB_WEBHOOK_SECRET }} - BUZZ_PRIVATE_KEY: ${{ env.BUZZ_PRIVATE_KEY }} + # The smoke check, without the deploy token: it fails until the Durable Object runs this commit. + # Its output is status codes, key names and the version only (scripts/smoke.mjs), never a body. + smoke: + needs: deploy + runs-on: setkernel-linux + timeout-minutes: 15 + permissions: + contents: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: - apiToken: ${{ env.CLOUDFLARE_API_TOKEN }} - secrets: | - GITHUB_WEBHOOK_SECRET - BUZZ_PRIVATE_KEY - # VERSION = this commit, so /api/state shows which code the Durable Object runs. - command: deploy --var VERSION:${{ github.sha }} - - # Runs without the deploy credentials: they are removed from its environment. + persist-credentials: false - name: Smoke check - id: smoke + run: node scripts/smoke.mjs --expect "${GITHUB_SHA}" + + # A failed smoke puts back the version that was live before this deploy, once. + rollback: + needs: [deploy, smoke] + if: ${{ !cancelled() && needs.smoke.result == 'failure' }} + runs-on: setkernel-linux + timeout-minutes: 10 + permissions: + contents: read + id-token: write # OIDC login to Infisical + env: + WRANGLER_SEND_METRICS: "false" + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - run: npm ci --ignore-scripts --no-audit --no-fund + - name: Fetch the deploy token from Infisical + uses: Infisical/secrets-action@d2e351f16c6ca20d17c85e6c992e04bdeb64e87d # v1.0.18 + with: + method: oidc + identity-id: 37b06681-c931-4bad-b3cf-4e76bfb633dc # setkernel-live-deploy + oidc-audience: https://github.com/setkernel + domain: https://secrets-api.setkernel.net + project-slug: ${{ github.event.repository.name }} + env-slug: prod + secret-path: / + export-type: env + extra-headers: | + CF-Access-Client-Id: ${{ secrets.CF_ACCESS_CLIENT_ID }} + CF-Access-Client-Secret: ${{ secrets.CF_ACCESS_CLIENT_SECRET }} + - name: Roll back + env: + VERSION: ${{ needs.deploy.outputs.previous-version }} run: | - set -o pipefail - env -u CLOUDFLARE_API_TOKEN -u GITHUB_WEBHOOK_SECRET -u BUZZ_PRIVATE_KEY \ - node scripts/smoke.mjs --expect "${{ github.sha }}" 2>&1 | tee smoke.log + if [ -z "$VERSION" ]; then + echo "::error::no version was live before ${GITHUB_SHA}, so there is nothing to roll back to" + exit 1 + fi + env -u ACTIONS_ID_TOKEN_REQUEST_URL -u ACTIONS_ID_TOKEN_REQUEST_TOKEN \ + npx --no-install wrangler rollback "$VERSION" --message "rollback from ${GITHUB_SHA::7}" --yes - - name: Roll back to the previous version - if: failure() && steps.smoke.outcome == 'failure' + # The same smoke check against the version rolled back to, without the token. + smoke-again: + needs: [deploy, rollback] + if: ${{ !cancelled() && needs.rollback.result == 'success' }} + runs-on: setkernel-linux + timeout-minutes: 15 + permissions: + contents: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Smoke check the version rolled back to env: - CLOUDFLARE_API_TOKEN: ${{ env.CLOUDFLARE_API_TOKEN }} - run: npx --no-install wrangler rollback --message "smoke check failed for ${{ github.sha }}" + EXPECT: ${{ needs.deploy.outputs.previous-sha }} + run: | + if [ -z "$EXPECT" ]; then + echo "::error::the version rolled back to names no commit (its message is not 'deploy '), so nothing can check it" + exit 1 + fi + node scripts/smoke.mjs --expect "$EXPECT" - # smoke.log holds status codes, key names and the version only (scripts/smoke.mjs), never a body. + # Any failure above, the checks included, opens one issue for the agents, and for the owner too when + # a failed smoke could not be rolled back or failed again after it. The runner image has curl and jq + # but no gh. + report: + needs: [check, deploy, smoke, rollback, smoke-again] + if: failure() + runs-on: setkernel-linux + timeout-minutes: 5 + permissions: + issues: write + steps: - name: Open an issue for the agents - if: failure() && steps.smoke.outcome == 'failure' env: GH_TOKEN: ${{ github.token }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + CHECK: ${{ needs.check.result }} + DEPLOY: ${{ needs.deploy.result }} + SMOKE: ${{ needs.smoke.result }} + ROLLBACK: ${{ needs.rollback.result }} + SMOKE_AGAIN: ${{ needs.smoke-again.result }} + VERSION: ${{ needs.deploy.outputs.previous-version }} + PREVIOUS: ${{ needs.deploy.outputs.previous-sha }} run: | - { - echo "The deploy of ${{ github.sha }} failed its smoke check and was rolled back to the previous version." - echo; echo "Run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" - echo; echo '```'; tail -n 60 smoke.log; echo '```' - echo; echo "README \"Reading a failed smoke\" says what each line means." - } > body.md - gh issue create --repo "${{ github.repository }}" --title "Deploy smoke check failed for ${GITHUB_SHA::7}" --label agent-ready --body-file body.md + more='[]' + case "${ROLLBACK}:${SMOKE_AGAIN}" in + success:success) + rolled="The smoke failed, so this run rolled back to ${VERSION} (${PREVIOUS}), and the smoke passed on that. main is unchanged: fix it there and the next push deploys again." ;; + success:*) + more='["owner"]' + if [ -z "$PREVIOUS" ]; then + rolled="The smoke failed, so this run rolled back to ${VERSION}, but that version's message names no commit, so no smoke could check it. main is unchanged." + else + rolled="The smoke failed, so this run rolled back to ${VERSION} (${PREVIOUS}), but the smoke failed on that too: the smoke or the environment is wrong, not ${GITHUB_SHA::7}. Nothing was rolled back again. main is unchanged." + fi ;; + failure:*) + more='["owner"]' + rolled="The smoke failed and the rollback failed (the job log says why), so ${GITHUB_SHA::7} may still be live. main is unchanged." ;; + *) + rolled="Nothing was rolled back: no smoke failed after a deploy." ;; + esac + body="The deploy of ${GITHUB_SHA} failed. Jobs: check ${CHECK}, deploy ${DEPLOY}, smoke ${SMOKE}, rollback ${ROLLBACK}, smoke-again ${SMOKE_AGAIN}. + + ${rolled} + + Run: ${RUN_URL} + + README \"Reading a failed smoke\" says what each smoke line means." + jq -n --arg title "Deploy failed for ${GITHUB_SHA::7}" --arg body "$body" --argjson more "$more" \ + '{title: $title, body: $body, labels: ["agent-ready"]} | .labels += $more' | + curl -fsS -X POST "${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/issues" \ + -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: application/vnd.github+json" \ + -H "X-GitHub-Api-Version: 2022-11-28" --data-binary @- > /dev/null diff --git a/AGENTS.md b/AGENTS.md index 99f79da..acb897a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,10 +26,15 @@ Public repo: everything here is visible to anyone. ``` ## Deploy -`.github/workflows/deploy.yml`, runs on every push to `main` (not for `*.md` or `docs/` only) and by hand -(workflow_dispatch); a failed smoke check rolls back and opens an `agent-ready` issue. It reads -`CLOUDFLARE_API_TOKEN` from Infisical (project `setkernel-live`, prod) through the OIDC identity -`setkernel-live-deploy`, which accepts only that workflow on `main`. The token itself is +`.github/workflows/deploy.yml`, runs on every push to `main` (docs-only ones too) and by hand +(workflow_dispatch). It is setkernel/hq's deploy contract template (`templates/repo/.github/workflows/deploy.yml` +at 6e8fdbd) and must keep passing hq `tools/check-deploy-contract.py --repo setkernel-live`: a failed +smoke rolls back to the version live before, smokes that, and opens an `agent-ready` issue (`owner` too +if the rollback or its smoke fails). Where it differs from the template, the header says why (`npm run +validate`, the identity and folder, `VERSION` and the Worker secrets in the deploy's one version). It +reads `CLOUDFLARE_API_TOKEN`, `GITHUB_WEBHOOK_SECRET` and `BUZZ_PRIVATE_KEY` from Infisical (project +`setkernel-live`, prod, `/`) through the OIDC identity `setkernel-live-deploy`, which accepts only that +workflow on `main`, in the `deploy` and `rollback` jobs only. The token itself is `setkernel-live-deploy` in setkernel-cloudflare `stacks/services/api_tokens.tf`. Never deploy from a laptop. diff --git a/README.md b/README.md index c4fc8f5..8854fdf 100644 --- a/README.md +++ b/README.md @@ -62,11 +62,16 @@ wall service ── filtered events ──▶ /ingest (Access: service token on ## Deploy -The Deploy workflow (`.github/workflows/deploy.yml`) runs on every push to `main`, so a merge -deploys. It runs `npm run validate`, deploys, runs the smoke check and, if that fails, rolls back. -Pushes that change only `*.md` files or `docs/` don't deploy. Runs never overlap; a waiting run is replaced by a newer one. To redeploy -`main` without a change, start the workflow by hand (workflow_dispatch). Never deploy from a laptop. `/api/state` carries `version`, the `VERSION` var from `wrangler.jsonc`: -`dev` unless the deploy sets the commit with `wrangler deploy --var VERSION:`. +The Deploy workflow (`.github/workflows/deploy.yml`) runs on every push to `main`, docs-only ones +too, so a merge deploys. It follows the org's deploy contract (setkernel/hq +`templates/repo/.github/workflows/deploy.yml`, checked by hq `tools/check-deploy-contract.py`): jobs +`check` (`npm run validate`), `deploy`, `smoke`, `rollback`, `smoke-again` and `report`. Only +`deploy` and `rollback` can get the deploy token; the checks and the smoke run without it. Runs +never overlap (`concurrency: deploy-setkernel-live`); a waiting run is replaced by a newer one. To +redeploy `main` without a change, start the workflow by hand (workflow_dispatch). Never deploy from +a laptop. Each deploy is one Worker version with the message `deploy `, carrying the two +Worker secrets from Infisical. `/api/state` carries `version`, the `VERSION` var from +`wrangler.jsonc`: `dev` unless the deploy sets the commit with `wrangler deploy --var VERSION:`. `node scripts/smoke.mjs --expect ` first polls `/api/state` until its `version` is that commit. The field comes from the Durable Object's own code, and for a few minutes after a deploy @@ -78,10 +83,15 @@ each route once: `/` 200, `/api/state` 200 with every key the page uses, `/api/r status codes, that host, key names and the version, never a response body, and exits 1 on failure (2 on bad arguments). -If the smoke fails, the workflow runs `wrangler rollback` to the previous version (message -"smoke check failed for ") and opens an issue titled "Deploy smoke check failed for -", labelled `agent-ready`, with the run link and the last 60 lines of the smoke -output. The issue exists because the rollback happened: fix the change and merge it again. +Before it deploys, `deploy` records the version serving all traffic and the commit its +`deploy ` message names. If the smoke fails, `rollback` runs `wrangler rollback` to that +version (message "rollback from "), and `smoke-again` runs the smoke with +`--expect` that commit. Any failed job, the checks included, opens one issue titled "Deploy failed +for ", labelled `agent-ready`, with each job's result and the run link; the smoke +output is in the run's log. It also gets `owner` when the rollback failed, no version was live, or +the version rolled back to failed its smoke too (or names no commit, as a version from before +setkernel-live#83 does): then nothing rolls back again. main is unchanged either way: fix the +change and merge it again. ### Reading a failed smoke @@ -94,7 +104,7 @@ Read the output from the top: - `FAIL GET /api/state: 200, version X, want Y`: the version changed after the wait had seen the right one. Either the Durable Object is still switching between old and new code, or someone rolled back or deployed by hand. The Deployments list shows which. A later Deploy run can't - overlap this one (`concurrency: deploy-production`). + overlap this one (`concurrency: deploy-setkernel-live`). - `FAIL GET /api/state: 200, missing keys: …`: the page's data changed shape; `view()` in `public/lib/state.js` and `STATE_KEYS` in `scripts/smoke.mjs` must agree (the worker tests check). - `FAIL GET /usage/data: 200, want …`: the route answered without a login, so its Access app is