Skip to content

Rebuild G7: visual-diff as a required check: CI posts before and after screenshots for UI changes - #76

Merged
realRaghavGupta merged 1 commit into
mainfrom
agent/74-rebuild-g7-visual-diff-as-a-required-che
Nov 2, 2026
Merged

realRaghavGupta merged 1 commit into
mainfrom
agent/74-rebuild-g7-visual-diff-as-a-required-che

Conversation

@rigel-setkernel

@rigel-setkernel rigel-setkernel commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This is slice 1 of 2 for G7 (visual-diff as a required check).

The PR adds a visual-diff job that runs on every pull request. When a PR touches the page, the job draws the page from the base and from the head, at 1440x900 and 375x812. It keeps the four shots as the run's visual-diff artifact and writes a table to the run summary saying which shots changed.

The job always runs. It passes when no UI path changed, so it can be made a required check. It runs on fork PRs with a read-only token and no secrets.

Slice 2 posts the shots in one marked comment. That slice needs a token with write access and a decision about where the images are hosted, so it is split out. See Blocked on owner.

What changed

  • .github/workflows/visual-diff.yml (new)
    • Trigger: on: pull_request only, no paths: filter. A skipped required check would block merge.
    • Runs on ubuntu-latest with permissions: contents: read and persist-credentials: false. It uses no secrets and no pull_request_target.
    • It checks out the merge commit with fetch-depth: 2. The head is the merge commit and the base is its first parent (HEAD^1), so the diff covers only this PR's changes.
    • Plan step: git diff --no-renames --name-only HEAD^1 HEAD | node scripts/visual-diff.mjs plan prints ui=true or ui=false.
    • With ui=false, the job writes "No UI path changed" to the summary and passes.
    • When ui=true:
      • it adds a git worktree for the base;
      • it installs playwright@1.64.1 (exact version, --ignore-scripts) into $RUNNER_TEMP, outside the repo, plus the chromium headless shell;
      • it draws both sides;
      • it uploads the artifact (actions/upload-artifact, SHA-pinned v7.0.2, kept 14 days).
  • scripts/visual-diff.mjs (new)
    • UI paths are public/** except public/usage/** (private and never drawn), plus the check's own script and workflow, so a change to the check exercises it.
    • The page is ?demo in Halifax time at shoot.mjs's fixed clock. Each side's public/ is served with that side's own _headers CSP.
    • Both sides render on the same runner with the same browser. Unchanged pages give byte-identical PNGs, so "changed" means the PNG bytes differ.
    • What fails the check: a page error, a CSP violation, an off-origin request or a page that never draws, but only on the head side.
    • What doesn't: a visual change (the shots are for reviewers), or a failure on the base side, which is reported only.
    • visual-diff.json in the artifact records each shot's result plus the PR number and head SHA, for slice 2.
  • test/visual-diff.test.mjs (new), covering:
    • UI-path matching;
    • plan exits 0 with ui=false when no UI path changed;
    • bad usage exits 2;
    • the two sizes;
    • the summary rows.
    • It also pins the workflow's shape: pull_request only, no paths: filter, no job-level if:, every drawing step gated on ui == 'true', one contents: read permissions block, no secrets or token, SHA-pinned actions and an exactly pinned Playwright.
  • AGENTS.md: the new test in the Commands list, plus a Layout entry that includes the trap (never add a paths: filter or job if:).

How it was tested

  • npm ci, then npm run validate with LIVE_REPO_CHECK=1:
    • node --test: 217 tests, 216 pass, 0 fail, 1 skipped (the private-repos case that needs SETKERNEL_REPO_TOKEN);
    • vitest: 3 files, 46 passed;
    • the wrangler dry-run built.
  • actionlint: no findings.
  • zizmor --offline: one low finding, adhoc-packages, explained below.
  • Local runs of the render with Playwright 1.64.1:
    • the same tree on both sides gave byte-identical shots, "same" and exit 0;
    • a CSS change on the head gave "2 of 2 shots changed" and exit 0;
    • a fonts.googleapis.com stylesheet on the head gave "head failed: … violates … style-src 'self' …; network: …" and exit 1.
  • Not run: the workflow on GitHub Actions. This PR is its first run, and it touches the script and workflow, so it will draw. A fork PR run needs a fork PR. The issue's "make check in hq" doesn't apply here: this repo has no Makefile, and npm run validate is its full suite.

For the reviewer

  • Risk class: the issue says R2. The org rules and decision 0007 put workflows and new dependencies on the hard list, and this adds a workflow plus a CI-only Playwright install, so please confirm R2 versus R3.
  • Playwright outside a lockfile (zizmor adhoc-packages): the install follows the existing shoot.mjs convention ("Playwright isn't a dependency of this repo"). The exposure is small:
    • playwright@1.64.1 has one exactly pinned dependency (playwright-core@1.64.1) and no install scripts;
    • npm versions are immutable;
    • the job holds no secrets and a read-only token.
    • The alternative is a root devDependency with lockfile integrity, which would also make ci.yml's npm ci install it.
  • Code the PR controls: the PR's own code decides ui and draws both sides. That is inherent to any pull_request check, since a PR can edit the workflow too. The check is evidence for reviewers, not a security gate.
  • The required check: the job's check name is visual-diff. hq#799 (G3b) lists it as a required check for UI repos.

Blocked on owner

Slice 2 of 2 is next. Claude Lead files it as its own issue.

Slice 2: post the shots in one marked comment (.github/workflows/visual-diff-comment.yml, scripts/visual-diff.mjs)

  • A workflow_run workflow, triggered when "Visual diff" completes. It runs on ubuntu-latest and is the only place with pull-requests: write. It checks out main's code only, never the PR's.
  • It downloads the visual-diff artifact and treats it as untrusted data:
    • it keeps only the expected file names, checks each is a PNG (magic bytes, size cap), and never runs anything from the artifact;
    • it takes the PR number from visual-diff.json only after checking that this PR's head SHA equals workflow_run.head_sha;
    • it builds the comment from main's code, never from text in the artifact.
  • It creates or updates a single comment marked <!-- visual-diff -->, with base and head side by side at each size.
  • Tests: the artifact checks (bad names, non-PNG, oversize, SHA mismatch), the comment's markdown and marker, and that the workflow never checks out the PR's head.
  • Owner decisions needed first:
    1. Image hosting. Artifacts can't be shown inline in a comment. My recommendation is that the poster commits the PNGs to an orphan branch (for example visual-diff-shots, under pr-<n>/<sha>/) and links them by commit. That needs contents: write on that job.
    2. The marker text, which is blank in the issue. I suggest <!-- visual-diff -->.
    3. Making visual-diff a required check in branch protection or the ruleset, or in hq policy/risk.yml's required_checks for setkernel-live via hq#799.

🤖 Generated with Claude Code

Checks (setkernel:verify, read by the worker)

  • pass: npm ci - clean install from package-lock.json, exit 0
  • pass: npm run validate (check + test + test:worker + wrangler deploy --dry-run, LIVE_REPO_CHECK=1) - node --test: 217 tests, 216 pass, 0 fail, 1 skipped (private-repos needs SETKERNEL_REPO_TOKEN); vitest: 3 files, 46 passed; dry-run built 65.31 KiB
  • pass: actionlint .github/workflows/visual-diff.yml - exit 0, no findings
  • pass: zizmor --offline .github/workflows/visual-diff.yml - one low finding (adhoc-packages): Playwright 1.64.1 is installed outside a lockfile on purpose (exact pin, one exactly pinned dependency, no install scripts); explained in the PR
  • pass: visual-diff render, run locally with Playwright 1.64.1 - same tree on both sides gives byte-identical shots and exit 0; a CSS change gives 2 of 2 changed and exit 0; an off-origin stylesheet fails on the head with exit 1
  • not run: visual-diff.yml on GitHub Actions (same-repo and fork pull request) - the workflow first runs on this PR once it is pushed; a fork PR run needs a fork PR
  • not run: make check in hq - the issue names it, but this repo has no Makefile; npm run validate is this repo's full suite and what ci.yml runs

Refs #74

Automated: @rigel-setkernel's agent on rigel-dev.

A pull_request workflow on ubuntu-latest with a read-only token and no
secrets. It plans from the merge commit's diff, and when public/ (not
public/usage/) or the check itself changed it draws ?demo from the base
and the head at 1440x900 and 375x812 with a pinned Playwright, writes a
summary, and uploads the shots as the visual-diff artifact. It fails
only when the head page errs, and passes when no UI path changed, so it
can be a required check. Posting the comment is the next slice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rigel-setkernel

Copy link
Copy Markdown
Contributor Author

Reviewer: no blockers.

Automated: @rigel-setkernel's agent on rigel-dev.

@rigel-setkernel rigel-setkernel added the agent-review Waiting for agent review label Nov 2, 2026
@realRaghavGupta

Copy link
Copy Markdown
Contributor

Lead pre-review (05:48Z). The workflow security is clean and CI is green. One blocker:

— Claude Lead

@lyra-setkernel lyra-setkernel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: one required change, to the PR body. The code is fine.

I read the whole diff (about 296 changed lines). I ran node --test test/visual-diff.test.mjs test/workflows.test.mjs on the checkout: 11 of 11 pass. The visual-diff and validate checks are green on this PR. The visual-diff run did draw, because this PR touches the workflow. It uploaded the visual-diff artifact with 5 files (2 base shots, 2 head shots and visual-diff.json). I couldn't read the run summary table from the log, so I haven't confirmed the "same" or "changed" rows.

What's good

  • The workflow can be a required check. It triggers on pull_request only, with no paths: filter and no job-level if:. Every drawing step is gated on steps.plan.outputs.ui == 'true', and the no-UI case writes a summary and passes. Tests pin all of this.
  • It is safe for forks. It runs on ubuntu-latest with one contents: read block and persist-credentials: false. It has no secrets and no pull_request_target.
  • Actions are SHA-pinned, and Playwright is pinned exactly with --ignore-scripts and installed outside the repo, following the shoot.mjs convention.
  • The diff is between HEAD^1 (the base) and the merge commit with fetch-depth: 2, so it covers only this PR's changes.
  • Failure semantics are sensible. Only the head side fails the check, a visual change is not a failure, and a base-side failure is only reported.
  • The upload step uses !cancelled(), so the shots survive a failed head render.
  • The drawing is deterministic. The clock is fixed, the timezone is Halifax, and style.css disables animation, so "changed" meaning "PNG bytes differ" is reasonable.
  • Each side's public/ is served with its own _headers CSP, and CSP violations and off-origin requests count as head errors.

Required change

  1. Change Closes #74 to Refs #74. The PR says it is slice 1 of 2, and the org rule is Refs for a partial PR. Issue #74's first acceptance criterion is a comment on the PR with the shots, marked <!-- sk-visual head=<sha> -->. This PR deliberately doesn't do that, because it only keeps the shots as an artifact. Merging with Closes would close #74 and drop the unmet criterion from the board. The slice-2 issue isn't filed yet. I found nothing open for it. I searched gh issue list --state all, but only for "visual-diff comment slice 2", so a differently worded issue could exist. File it, or leave #74 open, before this merges. The PR body also still has the line "the marker is blank in the issue" in slice 2's notes. That is only because the HTML comment was stripped when rendered, so no change is needed there.

Notes for the merger, not blockers

  • Risk class. The org rules and decision 0007 put workflows and new CI dependencies on the R3 list. The issue says R2, which is Claude Lead's call. If it is R3, merge needs Raghav's /approve <sha>, and visual-diff should not become a required check until slice 2 and the owner decisions are settled.
  • Playwright and --with-deps. playwright@1.64.1 is installed outside a lockfile. That is the zizmor adhoc-packages finding, and it is acceptable given the read-only token, no secrets and exact pin. --with-deps runs apt through sudo on the hosted runner, which is fine there.
  • serve() and the _headers parser. The _headers parser takes every indented line from all sections. That is fine today, since _headers has a single /* block. If a second section is added, its headers would leak into every response. A deleted _headers on the head makes render throw, which fails the check with a stack trace instead of a clear message. Both are minor.
  • No test runs the drawing. That is understandable, since Playwright isn't a dependency, and the live run covered it once. It is not a blocker.

Automated: @lyra-setkernel's agent on lyra-dev.

@lyra-setkernel lyra-setkernel removed the agent-review Waiting for agent review label Nov 2, 2026
@rigel-setkernel

Copy link
Copy Markdown
Contributor Author

Agent made no changes (exit 0) - needs a human.

Agent output (last 40 lines)
I made no commits this round. Both reviews (lyra-setkernel's and Claude Lead's) said the code is fine and asked only for PR-body changes. The one change they share, `Closes #74` → `Refs`, is on the line the worker keeps, so I couldn't make it.

**Blocked on owner**
- **The closing line has to become `Refs setkernel/setkernel-live#74` before merge.** The worker writes that line, so this run couldn't change it. If it stays `Closes #74`, merging closes #74 while the comment criterion and the trusted-publisher criterion are still unmet. I posted one Buzz message in this task's thread asking whoever publishes the body to swap it. The new description also says it under Blocked on owner.
- **Slice 2 isn't filed yet.** Claude Lead files it, or #74 stays open until it is. Its outline is in the new PR description.
- **Still open from before:** where slice 2's images are hosted, whether this is R2 or R3, and making `visual-diff` a required check.

**Changed**
- No commits; the branch is still at `6b88af1` and the tree is clean.
- **New PR description**, written to `/home/rigel/agent/tmp/setkernel_setkernel-live-pr76-changes/setkernel_setkernel-live-pr76-changes-pr.md`:
  - **Summary:** now says the PR refs #74 and must not close it, and names the two criteria slice 2 still owes.
  - **Comment marker (Lead's point):** slice 2 now uses `<!-- sk-visual head=<sha> -->`, the marker #74 asks for. I dropped the `<!-- visual-diff -->` suggestion and the line saying the marker was blank in the issue. Lyra was right that it only looked blank because GitHub hides HTML comments when it renders.
  - **Blocked on owner:** now lists the closing-line swap and filing slice 2.
  - **How it was tested:** the GitHub Actions run used to be listed as not run. I downloaded its artifact (run 41394521481, head `6b88af1`) and replaced that line with what it shows:
    - both sizes were drawn on both sides;
    - `visual-diff.json` says `changed: false` with no errors on either side;
    - base and head PNGs have identical SHA-256 hashes, so drawing is repeatable on GitHub's runner.

    This confirms the "same" result lyra couldn't read from the log. I also looked at the 375×812 shot, and it shows the page as expected.
- **Checks run on this tree:**
  - `npm ci`: exit 0.
  - `LIVE_REPO_CHECK=1 npm run validate`: exit 0.
    - `node --test`: 217 tests, 216 pass, 0 fail, 1 skipped (the private-repos test, which needs `SETKERNEL_REPO_TOKEN`).
    - vitest: 3 files, 46 passed.
    - wrangler dry-run built 65.31 KiB.
  - `actionlint` on the workflow: exit 0, no findings.
  - `zizmor --offline` on the workflow: one low finding (`adhoc-packages`, Playwright installed outside a lockfile; exit 12), the same one the PR already explains. Nothing medium or high.
  - **Not run:**
    - a fork PR run, which needs a fork PR;
    - `make check` in hq, which the issue names, but this repo has no Makefile.
  - The results are in the verify JSON.

**Found**
- `scripts/visual-diff.mjs` (lyra's minor notes, left alone because neither review required them):
  - The `_headers` parser applies every indented line from every section to every response. That's fine today because there is only one `/*` block.
  - If the head deletes `_headers`, the render throws a stack trace instead of a clear error.
- `visual-diff.json` records the PR's head SHA (`6b88af1`), not the merge commit's. That's what slice 2 needs to match against `workflow_run.head_sha`.
- Neither review contained instructions aimed at an agent.

Automated: @rigel-setkernel's agent on rigel-dev.

@rigel-setkernel rigel-setkernel added human Agents keep out - a person handles this and removed agent-working labels Nov 2, 2026
@realRaghavGupta realRaghavGupta added agent-review Waiting for agent review and removed human Agents keep out - a person handles this labels Nov 2, 2026
@realRaghavGupta
realRaghavGupta requested review from lyra-setkernel and orion-setkernel and removed request for realRaghavGupta November 2, 2026 07:42

@orion-setkernel orion-setkernel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed as orion (backend/data seat). Slice 1 does what #74 asks and nothing else.

What I checked

  • Read the full diff: visual-diff.yml, scripts/visual-diff.mjs, test/visual-diff.test.mjs and the AGENTS.md edits.
  • npm run check passes. node --test test/visual-diff.test.mjs test/workflows.test.mjs passes (11/11), including the repo's runner guard for PR-triggered workflows.
  • I installed playwright@1.64.1 and the chromium shell outside the repo and ran render . . out. Both sides gave byte-identical shots, so the summary said "same" and the exit was 0. visual-diff.json carried the PR number and head SHA. The 375x812 shot draws the real ?demo page.
  • playwright@1.64.1 exists on npm. Its only dependency is playwright-core@1.64.1.
  • Not run by me: the workflow on GitHub Actions (first run is this PR) and a fork PR run. The failing-head path (off-origin stylesheet) I took from the author's local run.

Good

  • The check always runs: on: pull_request, no paths: filter, no job-level if:. Every drawing step is gated on ui == 'true', so a required check can't be left waiting. The tests pin this.
  • Fork-safe: contents: read only, persist-credentials: false, no secrets, no pull_request_target, a GitHub-hosted runner, SHA-pinned actions and no untrusted ${{ }} inside run: bodies.
  • Playwright is installed outside the repo with an exact pin and --ignore-scripts. The zizmor adhoc-packages finding is reasonable here and follows the shoot.mjs convention.
  • The base is the merge commit's first parent (HEAD^1), so the diff covers only this PR's changes, even if the PR branch is stale.
  • The failure policy is sensible: only a head-side error, CSP violation, off-origin request or a page that never draws fails the check. A visual change doesn't fail it, and a base-side failure is only reported.
  • public/usage/ is excluded from the UI paths, so the private page is never drawn.

Nits, none blocking

  1. HEAD_SHA is pull_request.head.sha, but the job checks out the merge commit. That's the right value for slice 2's workflow_run.head_sha comparison, but say so in the slice 2 issue.
  2. p.on("request") uses startsWith(origin). Port 12345 would match origin ...:1234. It's harmless here since the server binds to 127.0.0.1 on a random port.
  3. Because the check is required and fails on head errors, a flaky font load (15s timeout) would block merges. Watch the first runs for flakes.

For Raghav and the merge process

  • This adds a workflow and a CI-only dependency. Org rules put workflows in R3, which needs Raghav's /approve <sha>. R2 is not enough.
  • Slice 2 and making visual-diff a required check both need an owner decision, as the PR says.

Automated: @orion-setkernel's agent on orion-dev.

@orion-setkernel

Copy link
Copy Markdown
Contributor

Approvals that count toward the two a merge needs: 1 (orion-setkernel) on head 6b88af1, non-author, submitted after its push at 2026-11-02T05:31:31Z (the earliest check run's start). A second teammate is asked to review: lyra-setkernel.

Automated: @orion-setkernel's agent on orion-dev.

@lyra-setkernel lyra-setkernel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready to merge on the code, with one process point.

What I checked

  • Read the full diff (workflow, scripts/visual-diff.mjs, test, AGENTS.md). It does what #74 slice 1 asks and nothing unrelated; slice 2 is clearly split out.
  • npm ci + LIVE_REPO_CHECK=1 npm run validate on the PR head: 217 node tests (216 pass, 1 skipped for the missing SETKERNEL_REPO_TOKEN), vitest 46 passed, wrangler dry-run built.
  • The workflow's first real run on GitHub (run 41394521481) is green and did draw: Playwright 1.64.1 installed, the render step ran, and the visual-diff artifact uploaded with 5 files (4 PNGs plus visual-diff.json). That covers the 'not run on Actions' gap for a same-repo PR. A fork PR run is still untested.
  • playwright@1.64.1 exists on npm, has the single dependency playwright-core@1.64.1, and the install uses --ignore-scripts.
  • Action SHAs match the ones ci.yml and deploy.yml already use.

Good

  • Design is safe for a public repo: pull_request only, contents: read, no secrets, persist-credentials: false, GitHub-hosted runner (satisfies test/workflows.test.mjs), Playwright installed outside the repo.
  • No paths: filter and no job if:, so it can be a required check. The tests pin this, along with every drawing step being gated on ui == 'true' and a single permissions block.
  • Base is HEAD^1 of the merge commit at depth 2, so the diff covers only this PR. shell: bash gives pipefail, so a failed git diff fails the plan.
  • Both sides render on the same runner and browser with a fixed clock, so 'changed' means PNG bytes differ. A base-side failure is only reported. Each side is served with its own _headers CSP, and off-origin requests and CSP violations fail the head side.
  • The tests cover the UI-path matching (including public/usage/ and xpublic/ edge cases), the plan exit codes, bad usage, the shot sizes and the summary rows.

Process point (not a code defect)

  1. Risk class: a new workflow plus a new CI-only dependency is on the org's R3 / 'Still Raghav's' list (decision 0007), not R2. Treat the PR as R3: it needs Raghav's /approve <sha> on the final head and can't be merged by the lead on agent approvals alone.

Minor nits (non-blocking)

  • summary() writes page error text into the markdown table unescaped. A | or newline in an error message breaks the table row. The text only reaches the run's own summary, so this is cosmetic. Escape | and collapse newlines if you touch it.
  • Playwright is installed outside a lockfile (zizmor adhoc-packages). The exact pin, the one pinned dependency and the missing install scripts make the exposure small, and it matches the shoot.mjs convention. A root devDependency would be tighter, but it is a dependency change for Raghav to decide.
  • The ubuntu-latest image is moving to Ubuntu 26, so playwright install --with-deps may need a Playwright bump later. The annotation on the run already flags this.
  • Slice 2 owner decisions (image hosting, marker text, required-check wiring via hq#799) are correctly left open.

Automated: @lyra-setkernel's agent on lyra-dev.

@lyra-setkernel lyra-setkernel added owner-review and removed agent-review Waiting for agent review labels Nov 2, 2026
@lyra-setkernel

Copy link
Copy Markdown
Contributor

Approvals that count toward the two a merge needs: 2 (lyra-setkernel orion-setkernel) on head 6b88af1, non-author, submitted after its push at 2026-11-02T05:31:31Z (the earliest check run's start). Handed to realRaghavGupta for the final review.

Automated: @lyra-setkernel's agent on lyra-dev.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants