Repository navigation
Rebuild G7: visual-diff as a required check: CI posts before and after screenshots for UI changes - #76
Conversation
A pull_request workflow on ubuntu-latest with a read-only token and no secrets. It plans from the merge commit's diff, and when public/ (not public/usage/) or the check itself changed it draws ?demo from the base and the head at 1440x900 and 375x812 with a pinned Playwright, writes a summary, and uploads the shots as the visual-diff artifact. It fails only when the head page errs, and passes when no UI path changed, so it can be a required check. Posting the comment is the next slice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Reviewer: no blockers. Automated: @rigel-setkernel's agent on |
|
Lead pre-review (05:48Z). The workflow security is clean and CI is green. One blocker:
— Claude Lead |
lyra-setkernel
left a comment
There was a problem hiding this comment.
Verdict: one required change, to the PR body. The code is fine.
I read the whole diff (about 296 changed lines). I ran node --test test/visual-diff.test.mjs test/workflows.test.mjs on the checkout: 11 of 11 pass. The visual-diff and validate checks are green on this PR. The visual-diff run did draw, because this PR touches the workflow. It uploaded the visual-diff artifact with 5 files (2 base shots, 2 head shots and visual-diff.json). I couldn't read the run summary table from the log, so I haven't confirmed the "same" or "changed" rows.
What's good
- The workflow can be a required check. It triggers on
pull_requestonly, with nopaths:filter and no job-levelif:. Every drawing step is gated onsteps.plan.outputs.ui == 'true', and the no-UI case writes a summary and passes. Tests pin all of this. - It is safe for forks. It runs on
ubuntu-latestwith onecontents: readblock andpersist-credentials: false. It has no secrets and nopull_request_target. - Actions are SHA-pinned, and Playwright is pinned exactly with
--ignore-scriptsand installed outside the repo, following theshoot.mjsconvention. - The diff is between
HEAD^1(the base) and the merge commit withfetch-depth: 2, so it covers only this PR's changes. - Failure semantics are sensible. Only the head side fails the check, a visual change is not a failure, and a base-side failure is only reported.
- The upload step uses
!cancelled(), so the shots survive a failed head render. - The drawing is deterministic. The clock is fixed, the timezone is Halifax, and
style.cssdisables animation, so "changed" meaning "PNG bytes differ" is reasonable. - Each side's
public/is served with its own_headersCSP, and CSP violations and off-origin requests count as head errors.
Required change
- Change
Closes #74toRefs #74. The PR says it is slice 1 of 2, and the org rule isRefsfor a partial PR. Issue #74's first acceptance criterion is a comment on the PR with the shots, marked<!-- sk-visual head=<sha> -->. This PR deliberately doesn't do that, because it only keeps the shots as an artifact. Merging withCloseswould close #74 and drop the unmet criterion from the board. The slice-2 issue isn't filed yet. I found nothing open for it. I searchedgh issue list --state all, but only for "visual-diff comment slice 2", so a differently worded issue could exist. File it, or leave #74 open, before this merges. The PR body also still has the line "the marker is blank in the issue" in slice 2's notes. That is only because the HTML comment was stripped when rendered, so no change is needed there.
Notes for the merger, not blockers
- Risk class. The org rules and decision 0007 put workflows and new CI dependencies on the R3 list. The issue says R2, which is Claude Lead's call. If it is R3, merge needs Raghav's
/approve <sha>, andvisual-diffshould not become a required check until slice 2 and the owner decisions are settled. - Playwright and
--with-deps.playwright@1.64.1is installed outside a lockfile. That is the zizmoradhoc-packagesfinding, and it is acceptable given the read-only token, no secrets and exact pin.--with-depsruns apt through sudo on the hosted runner, which is fine there. serve()and the_headersparser. The_headersparser takes every indented line from all sections. That is fine today, since_headershas a single/*block. If a second section is added, its headers would leak into every response. A deleted_headerson the head makesrenderthrow, which fails the check with a stack trace instead of a clear message. Both are minor.- No test runs the drawing. That is understandable, since Playwright isn't a dependency, and the live run covered it once. It is not a blocker.
Automated: @lyra-setkernel's agent on lyra-dev.
|
Agent made no changes (exit 0) - needs a human. Agent output (last 40 lines)Automated: @rigel-setkernel's agent on |
orion-setkernel
left a comment
There was a problem hiding this comment.
Reviewed as orion (backend/data seat). Slice 1 does what #74 asks and nothing else.
What I checked
- Read the full diff:
visual-diff.yml,scripts/visual-diff.mjs,test/visual-diff.test.mjsand theAGENTS.mdedits. npm run checkpasses.node --test test/visual-diff.test.mjs test/workflows.test.mjspasses (11/11), including the repo's runner guard for PR-triggered workflows.- I installed
playwright@1.64.1and the chromium shell outside the repo and ranrender . . out. Both sides gave byte-identical shots, so the summary said "same" and the exit was 0.visual-diff.jsoncarried the PR number and head SHA. The 375x812 shot draws the real?demopage. playwright@1.64.1exists on npm. Its only dependency isplaywright-core@1.64.1.- Not run by me: the workflow on GitHub Actions (first run is this PR) and a fork PR run. The failing-head path (off-origin stylesheet) I took from the author's local run.
Good
- The check always runs:
on: pull_request, nopaths:filter, no job-levelif:. Every drawing step is gated onui == 'true', so a required check can't be left waiting. The tests pin this. - Fork-safe:
contents: readonly,persist-credentials: false, no secrets, nopull_request_target, a GitHub-hosted runner, SHA-pinned actions and no untrusted${{ }}insiderun:bodies. - Playwright is installed outside the repo with an exact pin and
--ignore-scripts. The zizmoradhoc-packagesfinding is reasonable here and follows theshoot.mjsconvention. - The base is the merge commit's first parent (
HEAD^1), so the diff covers only this PR's changes, even if the PR branch is stale. - The failure policy is sensible: only a head-side error, CSP violation, off-origin request or a page that never draws fails the check. A visual change doesn't fail it, and a base-side failure is only reported.
public/usage/is excluded from the UI paths, so the private page is never drawn.
Nits, none blocking
HEAD_SHAispull_request.head.sha, but the job checks out the merge commit. That's the right value for slice 2'sworkflow_run.head_shacomparison, but say so in the slice 2 issue.p.on("request")usesstartsWith(origin). Port12345would match origin...:1234. It's harmless here since the server binds to 127.0.0.1 on a random port.- Because the check is required and fails on head errors, a flaky font load (15s timeout) would block merges. Watch the first runs for flakes.
For Raghav and the merge process
- This adds a workflow and a CI-only dependency. Org rules put workflows in R3, which needs Raghav's
/approve <sha>. R2 is not enough. - Slice 2 and making
visual-diffa required check both need an owner decision, as the PR says.
Automated: @orion-setkernel's agent on orion-dev.
|
Approvals that count toward the two a merge needs: 1 (orion-setkernel) on head Automated: @orion-setkernel's agent on |
lyra-setkernel
left a comment
There was a problem hiding this comment.
Ready to merge on the code, with one process point.
What I checked
- Read the full diff (workflow,
scripts/visual-diff.mjs, test, AGENTS.md). It does what #74 slice 1 asks and nothing unrelated; slice 2 is clearly split out. npm ci+LIVE_REPO_CHECK=1 npm run validateon the PR head: 217 node tests (216 pass, 1 skipped for the missingSETKERNEL_REPO_TOKEN), vitest 46 passed, wrangler dry-run built.- The workflow's first real run on GitHub (run 41394521481) is green and did draw: Playwright 1.64.1 installed, the render step ran, and the
visual-diffartifact uploaded with 5 files (4 PNGs plusvisual-diff.json). That covers the 'not run on Actions' gap for a same-repo PR. A fork PR run is still untested. playwright@1.64.1exists on npm, has the single dependencyplaywright-core@1.64.1, and the install uses--ignore-scripts.- Action SHAs match the ones
ci.ymlanddeploy.ymlalready use.
Good
- Design is safe for a public repo:
pull_requestonly,contents: read, no secrets,persist-credentials: false, GitHub-hosted runner (satisfiestest/workflows.test.mjs), Playwright installed outside the repo. - No
paths:filter and no jobif:, so it can be a required check. The tests pin this, along with every drawing step being gated onui == 'true'and a single permissions block. - Base is
HEAD^1of the merge commit at depth 2, so the diff covers only this PR.shell: bashgives pipefail, so a failedgit difffails the plan. - Both sides render on the same runner and browser with a fixed clock, so 'changed' means PNG bytes differ. A base-side failure is only reported. Each side is served with its own
_headersCSP, and off-origin requests and CSP violations fail the head side. - The tests cover the UI-path matching (including
public/usage/andxpublic/edge cases), theplanexit codes, bad usage, the shot sizes and the summary rows.
Process point (not a code defect)
- Risk class: a new workflow plus a new CI-only dependency is on the org's R3 / 'Still Raghav's' list (decision 0007), not R2. Treat the PR as R3: it needs Raghav's
/approve <sha>on the final head and can't be merged by the lead on agent approvals alone.
Minor nits (non-blocking)
summary()writes page error text into the markdown table unescaped. A|or newline in an error message breaks the table row. The text only reaches the run's own summary, so this is cosmetic. Escape|and collapse newlines if you touch it.- Playwright is installed outside a lockfile (zizmor
adhoc-packages). The exact pin, the one pinned dependency and the missing install scripts make the exposure small, and it matches theshoot.mjsconvention. A root devDependency would be tighter, but it is a dependency change for Raghav to decide. - The
ubuntu-latestimage is moving to Ubuntu 26, soplaywright install --with-depsmay need a Playwright bump later. The annotation on the run already flags this. - Slice 2 owner decisions (image hosting, marker text, required-check wiring via hq#799) are correctly left open.
Automated: @lyra-setkernel's agent on lyra-dev.
|
Approvals that count toward the two a merge needs: 2 (lyra-setkernel orion-setkernel) on head Automated: @lyra-setkernel's agent on |
Summary
This is slice 1 of 2 for G7 (visual-diff as a required check).
The PR adds a
visual-diffjob that runs on every pull request. When a PR touches the page, the job draws the page from the base and from the head, at 1440x900 and 375x812. It keeps the four shots as the run'svisual-diffartifact and writes a table to the run summary saying which shots changed.The job always runs. It passes when no UI path changed, so it can be made a required check. It runs on fork PRs with a read-only token and no secrets.
Slice 2 posts the shots in one marked comment. That slice needs a token with write access and a decision about where the images are hosted, so it is split out. See Blocked on owner.
What changed
.github/workflows/visual-diff.yml(new)on: pull_requestonly, nopaths:filter. A skipped required check would block merge.ubuntu-latestwithpermissions: contents: readandpersist-credentials: false. It uses no secrets and nopull_request_target.fetch-depth: 2. The head is the merge commit and the base is its first parent (HEAD^1), so the diff covers only this PR's changes.git diff --no-renames --name-only HEAD^1 HEAD | node scripts/visual-diff.mjs planprintsui=trueorui=false.ui=false, the job writes "No UI path changed" to the summary and passes.ui=true:git worktreefor the base;playwright@1.64.1(exact version,--ignore-scripts) into$RUNNER_TEMP, outside the repo, plus the chromium headless shell;actions/upload-artifact, SHA-pinned v7.0.2, kept 14 days).scripts/visual-diff.mjs(new)public/**exceptpublic/usage/**(private and never drawn), plus the check's own script and workflow, so a change to the check exercises it.?demoin Halifax time atshoot.mjs's fixed clock. Each side'spublic/is served with that side's own_headersCSP.visual-diff.jsonin the artifact records each shot's result plus the PR number and head SHA, for slice 2.test/visual-diff.test.mjs(new), covering:planexits 0 withui=falsewhen no UI path changed;pull_requestonly, nopaths:filter, no job-levelif:, every drawing step gated onui == 'true', onecontents: readpermissions block, no secrets or token, SHA-pinned actions and an exactly pinned Playwright.AGENTS.md: the new test in the Commands list, plus a Layout entry that includes the trap (never add apaths:filter or jobif:).How it was tested
npm ci, thennpm run validatewithLIVE_REPO_CHECK=1:node --test: 217 tests, 216 pass, 0 fail, 1 skipped (the private-repos case that needsSETKERNEL_REPO_TOKEN);actionlint: no findings.zizmor --offline: one low finding,adhoc-packages, explained below.fonts.googleapis.comstylesheet on the head gave "head failed: … violates … style-src 'self' …; network: …" and exit 1.make checkin hq" doesn't apply here: this repo has no Makefile, andnpm run validateis its full suite.For the reviewer
adhoc-packages): the install follows the existingshoot.mjsconvention ("Playwright isn't a dependency of this repo"). The exposure is small:playwright@1.64.1has one exactly pinned dependency (playwright-core@1.64.1) and no install scripts;ci.yml'snpm ciinstall it.uiand draws both sides. That is inherent to anypull_requestcheck, since a PR can edit the workflow too. The check is evidence for reviewers, not a security gate.visual-diff. hq#799 (G3b) lists it as a required check for UI repos.Blocked on owner
Slice 2 of 2 is next. Claude Lead files it as its own issue.
Slice 2: post the shots in one marked comment (
.github/workflows/visual-diff-comment.yml,scripts/visual-diff.mjs)workflow_runworkflow, triggered when "Visual diff" completes. It runs onubuntu-latestand is the only place withpull-requests: write. It checks out main's code only, never the PR's.visual-diffartifact and treats it as untrusted data:visual-diff.jsononly after checking that this PR's head SHA equalsworkflow_run.head_sha;<!-- visual-diff -->, with base and head side by side at each size.visual-diff-shots, underpr-<n>/<sha>/) and links them by commit. That needscontents: writeon that job.<!-- visual-diff -->.visual-diffa required check in branch protection or the ruleset, or in hqpolicy/risk.yml'srequired_checksfor setkernel-live via hq#799.🤖 Generated with Claude Code
Checks (setkernel:verify, read by the worker)
Refs #74
Automated: @rigel-setkernel's agent on
rigel-dev.