Skip to content

feat(verify): prefer make check-diff where a repo has it - #67

Merged
realRaghavGupta merged 2 commits into
mainfrom
lead/verify-check-diff
Nov 4, 2026
Merged

realRaghavGupta merged 2 commits into
mainfrom
lead/verify-check-diff

Conversation

@realRaghavGupta

Copy link
Copy Markdown
Contributor

What

  • Verify skill, step 0. Where the repo root has a check-diff target, commit the work first and run make check-diff. It picks the steps the committed diff against the default branch needs. A step it did not pick is reported as "not selected", never "passed". The full make check runs only on request, and in CI on the default branch. Repos without check-diff keep make check as before.
  • Reviewer. CI is already green on the PR, so the reviewer reruns only the checks that confirm or rule out a finding, preferring make check-diff.
  • tests/run.sh. New expect lines check the new text, that the verify skill names no model, CLI or vendor, and that the verify skill and the agents name no repo outside the allowed list.

Why

Agent runs ran the whole suite on every work and review step and hit their time limits. A scoped run checks what the diff can break; CI still runs everything.

Checks

step command result
tests (bash) bash tests/run.sh 803/803 passed
tests (dash) dash tests/run.sh 803/803 passed
generator bin/generate --check generated files are current
shellcheck shellcheck -S warning bin/generate plugins/setkernel/hooks/*.sh tests/run.sh exit 0

No version bump: plugin.json has none by design (AGENTS.md).

Merging releases this to every seat through plugin auto-update.

— Claude Lead

🤖 Generated with Claude Code

…r reruns only what confirms a finding

The verify skill sent every run to the full make check. Where the repo root has a
check-diff target, it now commits the work first and runs make check-diff, which picks
the steps the committed diff needs; a step it did not pick is "not selected", never
"passed". The full suite runs on request and in CI on the default branch. Repos
without check-diff keep make check.

The reviewer reruns only the checks that confirm or rule out a finding, preferring
make check-diff, since CI is already green on the PR.

tests/run.sh checks the new text, that the skill stays provider-neutral, and that the
verify skill and the agents name no repo outside the allowed list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@realRaghavGupta realRaghavGupta added the agent-review Waiting for agent review label Nov 4, 2026

@orion-setkernel orion-setkernel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No merge blockers. The diff matches its description and touches only the verify skill, the reviewer agent and tests/run.sh.

Checks I ran on the PR head: bash tests/run.sh gave 832/832 passed (the PR body says 803; the count has grown since, probably from the merge of main). bin/generate --check said the generated files are current. shellcheck -S warning over the AGENTS.md file list, including skills/request-review/*.sh, exited 0. I did not run dash or the JSON check.

What is good

  • Step 0 of the verify skill now prefers make check-diff. Repos that only have make check behave as before.
  • Committing first is needed for a committed-diff scope, and the skill says so. A step that was not picked is reported as "not selected", never "passed". That fits the org rule about reporting checks you did not run.
  • The new expect lines each check something that can break: the new text, no vendor names in the skill, and unlisted_repos on the verify skill and the agents (a function that already exists, line 100).

Nits (non-blocking)

  1. Step 2 of the verify skill still says that before anything reaches the default branch you run "the same steps CI runs". With step 0 that now reads as contradicting it. Add "unless make check-diff selected them" or similar.
  2. The reviewer text says CI is "already green". That holds for the PR review flow, but the reviewer agent is also used pre-PR through the review-blockers skill, where no CI has run. Consider "where CI is already green".
  3. The AGENTS.md traps section does not mention that the verify skill must name no vendor or model. The test enforces it, so this is optional.

Process note: this changes agent runtime behaviour on every seat once merged, through auto-update. Whether it needs R3 handling is for Raghav and the risk policy to say.

Automated: @orion-setkernel's agent on orion-dev.

@orion-setkernel

Copy link
Copy Markdown
Contributor

Approvals that count toward the two a merge needs: 1 (orion-setkernel) on head 6d4a679, non-author, submitted after its push at 2026-11-04T02:23:12Z (the earliest check run's start). A second teammate is asked to review: lyra-setkernel.

Automated: @orion-setkernel's agent on orion-dev.

@setkernel-merge-gate

Copy link
Copy Markdown

Handed back to @realRaghavGupta: the merge gate won't merge this PR because the merge gate is off in agent-plugins (its mode in hq's policy/autonomy.json).

@lyra-setkernel lyra-setkernel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers. The diff matches the PR description: the verify skill's step 0 now prefers make check-diff (commit first, unpicked steps reported as "not selected", full make check only on request), make check stays the fallback, and the reviewer reruns only checks that confirm or rule out a finding. The new expect lines in tests/run.sh are cheap text guards.

Checks I ran on the PR head (merged with current main): bash tests/run.sh 832/832 passed; dash tests/run.sh 832/832 passed; bin/generate --check "generated files are current"; shellcheck -S warning over the AGENTS.md file list exit 0. The PR body says 803, because main added tests after it was written.

Nits (non-blocking):

  1. The opt-in Stop gate (plugins/setkernel/hooks/verify-gate.sh:59, README.md:166) still runs the full make check, so it keeps the cost this PR cuts. It is outside this issue; a follow-up could add check-diff there.
  2. The skill now says the full make check runs "only when you are asked to", while the org rule says done means the repo's checks pass on the reported tree. Defensible, since check-diff is the repo's own scope, but a one-line note that CI on the default branch covers the rest would remove the tension.
  3. "Commit the work first" is a local commit only; consider saying "locally" so nobody reads it as a push.

Merging releases this to every seat via plugin auto-update (no version), so it's on the Raghav / merge-gate path as the PR says.

Automated: @lyra-setkernel's agent on lyra-dev.

@lyra-setkernel lyra-setkernel removed the agent-review Waiting for agent review label Nov 4, 2026
@lyra-setkernel

Copy link
Copy Markdown
Contributor

Approvals that count toward the two a merge needs: 2 (lyra-setkernel orion-setkernel) on head 6d4a679, non-author, submitted after its push at 2026-11-04T02:23:12Z (the earliest check run's start). Handed to realRaghavGupta for the final review.

Automated: @lyra-setkernel's agent on lyra-dev.

@realRaghavGupta
realRaghavGupta merged commit 3025407 into main Nov 4, 2026
1 check passed
@realRaghavGupta
realRaghavGupta deleted the lead/verify-check-diff branch November 4, 2026 02:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants