Skip to content

test: add tests/checks.sh and scale local checks with the change - #123

Merged
mmaclay merged 1 commit into
mainfrom
chore/efficient-checks
Nov 7, 2026
Merged

mmaclay merged 1 commit into
mainfrom
chore/efficient-checks

Conversation

@mmaclay

@mmaclay mmaclay commented Nov 7, 2026

Copy link
Copy Markdown
Member

Why

Transcripts from Nov 5 to 7 show where usage went. Long Opus agents and sessions took about 72% of it: one build agent ran 496 turns at up to 671K context, and one main session ran 1,257 turns. Sandbox runs took about 18%. A turn's cost grows with the context it carries, and much of those turns went to rerunning the full suite (about ten minutes) after each edit, polling it with sleep/pgrep, and grepping one thing at a time. CI already runs every check on every pull request, so a full local run after each edit duplicated CI.

What changes

  • tests/checks.sh runs the checks serially, one line per step; only a failing step prints output, its last 40 lines. A contributor tool; nothing in the plugin calls it.
    • Default: the tests that name a file changed against origin/main, and the tests of scripts that load it, plus check_static.py and codespell. Some changes run the whole suite instead: a data file that no test names, tests/fixtures/, pyproject.toml or .python-version. Skills, agents and docs get the static checks only. The header lists every rule.
    • NAME... runs the named test modules only. --all runs CI's steps except its gitleaks scan.
    • It runs with no global or system git config, as CI does. The tests' throwaway commits are then never signed, which cut the unit tests from 599s to 435s on a machine with commit.gpgsign=true.
  • .claude/CLAUDE.md gains a table of what each kind of change runs before review: static checks, mapped tests or --all, the sandbox, and adversarial review rounds. It also gains the session habits behind the cost: a fresh agent per phase, Sonnet for mechanical agents, long commands in the background rather than polled, and batched reads. The pre-PR bullet now names the runner.
  • README: a paragraph after CI's list of steps says how to run them locally.

Reviewing

  • The change-to-test mapping, tests/checks.sh lines 99-114, is the part to read. It deliberately selects more than it strictly needs: scripts/record.py selects 17 test files.
  • Not changed: CI, any test, and any skill, agent or hook. So the Origin:/Sandbox:/Canary: lines are left out.

Verification

  • bash tests/checks.sh --all: all 11 steps pass.
  • On this branch the default mode runs only the static checks, which is correct for a change to docs and the runner.
  • I checked the mapping with a copy of the runner whose test steps only print what they would run. It selected test_comment_list for record.py through its loader and test_capture for guidance_check.py. A harvest data file and a fixture each selected the whole suite. hooks.json selected the gate tests. A bad name exits 2 before anything runs.
  • One adversarial review round found 2 blocking and 5 should-fix issues, all fixed in this commit: loader tests, data files, renames, the README's placement and wording, and the canary and --check --repeat 2 requirements in the table. The review's reading of model ownership was declined: the habits bullet is about contributors' sessions, not the plugin's agent models.

Not in this PR

  • Running the tests in parallel. test_profile, test_window_merges and test_surface_diff share temporary state when run together, and that needs finding first.
  • Speeding up standard.sh's real partial-clone fetch, which accounts for most of test_profile's time.

🤖 Generated with Claude Code

Measured usage came from agents rerunning the full ten-minute suite after
every edit and polling it, not from the checks CI needs. tests/checks.sh
runs, by default, only the tests that name a changed file or a script
loading it, plus check_static and codespell; NAME... runs given modules
and --all reproduces CI's steps but gitleaks. It runs without the
person's git configuration, as CI does, so the tests' throwaway commits
are not signed: the unit tests drop from 599s to 435s locally.

.claude/CLAUDE.md gains a table of what each kind of change runs before
review, and the session habits that keep usage down.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mmaclay
mmaclay marked this pull request as ready for review November 7, 2026 17:16
@mmaclay
mmaclay merged commit 34489a5 into main Nov 7, 2026
1 check passed
@mmaclay
mmaclay deleted the chore/efficient-checks branch November 7, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant