Skip to content

fix(dojo-verify): speed tests measure CPU time; release-check reruns a mod run that only timed out - #18

Merged
DojoGenesis merged 1 commit into
mainfrom
fix/verify-speed-tests-cpu-time
Nov 9, 2026
Merged

DojoGenesis merged 1 commit into
mainfrom
fix/verify-speed-tests-cpu-time

Conversation

@DojoGenesis

Copy link
Copy Markdown
Owner

The PR 16 release check failed both dojo-verify rows at load ~550 (unittest:dojo-verify with 5 failures, mod:dojo-verify with exit 1). The same code on main passes 153/153 unit tests and 109/109 mod tests (the whole mod suite in about 4s) once the machine is quiet. Running the suites at background CPU priority brings the failures back on unchanged main. Nothing in dojo-verify's logic was wrong; both checks were timing the machine.

Cause

Row What failed Why
unittest TestSpeed and TestTailWindow limits (0.2s, 1.0s, 1.5s) They read time.time(), which also counts time spent waiting for a CPU
mod tests stalled past the kit's per-test 5000 ms The limit is fixed in the kit; a test can't raise it (an options timeout was probed and ignored)

At low load the slowest mod test takes 265 ms. Under load, a test that does almost nothing (DOJO_VERIFY_OFF=1) took 2.6s.

The fix

  • Unit speed tests read CPU time. tests/_helpers.py cpu_seconds() sums the CPU time of the test process and its finished children, and the hook runs as a waited subprocess. The limits are unchanged. Windows has no child CPU accounting there, so it keeps the wall clock.
  • release-check.sh reruns a mod run once when every failure is a kit timeout. The row then reads WARN with the first run's failure count and the load average, so a rerun never passes silently. A test that really hangs times out again and stays FAIL. A real failure beside a timeout is never rerun.

Checks

Check Result
dojo-verify unittest 153 tests, OK
release-check.sh --only unittest,mod --plugin dojo-verify both rows PASS
Release-check self-tests 46 OK, 3 of them new
Can the speed tests still fail? A quadratic loop added to detect_claim fails 18 speed checks at ~0.38s against 0.2s
Can the new self-tests fail? Against the old script, the two rerun tests fail; the no-rerun test passes on both, as it should

Still open: at background priority, which also lowers the clock speed, command_events with 3,000 if branches takes 1.09s of CPU against its 1.0s limit (0.35s at normal priority). Real load runs on full-speed cores, so the limit is unchanged, but it has the least headroom of the speed checks. I haven't reproduced load near 550 against this branch.

🤖 Generated with Claude Code

…a mod run that only timed out

On 2026-11-02 the release check for PR 16 failed both dojo-verify rows at load ~550. The same code
passed 153/153 and 109/109 (mod suite in 4s) once the machine was quiet; nothing in dojo-verify
was wrong.

- unittest: the TestSpeed and TestTailWindow limits read wall-clock time, which counts time spent
  waiting for a CPU. They now read CPU time of the test process and its finished children
  (tests/_helpers.py cpu_seconds). Windows has no child CPU accounting there and keeps the wall
  clock. Proven able to fail: a quadratic loop added to detect_claim fails 18 speed checks at
  ~0.38s against 0.2s.
- mod: the kit gives each test a fixed 5000 ms that a mod can't raise (probed: an options
  timeout is ignored). release-check now reruns a mod run once when every failure is that
  timeout, and the row reads WARN with the first run's count and the load average. A test that
  hangs times out again and stays FAIL; a real failure beside a timeout is never rerun. Three new
  self-tests; the two rerun tests fail against the old script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DojoGenesis
DojoGenesis merged commit 6eeb930 into main Nov 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants