Skip to content

Runner self-test, --resume, grep -q lint and the eval-warning gate - #33

Merged
andrewthomaslee merged 1 commit into
mainfrom
runner-checks-resume-warning-gate
Nov 13, 2026
Merged

andrewthomaslee merged 1 commit into
mainfrom
runner-checks-resume-warning-gate

Conversation

@andrewthomaslee

Copy link
Copy Markdown
Member

Summary

 flakeModules/
+├── eval-warnings.nix          # apps.flake-check + evalWarnings.deny
+├── vm-test-runner-test.sh     # stub-nix self-test of the runner
 ├── vm-tests.nix               # mkRunner; --resume; result.txt + test drv; grep -q lint;
 │                              # checks.vm-test-runner (vmTests.runnerSelfTest)
 └── dev.nix                    # imports eval-warnings.nix
 devenv/devenv.nix              # `check` = nix run .#flake-check (fallback: nix flake check)
 .github/workflows/org-ci.yml   # flake-check job: the same, with check-args
 dev/flake-module.nix           # runnerSelfTest.enable = true
 vm-tests/smoke.nix             # grep -q pipe -> grep >/dev/null
 documentation/docs/            # vm-tests.md, flake-standard.md, ci.md
flake-check
  nix flake check "$@" | tee log
  list unique "evaluation warning:" lines          # pass
  any matching evalWarnings.deny -> exit 1          # no longer has any effect | has no effect |
                                                    # has been removed | is insecure |
                                                    # known vulnerabilities | option .* does not exist

Evidence

  • Runner self-test: checks.vm-test-runner passes 16/16 cases.
    • Reintroducing the wait -n bug fixed in pkgs#31 makes it fail 2 cases.
    • The first draft failed all cases on a sandbox shebang. Proof it runs for real.
  • grep -q lint: with hello | grep -q … restored in smoke, vm-test --check printed smoke: | grep -q in a pipe … and 1 of 1 FAILED the check. modules and clanServices have no such pipe.
  • Warning gate:
    • Before: home passes nix flake check with 2 stdenv.is* deprecation warnings that nothing reports.
    • After: nix run .#flake-check (home, local override of org-pkgs) lists both and passes. With a temporary evalWarnings.deny matching one of them, it fails with FAILED: dangerous evaluation warnings (exit 1).
  • --resume, real run on euc-hel-0:
    • vm-test --all --remote --rebuild → PASS smoke 26s; result.txt reads passed 26 /nix/store/…-vm-test-run-smoke.drv.
    • vm-test --all --remote --rebuild --resume → PASS smoke 26s (earlier run, --resume), nothing booted.
  • nix run .#flake-check (pkgs): all checks passed.

Merge Danger

Door: two-way

Blast Radius: org CI

Every repo's CI switches to nix run .#flake-check once it bumps org-pkgs. Before that bump, the job falls back to plain nix flake check. A repo that has a dangerous warning today would go red on its bump; none of the core repos have one. Merge order: this PR, then skills, clanServices, modules, home.

🤖 Generated with Claude Code

From the clanServices modernization retro:

- checks.vm-test-runner (vmTests.runnerSelfTest, on in pkgs): the
  runner, now built by mkRunner, runs for two stub tests against a fake
  nix: PASS, PASS(!), FAIL alone/also alone, preparation retry, CACHED,
  --summary, --resume, --rebuild. The two runner bugs of pkgs#31/#32
  reached real VM runs first; reintroducing the wait -n one fails it.
- vm-test --all --resume continues an interrupted run: result.txt now
  records the test derivation, and a test that passed for the same one
  is skipped.
- The per-test check and vm-tests-resources fail on `| grep -q` in a
  test script (SIGPIPE under pipefail), a rule that was prose only.
- apps.flake-check (flakeModules.default), used by the devenv `check`
  script and org-ci: nix flake check, then evaluation warnings listed;
  those matching evalWarnings.deny (ignored, removed or insecure
  config) fail. Ordinary warnings pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andrewthomaslee
andrewthomaslee enabled auto-merge (squash) November 13, 2026 22:57
@andrewthomaslee
andrewthomaslee merged commit 2fa0b28 into main Nov 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant