Skip to content

docs(test): define when a test may be mock-only - #241

Closed
meyer9 wants to merge 1 commit into
mainfrom
docs/mock-only-test-criteria
Closed

meyer9 wants to merge 1 commit into
mainfrom
docs/mock-only-test-criteria

Conversation

@meyer9

@meyer9 meyer9 commented Oct 21, 2026

Copy link
Copy Markdown

Summary

A test marks itself mock-only by opening with vm.skip(livePrecompiles), but nothing says when that's allowed: the reasons live only in per-test comments. This PR documents the criteria on the livePrecompiles flag in test/lib/BaseTest.sol, and adds a row pointing to them in LIVE_PRECOMPILE_TESTING.md. A test is mock-only when it:

  1. writes storage at a fixed precompile address, which forge rejects with vm.store: cannot use precompile ... as an argument;
  2. asserts a mock storage slot that the Rust implementation doesn't have, such as MockB20Storage.initializedSlot(); or
  3. needs a state that no real sequence of calls can reach and that only mock storage writes can build.

It also says to branch on livePrecompiles when a behavior can be checked in both worlds. A test waiting on a base/base change should be gated behind an environment flag, like POLICY_DISPATCH_FIX, instead of this guard. base/base's live-precompile CI now leaves mock-only skips out of its skipped count (base/base#6154), so misusing the guard would hide the test there.

How the current skips fit

I removed every vm.skip(livePrecompiles) and ran the tests against the live precompiles (base-anvil at 541626fd, Denim, base-std 7321d891):

Criterion Tests Result with the skip removed
1: vm.store on the PolicyRegistry precompile address 4: both counterOverflow tests, createPolicy_revertOrder, pendingPolicyAdmin storage poison fail: forge rejects the store
2: mock-only slot 2: MockB20SlotHelpers initializedSlot fails. adminCountSlot_disjoint passes, but it only compares library constants
3: unreachable bootstrap state 5: the privileged transferFrom and transferFromWithMemo tests 4 pass by accident: the store hits a slot that doesn't exist in Rust, so the window never reopens and the privileged path never runs. skipsExecutorPolicy fails with PolicyForbids, which confirms it
None: waiting on base/base#3362 2: B20FactoryNonPayable both pass

The two B20FactoryNonPayable skips don't meet the criteria. base/base#3362 has merged, and both tests pass on Beryl and Denim. I left them alone to keep this PR to documentation; removing them is the follow-up.

Verification

forge fmt --check test/lib/BaseTest.sol and forge build pass, and the PolicyRegistryCreatePolicyTest and MockB20SlotHelpersTest suites pass in reference mode. The change is comments and docs only.

Tests mark themselves mock-only by opening with vm.skip(livePrecompiles),
but the criteria lived only in per-test comments. Document them on the
livePrecompiles flag: a test is mock-only when it writes storage at a
fixed precompile address, asserts a mock slot the Rust implementation
lacks, or needs a state that only mock storage writes can build.

Also say to branch on livePrecompiles when a behavior can be checked in
both worlds, and to gate tests that wait on a base/base change behind an
environment flag, so those skips stay visible in base/base's live
precompile CI, which leaves mock-only skips out of its skipped count.

Amp-Thread-ID: https://ampcode.com/threads/T-f04cecba-e4dc-42c5-94de-8846dbf5444c
Co-authored-by: Julian Meyer <julian.meyer@coinbase.com>
@github-actions

Copy link
Copy Markdown

Interface Coverage

✅ All interface functions have test coverage.

@meyer9 meyer9 closed this Oct 21, 2026
@github-actions

Copy link
Copy Markdown

📊 Forge Coverage (src/lib/)

🟡 ≥95% across all metrics — some metrics below 99%.

File Lines Stmts Branches Funcs
🟡 B20FactoryLib.sol 97.70% 98.00% 100.00% 95.00%
🔴 test/lib/ForceFeeder.sol 0.00% 0.00% 100.00% 0.00%
🔴 test/lib/PrecompileProbe.sol 0.00% 0.00% 0.00% 0.00%
🟢 MockActivationRegistry.sol 100.00% 100.00% 100.00% 100.00%
🟢 MockActivationRegistryStorage.sol 100.00% 100.00% 100.00% 100.00%
🟢 MockB20.sol 100.00% 99.68% 100.00% 100.00%
🟢 MockB20Asset.sol 100.00% 100.00% 100.00% 100.00%
🟡 MockB20Factory.sol 98.96% 99.10% 100.00% 100.00%
🟢 MockB20Stablecoin.sol 100.00% 100.00% 100.00% 100.00%
🟢 MockB20Storage.sol 100.00% 100.00% 100.00% 100.00%
🟢 MockBaseTime.sol 100.00% 100.00% 100.00% 100.00%
🟡 MockPolicyRegistry.sol 100.00% 99.58% 97.78% 100.00%
🟢 MockPolicyRegistryStorage.sol 100.00% 100.00% 100.00% 100.00%
Total 97.14% 97.51% 98.21% 97.07%

Full report: download artifact. To browse locally: make coverage (runs forge coverage + genhtml + opens the HTML report).

@github-actions

Copy link
Copy Markdown

❌ Fork tests did not complete

No completed, non-empty suite was recorded. Check the workflow logs for build, setup, or execution errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants