Skip to content

test(ctxd,dispatch,store): the clock-, scheduler- and planner-bound tests of #369 read events and counts (meter on a frozen clock, W6 judges buffers, the hop plan pin holds after a vacuum) - #489

Merged
GottZ merged 6 commits into
rootfrom
issue/369-clock-scheduler-planner-tests
Nov 22, 2026

Conversation

@GottZ

@GottZ GottZ commented Nov 22, 2026

Copy link
Copy Markdown
Owner

What this changes

Test code for the tests of #369 (comments since 19.11.) and #474 point 6 that depended on the clock, the Go scheduler or the planner's statistics state, plus one test seam:

  • Meter clock seam (production, go/cmd/ctxd/gateway.go, server.go). newModelGateway takes an optional clock for its meter (keyquota.New(now), the form test(ctxd): TestMeterLimiterMapsTheContract asserts a Retry-After window against the real clock (one red run in CI) #461's meter tests use), forwarded by buildRouter. Only Go code reaches it; newRouter (the production path) passes none, a nil clock counts as none, and three tests pin time.Now: TestModelGatewayMeterClock, the loop-guard inventory fixture on newRouter's gateway, and the meter rig for every rig built without a clock.
  • TestV1ChatMeterOnTheRealRouter runs the production router's meter on a frozen clock and asserts Retry-After 30 and retry-after-ms 30000 exactly (was [1, 30] / [1, 30000]).
  • cmd/ctxd wall-clock bounds → events: the stuck flusher tick reads context.Canceled; the settle timeout reads its incomplete-settle WARN; "gateway stop alongside the first settle" reads the order of the two events; the locked-key final flush releases its lock after the first attempt returned and reads the stop step's WARN line instead of took < 5 s; the herald probe uses TryAcquire; the L1 concurrency gates hold the engine until the parallel request was answered. The foreign-load ticker's 6-s deadline is a 30-s guard.
  • TestExternalReservationServesTheFairnessFirstHead: the background leases end only once the test opens a gate, after both heads are queued (the queued light head was a state the next background release ended on its own).
  • TestTemporalWritersAgainstHardDeletes: a quota per actor and a built overlap instead of one shared budget.
  • TestStructuralHop_IndexPlan: 100 000 filler blocks, autovacuum off for the two tables, the plan pinned after ANALYZE and after VACUUM ANALYZE.
  • TestW6Explain_Integration: gate (E) sets its planner switches with SET LOCAL in a rolled-back transaction (a plain SET stayed on the pool connection); gate (F) judges the list paths' buffers in the custom and the generic plan (at most a tenth of the table's pages) and logs the P95.

Why

Part of #369 (the comments of 19.11.–21.11.; the HNSW recall question has its own issue, #381). Part of #474 (point 6; points 1–5 belong to other changes).

Each location was measured on the base first (binary and one busy loop pinned to one core, with and without -race) and changed only where its form measured the clock, the scheduler or the planner state. The two causes the measurement turned up are not load: W6 gate (F) measured a forced sequential scan because gate (E) leaked enable_indexscan=off onto a pool connection (P95 60–69 ms unpinned against 0.06–6 ms for the indexed statements; 1–5 ms after the fix), and the structural-hop plan flips with the visibility map of context_structural_links (20 000 filler blocks: index order before a vacuum, an explicit Sort in the reverse leg after one; 50 000 and more: index order in both states).

Rejected alternatives: widening the windows (a wider window keeps the dependency and lets a meter that answers 1 ms short through); a relative wall-clock bound for W6 (the buffer count does not depend on the host at all); setting planner GUCs in the structural-hop pin (the pin is about the production plan).

Behaviour changes (intended)

None for callers in production: newModelGateway and buildRouter gain a variadic, test-only clock; without it (every production call) the meter runs on time.Now as before.

Consumers of the changed contracts

Changed Callers and readers Covered by
newModelGateway(pool, cfg, meterClock ...) buildRouter (server.go), gateway_seed_integration_test.go ×5, shutdown_settle_integration_test.go, gateway_meter_test.go TestModelGatewayMeterClock, the packages' integration run
buildRouter(…, wrapRoutes, meterClock ...) newRouter (production, no clock), 8 integration fixtures inventory fixture pin (newRouter), meter rig pin
v1ChatEngine.hold → holdUntilReleased the three L1 gates (only users of hold) the L1 gates
backgroundLoad → gatedBackgroundLoad backgroundLoad callers unchanged (nil gate) internal/dispatch suite
w6GenericPlan → w6GenericExplain gate (B2) unchanged TestW6Explain_Integration

Cost

  • TestStructuralHop_IndexPlan: 100 000 instead of 20 000 filler blocks, about 15 s instead of 5 s per run (measured on the test container).
  • TestTemporalWritersAgainstHardDeletes: at least the former 2400 operations (writers run until the deleter's 480 turns are done).
  • W6 gate (F): six EXPLAIN (ANALYZE, BUFFERS) added; the 120 store calls for the P95 log line stay.

Verification

Stress form for every series: the test binary and one busy loop pinned to core 0 (taskset -c 0; the temporal load test on -c 0-1), host load 5–33 (1 min) from other work, logged per series. Base = binaries built from 3f5a9261 before any edit.

Test Base After
TestV1ChatMeterOnTheRealRouter 0/60 (retry-after-ms 29 423–29 994) 0/60, 30 / 30000 in 60 of 60
TestRejectionFlusherFinalCancelsAStuckTick 0/400 0/400, +200 on the final code
TestSettleLLMLogRespectsTimeout 0/60 0/120
TestShutdownGatewayStopRunsAlongsideTheSettle 0/60 (5.1–5.8 s) 0/120, the two events as wanted in 120 of 120
TestV1ModelKeyUsageFinalFlushWaitsForALockedKey 1/60 (read after the release) 0/120
TestForeignLoadTickerHeadless_Integration 0/60 0/60
TestV1DoesNotRaiseHerald 0/60 0/60
three L1 concurrency gates 0/60 0/60
TestExternalReservationServesTheFairnessFirstHead 0/400 alone, 0/1040 in the issue's 52-function pass 0/400 alone, 0/1040; admission 10–17 ms (bound 70 ms)
TestTemporalWritersAgainstHardDeletes 0/60 (277–420 deletes) 0/60, 480 deletes in every run
TestStructuralHop_IndexPlan 0/30 isolated; the CI shape reproduced by state (Sort after VACUUM at 20 000 fillers) 0/60 isolated; full internal/store on the CI image (18.7 / 2.32.1) ×2: green, base ×2: green
TestW6Explain_Integration 2/20 (P95 41–158 ms) 0/60, buffers 6 / 11 / 117 in every run, P95 1.0–11.6 ms (logged)

Mutants (production file changed, binary built, file restored and compared byte for byte), 21 of 21 red: meter ignores the handed clock · default clock not time.Now · meter 1 ms short (retry-after-ms 29999, inside the former window) · final without cancel · settle without budget · gateway stop before / after the first settle · final flush without retry · flush window past the stop budget · background ignores the herald term · L1 off · bulk share off · no headless foreign-load ticker · reservation for the lightest head · d2 off · gone temporal write as an error · temporal block lock removed (5/5) · hop leg ordered by a key the index cannot give · test-side: hop fixture at 20 000 fillers, gate (E) on plain SET (gate F at 11 820 buffers). The status statement off its index is red at gate (A); gate (F) reads 930–967 buffers for it, under its bound (set for reads of the backdrop).

Gates: bash go/gates.sh wave --final rc 0 (build, vet, lint, short, census-tests, race, integration for internal/store and cmd/ctxd, python, deadcode, deadcode-testonly); go test -tags=integration -p 1 -count=1 ./cmd/ctxd/ ./internal/store/ ./internal/dispatch/ ./internal/keyquota/ rc 0 without and with -race; deadcode.sh and deadcode.sh testonly rc 0; golangci-lint 0 issues on every commit (pre-commit). A fresh reviewer agent read the diff; its findings that held (a multi-statement VACUUM that cannot run in the implicit transaction, a check made vacuous by reading under the lock, the unpinned newRouter link, a nil clock, a missing hang guard, a stale file header) are fixed.

Not verified

  • The CI runner itself: all rates are from this host under the stress above; the structural-hop flake did not recur in 4 full package runs (2 base, 2 after), so the after side of that CI sample rests on the reproduction by state.
  • The production effect of the abandoned final-flush statement (a stamp the stop step reports as unwritten lands afterwards): one observation, not changed.
  • The dispatcher's timer-accuracy bounds and the foreign-load ticker's post-cancel window: read and measured without failure, not changed (no clock seam).

🤖 Generated with Claude Code

GottZ and others added 6 commits November 21, 2026 22:17
…und leases until both heads queued

TestExternalReservationServesTheFairnessFirstHead polled for "the light head
is queued" under 16 free-running background workers. Alone in the queue the
light head fits the next freed unit (XU is woken before background), so the
state ended at the next background release without the test doing anything;
a poll that came more than about 18 ms late never saw it (1 timeout in 1040
runs, #369). The background workers now end their leases only once the test
closes a gate (gatedBackgroundLoad; backgroundLoad is it without a gate),
after both heads are queued, and the bound (one hold time + 50 ms) is
measured from that moment. The sweep over the package's waitFor/waitQueued
calls found no other poll of a state the load ends on its own.

Part of #369.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestV1ChatMeterOnTheRealRouter read the meter of the production router on
the real clock: the remaining requests and the Retry-After window [1, 30]
broke after more than 29 s between the first and the third request, and the
refusal's milliseconds could not be pinned. newModelGateway takes an
optional clock for its meter (forwarded by buildRouter, the form of #461's
meter tests: keyquota.New over a test clock); no setting or environment
variable reaches it, and the production path (newRouter) passes none.
TestModelGatewayMeterClock pins that the gateway built without a clock (or
with a nil one) runs its meter on time.Now and one built with a clock on
that clock; the loop-guard inventory fixture pins time.Now on newRouter's
gateway, and the meter rig on every rig built without a clock. Frozen, the third
request of RPM 2 waits exactly 30 s: Retry-After 30, retry-after-ms 30000.

Part of #369.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ents instead of the wall clock

The cmd/ctxd wall-clock bounds of the 21.11. sweep, each by what it meant:

- TestRejectionFlusherFinalCancelsAStuckTick: the stuck write reads why
  its context ended (context.Canceled = final's cancel) instead of
  final < rejectionFlushTimeout/2.
- TestSettleLLMLogRespectsTimeout: the outcome line is the
  incomplete-settle WARN with the budget's deadline; "returned within 2 s"
  could not tell it from a settle that waits for all rows (650 ms) and is
  a guard at the production budget (5 s) now.
- TestShutdownGatewayStopRunsAlongsideTheSettle: the first settle has not
  ended when the gateway stop step begins and ends while it is open,
  instead of "the sequence took under 8 s". signalSettleEnd uses a handler
  of its own (wrapping the default handler loops through the log package).
- TestV1ModelKeyUsageFinalFlushWaitsForALockedKey: the lock goes once the
  first flush attempt has returned (not after 200 ms); the stop step's
  WARN line stands for "Run answered before the stop's timer" (the
  took < gatewayStopTimeout bound is gone) and, with stamps=1, for "the
  stamp stayed unwritten"; last_used_at is no longer read after the release
  (an abandoned last attempt wrote it there once in 30 runs on the base).
- TestForeignLoadTickerHeadless_Integration: the 6-s deadline for two
  rounds is a 30-s guard; the loop ends at the events.
- TestV1DoesNotRaiseHerald: the background acquire uses the dispatcher's
  try door (same predicate as Acquire, answers at once) instead of Acquire
  on a 50-ms context.
- The three L1 concurrency gates: the engine holds the admitted requests
  until the parallel one was answered (holdUntilReleased) instead of a fixed
  1-2 s hold; a request admitted instead of refused releases them, so a
  broken limit fails the gate instead of hanging it.

Part of #369.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d builds the overlap

TestTemporalWritersAgainstHardDeletes shared one budget of 2400 operations
between four writers and the deleter, then demanded at least 120 hard
deletes and one write that met its block gone: a deleter the scheduler
starved failed it with every database operation intact (CodeRabbit on
#470). Each writer now has 480 writes, the deleter 480 deletes with their
re-stores; the deleter starts once every writer has written, the writers go
on past their quota until the deleter is done, and the first delete waits
before its re-store until a writer has met the deleted block. The test then
wants exactly 480 deletes and at least one write that found its block gone.

Part of #369.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the vacuumed state

TestStructuralHop_IndexPlan failed once in CI (PR #400) with an explicit
Sort over sl.created_at in a leg. Reproduced by state, not by luck: with the
fixture's 20 000 filler blocks the plan takes the index order while the
visibility map of context_structural_links is empty (freshly written, as in
an isolated run) and puts a Sort into the reverse leg once the table is
vacuumed (a merge join over the whole context_blocks primary key), which is
what autovacuum's insert-triggered vacuum leaves when it reaches the table
before the EXPLAIN. With 50 000, 100 000 and 200 000 filler blocks both
states take the index order and the red probe still sorts. The test now
seeds 100 000 filler blocks, turns autovacuum off for the two tables and
pins the plan after ANALYZE and again after VACUUM ANALYZE (the steady state
of a table at the target).

Part of #369.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… no longer leaves its planner switches on a pool connection

Gate (E) turned index and bitmap scans off with plain SET on an acquired
pool connection and released it with the switches still set; gate (F), next
on that connection, measured the forced sequential scan: P95 60-130 ms
where the indexed statements run in 0.1-6 ms (#474 point 6 saw 119-143 ms
under load against the 100-ms budget). (E) now sets them with SET LOCAL in
a transaction that rolls back.

Gate (F) judges work instead of wall clock: each list path's statement (the
one its store function runs, first-page binds) is EXPLAINed with ANALYZE and
BUFFERS in the custom and in the generic plan form and may read at most a
tenth of context_blocks' pages (w6BufferShare); the paths read 6-117 of
11 820. The P95 over 40 calls per store function is logged against the
budget and judges nothing.

Part of #369. Part of #474 (point 6).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GottZ
GottZ requested a balanced review from Copilot November 22, 2026 00:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@coderabbitai

coderabbitai Bot commented Nov 22, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 30 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 62c0d9f2-4246-4c84-9e44-34c9150d1eda
Commits

Reviewing files that changed from the base of the PR and between 53b8f59 and b6752f5.

Files selected for processing: 17

  • go/cmd/ctxd/foreign_load_ticker_integration_test.go
  • go/cmd/ctxd/gateway.go
  • go/cmd/ctxd/gateway_ledger_test.go
  • go/cmd/ctxd/gateway_meter_test.go
  • go/cmd/ctxd/loopguard_inventory_integration_test.go
  • go/cmd/ctxd/server.go
  • go/cmd/ctxd/shutdown_settle_integration_test.go
  • go/cmd/ctxd/v1_admission_integration_test.go
  • go/cmd/ctxd/v1_chat_integration_test.go
  • go/cmd/ctxd/v1_chat_l1_integration_test.go
  • go/cmd/ctxd/v1_chat_meter_integration_test.go
  • go/cmd/ctxd/v1_mount_integration_test.go
  • go/internal/dispatch/fairness_gates_test.go
  • go/internal/dispatch/fairness_hardening_test.go
  • go/internal/store/graph_structuralhop_integration_test.go
  • go/internal/store/issues_read_explain_integration_test.go
  • go/internal/store/temporal_delete_integration_test.go
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@GottZ
GottZ merged commit 138cfb7 into root Nov 22, 2026
25 checks passed
@GottZ
GottZ deleted the issue/369-clock-scheduler-planner-tests branch November 22, 2026 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants