Repository navigation
test(ctxd,dispatch,store): the clock-, scheduler- and planner-bound tests of #369 read events and counts (meter on a frozen clock, W6 judges buffers, the hop plan pin holds after a vacuum) - #489
Merged
Conversation
…und leases until both heads queued TestExternalReservationServesTheFairnessFirstHead polled for "the light head is queued" under 16 free-running background workers. Alone in the queue the light head fits the next freed unit (XU is woken before background), so the state ended at the next background release without the test doing anything; a poll that came more than about 18 ms late never saw it (1 timeout in 1040 runs, #369). The background workers now end their leases only once the test closes a gate (gatedBackgroundLoad; backgroundLoad is it without a gate), after both heads are queued, and the bound (one hold time + 50 ms) is measured from that moment. The sweep over the package's waitFor/waitQueued calls found no other poll of a state the load ends on its own. Part of #369. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestV1ChatMeterOnTheRealRouter read the meter of the production router on the real clock: the remaining requests and the Retry-After window [1, 30] broke after more than 29 s between the first and the third request, and the refusal's milliseconds could not be pinned. newModelGateway takes an optional clock for its meter (forwarded by buildRouter, the form of #461's meter tests: keyquota.New over a test clock); no setting or environment variable reaches it, and the production path (newRouter) passes none. TestModelGatewayMeterClock pins that the gateway built without a clock (or with a nil one) runs its meter on time.Now and one built with a clock on that clock; the loop-guard inventory fixture pins time.Now on newRouter's gateway, and the meter rig on every rig built without a clock. Frozen, the third request of RPM 2 waits exactly 30 s: Retry-After 30, retry-after-ms 30000. Part of #369. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ents instead of the wall clock The cmd/ctxd wall-clock bounds of the 21.11. sweep, each by what it meant: - TestRejectionFlusherFinalCancelsAStuckTick: the stuck write reads why its context ended (context.Canceled = final's cancel) instead of final < rejectionFlushTimeout/2. - TestSettleLLMLogRespectsTimeout: the outcome line is the incomplete-settle WARN with the budget's deadline; "returned within 2 s" could not tell it from a settle that waits for all rows (650 ms) and is a guard at the production budget (5 s) now. - TestShutdownGatewayStopRunsAlongsideTheSettle: the first settle has not ended when the gateway stop step begins and ends while it is open, instead of "the sequence took under 8 s". signalSettleEnd uses a handler of its own (wrapping the default handler loops through the log package). - TestV1ModelKeyUsageFinalFlushWaitsForALockedKey: the lock goes once the first flush attempt has returned (not after 200 ms); the stop step's WARN line stands for "Run answered before the stop's timer" (the took < gatewayStopTimeout bound is gone) and, with stamps=1, for "the stamp stayed unwritten"; last_used_at is no longer read after the release (an abandoned last attempt wrote it there once in 30 runs on the base). - TestForeignLoadTickerHeadless_Integration: the 6-s deadline for two rounds is a 30-s guard; the loop ends at the events. - TestV1DoesNotRaiseHerald: the background acquire uses the dispatcher's try door (same predicate as Acquire, answers at once) instead of Acquire on a 50-ms context. - The three L1 concurrency gates: the engine holds the admitted requests until the parallel one was answered (holdUntilReleased) instead of a fixed 1-2 s hold; a request admitted instead of refused releases them, so a broken limit fails the gate instead of hanging it. Part of #369. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d builds the overlap TestTemporalWritersAgainstHardDeletes shared one budget of 2400 operations between four writers and the deleter, then demanded at least 120 hard deletes and one write that met its block gone: a deleter the scheduler starved failed it with every database operation intact (CodeRabbit on #470). Each writer now has 480 writes, the deleter 480 deletes with their re-stores; the deleter starts once every writer has written, the writers go on past their quota until the deleter is done, and the first delete waits before its re-store until a writer has met the deleted block. The test then wants exactly 480 deletes and at least one write that found its block gone. Part of #369. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the vacuumed state TestStructuralHop_IndexPlan failed once in CI (PR #400) with an explicit Sort over sl.created_at in a leg. Reproduced by state, not by luck: with the fixture's 20 000 filler blocks the plan takes the index order while the visibility map of context_structural_links is empty (freshly written, as in an isolated run) and puts a Sort into the reverse leg once the table is vacuumed (a merge join over the whole context_blocks primary key), which is what autovacuum's insert-triggered vacuum leaves when it reaches the table before the EXPLAIN. With 50 000, 100 000 and 200 000 filler blocks both states take the index order and the red probe still sorts. The test now seeds 100 000 filler blocks, turns autovacuum off for the two tables and pins the plan after ANALYZE and again after VACUUM ANALYZE (the steady state of a table at the target). Part of #369. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… no longer leaves its planner switches on a pool connection Gate (E) turned index and bitmap scans off with plain SET on an acquired pool connection and released it with the switches still set; gate (F), next on that connection, measured the forced sequential scan: P95 60-130 ms where the indexed statements run in 0.1-6 ms (#474 point 6 saw 119-143 ms under load against the 100-ms budget). (E) now sets them with SET LOCAL in a transaction that rolls back. Gate (F) judges work instead of wall clock: each list path's statement (the one its store function runs, first-page binds) is EXPLAINed with ANALYZE and BUFFERS in the custom and in the generic plan form and may read at most a tenth of context_blocks' pages (w6BufferShare); the paths read 6-117 of 11 820. The P95 over 40 calls per store function is logged against the budget and judges nothing. Part of #369. Part of #474 (point 6). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 30 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Test code for the tests of #369 (comments since 19.11.) and #474 point 6 that depended on the clock, the Go scheduler or the planner's statistics state, plus one test seam:
go/cmd/ctxd/gateway.go,server.go).newModelGatewaytakes an optional clock for its meter (keyquota.New(now), the form test(ctxd): TestMeterLimiterMapsTheContract asserts a Retry-After window against the real clock (one red run in CI) #461's meter tests use), forwarded bybuildRouter. Only Go code reaches it;newRouter(the production path) passes none, a nil clock counts as none, and three tests pintime.Now:TestModelGatewayMeterClock, the loop-guard inventory fixture onnewRouter's gateway, and the meter rig for every rig built without a clock.TestV1ChatMeterOnTheRealRouterruns the production router's meter on a frozen clock and asserts Retry-After30and retry-after-ms30000exactly (was[1, 30]/[1, 30000]).context.Canceled; the settle timeout reads its incomplete-settle WARN; "gateway stop alongside the first settle" reads the order of the two events; the locked-key final flush releases its lock after the first attempt returned and reads the stop step's WARN line instead oftook < 5 s; the herald probe usesTryAcquire; the L1 concurrency gates hold the engine until the parallel request was answered. The foreign-load ticker's 6-s deadline is a 30-s guard.TestExternalReservationServesTheFairnessFirstHead: the background leases end only once the test opens a gate, after both heads are queued (the queued light head was a state the next background release ended on its own).TestTemporalWritersAgainstHardDeletes: a quota per actor and a built overlap instead of one shared budget.TestStructuralHop_IndexPlan: 100 000 filler blocks, autovacuum off for the two tables, the plan pinned afterANALYZEand afterVACUUM ANALYZE.TestW6Explain_Integration: gate (E) sets its planner switches withSET LOCALin a rolled-back transaction (a plainSETstayed on the pool connection); gate (F) judges the list paths' buffers in the custom and the generic plan (at most a tenth of the table's pages) and logs the P95.Why
Part of #369 (the comments of 19.11.–21.11.; the HNSW recall question has its own issue, #381). Part of #474 (point 6; points 1–5 belong to other changes).
Each location was measured on the base first (binary and one busy loop pinned to one core, with and without
-race) and changed only where its form measured the clock, the scheduler or the planner state. The two causes the measurement turned up are not load: W6 gate (F) measured a forced sequential scan because gate (E) leakedenable_indexscan=offonto a pool connection (P95 60–69 ms unpinned against 0.06–6 ms for the indexed statements; 1–5 ms after the fix), and the structural-hop plan flips with the visibility map ofcontext_structural_links(20 000 filler blocks: index order before a vacuum, an explicit Sort in the reverse leg after one; 50 000 and more: index order in both states).Rejected alternatives: widening the windows (a wider window keeps the dependency and lets a meter that answers 1 ms short through); a relative wall-clock bound for W6 (the buffer count does not depend on the host at all); setting planner GUCs in the structural-hop pin (the pin is about the production plan).
Behaviour changes (intended)
None for callers in production:
newModelGatewayandbuildRoutergain a variadic, test-only clock; without it (every production call) the meter runs ontime.Nowas before.Consumers of the changed contracts
newModelGateway(pool, cfg, meterClock ...)buildRouter(server.go),gateway_seed_integration_test.go×5,shutdown_settle_integration_test.go,gateway_meter_test.goTestModelGatewayMeterClock, the packages' integration runbuildRouter(…, wrapRoutes, meterClock ...)newRouter(production, no clock), 8 integration fixturesnewRouter), meter rig pinv1ChatEngine.hold→holdUntilReleasedhold)backgroundLoad→gatedBackgroundLoadbackgroundLoadcallers unchanged (nil gate)internal/dispatchsuitew6GenericPlan→w6GenericExplainTestW6Explain_IntegrationCost
TestStructuralHop_IndexPlan: 100 000 instead of 20 000 filler blocks, about 15 s instead of 5 s per run (measured on the test container).TestTemporalWritersAgainstHardDeletes: at least the former 2400 operations (writers run until the deleter's 480 turns are done).EXPLAIN (ANALYZE, BUFFERS)added; the 120 store calls for the P95 log line stay.Verification
Stress form for every series: the test binary and one busy loop pinned to core 0 (
taskset -c 0; the temporal load test on-c 0-1), host load 5–33 (1 min) from other work, logged per series. Base = binaries built from3f5a9261before any edit.TestV1ChatMeterOnTheRealRouter30/30000in 60 of 60TestRejectionFlusherFinalCancelsAStuckTickTestSettleLLMLogRespectsTimeoutTestShutdownGatewayStopRunsAlongsideTheSettleTestV1ModelKeyUsageFinalFlushWaitsForALockedKeyTestForeignLoadTickerHeadless_IntegrationTestV1DoesNotRaiseHeraldTestExternalReservationServesTheFairnessFirstHeadTestTemporalWritersAgainstHardDeletesTestStructuralHop_IndexPlanVACUUMat 20 000 fillers)internal/storeon the CI image (18.7 / 2.32.1) ×2: green, base ×2: greenTestW6Explain_IntegrationMutants (production file changed, binary built, file restored and compared byte for byte), 21 of 21 red: meter ignores the handed clock · default clock not
time.Now· meter 1 ms short (retry-after-ms 29999, inside the former window) ·finalwithout cancel · settle without budget · gateway stop before / after the first settle · final flush without retry · flush window past the stop budget · background ignores the herald term · L1 off · bulk share off · no headless foreign-load ticker · reservation for the lightest head · d2 off · gone temporal write as an error · temporal block lock removed (5/5) · hop leg ordered by a key the index cannot give · test-side: hop fixture at 20 000 fillers, gate (E) on plainSET(gate F at 11 820 buffers). The status statement off its index is red at gate (A); gate (F) reads 930–967 buffers for it, under its bound (set for reads of the backdrop).Gates:
bash go/gates.sh wave --finalrc 0 (build, vet, lint, short, census-tests, race, integration forinternal/storeandcmd/ctxd, python, deadcode, deadcode-testonly);go test -tags=integration -p 1 -count=1 ./cmd/ctxd/ ./internal/store/ ./internal/dispatch/ ./internal/keyquota/rc 0 without and with-race;deadcode.shanddeadcode.sh testonlyrc 0; golangci-lint 0 issues on every commit (pre-commit). A fresh reviewer agent read the diff; its findings that held (a multi-statementVACUUMthat cannot run in the implicit transaction, a check made vacuous by reading under the lock, the unpinnednewRouterlink, a nil clock, a missing hang guard, a stale file header) are fixed.Not verified
🤖 Generated with Claude Code