Skip to content

[AI-3471] Give the api suite's routers writers with a small queue - #2171

Draft
kanat wants to merge 2 commits into
mainfrom
kanat/ai-3471-api-test-memory
Draft

kanat wants to merge 2 commits into
mainfrom
kanat/ai-3471-api-test-memory

Conversation

@kanat

@kanat kanat commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

Problem

CI job integration (api) dies with "The runner has received a shutdown signal" (exit 143). It happens on main and on PR branches. The internal/api test binary needs more memory than the runner has.

Each router the suite builds makes its own routing.Writer. Each writer has two channels of 20,000 rows (internal/routing/writer.go:22,152-153). The suite closes its routers. But a finished suite stays in memory until the package ends, because goroutines of its nodes never end:

goroutine that never ends                          holds
api.(*Server).expireProxies (ctx = Background)  ──> *Server ──> routers ──> Writer ──> 2 x 20,000-row channels
relay.(*Bus).subscribe, blocked at bus.go:219    ──> deliver closure ──> *Server ...

Evidence (goroutine dump after m.Run, suites ^Test[A-C]): 1,024 goroutines are still running, and the heap still holds 2.37 GB. Among them: 96 api.(*Server).expireProxies (relayws.go:452, started with context.Background() at server.go:680), and 192 relay.(*Bus).subscribe blocked at bus.go:219. A heap profile written after the last test of a full run holds 60.88 GB inuse in routing.NewWriter (virtual: most of it is never touched, but the GC counts it as live heap).

What this PR does (test-side only)

RouterSuite.routerWriter builds the writer the router would build for itself (routing.newRecorder: Store, Live, Logger, and Counted from the gate), with QueueSize: suiteQueueSize (1,000). The writer is closed when the suite ends, as the router's Close would do. Users:

  • every router in RouterSuite.routers: STT, TTS, both recordings routers, search, decision, image, detection, OCR, the STS stub, and the LLM router when the suite does not ship;
  • telephonyRecorder, meteringRecorder, the session managers in sessionManager and otherNodeHolding;
  • shippingWriter gets the same QueueSize.

Production defaults are unchanged.

Measurements

Command: scripts/run-api.sh <binary> <worktree> <db> <out>. It runs the prebuilt internal/api binary with CI's skip list, -test.count=1 -test.timeout=30m, GOMAXPROCS=4 (so -parallel is 4, as on CI's 4 vCPU), under /usr/bin/time -l. Machine: local macOS (14 CPU, 48 GB). One run per row.

Build maximum resident set size wall fails
main 175bce1 11.91 GB 327 s 0
this PR, queue 1,000 (47e9da0) 9.97 GB 330 s 0
mutant: this PR with the queue at 20,000 11.15 GB 382 s 0
mutant: this PR without the writer's Close 10.45 GB 350 s 0
variant: queue 200 8.59 GB 299 s 1 (TestTelnyxUnpaidSuite/TestAThreadTaskWhoseSessionEndsMidTurnSettles)
probe: main + overwrite testing's leftover cleanups 13.28 GB 377 s 0
  • main and this PR: 4,461 === RUN, 4,409 PASS, 52 SKIP each, and no fails. The fail lists are the same (both empty).
  • A TestMain probe (not committed) printed no router_usage_lost_total series after the PR's run, so the smaller queue lost no row.
  • The cut is about 16% on one run per side. Runs with the same writer setup differ by up to about 2 GB (11.15 to 13.28 GB). See Questions.

Mutations

Mutant Result
suiteQueueSize = 20_000 (the leak back) peak RSS back to 11.15 GB (from 9.97 GB)
routerWriter without its Close cleanup survived: 0 fails. Rows are written as they come in, and no test reads a row that only the final drain writes
Counted not passed from the gate survived on Admission, ShadowAdmission, Limits, Balance, Cost, UnbilledCost, TelephonyBilling and TelnyxUnpaid. It is kept so the writer stays the same as the one newRecorder builds

Questions

  • This only shrinks the leak; it does not remove it. The real fix is in production code: NewServer subscribes to the relay and starts expireProxies on context.Background() (server.go:676-682), and nothing stops them. The comment there says closing the Redis client ends them, but expireProxies only watches its context. A Server.Close that cancels that context would let every finished suite go. Should that be a follow-up ticket?
  • Whether 9.97 GB is enough for CI's runner (16 GB, with Postgres on tmpfs) is unverified until CI runs it.

🤖 Generated with Claude Code

kanat and others added 2 commits November 21, 2026 07:23
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant