Repository navigation
Description
Why
Under edit load the SQLite WAL stays well above the point where the reclaim counts as urgent. In the edit benchmarks (agents editing files and reading them back, with 1, 4 and 8 agents) the WAL peaks at 1,007 / 1,059 / 960 MiB, and at 760 MiB with 6 agents editing 6 repos. Read-only loads stay at 0.3 MiB. Reclaim starts at 256 MiB, and from 512 MiB an attempt counts as urgent. An urgent attempt behaves differently from a normal one:
- it does not yield to writes already waiting for the writer;
- it may hold the application writer for up to 2 s in total, where a normal hold is 50 ms;
- the fallback paths that close the read gate can pause every new read for up to 250 ms while readers drain, and a closed-gate TRUNCATE can add another 250 ms.
During an edit burst, that means the reclaim can add up to 2 s of writer wait to the edit's file-graph and relations updates. It can also add up to 0.5 s of paused reads to the edit acknowledgement and to the read-back of the edited file, which are the two steps an agent waits on. None of these holds or pauses are measured today.
v0.64.10 has no WAL reclaim at all, so we presume this is a regression from the store rewrite in #806. Comparing edit-path tails against v0.64.10 will confirm it or not. The fix ships either way.
Current state
The reclaim already stands down for edits in several places. On main it pauses its copy during an edit and keeps its progress (0555af2), keeps background WAL work out of edit cycles (d55d6f9), and resets the WAL with short 50 ms writer holds instead of long blocks (0174a67). The remaining gaps are the urgent, pressure and closed-gate paths. All files below are in internal/graph/store_sqlite/:
wal_reclaim.goconstants:defaultWALReclaimThresholdBytes= 256 MiB,walReclaimUrgentFactor= 2,walReclaimMaxWriterHold= 2 s,defaultWALReclaimDrainDeadline= 250 ms andwalReclaimTruncateBudget= 250 ms. The ceiling is 8 x threshold (2 GiB). The hard cap (walReclaimHardCapFactor) and the last resort (walReclaimLastResortFactor) both sit at 4 x ceiling (8 GiB).reclaimWALAttemptsetsres.urgentfrom the WAL size. In the open-gate reset loop it admits the reset whenres.urgent || !s.writeWanted(). So an urgent attempt takes the writer even while a write is queued for it.reclaimWALInLaneskips its "writer waiting" early exit when the attempt is urgent.reclaimWALResetHoldOnceskips the same check. Both setwriter.yieldToWriters = !res.urgentand starts.yieldToWriters(...)only when the attempt is not urgent. An urgent hold therefore ignores queued writers.- Step 6 of
reclaimWALInLaneis the closed-gate path:s.readGate.quiesce(...)followed by TRUNCATE. Last-resort and over-ceiling bulk-completion attempts reach it, and so do tests that disable the open-gate stages (walReclaimSkipOpenGate, never set in production). Those holds are allowed up towalReclaimMaxWriterHold. wal_reclaim_pressure.gohandles attempts above the pressure mark, or after a writer has asked for a reclaim, and these run inside a busy lane (an edit in progress). When the 50 ms reset (walReclaimPressureHold, aliased aswalReclaimResetHold) is blocked by readers,reclaimWALPressureResetWithDrainfalls through toreclaimWALPressureResetAfterReaderDrain. That function checkswriteWanted()first, but then closes the read gate (readGate.quiesce) for up todefaultWALReclaimDrainDeadline, even while an edit cycle holds the lane. The result is a read-gate closure during an edit burst.wal_reclaim_adaptive.go:takeAdaptiveWriterBudgetlets an urgent attempt with a proven slow-copy tail take one adaptive writer slice of up to 2 s. Both the reset-hold path and the pressure path call it, including inside a busy lane.- An edit burst can already be detected.
buildLaneBusy()(checkpoint_cycle_yield.go) covers a held build lane, andeditIntentActive()(wal_reclaim.go) covers an announced edit. Queued writers show up throughwriteWanted()andAnnounceWrite(). - The read gate (
sqlite_read_gate.go) knows nothing about what a read is for. It cannot tell an edit's acknowledgement or its read-back apart from a long analysis read. WALReclaimStatsrecordsWriterHoldMaxandAdaptiveWriterHoldMax, and records gate pauses throughnotePause. None of these are split by whether an edit burst was active, and no p95 can be derived from them.
What to do
-
Urgent attempts yield to edits. While an edit burst is active (
buildLaneBusy()oreditIntentActive()) or a writer is announced or queued (writeWanted()), an urgent attempt behaves like a non-urgent one. That means it keeps the "writer waiting" early exit,yieldToWritersstays on, and the open-gate reset is admitted only when!writeWanted(). Outside bursts, urgency still shortens spacing and picks the converge-first path. It no longer overrides a waiting edit. -
Cap writer holds during edit bursts at
walReclaimResetHold(50 ms). While a burst is active, refuse the adaptive slow-copy slice (takeAdaptiveWriterBudget) and any completion hold longer than 50 ms, and defer them to the next gap between edits. The only exceptions are the over-ceiling bulk completion (2 GiB) and the hard-cap / last-resort step (8 GiB). Both keep their current logging and are counted as exceptions. -
No read-gate closure during an edit burst. In that case
reclaimWALPressureResetAfterReaderDrainand the step-6 closed-gate path ofreclaimWALInLanereturn a "deferred: edit burst" outcome instead of callingreadGate.quiesce. The attempt then waits for a gap, or for the hard cap. Outside edit bursts nothing changes. -
Yield to an edit's acknowledgement and read-back. Add a cheap store-level marker for these interactive reads, similar to
AnnounceWrite: a counter, or a context tag the daemon sets while it serves the edit acknowledgement and the read of the edited file that follows. While such a read is in flight, the reclaim must not start a reset hold or close the gate. Wire the marker from the edit tool path and from the read-back path in the MCP layer. -
Keep the two yield conditions separable. Yielding to edit tickets and yielding to interactive reads should be distinct conditions in code. If the nightly edit-load WAL goes past 2 GiB, the fallback is to keep only the yield to interactive reads (a small code change, not a new runtime switch) until the edit-path WAL holders are converted.
-
Counters. Add to
WALReclaimStats:- writer-hold samples taken during edit bursts, enough to compute a p95 (a small histogram or ring buffer);
- read-gate closures during edit bursts (expected 0);
- deferrals caused by an edit burst and by an interactive read, counted separately;
- urgent attempts;
- exception holds (bulk completion, hard cap / last resort).
Expose them wherever the existing reclaim stats are already reported.
-
Update the constant comments in
wal_reclaim.goandwal_reclaim_pressure.go, which today say an urgent attempt does not yield to writers, so they describe the new contract.
Out of scope
- Getting the edit-load WAL below 512 MiB so that no urgent reclaim happens at all. That needs the edit-path WAL holders converted, and it is tracked separately.
- WAL checkpoint counters and reclaim-pause attribution inside the edit-loop measurement laps, and the CI gate on reclaim holds in the edit benchmarks. Both are separate items. This issue adds only the reclaim-side counters it needs.
- The search and read tails under edits that come from the search rerank taking the store write lock and from edits holding their dispatch slot. Those are separate fixes.
- Cold-index WAL volume, and enforcing the 2 GiB ceiling outside the cold window.
- Changing the threshold, the ceiling, the hard cap or the last-resort marks.
Done when
- Unit tests in
internal/graph/store_sqliteshow the following, with an edit burst active (lane busy, or an announced edit) and the WAL above the urgent mark:- (a) an urgent attempt does not take the writer while a write is queued;
- (b) no reclaim writer hold exceeds 50 ms;
- (c)
readGate.quiesceis never called from the pressure reader-drain path or from the step-6 closed-gate path; - (d) the adaptive slice is not taken;
- (e) once the burst ends, the reset goes through on the next attempt.
- A unit test shows that no reset hold and no gate closure starts while an edit acknowledgement or read-back is marked in flight.
- A unit test shows that the hard-cap / last-resort step still bounds the log when edit bursts never leave a gap.
- Measured from the new counters in benchmark runs with 4 and with 8 agents editing files and reading them back (loadedit-4, loadedit-8) on the reference host:
- reclaim writer-hold p95 during edit bursts ≤ 50 ms;
- read-gate closures during edit bursts = 0.
- Edit-load WAL peak ≤ 2 GiB in the same runs (about 1.06 GiB today).
- No regression in loadedit-4 and loadedit-8, compared against a main baseline run on the same host:
- edit acknowledgement p95 is no worse than the baseline;
- p95 of the read right after the edit is no worse than the baseline;
- WAL resets still happen (resets > 0 per run).
-
go test -race ./internal/graph/store_sqlite/...passes.
How to verify
go test -race -run 'WALReclaim|Pressure' ./internal/graph/store_sqlite/runs the new and the existing reclaim tests. The opt-in timed reclaim load tests must also stay green.- Run the edit benchmark with 4 and with 8 agents editing and reading back (loadedit-4, loadedit-8), once on main and once on this branch. At the end of each run, collect the WAL reclaim stats: edit-burst hold p95, edit-burst gate closures, deferrals by cause, urgent attempts, exception holds, resets and WAL peak. Collect edit acknowledgement p95 and read-back p95 from the same runs.
- Check the daemon log for the "last resort" and "bulk completion" reclaim lines. They should appear only when the WAL is over those marks.
Dependencies
Blocked by (must land first):
- Measure the edit loop: per-phase daemon laps, CPU per edit, WAL reclaim cancels, edit_and_probe #1005 — The week-2 WAL check from the edit-loop laps confirms the scope of the reclaim yield.
- Add WAL checkpoint counters and per-stage reclaim pause attribution to the edit-loop timings #1006 — The reclaim yield is sized and verified with the reclaim-pause attribution counters.
Blocks: - Move analysis readers off writeMu and bound edit-path WAL holders #1047 (merges after this)
Related issues
- Addresses: WAL checkpoint path appears to stall on large stores, forcing full re-index far below documented throughput #843 (close those when the targets below pass and the reporters confirm)
- PRs: Improve incremental indexing and resumable contract analysis #806
Planning
- Milestone: v0.64.12
- Priority: latency (first priority)
- Estimate: ~2 engineer-days
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsDone
Activity
zzet commentedon Nov 2, 2026
The WAL source changes landed through #1258 after all 12 hosted test partitions and required checks passed. The merge preserves the 29 qualified WAL-owned files and current main outside that scope.
The reference-host four/eight-editor comparisons and performance acceptance remain unrun/open. This source merge does not close #1007; it remains In Progress.
zzet commentedon Nov 13, 2026
Shared current-product evidence is available from the 4-editor and 8-editor reference-host comparisons. Actual candidate source is
8926a57; the later221a7e5delta is benchmark/test-only relative to its product/module closure.The candidate's timed edit-window WAL peaks were 467.147 MiB (4 editors) and 1022.358 MiB (8 editors), each below 2 GiB. These are the edit-window observations, not a bound on cold-index warmup WAL. Both windows include indexed read-back and actual resets (three per run); the coupled rotating-request method and raw outcomes are linked above.
This supplies the narrow peak measurement for AC5. It does not establish AC4: the edit-burst-specific hold population is empty, so its percentile remains unavailable; the wider all-hold bound is not substituted for burst p95. It also does not establish AC6's immediate-own-ACK read-back regression gate: this driver waits for the paired ACK group before its indexed probes. The 8-editor candidate's three caller refusals and the tag failures remain genuine incomplete outcomes, with no acceptance waiver. Issue and remaining criteria stay open.
zzet commentedon Nov 13, 2026
Checked only AC5 (timed edit-window WAL peak) from the shared 4/8-editor evidence: 467.147 / 1022.358 MiB, both below 2 GiB, on actual candidate
8926a57and the admitted reference profile. This is the edit-window peak, not a cold-index warmup bound.All other criteria and the issue remain open. Empty edit-burst hold populations still have unavailable percentiles; they are not replaced by all-hold bounds. The paired driver does not prove the immediate-own-ACK read-back gate. Candidate/tag incomplete outcomes and original raw errors remain unchanged; no further pass or whole-issue closure is inferred.
zzet commentedon Nov 13, 2026
Checked only AC1, AC2 and AC7 against main commit
38427f06:AC3 stays unchecked: the ordinary hard-cap test proves admission/reset/spacing over a fixed WAL under a held lane, not the continuously growing-WAL bound. The separate opt-in continuous-churn tests are not credited from default CI. AC4 and AC6 remain unchecked; no empty-histogram p95 or immediate own-ACK comparison is adopted. Existing AC5 is unchanged, and the issue remains open.
zzet commentedon Nov 13, 2026
Checked AC3 after running the existing opt-in permanent-announcement continuous-WAL test. This adds the previously missing growing-WAL evidence to the default held-lane admission/reset proof.
The single named test passed (2.11 s; package 4.852 s) with
make test,-tags=performance,-raceand the repository's SQLite/libc instrumentation exclusions. Actual checkout was29e83330; the store sources are byte-identical to main38427f06. Execution was native Darwin/ARM64 with Go 1.27.3 and safe CGO flags.The fixture keeps an actual write announcement active while a continuous writer and two rotating readers run. Its threshold/ceiling/hard cap are 4/8/32 MiB. Observed peak was 45.2 MiB ≤ 64 MiB, with 3 hard-cap runs, 3 resets and 2 refusals. Maximum writer hold was 259.567 ms in the hard-cap exception class.
This is scaled unit evidence only. It gives no ordinary-burst 50 ms p95, timed Linux load, production 8 GiB threshold, baseline, or broader opt-in-suite credit. AC4 and AC6 remain unchecked; AC5 and the adopted source/unit rows are unchanged. The issue remains open.
zzet commentedon Nov 13, 2026
AC4 is adopted as a qualified performance guard: no observed edit-burst hold exposure; p95 remains undefined. This is not a numerical p95 pass or a change to the 50 ms bound.
The retained 4- and 8-editor reference runs positively expose the real counters. Their edit-burst hold counts are respectively 11→11 and 8→8, with all 13 histogram buckets, totals and maxima unchanged; read-gate closures are 0→0 in both. These are supported nonzero lifetime histograms, not missing fields or a reset-to-zero result. The same owned daemon PIDs (2016 and 2025), monotone counters, and full snapshot brackets are retained. WAL reset deltas are +3 in each run.
Hold accounting happens on write-gate unlock and has no separate in-flight gauge, so equal snapshots alone were not treated as complete proof. The independent check also inspected the complete, untruncated candidate logs through joined normal shutdown: the last successful resets/shrink completed before the phase-end captures, with no later begun reclaim failure, defer, abandonment or shrink tail. There is no observed hidden tail hold. The previously adopted real-unit policy tests cover the ordinary ≤50 ms bound when a burst hold is used.
Both raw per-window burst p95/max fields remain null/unavailable, never 0 ms. Hard-cap/bulk exceptions are separate: the observed interval deltas are zero; the retained startup exception is not erased or included in the ordinary guard. Generic hold histograms are not substituted for burst histograms.
The original coupled reports remain unchanged, including the 8-editor candidate's three read errors and the tag errors. This does not certify clean workload success, independent capacity, causal latency attribution, or the immediate-own-ACK comparison in AC6. AC6 stays unchecked and the issue stays open.
zzet commentedon Nov 14, 2026
The approved AC6 comparison unit is the whole matched transaction: edit → that build’s own ACK → immediate file read → source read. Build order rotates at the transaction boundary; the first build’s transaction finishes before the next build receives the same logical edit. The 4/8 editor streams remain concurrent.
ACK and individual file/source RPC latencies remain separate measurements. Existing deadlines, correctness oracles, errors, refusals, retries, phase-end censors and no-regression limits remain unchanged; censored or failed transactions are not silently dropped or counted as successful matches. The opt-in method and workload identity will be explicit in the report.
This records the approved method only. Implementation, validation and matched measurements are still pending; AC6 remains unchecked and this issue remains open.
zzet commentedon Nov 16, 2026
The 4-editor portion of the remaining latency criterion now has an authenticated, unfiltered measurement. The combined 4/8 checkbox remains unchecked because EDIT8 has not run.
This paired run used the retained historical main
4ebc561, frozen candidate3e7838e, the same reference host/profile, and private LOAD66865ee1with the declared lossless evidence method. It is an as-tested comparison, not a claim about PR #1480 or current moving main.Each build contributes all 100 acknowledgement and immediate file-read measurements, with 100 successes, no missing/error/censored rows, and no rows filtered by later source-query or diagnostic outcomes. The same coupled measurement lasted 510.051945 s; reset deltas are joined to the same respective daemon PIDs and before/after window.
These are the three literal numerical subpredicates for EDIT4 only. The broader original LOAD run still exited 1 and its stronger whole-report qualifier rejected it: candidate source queries included three safe pending refusals and one explicitly pending reply without source; one baseline source reply had stale context/range metadata. Additional phase/edit-proof deadline and diagnostic failures remain recorded. This note does not turn that run into a full-method or correctness PASS, waive those outcomes, or qualify EDIT8.
Evidence:
LITERAL-C6-EDIT4-ALL100-METRICS.json, SHA-2566d2cfc3257df7bca3b55b438226a0ffeb1263a991afd70e8b96eabb89e3d5a87; original-report disposition05bbfa1230130fd153faad375abb0eb82ad47ed1b41f10620aea6d97da1ee231; source-body/order dispositionf0b1177c37f5c1d51a81e3745ccca85eafccce58602a4550febaa2e3aab1ac95. Original report and five selected spools were recovered read-only and authenticated; no workload was rerun for this table.zzet commentedon Nov 16, 2026
C6 is now accepted with an explicit conservative rank-bound qualification, and all seven existing Done-when criteria are addressed. This issue is complete; this does not declare the broader release or the paired harness's stronger whole-method checks complete.
EDIT4 has all 100 successful ACK and immediate-file observations per build. EDIT8 has all 100 successful candidate ACK and file observations. Its historical baseline has 99 successful ACKs and 94 successful file observations, with the remaining failed, censored or absent slots retained. The exact all-100 baseline p95 is unavailable and is not estimated here.
The EDIT8 inequalities nevertheless follow from actual completed-success witnesses: six distinct baseline ACKs are at least 67590.363025 ms, and six distinct baseline file reads are at least 72200.603880 ms. For nearest rank 95 of 100 planned slots, these six fixed observations force the 95th value to be no smaller than those bounds, regardless of how fast any missing values could be. No zero/Infinity values are inserted, no rows are filtered to manufacture a baseline p95, and no failed/censored elapsed value is used as a completed-success witness. The candidate p95s are below both conservative bounds. All successful file calls used in the proof follow their own ACK; reset deltas use stable per-arm PIDs and measured before/after windows.
The comparison uses the retained historical main
4ebc561, frozen candidate3e7838e, private LOAD66865ee1, and the same reference host/profile. EDIT8 used a fresh same-60-GiB scratch allocation; no IO-equivalence across allocations or current-main/PR #1480 result is claimed. Existing data was preserved.Original LOAD exit 1/incomplete outcomes, whole-method qualifier rejections, source-read availability/correctness labels, phase diagnostics and baseline errors remain recorded separately. Accepting the stated ACK/file/reset inequalities does not turn those outcomes into a whole-method or source-correctness PASS, or credit #1005/#1010. The literal C6 adds no complete-baseline/source/P/B requirement; its numerical limits and denominators are unchanged.
Evidence: EDIT4 metrics SHA-256
6d2cfc3257df7bca3b55b438226a0ffeb1263a991afd70e8b96eabb89e3d5a87; EDIT8 authenticated disposition5612083f72b592366b613e0257190f27693cd270ce92d19d58f6a9392e9d942e; rank proofe2f10ed37ebebfb81081bbd3937156cd9951ccf9063ab6634363192af89d335a; original-record joine378266ed18c425b1d92b1f90fbc5c9179d01e01d3ca6dbc787183e5c8244607; independent review certificate72dc8da52f7c35377c0bed4910766ebdb3a296dafa0c6db5f8a80325d9994ed8.