Skip to content

Make the WAL reclaim yield to edits and their read-backs during edit bursts #1007

Closed

Description

@zzet

Why

Under edit load the SQLite WAL stays well above the point where the reclaim counts as urgent. In the edit benchmarks (agents editing files and reading them back, with 1, 4 and 8 agents) the WAL peaks at 1,007 / 1,059 / 960 MiB, and at 760 MiB with 6 agents editing 6 repos. Read-only loads stay at 0.3 MiB. Reclaim starts at 256 MiB, and from 512 MiB an attempt counts as urgent. An urgent attempt behaves differently from a normal one:

  • it does not yield to writes already waiting for the writer;
  • it may hold the application writer for up to 2 s in total, where a normal hold is 50 ms;
  • the fallback paths that close the read gate can pause every new read for up to 250 ms while readers drain, and a closed-gate TRUNCATE can add another 250 ms.

During an edit burst, that means the reclaim can add up to 2 s of writer wait to the edit's file-graph and relations updates. It can also add up to 0.5 s of paused reads to the edit acknowledgement and to the read-back of the edited file, which are the two steps an agent waits on. None of these holds or pauses are measured today.

v0.64.10 has no WAL reclaim at all, so we presume this is a regression from the store rewrite in #806. Comparing edit-path tails against v0.64.10 will confirm it or not. The fix ships either way.

Current state

The reclaim already stands down for edits in several places. On main it pauses its copy during an edit and keeps its progress (0555af2), keeps background WAL work out of edit cycles (d55d6f9), and resets the WAL with short 50 ms writer holds instead of long blocks (0174a67). The remaining gaps are the urgent, pressure and closed-gate paths. All files below are in internal/graph/store_sqlite/:

  • wal_reclaim.go constants: defaultWALReclaimThresholdBytes = 256 MiB, walReclaimUrgentFactor = 2, walReclaimMaxWriterHold = 2 s, defaultWALReclaimDrainDeadline = 250 ms and walReclaimTruncateBudget = 250 ms. The ceiling is 8 x threshold (2 GiB). The hard cap (walReclaimHardCapFactor) and the last resort (walReclaimLastResortFactor) both sit at 4 x ceiling (8 GiB).
  • reclaimWALAttempt sets res.urgent from the WAL size. In the open-gate reset loop it admits the reset when res.urgent || !s.writeWanted(). So an urgent attempt takes the writer even while a write is queued for it.
  • reclaimWALInLane skips its "writer waiting" early exit when the attempt is urgent. reclaimWALResetHoldOnce skips the same check. Both set writer.yieldToWriters = !res.urgent and start s.yieldToWriters(...) only when the attempt is not urgent. An urgent hold therefore ignores queued writers.
  • Step 6 of reclaimWALInLane is the closed-gate path: s.readGate.quiesce(...) followed by TRUNCATE. Last-resort and over-ceiling bulk-completion attempts reach it, and so do tests that disable the open-gate stages (walReclaimSkipOpenGate, never set in production). Those holds are allowed up to walReclaimMaxWriterHold.
  • wal_reclaim_pressure.go handles attempts above the pressure mark, or after a writer has asked for a reclaim, and these run inside a busy lane (an edit in progress). When the 50 ms reset (walReclaimPressureHold, aliased as walReclaimResetHold) is blocked by readers, reclaimWALPressureResetWithDrain falls through to reclaimWALPressureResetAfterReaderDrain. That function checks writeWanted() first, but then closes the read gate (readGate.quiesce) for up to defaultWALReclaimDrainDeadline, even while an edit cycle holds the lane. The result is a read-gate closure during an edit burst.
  • wal_reclaim_adaptive.go: takeAdaptiveWriterBudget lets an urgent attempt with a proven slow-copy tail take one adaptive writer slice of up to 2 s. Both the reset-hold path and the pressure path call it, including inside a busy lane.
  • An edit burst can already be detected. buildLaneBusy() (checkpoint_cycle_yield.go) covers a held build lane, and editIntentActive() (wal_reclaim.go) covers an announced edit. Queued writers show up through writeWanted() and AnnounceWrite().
  • The read gate (sqlite_read_gate.go) knows nothing about what a read is for. It cannot tell an edit's acknowledgement or its read-back apart from a long analysis read.
  • WALReclaimStats records WriterHoldMax and AdaptiveWriterHoldMax, and records gate pauses through notePause. None of these are split by whether an edit burst was active, and no p95 can be derived from them.

What to do

  • Urgent attempts yield to edits. While an edit burst is active (buildLaneBusy() or editIntentActive()) or a writer is announced or queued (writeWanted()), an urgent attempt behaves like a non-urgent one. That means it keeps the "writer waiting" early exit, yieldToWriters stays on, and the open-gate reset is admitted only when !writeWanted(). Outside bursts, urgency still shortens spacing and picks the converge-first path. It no longer overrides a waiting edit.

  • Cap writer holds during edit bursts at walReclaimResetHold (50 ms). While a burst is active, refuse the adaptive slow-copy slice (takeAdaptiveWriterBudget) and any completion hold longer than 50 ms, and defer them to the next gap between edits. The only exceptions are the over-ceiling bulk completion (2 GiB) and the hard-cap / last-resort step (8 GiB). Both keep their current logging and are counted as exceptions.

  • No read-gate closure during an edit burst. In that case reclaimWALPressureResetAfterReaderDrain and the step-6 closed-gate path of reclaimWALInLane return a "deferred: edit burst" outcome instead of calling readGate.quiesce. The attempt then waits for a gap, or for the hard cap. Outside edit bursts nothing changes.

  • Yield to an edit's acknowledgement and read-back. Add a cheap store-level marker for these interactive reads, similar to AnnounceWrite: a counter, or a context tag the daemon sets while it serves the edit acknowledgement and the read of the edited file that follows. While such a read is in flight, the reclaim must not start a reset hold or close the gate. Wire the marker from the edit tool path and from the read-back path in the MCP layer.

  • Keep the two yield conditions separable. Yielding to edit tickets and yielding to interactive reads should be distinct conditions in code. If the nightly edit-load WAL goes past 2 GiB, the fallback is to keep only the yield to interactive reads (a small code change, not a new runtime switch) until the edit-path WAL holders are converted.

  • Counters. Add to WALReclaimStats:

    • writer-hold samples taken during edit bursts, enough to compute a p95 (a small histogram or ring buffer);
    • read-gate closures during edit bursts (expected 0);
    • deferrals caused by an edit burst and by an interactive read, counted separately;
    • urgent attempts;
    • exception holds (bulk completion, hard cap / last resort).

    Expose them wherever the existing reclaim stats are already reported.

  • Update the constant comments in wal_reclaim.go and wal_reclaim_pressure.go, which today say an urgent attempt does not yield to writers, so they describe the new contract.

Out of scope

  • Getting the edit-load WAL below 512 MiB so that no urgent reclaim happens at all. That needs the edit-path WAL holders converted, and it is tracked separately.
  • WAL checkpoint counters and reclaim-pause attribution inside the edit-loop measurement laps, and the CI gate on reclaim holds in the edit benchmarks. Both are separate items. This issue adds only the reclaim-side counters it needs.
  • The search and read tails under edits that come from the search rerank taking the store write lock and from edits holding their dispatch slot. Those are separate fixes.
  • Cold-index WAL volume, and enforcing the 2 GiB ceiling outside the cold window.
  • Changing the threshold, the ceiling, the hard cap or the last-resort marks.

Done when

  • Unit tests in internal/graph/store_sqlite show the following, with an edit burst active (lane busy, or an announced edit) and the WAL above the urgent mark:
    • (a) an urgent attempt does not take the writer while a write is queued;
    • (b) no reclaim writer hold exceeds 50 ms;
    • (c) readGate.quiesce is never called from the pressure reader-drain path or from the step-6 closed-gate path;
    • (d) the adaptive slice is not taken;
    • (e) once the burst ends, the reset goes through on the next attempt.
  • A unit test shows that no reset hold and no gate closure starts while an edit acknowledgement or read-back is marked in flight.
  • A unit test shows that the hard-cap / last-resort step still bounds the log when edit bursts never leave a gap.
  • Measured from the new counters in benchmark runs with 4 and with 8 agents editing files and reading them back (loadedit-4, loadedit-8) on the reference host:
    • reclaim writer-hold p95 during edit bursts ≤ 50 ms;
    • read-gate closures during edit bursts = 0.
  • Edit-load WAL peak ≤ 2 GiB in the same runs (about 1.06 GiB today).
  • No regression in loadedit-4 and loadedit-8, compared against a main baseline run on the same host:
    • edit acknowledgement p95 is no worse than the baseline;
    • p95 of the read right after the edit is no worse than the baseline;
    • WAL resets still happen (resets > 0 per run).
  • go test -race ./internal/graph/store_sqlite/... passes.

How to verify

  • go test -race -run 'WALReclaim|Pressure' ./internal/graph/store_sqlite/ runs the new and the existing reclaim tests. The opt-in timed reclaim load tests must also stay green.
  • Run the edit benchmark with 4 and with 8 agents editing and reading back (loadedit-4, loadedit-8), once on main and once on this branch. At the end of each run, collect the WAL reclaim stats: edit-burst hold p95, edit-burst gate closures, deferrals by cause, urgent attempts, exception holds, resets and WAL peak. Collect edit acknowledgement p95 and read-back p95 from the same runs.
  • Check the daemon log for the "last resort" and "bulk completion" reclaim lines. They should appear only when the WAL is over those marks.

Dependencies

Blocked by (must land first):

Related issues

Planning

  • Milestone: v0.64.12
  • Priority: latency (first priority)
  • Estimate: ~2 engineer-days

Activity

  1. added this to the v0.64.12 milestone on Oct 25, 2026
  2. added
    daemonDaemon lifecycle, sessions and repo registry
    editingedit/batch_edit and mutation admission
    performanceLatency, memory and disk cost
    p1: latencyResponsiveness: tool-call, edit-loop, concurrency, restart/stop timing
    on Oct 25, 2026
  3. moved this to Todo in Gortexon Oct 25, 2026
  4. moved this from Todo to In Progress in Gortexon Nov 1, 2026
  5. zzet commented on Nov 2, 2026

    @zzet
    OwnerAuthor

    The WAL source changes landed through #1258 after all 12 hosted test partitions and required checks passed. The merge preserves the 29 qualified WAL-owned files and current main outside that scope.

    The reference-host four/eight-editor comparisons and performance acceptance remain unrun/open. This source merge does not close #1007; it remains In Progress.

  6. zzet commented on Nov 13, 2026

    @zzet
    OwnerAuthor

    Shared current-product evidence is available from the 4-editor and 8-editor reference-host comparisons. Actual candidate source is 8926a57; the later 221a7e5 delta is benchmark/test-only relative to its product/module closure.

    The candidate's timed edit-window WAL peaks were 467.147 MiB (4 editors) and 1022.358 MiB (8 editors), each below 2 GiB. These are the edit-window observations, not a bound on cold-index warmup WAL. Both windows include indexed read-back and actual resets (three per run); the coupled rotating-request method and raw outcomes are linked above.

    This supplies the narrow peak measurement for AC5. It does not establish AC4: the edit-burst-specific hold population is empty, so its percentile remains unavailable; the wider all-hold bound is not substituted for burst p95. It also does not establish AC6's immediate-own-ACK read-back regression gate: this driver waits for the paired ACK group before its indexed probes. The 8-editor candidate's three caller refusals and the tag failures remain genuine incomplete outcomes, with no acceptance waiver. Issue and remaining criteria stay open.

  7. zzet commented on Nov 13, 2026

    @zzet
    OwnerAuthor

    Checked only AC5 (timed edit-window WAL peak) from the shared 4/8-editor evidence: 467.147 / 1022.358 MiB, both below 2 GiB, on actual candidate 8926a57 and the admitted reference profile. This is the edit-window peak, not a cold-index warmup bound.

    All other criteria and the issue remain open. Empty edit-burst hold populations still have unavailable percentiles; they are not replaced by all-hold bounds. The paired driver does not prove the immediate-own-ACK read-back gate. Candidate/tag incomplete outcomes and original raw errors remain unchanged; no further pass or whole-issue closure is inferred.

  8. zzet commented on Nov 13, 2026

    @zzet
    OwnerAuthor

    Checked only AC1, AC2 and AC7 against main commit 38427f06:

    • AC1: the burst-yield tests cover queued-writer refusal despite urgency, the 50 ms nonexception hold, pressure/closed-gate deferral, rejected adaptive completion, and a successful next reset after demand ends. These use real SQLite operations with explicit urgency and simulated lane demand.
    • AC2: the same store tests block reset/gate admission for an active interactive-read marker, including late-arriving markers and exception classes. MCP wiring tests check ACK/read-back marker scope using a spy; they do not measure ACK/read-back latency.
    • AC7: the explicit Linux store partition passed in the full-push CI run, with the repository's standard race settings (transpiled SQLite/libc C excluded from instrumentation).

    AC3 stays unchecked: the ordinary hard-cap test proves admission/reset/spacing over a fixed WAL under a held lane, not the continuously growing-WAL bound. The separate opt-in continuous-churn tests are not credited from default CI. AC4 and AC6 remain unchecked; no empty-histogram p95 or immediate own-ACK comparison is adopted. Existing AC5 is unchanged, and the issue remains open.

  9. zzet commented on Nov 13, 2026

    @zzet
    OwnerAuthor

    Checked AC3 after running the existing opt-in permanent-announcement continuous-WAL test. This adds the previously missing growing-WAL evidence to the default held-lane admission/reset proof.

    The single named test passed (2.11 s; package 4.852 s) with make test, -tags=performance, -race and the repository's SQLite/libc instrumentation exclusions. Actual checkout was 29e83330; the store sources are byte-identical to main 38427f06. Execution was native Darwin/ARM64 with Go 1.27.3 and safe CGO flags.

    The fixture keeps an actual write announcement active while a continuous writer and two rotating readers run. Its threshold/ceiling/hard cap are 4/8/32 MiB. Observed peak was 45.2 MiB ≤ 64 MiB, with 3 hard-cap runs, 3 resets and 2 refusals. Maximum writer hold was 259.567 ms in the hard-cap exception class.

    This is scaled unit evidence only. It gives no ordinary-burst 50 ms p95, timed Linux load, production 8 GiB threshold, baseline, or broader opt-in-suite credit. AC4 and AC6 remain unchecked; AC5 and the adopted source/unit rows are unchanged. The issue remains open.

  10. zzet commented on Nov 13, 2026

    @zzet
    OwnerAuthor

    AC4 is adopted as a qualified performance guard: no observed edit-burst hold exposure; p95 remains undefined. This is not a numerical p95 pass or a change to the 50 ms bound.

    The retained 4- and 8-editor reference runs positively expose the real counters. Their edit-burst hold counts are respectively 11→11 and 8→8, with all 13 histogram buckets, totals and maxima unchanged; read-gate closures are 0→0 in both. These are supported nonzero lifetime histograms, not missing fields or a reset-to-zero result. The same owned daemon PIDs (2016 and 2025), monotone counters, and full snapshot brackets are retained. WAL reset deltas are +3 in each run.

    Hold accounting happens on write-gate unlock and has no separate in-flight gauge, so equal snapshots alone were not treated as complete proof. The independent check also inspected the complete, untruncated candidate logs through joined normal shutdown: the last successful resets/shrink completed before the phase-end captures, with no later begun reclaim failure, defer, abandonment or shrink tail. There is no observed hidden tail hold. The previously adopted real-unit policy tests cover the ordinary ≤50 ms bound when a burst hold is used.

    Both raw per-window burst p95/max fields remain null/unavailable, never 0 ms. Hard-cap/bulk exceptions are separate: the observed interval deltas are zero; the retained startup exception is not erased or included in the ordinary guard. Generic hold histograms are not substituted for burst histograms.

    The original coupled reports remain unchanged, including the 8-editor candidate's three read errors and the tag errors. This does not certify clean workload success, independent capacity, causal latency attribution, or the immediate-own-ACK comparison in AC6. AC6 stays unchecked and the issue stays open.

  11. zzet commented on Nov 14, 2026

    @zzet
    OwnerAuthor

    The approved AC6 comparison unit is the whole matched transaction: edit → that build’s own ACK → immediate file read → source read. Build order rotates at the transaction boundary; the first build’s transaction finishes before the next build receives the same logical edit. The 4/8 editor streams remain concurrent.

    ACK and individual file/source RPC latencies remain separate measurements. Existing deadlines, correctness oracles, errors, refusals, retries, phase-end censors and no-regression limits remain unchanged; censored or failed transactions are not silently dropped or counted as successful matches. The opt-in method and workload identity will be explicit in the report.

    This records the approved method only. Implementation, validation and matched measurements are still pending; AC6 remains unchecked and this issue remains open.

  12. zzet commented on Nov 16, 2026

    @zzet
    OwnerAuthor

    The 4-editor portion of the remaining latency criterion now has an authenticated, unfiltered measurement. The combined 4/8 checkbox remains unchecked because EDIT8 has not run.

    This paired run used the retained historical main 4ebc561, frozen candidate 3e7838e, the same reference host/profile, and private LOAD 66865ee1 with the declared lossless evidence method. It is an as-tested comparison, not a claim about PR #1480 or current moving main.

    Literal C6 measurement Main baseline Candidate Candidate / baseline
    Edit acknowledgement p95 7495.249803 ms 3138.495348 ms 0.4187312539
    Immediate file read-back p95 4392.819474 ms 2389.238255 ms 0.5438962992
    WAL reset delta in the measured window 11 4 —

    Each build contributes all 100 acknowledgement and immediate file-read measurements, with 100 successes, no missing/error/censored rows, and no rows filtered by later source-query or diagnostic outcomes. The same coupled measurement lasted 510.051945 s; reset deltas are joined to the same respective daemon PIDs and before/after window.

    These are the three literal numerical subpredicates for EDIT4 only. The broader original LOAD run still exited 1 and its stronger whole-report qualifier rejected it: candidate source queries included three safe pending refusals and one explicitly pending reply without source; one baseline source reply had stale context/range metadata. Additional phase/edit-proof deadline and diagnostic failures remain recorded. This note does not turn that run into a full-method or correctness PASS, waive those outcomes, or qualify EDIT8.

    Evidence: LITERAL-C6-EDIT4-ALL100-METRICS.json, SHA-256 6d2cfc3257df7bca3b55b438226a0ffeb1263a991afd70e8b96eabb89e3d5a87; original-report disposition 05bbfa1230130fd153faad375abb0eb82ad47ed1b41f10620aea6d97da1ee231; source-body/order disposition f0b1177c37f5c1d51a81e3745ccca85eafccce58602a4550febaa2e3aab1ac95. Original report and five selected spools were recovered read-only and authenticated; no workload was rerun for this table.

  13. zzet commented on Nov 16, 2026

    @zzet
    OwnerAuthor

    C6 is now accepted with an explicit conservative rank-bound qualification, and all seven existing Done-when criteria are addressed. This issue is complete; this does not declare the broader release or the paired harness's stronger whole-method checks complete.

    Reference-host case Candidate ACK p95 Main ACK comparison Candidate immediate-file p95 Main file comparison Resets candidate / main
    4 editors 3138.495348 ms 7495.249803 ms 2389.238255 ms 4392.819474 ms 4 / 11
    8 editors 3074.435269 ms at least 67590.363025 ms 2256.648463 ms at least 72200.603880 ms 1 / 8

    EDIT4 has all 100 successful ACK and immediate-file observations per build. EDIT8 has all 100 successful candidate ACK and file observations. Its historical baseline has 99 successful ACKs and 94 successful file observations, with the remaining failed, censored or absent slots retained. The exact all-100 baseline p95 is unavailable and is not estimated here.

    The EDIT8 inequalities nevertheless follow from actual completed-success witnesses: six distinct baseline ACKs are at least 67590.363025 ms, and six distinct baseline file reads are at least 72200.603880 ms. For nearest rank 95 of 100 planned slots, these six fixed observations force the 95th value to be no smaller than those bounds, regardless of how fast any missing values could be. No zero/Infinity values are inserted, no rows are filtered to manufacture a baseline p95, and no failed/censored elapsed value is used as a completed-success witness. The candidate p95s are below both conservative bounds. All successful file calls used in the proof follow their own ACK; reset deltas use stable per-arm PIDs and measured before/after windows.

    The comparison uses the retained historical main 4ebc561, frozen candidate 3e7838e, private LOAD 66865ee1, and the same reference host/profile. EDIT8 used a fresh same-60-GiB scratch allocation; no IO-equivalence across allocations or current-main/PR #1480 result is claimed. Existing data was preserved.

    Original LOAD exit 1/incomplete outcomes, whole-method qualifier rejections, source-read availability/correctness labels, phase diagnostics and baseline errors remain recorded separately. Accepting the stated ACK/file/reset inequalities does not turn those outcomes into a whole-method or source-correctness PASS, or credit #1005/#1010. The literal C6 adds no complete-baseline/source/P/B requirement; its numerical limits and denominators are unchanged.

    Evidence: EDIT4 metrics SHA-256 6d2cfc3257df7bca3b55b438226a0ffeb1263a991afd70e8b96eabb89e3d5a87; EDIT8 authenticated disposition 5612083f72b592366b613e0257190f27693cd270ce92d19d58f6a9392e9d942e; rank proof e2f10ed37ebebfb81081bbd3937156cd9951ccf9063ab6634363192af89d335a; original-record join e378266ed18c425b1d92b1f90fbc5c9179d01e01d3ca6dbc787183e5c8244607; independent review certificate 72dc8da52f7c35377c0bed4910766ebdb3a296dafa0c6db5f8a80325d9994ed8.

  14. moved this from In Progress to Done in Gortexon Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    daemonDaemon lifecycle, sessions and repo registryeditingedit/batch_edit and mutation admissionp1: latencyResponsiveness: tool-call, edit-loop, concurrency, restart/stop timingperformanceLatency, memory and disk cost

    Projects

    • Status
      Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions