Skip to content

fix: Fix three deadlocks with coexisting sessions - #971

Merged
rphmeier merged 3 commits into
thrumdev:masterfrom
citizen-stig:nikolai/session-deadlock-fixes
Nov 5, 2026
Merged

rphmeier merged 3 commits into
thrumdev:masterfrom
citizen-stig:nikolai/session-deadlock-fixes

Conversation

@citizen-stig

@citizen-stig citizen-stig commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

begin_session documents that multiple sessions may coexist, but three patterns of coexisting sessions deadlock. Each commit fixes one and adds a regression test that hangs on master.

Found by #970

1. Rollback: a third live session can't finish

Each session's reverse-delta worker runs until the session is finished or dropped, but it ran on the 2-thread rollback-worker pool. With three live sessions the third worker never starts, and that session's finish blocks forever in ReverseDeltaBuilder::finalize.

Fix: the worker runs on a dedicated thread (task::spawn_thread_task, the thread counterpart of spawn_task), and the pool is removed.

2. Warm-up: a session can't finish while others are live

With warm_up enabled, a session's warm-up task runs until the session finishes, but it ran on the nomt-commit pool (commit_concurrency threads). Once that many sessions are live, a later session's warm-up never starts and its finish blocks forever joining it. With the default commit_concurrency(1), two sessions are enough.

Fix: the warm-up runs on a dedicated thread, leaving the pool to the update workers.

3. A thread holding a session opens another while a commit is queued

parking_lot's RwLock is fair: once a commit or rollback queues for the write side of access_lock, new readers wait behind it. The commit waits for every live session, so a thread that holds a session and opens a second one (or calls Nomt::read) deadlocks.

Fix: live sessions are counted per creating thread. A thread that already has one takes the read lock recursively, past the queued writer; every other thread still queues behind it. Making every session's read recursive also removes the deadlock, but it lets overlapping sessions from other threads starve commits (numbers below). Known limitation: a session moved to another thread still counts for the thread that created it, so a thread holding a session it didn't create can still deadlock this way.

Cost

master this PR
begin_session → finish → drop, rollback on 36 µs 56 µs
same, warm-up on 40 µs 71 µs
same, neither 29 µs 34 µs (95% CI includes 0)
benchtop transfer throughput (base / rollback / warm-up), n=15 interleaved¹ −1.2% / +1.1% / −0.5%, all 95% CIs include 0
commits/s, 4 threads looping 64-read sessions, every session read recursive (rejected variant)² 626 0.6
same workload, this PR 533 544 (95% CI includes 0; likewise for 1, 2 and 8 threads, with and without reads)

¹ Measured with the rejected recursive-read variant of fix 3. Benchtop opens one session at a time on one thread, so the final version takes the same fair-lock path there as master.
² Separate run, hence the different master baseline.

Testing

  • nomt/tests/coexisting_sessions.rs: many_coexisting_sessions_with_rollback and many_coexisting_sessions_with_warm_up (4 live sessions, finish the newest).
  • nomt/src/lib.rs: session_opens_while_commit_waits_for_another_session (deadlock 3) and session_on_another_thread_waits_for_queued_commit (commits are not starved by sessions on other threads). Both wait for the commit to queue by polling try_read; there are no sleeps.
  • The three deadlock tests hang on master and pass 50/50 here. cargo test -p nomt passes in release and debug, and the loom tests (--cfg loom) pass.

🤖 Generated with Claude Code

citizen-stig and others added 3 commits November 2, 2026 11:11
A session's reverse-delta worker runs until the session is finished or
dropped, but it ran on the 2-thread `rollback-worker` pool. With three
coexisting sessions the third worker never started, and that session's
`finish` blocked forever in `ReverseDeltaBuilder::finalize` waiting on it,
although coexisting sessions are allowed.

Spawn the worker on a dedicated thread instead (`spawn_thread_task`, the
thread counterpart of `spawn_task`) and drop the pool. This costs one
thread spawn per session when rollback is enabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With `warm_up` enabled, a session's warm-up task runs until the session
finishes, but it ran on the `nomt-commit` pool, which has
`commit_concurrency` threads. Once that many sessions coexisted, a later
session's warm-up never started, and its `finish` blocked forever joining
it (with the default `commit_concurrency(1)`, two sessions suffice).

Spawn the warm-up on a dedicated thread, leaving the pool to the update
workers. This costs one thread spawn per session when warm-up is enabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
parking_lot's RwLock is fair: once a commit (or rollback) queues for the
write side of `access_lock`, new readers wait behind it. The commit itself
waits for every live session, so a thread that holds a session and opens
another one (or calls `Nomt::read`) deadlocked whenever a commit was queued
in between.

Track live sessions per creating thread. A thread that has one takes the
read lock recursively, past the queued writer; every other thread still
queues behind it. Taking every session's lock recursively also fixes the
deadlock, but lets overlapping sessions from other threads starve commits:
with 4 threads each opening sessions of 64 reads in a loop, commits fell
from ~630/s to ~0.6/s with multi-second latency.

A session moved to another thread still counts for the thread that created
it, so a thread holding a session it didn't create can still deadlock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@citizen-stig
citizen-stig marked this pull request as ready for review November 2, 2026 11:14
@rphmeier
rphmeier merged commit 0e2dd95 into thrumdev:master Nov 5, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants