Skip to content

Retry a busy replica for a few tens of milliseconds before skipping a heartbeat - #685

Open
matrixarkai wants to merge 2 commits into
mainfrom
fix/heartbeat-retries-a-busy-replica-clustergw
Open

matrixarkai wants to merge 2 commits into
mainfrom
fix/heartbeat-retries-a-busy-replica-clustergw

Conversation

@matrixarkai

Copy link
Copy Markdown
Owner

What

The chunk server's heartbeat takes the replica lock with try_lock, which a request thread holds while it applies a write. When the lock was busy the report failed ("replica health is busy applying state") and the next try was a whole heartbeat interval away (3 s by default). Under sustained write load a node could skip report after report, and with a TTL of a few intervals it dropped out of every group: the rig saw chunk heartbeat failed ... busy applying state / recovered alternating continuously under load.

The report now waits in 30 ms steps for up to five tries (lock_for_report, BUSY_RETRY_STEP, BUSY_TRIES), stopping at once on shutdown, before it gives up on the interval. The report needs the replica for a moment only (used bytes, capacity, online flag, the journal health check, and for dynamic members the applied membership), so a short wait gets it.

Tests

  • a_busy_replica_delays_the_report_by_the_wait_and_not_by_a_whole_interval: the replica lock is held for 60 ms when the first report is due, with a heartbeat interval of 30 s; the node's first heartbeat must land within 3 s.
  • busy_replica_state_does_not_block_heartbeat_shutdown (existing) still passes: shutdown is not delayed by the wait.
  • Planted defects, each makes the first test fail (source restored after every run): a single try (no retry); no wait between tries; two tries (too few for the 60 ms hold).
  • The other heartbeat tests pass (8 in total); cargo check --all-targets with -D warnings.

Not in this change

  • The report still gives up after five tries (150 ms) and waits the interval; a replica busy for longer than that for every report still skips. A lock-free health snapshot would remove the dependency on the lock entirely; that is a separate, larger change.
  • Effect under sustained load is measured separately (8 appenders, before/after) and will be added as a comment on this PR.

🤖 Generated with Claude Code

… heartbeat

The heartbeat takes the replica lock with try_lock, which a request thread holds while it applies a
write. When it was busy the report failed and the next try was a whole heartbeat interval away
(3 s by default), so under write load a node could skip report after report and, with a TTL of
a few intervals, drop out of every group. The report now waits in 30 ms steps for up to five tries
(stopping at once on shutdown) before it gives up on the interval.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@matrixarkai

Copy link
Copy Markdown
Owner Author

Measurement under sustained load (run after the tests, on a shared box):

Release binaries of main (b9a1835) and of this change (same code as 4ebd8b2), the rig's bring-up defaults (chunk heartbeat 3000 ms, heartbeat TTL 15000 ms, 64 gateway threads, compact append), 8 concurrent appenders of 1 KiB on 8 objects for 120 s, alternated, my own ports. load is the 1-minute load average at the end of the run.

run load 200 503 ok/s p50 ms p99 ms chunk heartbeat failures (busy applying)
main 1 12.2 8706 557 73 54 557 23 (19)
fix 1 21.9 6306 298 53 63 1783 31 (28)
main 2 21.6 6865 749 57 49 1289 26 (23)
fix 2 18.5 9331 452 78 51 968 31 (28)

Reading: the retry did not reduce the number of heartbeat failures (23 and 26 on main, 31 and 31 with the change, out of about 120 heartbeats per run), and the 503 share moved from 6.0% and 9.8% to 4.5% and 4.6%, which is inside the noise of loads from 12 to 22. So under this load the replica lock is held for longer than the 150 ms the retry covers on roughly one try in five; a wait of a few tens of milliseconds does not take the heartbeat off that lock. This change is correct and cheap but it is not the fix for the health lapses; the heartbeat has to stop depending on the replica lock (a health snapshot readable without it).

The guard counts the tests of cluster::server::heartbeat::tests; the busy replica test added
for the quick retry is one of them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants