Repository navigation
Retry a busy replica for a few tens of milliseconds before skipping a heartbeat - #685
matrixarkai wants to merge 2 commits into
Conversation
… heartbeat The heartbeat takes the replica lock with try_lock, which a request thread holds while it applies a write. When it was busy the report failed and the next try was a whole heartbeat interval away (3 s by default), so under write load a node could skip report after report and, with a TTL of a few intervals, drop out of every group. The report now waits in 30 ms steps for up to five tries (stopping at once on shutdown) before it gives up on the interval. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Measurement under sustained load (run after the tests, on a shared box): Release binaries of main (b9a1835) and of this change (same code as 4ebd8b2), the rig's bring-up defaults (chunk heartbeat 3000 ms, heartbeat TTL 15000 ms, 64 gateway threads, compact append), 8 concurrent appenders of 1 KiB on 8 objects for 120 s, alternated, my own ports.
Reading: the retry did not reduce the number of heartbeat failures (23 and 26 on main, 31 and 31 with the change, out of about 120 heartbeats per run), and the 503 share moved from 6.0% and 9.8% to 4.5% and 4.6%, which is inside the noise of loads from 12 to 22. So under this load the replica lock is held for longer than the 150 ms the retry covers on roughly one try in five; a wait of a few tens of milliseconds does not take the heartbeat off that lock. This change is correct and cheap but it is not the fix for the health lapses; the heartbeat has to stop depending on the replica lock (a health snapshot readable without it). |
The guard counts the tests of cluster::server::heartbeat::tests; the busy replica test added for the quick retry is one of them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
What
The chunk server's heartbeat takes the replica lock with
try_lock, which a request thread holds while it applies a write. When the lock was busy the report failed ("replica health is busy applying state") and the next try was a whole heartbeat interval away (3 s by default). Under sustained write load a node could skip report after report, and with a TTL of a few intervals it dropped out of every group: the rig sawchunk heartbeat failed ... busy applying state/recoveredalternating continuously under load.The report now waits in 30 ms steps for up to five tries (
lock_for_report,BUSY_RETRY_STEP,BUSY_TRIES), stopping at once on shutdown, before it gives up on the interval. The report needs the replica for a moment only (used bytes, capacity, online flag, the journal health check, and for dynamic members the applied membership), so a short wait gets it.Tests
a_busy_replica_delays_the_report_by_the_wait_and_not_by_a_whole_interval: the replica lock is held for 60 ms when the first report is due, with a heartbeat interval of 30 s; the node's first heartbeat must land within 3 s.busy_replica_state_does_not_block_heartbeat_shutdown(existing) still passes: shutdown is not delayed by the wait.cargo check --all-targetswith-D warnings.Not in this change
🤖 Generated with Claude Code