Skip to content

Answer a request no metadata leader admitted as 503 and refuse a heartbeat TTL under three intervals - #684

Merged
matrixarkai merged 1 commit into
mainfrom
fix/placement-unavailable-is-503-clustergw
Nov 18, 2026
Merged

matrixarkai merged 1 commit into
mainfrom
fix/placement-unavailable-is-503-clustergw

Conversation

@matrixarkai

Copy link
Copy Markdown
Owner

Follow-up to #682.

What

  • A PreconditionFailed saying the request "was not admitted by a leader" (an election, or the first seconds of a cluster, which the QPS rig saw right after a cluster came up) took the same lost-race path as the placement refusal fixed in Answer a placement with no healthy group as 503 and keep the heartbeat TTL above one interval #682: the gateway retried it at once and then answered 412 "the object kept changing under this append". It is now a retryable 503 with the reason (leader_unavailable).
  • scripts/cluster_gateway_bringup.sh takes the chunk heartbeat interval as CHUNK_HEARTBEAT_MS (default 200 in this script, 3000 in the product) and stops with an error when HEARTBEAT_TTL_MS is under three intervals. Checked: TTL 5000 with an interval of 3000 stops with "HEARTBEAT_TTL_MS=5000 is under three chunk heartbeat intervals (3000 ms)"; the defaults start.

Tests

  • a_request_no_leader_admitted_is_a_503_and_not_a_lost_race (a plane refusing one append that way: 503, the reason in the body, nothing written, the retry lands).
  • Planted defects, each fails that test (source restored after every run): counted as a lost race again; not mapped to 503.
  • The existing consensus, stale allocation, staged mismatch and placement tests pass; cargo check --all-targets with -D warnings.

Not changed

The chunk server's heartbeat itself (it skips a whole interval when the replica lock is busy) is a separate finding, not touched here.

🤖 Generated with Claude Code

… heartbeat TTL under three intervals

A PreconditionFailed saying the request was not admitted by a leader (an election, or the first
seconds of a cluster) took the lost-race path as well: retried at once, then 412. It is now a
retryable 503 like the placement refusal. The bring-up script takes the chunk heartbeat interval
as a variable and stops when the heartbeat TTL is under three intervals.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@matrixarkai
matrixarkai merged commit b9a1835 into main Nov 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants