Skip to content

Answer a placement with no healthy group as 503 and keep the heartbeat TTL above one interval - #682

Merged
matrixarkai merged 2 commits into
mainfrom
fix/placement-unavailable-is-503-clustergw
Nov 18, 2026
Merged

matrixarkai merged 2 commits into
mainfrom
fix/placement-unavailable-is-503-clustergw

Conversation

@matrixarkai

@matrixarkai matrixarkai commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

What

A sequential writer appending 1 KiB at a time through the gateway got "412 the object kept changing under this append; gave up after several attempts" in bursts, with the object size equal to the offset sent, so no race. The cause was not a race:

  • The 412 now carries the refusal that made the last attempt retry (first commit). On a build with it, the rig's probe printed: (the last refusal: precondition failed: precondition failed: root placement could not satisfy replica count) (services/meta/placement/object_placement.rs): the metadata plane found no healthy data group.
  • Why no healthy group: the bring-up script (copied into the rig) set MATRIXOBJECT_METASERVER_HEARTBEAT_TTL_MS=5000 while chunk servers heartbeat every 3000 ms (the default). A TTL under two intervals expires a node on one late heartbeat; the chunk logs show chunk heartbeat ... failed: replica health is busy applying state. Until the next heartbeat there is no healthy group and placement refuses every write. The metaserver's own default TTL is five intervals (15000 ms).
  • The gateway made it worse: it treated that PreconditionFailed as a lost race, retried it at once (4 attempts, all refused) and answered 412, which tells the client its offset is wrong.

Evidence (release binaries from the first commit, the rig's own bring-up, THREADS=64, one sequential client)

  • TTL 5000 / heartbeat 3000: 2028 appends in 90 s with bursts of 1 to 4 failures every 5 to 24 s (412 "root placement could not satisfy replica count", and a 503 "publication unconfirmed" at the start of a burst), each followed by an immediate recovery. A probe that stops after five failures reads as a permanent wedge.
  • Same cluster, TTL 15000: 1994 appends in 120 s, zero failures, one run of 200s.
  • On my own cluster with the script's 200 ms heartbeat (TTL 5000 is 25 intervals): 3000 appends, no failure.

Changes

  • placement_unavailable(error): a PreconditionFailed containing "could not satisfy replica count" is not a lost race (lost_race), and error_response answers it as 503 ServiceUnavailable with the reason. The client retries after a moment; the gateway no longer retries at once.
  • scripts/cluster_gateway_bringup.sh: HEARTBEAT_TTL_MS default 15000 (documented), instead of the fixed 5000.
  • (First commit) the 412 "kept changing" message ends with the last refusal, so a bound or a cluster condition is no longer reported as a race.

Tests

  • a_placement_with_no_healthy_group_is_a_503_and_not_a_lost_race (gateway with a plane that refuses one append with that error): the append answers 503, nothing is written, and the client's retry at the same offset lands.
  • a_real_lost_race_is_still_a_412.
  • Planted defects, each fails the test, source restored after every run: a placement refusal counted as a lost race again; the answer not mapped to 503; the refusal not recognised.
  • Existing consensus, stale allocation and staged mismatch tests pass; cargo check --all-targets with -D warnings (default and all features); no pinned --lib filter matches the new names.

Not changed / open

  • The heartbeat itself: replica health is busy applying state makes a chunk server refuse to report while it is applying, so with a short TTL it expires. A TTL of five intervals tolerates it; whether the chunk server should heartbeat more tolerantly is a separate question.
  • The 503 publication unconfirmed at the start of each burst (a publish that failed while the cluster was lapsing) is the existing path and resolves by reading.
  • Earlier rig runs with TTL 5000 and the 3000 ms heartbeat include these bursts.

🤖 Generated with Claude Code

superhaiou and others added 2 commits November 17, 2026 22:15
The 412 the gateway answers when every attempt of an append was refused as a lost race did
not carry the refusal, so a failure that was not a race (for example a bound) looked like
one. The message now ends with the last refusal.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…eartbeat TTL above one interval

When every chunk server has lapsed (one late heartbeat expires a node when the TTL is under two
heartbeat intervals) the metadata plane refuses a write with root placement could not satisfy
replica count. The gateway took that PreconditionFailed for a lost race, retried it at once and
answered 412 the object kept changing under this append, which told the client its offsets were
wrong. It is the cluster being briefly unable: a retryable 503 with the reason, and no
immediate retry. The bring-up script set the heartbeat TTL to 5000 ms against the default 3000 ms
interval; the default is now 15000 ms, five intervals.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@matrixarkai
matrixarkai merged commit 2aa2192 into main Nov 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants