Repository navigation
Answer a placement with no healthy group as 503 and keep the heartbeat TTL above one interval - #682
Merged
Conversation
The 412 the gateway answers when every attempt of an append was refused as a lost race did not carry the refusal, so a failure that was not a race (for example a bound) looked like one. The message now ends with the last refusal. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…eartbeat TTL above one interval When every chunk server has lapsed (one late heartbeat expires a node when the TTL is under two heartbeat intervals) the metadata plane refuses a write with root placement could not satisfy replica count. The gateway took that PreconditionFailed for a lost race, retried it at once and answered 412 the object kept changing under this append, which told the client its offsets were wrong. It is the cluster being briefly unable: a retryable 503 with the reason, and no immediate retry. The bring-up script set the heartbeat TTL to 5000 ms against the default 3000 ms interval; the default is now 15000 ms, five intervals. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A sequential writer appending 1 KiB at a time through the gateway got "412 the object kept changing under this append; gave up after several attempts" in bursts, with the object size equal to the offset sent, so no race. The cause was not a race:
(the last refusal: precondition failed: precondition failed: root placement could not satisfy replica count)(services/meta/placement/object_placement.rs): the metadata plane found no healthy data group.MATRIXOBJECT_METASERVER_HEARTBEAT_TTL_MS=5000while chunk servers heartbeat every 3000 ms (the default). A TTL under two intervals expires a node on one late heartbeat; the chunk logs showchunk heartbeat ... failed: replica health is busy applying state. Until the next heartbeat there is no healthy group and placement refuses every write. The metaserver's own default TTL is five intervals (15000 ms).PreconditionFailedas a lost race, retried it at once (4 attempts, all refused) and answered 412, which tells the client its offset is wrong.Evidence (release binaries from the first commit, the rig's own bring-up, THREADS=64, one sequential client)
Changes
placement_unavailable(error): aPreconditionFailedcontaining "could not satisfy replica count" is not a lost race (lost_race), anderror_responseanswers it as 503 ServiceUnavailable with the reason. The client retries after a moment; the gateway no longer retries at once.scripts/cluster_gateway_bringup.sh:HEARTBEAT_TTL_MSdefault 15000 (documented), instead of the fixed 5000.Tests
a_placement_with_no_healthy_group_is_a_503_and_not_a_lost_race(gateway with a plane that refuses one append with that error): the append answers 503, nothing is written, and the client's retry at the same offset lands.a_real_lost_race_is_still_a_412.cargo check --all-targetswith-D warnings(default and all features); no pinned--libfilter matches the new names.Not changed / open
replica health is busy applying statemakes a chunk server refuse to report while it is applying, so with a short TTL it expires. A TTL of five intervals tolerates it; whether the chunk server should heartbeat more tolerantly is a separate question.503 publication unconfirmedat the start of each burst (a publish that failed while the cluster was lapsing) is the existing path and resolves by reading.🤖 Generated with Claude Code