Repository navigation
feat(raft): multi-record conditional publishes on Raft shards (#2751) - #2906
Merged
Merged
Conversation
… the publish path (#2751)
…cluster (#2751) A conditional publish of several records on a Raft shard is taken once the fleet finalizes raft_conditional_batches, and on a lone broker from the start. Tests: a batch through the control stream is written or refused whole, and across real brokers a leader killed as a batch lands leaves all of it or none.
…#2751) Spec-Unaffected: the read rule is unchanged; FelixRaftCond already models batches of several records (FelixRaftCondBatch.cfg), and raft_conditional_batches is a fleet gate.
Document raft_conditional_batches in the upgrade guide and the reported feature list, correct publish_at's doc, and make the cluster test kill the leader while batches stream so a batch is likely in flight, retrying the first batch at whatever tail a refusal names.
… kill (#2751) Spec-Unaffected: test only.
el-bri-ga
enabled auto-merge (squash)
November 21, 2026 16:44
Performance comparison (fast subset)❔ 3 large change(s) in the worse direction these trials can neither confirm nor rule outWelch's t-test, p < 0.05, minimum practical effect 3.0%. A worse change of 10.0% or more that is not significant is inconclusive: rerun the cell with more trials before trusting it. Baseline and candidate trials alternated on the same runner instance in this job, with the side that goes first flipping each round, to control for ambient noise. This check is advisory only and does not block merging.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Fixes #2751
Turns on conditional publishes of several records on Raft shards. The storage (#2880), codec (#2892) and read paths (#2893) are on main; this reports
raft_conditional_batches, finalizes it on a new cluster, opens the lone-broker gate, and adds the tests the last two issue comments asked for.Changes
raft_conditional_batchesjoinsIMPLEMENTEDand the control plane'sNEW_CLUSTER(18 -> 19). ClassifiedFailsClosedinevery_feature_is_safe_to_read_as_off_before_registration: off, a batch of several is refused by name.node/startup.rs: a lone broker takes such batches from the start, like the other Raft gates.docs/per-shard-raft.md("Conditional publishes"),docs/protocol.md,docs/semantics.md, the pubsub feature page and the per-shard Raft status row. A refused batch consumes one offset per record. Changelog fragment added.Tests
broker/publish/proposed/tests.rsa_refused_conditional_batch_releases_every_offsetpropose_publish_atand the release task, movesnext_seqfrom 1 to 4 and keeps the ring; the next record lands at 4next_seq2) withHeldBatch::blank(first_offset, 1)in proposed.rs, and withpass_blank(batch.first_offset, 1)in completion.rsserving/quic/streams/tests/raft_publish_if.rsa_publish_if_batch_to_a_raft_shard_is_written_or_refused_wholefelix-cluster/tests/raft_shards.rsa_conditional_batch_on_a_raft_shard_is_read_whole_or_not_at_allDesign calls
raft_conditional_batches.check_finalizerefuses to finalize a feature any serving broker lacks, andadmitrefuses a broker that lacks an enabled feature, so the gate opens only once every member reports it and stays sound after. A seeded but unadopted gate reads it as off, which refuses (FailsClosed).publish_ifstays ungated as before.a_leader_change_that_cuts_a_batch_leaves_no_bound_inside_it,a_lapsed_lease_mid_batch_keeps_readers_out_of_it). The cluster test races a kill against a batch in flight, so which outcome it hits varies by run; it asserts the invariant that holds in all of them.Spec-Unaffected: the read rule is unchanged;
FelixRaftCondBatch.cfgalready models batches of several records, andraft_conditional_batchesis a fleet gate (commit cc8d296).