Skip to content

feat(raft): multi-record conditional publishes on Raft shards (#2751) - #2906

Merged
el-bri-ga merged 5 commits into
mainfrom
feat/raft-cond-batches-on-2751
Nov 21, 2026
Merged

el-bri-ga merged 5 commits into
mainfrom
feat/raft-cond-batches-on-2751

Conversation

@el-bri-ga

@el-bri-ga el-bri-ga commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2751

Turns on conditional publishes of several records on Raft shards. The storage (#2880), codec (#2892) and read paths (#2893) are on main; this reports raft_conditional_batches, finalizes it on a new cluster, opens the lone-broker gate, and adds the tests the last two issue comments asked for.

Changes

  • raft_conditional_batches joins IMPLEMENTED and the control plane's NEW_CLUSTER (18 -> 19). Classified FailsClosed in every_feature_is_safe_to_read_as_off_before_registration: off, a batch of several is refused by name.
  • node/startup.rs: a lone broker takes such batches from the start, like the other Raft gates.
  • Docs: the refusal is gone from docs/per-shard-raft.md ("Conditional publishes"), docs/protocol.md, docs/semantics.md, the pubsub feature page and the per-shard Raft status row. A refused batch consumes one offset per record. Changelog fragment added.

Tests

Test What it checks Revert check
broker/publish/proposed/tests.rs a_refused_conditional_batch_releases_every_offset A blanked batch of 3, through propose_publish_at and the release task, moves next_seq from 1 to 4 and keeps the ring; the next record lands at 4 Fails (next_seq 2) with HeldBatch::blank(first_offset, 1) in proposed.rs, and with pass_blank(batch.first_offset, 1) in completion.rs
serving/quic/streams/tests/raft_publish_if.rs a_publish_if_batch_to_a_raft_shard_is_written_or_refused_whole Through the control stream on a Raft shard: a batch of 3 at the tail is written at the tail, a batch of 2 at the same offset is refused with tail + 5, and a subscriber gets exactly the 3 records n/a (new behaviour)
felix-cluster/tests/raft_shards.rs a_conditional_batch_on_a_raft_shard_is_read_whole_or_not_at_all Real brokers: written whole, refused whole, then batches of 4 stream to the leader back to back and it is killed after three land; on every survivor each batch is all 4 in order or none, and an acknowledged batch missing fails Runs in this PR's "Cluster tests (Raft shards)" job

Design calls

  • Mixed versions keep the gate closed. An older build does not report raft_conditional_batches. check_finalize refuses to finalize a feature any serving broker lacks, and admit refuses a broker that lacks an enabled feature, so the gate opens only once every member reports it and stays sound after. A seeded but unadopted gate reads it as off, which refuses (FailsClosed).
  • No prerequisite features. Pieces need storage format v9 records and Raft entry tag 10, both covered by this gate; a one-record publish_if stays ungated as before.
  • Leader change mid-batch. The deterministic cut-prefix cases are the engine tests from feat(raft): readers never see part of a conditional batch (#2751) #2893 (a_leader_change_that_cuts_a_batch_leaves_no_bound_inside_it, a_lapsed_lease_mid_batch_keeps_readers_out_of_it). The cluster test races a kill against a batch in flight, so which outcome it hits varies by run; it asserts the invariant that holds in all of them.
  • Lone broker opens the gate. It has no older member to refuse a piece, and every read path applies the whole-batch rule since feat(raft): readers never see part of a conditional batch (#2751) #2893.

Spec-Unaffected: the read rule is unchanged; FelixRaftCondBatch.cfg already models batches of several records, and raft_conditional_batches is a fleet gate (commit cc8d296).

…cluster (#2751)

A conditional publish of several records on a Raft shard is taken once the
fleet finalizes raft_conditional_batches, and on a lone broker from the start.
Tests: a batch through the control stream is written or refused whole, and
across real brokers a leader killed as a batch lands leaves all of it or none.
…#2751)

Spec-Unaffected: the read rule is unchanged; FelixRaftCond already models batches of several records (FelixRaftCondBatch.cfg), and raft_conditional_batches is a fleet gate.
Document raft_conditional_batches in the upgrade guide and the reported
feature list, correct publish_at's doc, and make the cluster test kill the
leader while batches stream so a batch is likely in flight, retrying the
first batch at whatever tail a refusal names.
@el-bri-ga
el-bri-ga enabled auto-merge (squash) November 21, 2026 16:44
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Performance comparison (fast subset)

❔ 3 large change(s) in the worse direction these trials can neither confirm nor rule out

Welch's t-test, p < 0.05, minimum practical effect 3.0%. A worse change of 10.0% or more that is not significant is inconclusive: rerun the cell with more trials before trusting it. Baseline and candidate trials alternated on the same runner instance in this job, with the side that goes first flipping each round, to control for ambient noise. This check is advisory only and does not block merging.

Config Metric Baseline (median) Candidate (median) Change p-value CV base/cand
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B p50_us 151.0 us 151.0 us +0.0% 0.4008 0%/4% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B p99_us 204.0 us 202.5 us -0.7% 0.3750 1%/76% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B p999_us 224.0 us 233.0 us +4.0% 0.3449 5%/100% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B throughput 6576.2 msg/s 6598.0 msg/s +0.3% 0.4868 0%/3% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B delivered_throughput 6576.2 msg/s 6598.0 msg/s +0.3% 0.4868 0%/3% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B p50_us 150.5 us 151.5 us +0.7% 0.3412 1%/2% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B p99_us 204.0 us 204.5 us +0.2% 0.4662 3%/8% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B p999_us 310.5 us 278.5 us -10.3% 0.3253 45%/18% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B throughput 6604.2 msg/s 6557.4 msg/s -0.7% 0.2779 1%/2% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B delivered_throughput 6604.2 msg/s 6557.4 msg/s -0.7% 0.2779 1%/2% ✅
balanced/P8_hash fanout=1 batch=64 payload=1024B p50_us 31702.5 us 43988.0 us +38.8% 0.5077 31%/49% ❔ inconclusive
balanced/P8_hash fanout=1 batch=64 payload=1024B p99_us 76902.5 us 97221.0 us +26.4% 0.3152 29%/35% ❔ inconclusive
balanced/P8_hash fanout=1 batch=64 payload=1024B p999_us 78880.0 us 102746.0 us +30.3% 0.3422 29%/34% ❔ inconclusive
balanced/P8_hash fanout=1 batch=64 payload=1024B throughput 358755.1 msg/s 350831.8 msg/s -2.2% 0.1986 2%/3% ✅
balanced/P8_hash fanout=1 batch=64 payload=1024B delivered_throughput 358755.1 msg/s 350831.8 msg/s -2.2% 0.1986 2%/3% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B p50_us 199.0 us 199.5 us +0.3% 0.5265 2%/1% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B p99_us 393.5 us 390.0 us -0.9% 0.3694 12%/4% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B p999_us 1039.5 us 541.5 us -47.9% 0.3802 61%/80% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B throughput 5037.0 msg/s 5003.5 msg/s -0.7% 0.6447 2%/1% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B delivered_throughput 50369.7 msg/s 50034.7 msg/s -0.7% 0.6447 2%/1% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B p50_us 714496.0 us 724140.0 us +1.3% 0.5447 3%/6% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B p99_us 1055794.0 us 1047404.5 us -0.8% 0.2204 7%/7% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B p999_us 1058431.5 us 1069985.5 us +1.1% 0.2862 7%/6% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B throughput 80426.4 msg/s 81528.9 msg/s +1.4% 0.2308 1%/1% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B delivered_throughput 804264.3 msg/s 815289.2 msg/s +1.4% 0.2308 1%/1% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B p50_us 54457.0 us 43640.0 us -19.9% 0.6069 21%/49% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B p99_us 103997.0 us 95945.0 us -7.7% 0.9417 43%/52% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B p999_us 106898.5 us 98093.0 us -8.2% 0.9424 42%/52% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B throughput 1513.7 msg/s 1527.3 msg/s +0.9% 0.3019 2%/2% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B delivered_throughput 15136.7 msg/s 15273.0 msg/s +0.9% 0.3020 2%/2% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B p50_us 1048.5 us 1033.0 us -1.5% 0.0110 1%/1% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B p99_us 2853.0 us 2702.0 us -5.3% 0.1515 7%/8% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B p999_us 4097.5 us 3946.5 us -3.7% 0.1811 6%/6% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B throughput 1257.8 msg/s 1264.6 msg/s +0.5% 0.2597 1%/1% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B delivered_throughput 125785.0 msg/s 126460.1 msg/s +0.5% 0.2595 1%/1% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B p50_us 151.0 us 151.0 us +0.0% 0.4734 1%/1% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B p99_us 202.5 us 201.5 us -0.5% 1.0000 1%/2% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B p999_us 229.0 us 238.0 us +3.9% 0.9169 13%/9% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B throughput 6570.4 msg/s 6584.8 msg/s +0.2% 0.4748 1%/1% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B delivered_throughput 6570.4 msg/s 6584.8 msg/s +0.2% 0.4748 1%/1% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B p50_us 1057.0 us 1049.0 us -0.8% 0.0678 1%/1% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B p99_us 3021.0 us 3003.0 us -0.6% 0.7487 7%/8% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B p999_us 4279.0 us 4160.5 us -2.8% 0.7381 5%/5% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B throughput 1243.5 msg/s 1256.3 msg/s +1.0% 0.0483 1%/1% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B delivered_throughput 124347.2 msg/s 125633.5 msg/s +1.0% 0.0483 1%/1% ✅

@sonarqubecloud

Copy link
Copy Markdown

@el-bri-ga
el-bri-ga merged commit ebf7361 into main Nov 21, 2026
35 checks passed
@el-bri-ga
el-bri-ga deleted the feat/raft-cond-batches-on-2751 branch November 21, 2026 17:46
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

raft: conditional publishes of more than one record on a Raft shard

1 participant