Skip to content

test(cluster): wait for a settled Raft leader before the Kafka epoch-bump test sends - #2908

Open
el-bri-ga wants to merge 1 commit into
mainfrom
test/kafka-epoch-leader-2907
Open

el-bri-ga wants to merge 1 commit into
mainfrom
test/kafka-epoch-leader-2907

Conversation

@el-bri-ga

Copy link
Copy Markdown
Collaborator

Fixes #2907

leader_among in the failures suite could name the group-start leader before the transfer to placement's leader. It now requires the same single broker to lead on two reads 2 s apart, within a bounded wait. send_ok retries retriable errors, and the first send's result is asserted. Test only.

…bump test sends (#2907)

leader_among now names a leader only once the same broker still leads two seconds later, so the group-start leader is not picked. send_ok retries retriable errors and the first send's result is asserted.

Spec-Unaffected: test only.
@el-bri-ga
el-bri-ga enabled auto-merge (squash) November 21, 2026 16:54
@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

Performance comparison (fast subset)

❔ 3 large change(s) in the worse direction these trials can neither confirm nor rule out

Welch's t-test, p < 0.05, minimum practical effect 3.0%. A worse change of 10.0% or more that is not significant is inconclusive: rerun the cell with more trials before trusting it. Baseline and candidate trials alternated on the same runner instance in this job, with the side that goes first flipping each round, to control for ambient noise. This check is advisory only and does not block merging.

Config Metric Baseline (median) Candidate (median) Change p-value CV base/cand
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B p50_us 72.0 us 71.5 us -0.7% 0.6280 2%/2% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B p99_us 120.5 us 118.5 us -1.7% 0.3131 2%/3% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B p999_us 141.0 us 137.0 us -2.8% 0.3997 5%/88% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B throughput 14947.5 msg/s 14952.9 msg/s +0.0% 0.6867 1%/1% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B delivered_throughput 14947.5 msg/s 14952.9 msg/s +0.0% 0.6867 1%/1% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B p50_us 69.5 us 70.5 us +1.4% 0.3612 1%/1% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B p99_us 120.5 us 124.0 us +2.9% 0.2691 2%/2% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B p999_us 148.0 us 151.0 us +2.0% 0.3209 41%/67% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B throughput 15154.5 msg/s 14970.4 msg/s -1.2% 0.0561 1%/1% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B delivered_throughput 15154.5 msg/s 14970.4 msg/s -1.2% 0.0561 1%/1% ✅
balanced/P8_hash fanout=1 batch=64 payload=1024B p50_us 80665.5 us 95435.5 us +18.3% 0.5535 22%/40% ❔ inconclusive
balanced/P8_hash fanout=1 batch=64 payload=1024B p99_us 112173.5 us 123922.5 us +10.5% 0.3179 19%/21% ❔ inconclusive
balanced/P8_hash fanout=1 batch=64 payload=1024B p999_us 112592.5 us 125959.5 us +11.9% 0.3082 19%/22% ❔ inconclusive
balanced/P8_hash fanout=1 batch=64 payload=1024B throughput 529968.1 msg/s 505575.4 msg/s -4.6% 0.2584 6%/6% ✅
balanced/P8_hash fanout=1 batch=64 payload=1024B delivered_throughput 529968.1 msg/s 505575.4 msg/s -4.6% 0.2584 6%/6% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B p50_us 87.0 us 86.5 us -0.6% 0.1226 1%/1% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B p99_us 186.5 us 184.0 us -1.3% 0.2105 15%/2% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B p999_us 1330.5 us 287.0 us -78.4% 0.0433 69%/14% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B throughput 11964.9 msg/s 12011.2 msg/s +0.4% 0.2287 2%/1% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B delivered_throughput 119649.1 msg/s 120112.2 msg/s +0.4% 0.2287 2%/1% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B p50_us 435991.5 us 447154.0 us +2.6% 0.8174 6%/7% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B p99_us 667154.0 us 663112.0 us -0.6% 0.9894 5%/5% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B p999_us 671805.5 us 681635.0 us +1.5% 0.5916 4%/6% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B throughput 124958.0 msg/s 124170.1 msg/s -0.6% 0.4913 1%/1% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B delivered_throughput 1249580.2 msg/s 1241701.3 msg/s -0.6% 0.4913 1%/1% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B p50_us 52606.5 us 33361.5 us -36.6% 0.1272 87%/21% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B p99_us 176489.0 us 131044.0 us -25.7% 0.1306 66%/34% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B p999_us 179547.0 us 133728.5 us -25.5% 0.1228 64%/34% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B throughput 2337.4 msg/s 2328.7 msg/s -0.4% 0.7093 2%/2% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B delivered_throughput 23374.4 msg/s 23287.0 msg/s -0.4% 0.7092 2%/2% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B p50_us 596.0 us 590.0 us -1.0% 0.1699 1%/1% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B p99_us 1587.0 us 1618.5 us +2.0% 0.7120 11%/9% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B p999_us 2529.0 us 2596.0 us +2.6% 0.4872 10%/9% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B throughput 2164.6 msg/s 2183.2 msg/s +0.9% 0.0124 0%/1% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B delivered_throughput 216461.3 msg/s 218320.5 msg/s +0.9% 0.0124 0%/1% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B p50_us 71.0 us 71.5 us +0.7% 0.6215 2%/1% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B p99_us 121.5 us 119.5 us -1.6% 0.4681 2%/1% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B p999_us 138.5 us 137.5 us -0.7% 0.9222 4%/4% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B throughput 15052.6 msg/s 15005.6 msg/s -0.3% 0.8085 1%/1% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B delivered_throughput 15052.6 msg/s 15005.6 msg/s -0.3% 0.8085 1%/1% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B p50_us 586.0 us 589.0 us +0.5% 0.5845 1%/1% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B p99_us 1772.5 us 1665.0 us -6.1% 0.5876 12%/7% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B p999_us 2699.0 us 2777.5 us +2.9% 0.4248 10%/7% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B throughput 2191.6 msg/s 2194.4 msg/s +0.1% 0.9194 1%/0% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B delivered_throughput 219155.7 msg/s 219443.6 msg/s +0.1% 0.9190 1%/0% ✅

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(cluster): kafka epoch-bump failover test can pick a bootstrap leader and fail before the kill

1 participant