Skip to content

fix(auth): end streams and connections when their credentials expire (#2817) - #2900

Merged
el-bri-ga merged 4 commits into
mainfrom
fix/close-on-credential-expiry-2817
Nov 21, 2026
Merged

el-bri-ga merged 4 commits into
mainfrom
fix/close-on-credential-expiry-2817

Conversation

@el-bri-ga

@el-bri-ga el-bri-ga commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2817

Invariant restored: no connection or long-lived stream outlives the earliest expiry of the credentials it was authorized with (token exp + its acceptance leeway, client certificate chain notAfter).

Mechanism

One end time per authenticated credential, not a timer per stream. Revocable already had an end time that only lease tokens set; every waiter on revocation already races Revocable::revoked(). Now:

  • BrokerAuth::authenticate sets the end time for every token: exp + 60 s (the verifier's leeway; lease tokens keep 2 s).
  • authenticate_peer and authenticate_forwarded cap it at the chain's end (Revocable::ending_at keeps the minimum), and refuse a stream opened after the certificate expired.
  • A chain's end comes from felix_common::tls::ChainExpiry: the leaf's notAfter and the intermediates on the path webpki rebuilds to the configured roots (verify_for_usage). The trust anchor and certificates off the path never count. With no path now (an intermediate has since expired, or an unverified forwarded chain) it counts every presented cert that is neither self-signed nor an anchor, which can only close earlier. Never later than the leaf. A leaf x509-parser cannot read refuses the credential (QUIC/peer connections close at once).
  • The QUIC client accept loop and the internal peer transport (both sides) close the whole connection at the chain's end. Peer connections close at a random point in [notAfter - min(30 s, left/4), notAfter].

Every end, stream or connection, waits through sleep_until_unix, which re-checks the wall clock at least once a minute, so a host suspend or a stepped clock cannot keep anything open much past its end.

Entry points

Path Where How it is covered Test
QUIC control stream (publish, cache ops, consumer groups, gateway connections) serving/quic/streams/control.rs:160, :1236; auth in control/session.rs:77 authenticate_peer end time -> revoked arm sends revoked and closes tests/revocation.rs (existing revoke tests use the same arm)
QUIC uni publish streams serving/quic/streams/uni.rs:73, :148 same Revocable same arm as revocation
Subscriptions (incl. replay) quic/handlers/subscribe/feeder.rs:65,92,107; subscribe/replay.rs:364 same Revocable, last frame revoked a_subscription_ends_when_its_token_expires
Cache watches quic/handlers/cache_watch.rs:683 same Revocable existing a_watch_ends_when_its_credential_is_revoked covers the arm
QUIC client connection (cert) serving/quic/conn.rs:303 (client_chain_ends) closes at the chain's end, code 2; unreadable leaf closes at once a_connection_closes_when_its_client_certificate_expires (checks close code 2 on a bare connection, 20 s cert)
Chain end (leaf + verified-path intermediates) felix-common/src/tls/expiry.rs (ChainExpiry::not_after) anchor and off-path certs excluded; leaf always bounds tls::expiry::tests::* (expired root in chain, expired intermediate-as-anchor, stray expired cert, no-path fallback, unreadable leaf/extra)
New stream on a connection whose cert expired serving/auth.rs:154 (bound_by_chain) refused; unreadable leaf refused a_client_certificate_ends_what_its_token_opens_when_it_expires_first, a_client_certificate_that_does_not_parse_is_refused
MQTT ingress serving/mqtt.rs:84 -> felix-mqtt/src/service/connection.rs:88 end time on the wall clock (re-checked sleep_until_unix) -> Disconnect(NotAuthorized) connection_ends_when_the_token_expires (real time)
Kafka listener serving/kafka.rs:133 (Principal::with_end) -> felix-kafka/src/service/connection.rs:96,149 per request, and the connection closes at the end time while idle or inside a long-poll fetch api::tests::sasl::a_connection_closes_when_its_credential_ends
Forwarded requests (internal port, owner side) serving/auth.rs:237 (authenticate_forwarded) token end time + forwarded chain end; unreadable leaf refused when binding, ignored otherwise; per-request serving:: lib tests (client_certs)
Internal peer / Raft connections, listener side felix-replication/src/peer/server.rs:496 closes at the dialer chain's end, spread up to 30 s early, code 5 peer::tests::mtls::a_connection_closes_when_a_certificate_on_it_expires
Internal peer / Raft connections, dialer side felix-replication/src/peer/pool/connection.rs:196 closes at the listener chain's end, spread up to 30 s early same test (dialer_expires = false)
Control-plane REST and Raft HTTP listeners — out of scope: HTTP keep-alive, per-request tokens follow-up #2899

Revert checks: with the token end time back to lease-only, a_subscription_ends_when_its_token_expires fails (8 s timeout). With the connection close removed, a_connection_closes_when_its_client_certificate_expires fails; bound_by_certificate has its own unit test. The peer test fails with either the listener-side or the dialer-side close removed alone. With ChainExpiry::not_after back to the minimum over every presented cert, 5 of the 9 tls::expiry tests fail (expired root, stray cert, intermediate-as-anchor, fallback, on-path intermediate).

Design calls

  • Reuse code revoked for expiry rather than add a new error code; it is what lease-token expiry already sends, and a new code needs negotiation. Docs say it covers expiry.
  • New QUIC close code 2 (credential expired) for the client connection, and 5 on the internal port (0-4 were taken).
  • Leeway: a token ends at exp + 60 s, the same leeway the verifier accepts it under, so the broker never ends a stream on a token it would still accept. Certificates get no leeway (rustls checks notAfter without one).
  • No re-auth on a live stream. The protocol already refuses a second Auth; refresh means a new connection. Docs say so.
  • Forwarded chains shorten the end time before they are verified (and when the owner does not bind). Only ever earlier, so an untrusted chain can only end its own request.
  • Leaf plus verified-path intermediates, rebuilt with webpki against the same roots the handshake used, at the current time. Anchors are never counted (webpki does not check their validity). If no path builds now, fall back to every non-self-signed, non-anchor presented cert: errs toward closing early.
  • Fail closed on an unreadable leaf (x509-parser): refuse the credential, close the connection at once. An unreadable non-leaf cert is ignored. For forwarded chains, refuse only when the owner binds subjects (a non-binding owner ignores the chain by contract).
  • Peer close spread: spread_before draws from felix_common::jitter (seeded under the simulator), up to 30 s and at most a quarter of the time left, so a connection made on a nearly expired cert is not closed instantly. Never after notAfter. Docs say to rotate node certs before notAfter; the Raft peer transport's reload is Raft peer TLS: re-read the certificate and key without a restart #2815.
  • Kafka closes on the credential's end, not only at the next request: Principal::with_end hands the connection Revocable::revoked(), which also fires on revocation, so a revoked idle Kafka connection closes too.
  • A closing Kafka connection waits at most 5 s for its queued answers (credential end, or any reader error), so a client that stops reading cannot hold it open. The anonymous-tenant dev switch has no credential and is never ended; documented.
  • MQTT test runs in real time: the end is wall-clock, which a paused runtime does not move.

Spec-Unaffected: authentication and connection lifetime; no TLA spec models credentials.

…2817)

Every token now ends what it opened at its exp plus the leeway it was
accepted under, through the same Revocable end time lease tokens already
used, so every waiter on revocation (control and uni streams,
subscriptions, replays, cache watches, MQTT, Kafka) ends at expiry with
no new timers per stream. A client certificate's notAfter caps that end
time in authenticate_peer and authenticate_forwarded, and a stream opened
on a connection whose certificate has already expired is refused.

The QUIC client accept loop closes the connection at the certificate's
notAfter (close code 2, "credential expired"), and both sides of an
internal peer connection close it when the other side's certificate
expires (close code 5), so the next dial presents the renewed one.

Spec-Unaffected: authentication and connection lifetime; no TLA spec models credentials.
Stream ends wait on the wall clock (re-checked), like the connection
closes, so a suspended host or a stepped clock cannot move them. The
QUIC test checks the connection close code on a bare connection, so each
certificate bound is guarded alone; a unit test covers the stream cap.
Wider timing windows in the new tests. Kafka reports an expired client
certificate as a refusal, not a transient failure. The auth doc notes
MQTT ends at the bare exp.

Spec-Unaffected: authentication and connection lifetime; no TLA spec models credentials.
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Performance comparison (fast subset)

✅ No statistically significant performance regressions

Welch's t-test, p < 0.05, minimum practical effect 3.0%. A worse change of 10.0% or more that is not significant is inconclusive: rerun the cell with more trials before trusting it. Baseline and candidate trials alternated on the same runner instance in this job, with the side that goes first flipping each round, to control for ambient noise. This check is advisory only and does not block merging.

Config Metric Baseline (median) Candidate (median) Change p-value CV base/cand
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B p50_us 114.0 us 115.0 us +0.9% 0.4046 3%/2% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B p99_us 173.5 us 173.0 us -0.3% 0.6221 3%/2% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B p999_us 200.0 us 196.5 us -1.8% 0.3644 130%/6% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B throughput 9342.6 msg/s 9277.7 msg/s -0.7% 0.5791 3%/2% ✅
balanced/P1_conn_per_sub fanout=1 batch=1 payload=256B delivered_throughput 9342.6 msg/s 9277.7 msg/s -0.7% 0.5791 3%/2% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B p50_us 106.0 us 111.5 us +5.2% 0.1591 8%/2% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B p99_us 167.0 us 163.5 us -2.1% 0.5346 6%/3% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B p999_us 318.0 us 201.5 us -36.6% 0.3005 112%/64% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B throughput 9890.1 msg/s 9489.3 msg/s -4.1% 0.2006 8%/2% ✅
balanced/P1_hash fanout=1 batch=1 payload=256B delivered_throughput 9890.1 msg/s 9489.3 msg/s -4.1% 0.2006 8%/2% ✅
balanced/P8_hash fanout=1 batch=64 payload=1024B p50_us 76052.5 us 75768.5 us -0.4% 0.6675 62%/48% ✅
balanced/P8_hash fanout=1 batch=64 payload=1024B p99_us 121708.0 us 126187.5 us +3.7% 0.8745 25%/26% ✅
balanced/P8_hash fanout=1 batch=64 payload=1024B p999_us 122314.0 us 126388.0 us +3.3% 0.8550 25%/26% ✅
balanced/P8_hash fanout=1 batch=64 payload=1024B throughput 436407.0 msg/s 432806.9 msg/s -0.8% 0.9894 6%/6% ✅
balanced/P8_hash fanout=1 batch=64 payload=1024B delivered_throughput 436407.0 msg/s 432806.9 msg/s -0.8% 0.9894 6%/6% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B p50_us 119.0 us 117.5 us -1.3% 0.1527 1%/1% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B p99_us 253.0 us 249.0 us -1.6% 0.2489 7%/2% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B p999_us 383.5 us 421.0 us +9.8% 0.9536 47%/30% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B throughput 8634.6 msg/s 8742.1 msg/s +1.2% 0.1707 2%/2% ✅
balanced/P1_hash fanout=10 batch=1 payload=256B delivered_throughput 86346.1 msg/s 87420.9 msg/s +1.2% 0.1707 2%/2% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B p50_us 553560.0 us 548393.5 us -0.9% 0.4991 4%/9% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B p99_us 804129.5 us 840732.0 us +4.6% 0.1038 3%/7% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B p999_us 822345.0 us 848591.0 us +3.2% 0.1345 3%/7% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B throughput 102992.9 msg/s 102584.0 msg/s -0.4% 0.3971 1%/3% ✅
balanced/P8_hash fanout=10 batch=64 payload=1024B delivered_throughput 1029928.9 msg/s 1025840.4 msg/s -0.4% 0.3971 1%/3% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B p50_us 41964.5 us 35434.5 us -15.6% 0.3345 115%/25% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B p99_us 91642.5 us 87216.0 us -4.8% 0.4461 99%/31% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B p999_us 100637.0 us 88584.5 us -12.0% 0.4197 95%/30% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B throughput 1899.0 msg/s 1854.5 msg/s -2.3% 0.0402 1%/2% ✅
balanced/P8_hash fanout=10 batch=64 payload=65536B delivered_throughput 18990.2 msg/s 18545.2 msg/s -2.3% 0.0402 1%/2% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B p50_us 716.5 us 722.0 us +0.8% 0.9330 2%/1% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B p99_us 2227.5 us 2168.5 us -2.6% 0.5264 8%/10% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B p999_us 3219.0 us 3359.0 us +4.3% 0.5921 21%/11% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B throughput 1801.6 msg/s 1794.6 msg/s -0.4% 0.8382 3%/1% ✅
balanced/P1_conn_per_sub fanout=100 batch=1 payload=256B delivered_throughput 180156.3 msg/s 179459.3 msg/s -0.4% 0.8382 3%/1% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B p50_us 115.5 us 114.5 us -0.9% 0.3318 3%/2% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B p99_us 170.0 us 170.5 us +0.3% 0.3737 31%/2% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B p999_us 193.0 us 194.0 us +0.5% 0.3788 35%/1% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B throughput 9328.0 msg/s 9327.5 msg/s -0.0% 0.2080 3%/2% ✅
drop_new/P1_conn_per_sub fanout=1 batch=1 payload=256B delivered_throughput 9328.0 msg/s 9327.5 msg/s -0.0% 0.2080 3%/2% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B p50_us 722.0 us 726.0 us +0.6% 0.5422 1%/1% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B p99_us 2174.0 us 2130.5 us -2.0% 0.7788 2%/9% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B p999_us 3243.5 us 3180.0 us -2.0% 0.2968 3%/4% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B throughput 1799.9 msg/s 1783.8 msg/s -0.9% 0.0396 0%/1% ✅
drop_new/P1_conn_per_sub fanout=100 batch=1 payload=256B delivered_throughput 179990.1 msg/s 178385.3 msg/s -0.9% 0.0396 0%/1% ✅

…er closes (#2817)

A chain's end is now the leaf and the intermediates on the path webpki
builds to a trust anchor; the anchor and stray certificates no longer
count. A leaf that does not parse refuses the credential. Peer
connections close at a random time in the last 30 s before notAfter.
MQTT waits on the wall clock, and Kafka closes idle and long-polling
connections when the credential ends.
A client that stops reading could keep a connection open past its
credential's end while the queued answers waited to be written.
@sonarqubecloud

Copy link
Copy Markdown

@el-bri-ga
el-bri-ga merged commit 813b191 into main Nov 21, 2026
35 checks passed
@el-bri-ga
el-bri-ga deleted the fix/close-on-credential-expiry-2817 branch November 21, 2026 17:22
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Close connections and streams when their token or client certificate expires

1 participant