Repository navigation
Pub/Sub: don't orphan a subscribe that races the last unsubscribe of the same channel - #3363
Open
aradalvand wants to merge 1 commit into
Open
aradalvand wants to merge 1 commit into
aradalvand wants to merge 1 commit into
Conversation
…the same channel Subscribing (looking up the channel's subscription, then adding the handler) and unsubscribing the last handler (removing it, removing the now-empty subscription from the multiplexer, then unsubscribing from the server) were not atomic with respect to each other. Interleaved, the handler could be added to a subscription that was being removed: nothing routes messages to it anymore, and the server is told to stop sending them; yet the subscribe completes successfully, so the handler silently never fires. A per-subscription registration lock now makes adding a handler and retiring the subscription mutually exclusive; a subscribe that finds the subscription retired looks up a fresh one instead. The retiring side also issues the unsubscribe command to the server (and only then removes the subscription from the multiplexer) while holding the lock, so that the racing subscribe's command is issued after it, rather than being able to overtake it and leave the server unsubscribed while the client considers itself subscribed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XUNs4qcAQaGCRpkwmr8LnY
aradalvand
marked this pull request as ready for review
November 19, 2026 17:47
Author
|
Some additional evidence, in case it helps the review: I hit both variants through a higher-level usage pattern, not just the minimal repro in #3362, and measured the effect of this branch. Setup
Results
Every miss had one of two fingerprints. They match the two variants described in this PR:
The second variant reproduces on the released 3.4.20 and comes in bursts. Across three 1,500-iteration runs, it accounted for 41 of 42 misses in one run, none in another, and the bulk of a third. In the 1,800 iterations against Two more observations:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3362.
Subscribing a handler, and unsubscribing a channel's last handler, each take several steps that weren't atomic with respect to each other. When they interleave on the same channel, the new handler can be added to a
Subscriptionthat is concurrently being removed. Nothing routes messages to it anymore, and the server has been told to stop sending them.SubscribeAsyncstill completes successfully, so the handler silently never fires.Changes
Registration lock.
Subscriptiongets a registration lock and a "retired" flag.TryAdd) and retiring the subscription after its last handler is removed (RetireIfEmpty) are now mutually exclusive.ConnectionMultiplexer.AddToSubscription).Ordering on the wire. While still holding that lock, the retiring side first issues the unsubscribe command to the server, and only then removes the subscription from the multiplexer.
Multi-node unsubscribe.
MultiNodeSubscription.UnsubscribeFromServerAsyncnow issues the command to every server before awaiting any of them, so that all of them are issued while the lock is held.Unsubscribe-all.
UnsubscribeAll/UnsubscribeAllAsyncgo through the same retire-then-remove path.Things worth a reviewer's eye
Unsubscribeholds the registration lock while it waits for the server's reply. The lock is per subscription, so this only holds up concurrent subscribe/unsubscribe calls on the same channel. None of the reply-processing paths (TrackSubscriptionsProcessor, message dispatch,EnsureSubscriptions,ResubscribeToServer) take that lock, so it can't deadlock. The async paths only issue the command under the lock; they don't await it there.Subscription.cs,RedisSubscriber.cs), but the two address different races and should compose. Fix duplicate Pub/Sub subscription restoration after Sentinel failover #3337's restoration logic lives insideEnsureSubscribedToServer*, while this change is about whichSubscriptiona handler is added to, and the order of unsubscribe versus subscribe.Testing
SubscriptionRegistrationRaceUnitTests. It uses the in-process test server: 300 attempts at unsubscribing a channel's last handler while another handler subscribes to the same channel, then publishing and checking that it arrives.main, it consistently misses some messages.*UnitTestsclasses pass (1221), and all*PubSub*classes pass against a local primary/replica (91 passed, 93 skipped, needing cluster or sentinel). Both onnet10.0.Checklist