Repository navigation
Keep log forwarding running when one client or worker on a runtime is dropped - #1998
Open
parbhatkapila4 wants to merge 3 commits into
Open
parbhatkapila4 wants to merge 3 commits into
parbhatkapila4 wants to merge 3 commits into
Conversation
Contributor
Author
|
The CI timeouts were real, sorry about that. Keeping the forwarder alive until the last clone exposed a deadlock at teardown: Python drops the last runtime clone while holding the GIL, Tokio's shutdown waits for its tasks, and the forwarder was one of them, waiting for the GIL. I caught it with gdb on a hung run. The second commit shuts Tokio down with the GIL released. test_runtime.py hung in 5 of 10 runs before it and 0 of 60 after. |
Contributor
|
Oooh, not loving a use of unsafe to do this though. |
Contributor
Author
|
Fair. Pushed a version without it: the Core runtime now lives in a small wrapper whose Drop releases the GIL, so dropping the last Arc shuts Tokio down with the GIL released. No ManualDrop and no unsafe. 40 runs of test_runtime.py with no hang. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1988
Clients and workers each hold a clone of the bridge
Runtime, and every clone shares the log forwarder task.Drop for Runtimeaborted that task whenever any clone was dropped, so dropping one short-lived client, or a worker after shutdown, stopped log forwarding for everything else on the runtime with no error.Now the forwarder is aborted only when the last clone is dropped.
Arc::into_innergives the handle to exactly one clone, so this holds even when two clones drop at the same time.Keeping the forwarder alive exposed a deadlock: when Python drops the last clone it holds the GIL while Tokio shuts down and waits for its tasks, and the forwarder can be one of them, waiting for the GIL. The last clone now shuts Tokio down with the GIL released.
Tested:
tests/test_runtime.pydrops a client, then checks that a Core log still gets forwarded. It fails on main and passes with the fix.tests/test_runtime.pyhung at teardown in 5 of 10 runs with only the first commit, and in 0 of 40 runs with the final version.cargo clippy -- -D warningsandcargo fmt --checkare clean.