Skip to content

Keep log forwarding running when one client or worker on a runtime is dropped - #1998

Open
parbhatkapila4 wants to merge 3 commits into
temporalio:mainfrom
parbhatkapila4:fix/log-forwarding-runtime-drop
Open

parbhatkapila4 wants to merge 3 commits into
temporalio:mainfrom
parbhatkapila4:fix/log-forwarding-runtime-drop

Conversation

@parbhatkapila4

@parbhatkapila4 parbhatkapila4 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1988

Clients and workers each hold a clone of the bridge Runtime, and every clone shares the log forwarder task. Drop for Runtime aborted that task whenever any clone was dropped, so dropping one short-lived client, or a worker after shutdown, stopped log forwarding for everything else on the runtime with no error.

Now the forwarder is aborted only when the last clone is dropped. Arc::into_inner gives the handle to exactly one clone, so this holds even when two clones drop at the same time.

Keeping the forwarder alive exposed a deadlock: when Python drops the last clone it holds the GIL while Tokio shuts down and waits for its tasks, and the forwarder can be one of them, waiting for the GIL. The last clone now shuts Tokio down with the GIL released.

Tested:

  • New test in tests/test_runtime.py drops a client, then checks that a Core log still gets forwarded. It fails on main and passes with the fix.
  • Dropping a worker after shutdown had the same effect on main and is fixed too.
  • tests/test_runtime.py hung at teardown in 5 of 10 runs with only the first commit, and in 0 of 40 runs with the final version.
  • cargo clippy -- -D warnings and cargo fmt --check are clean.

@parbhatkapila4
parbhatkapila4 requested a review from a team as a code owner October 21, 2026 09:01
@parbhatkapila4

Copy link
Copy Markdown
Contributor Author

The CI timeouts were real, sorry about that. Keeping the forwarder alive until the last clone exposed a deadlock at teardown: Python drops the last runtime clone while holding the GIL, Tokio's shutdown waits for its tasks, and the forwarder was one of them, waiting for the GIL. I caught it with gdb on a hung run. The second commit shuts Tokio down with the GIL released. test_runtime.py hung in 5 of 10 runs before it and 0 of 60 after.

@tconley1428

Copy link
Copy Markdown
Contributor

Oooh, not loving a use of unsafe to do this though.

@parbhatkapila4

Copy link
Copy Markdown
Contributor Author

Fair. Pushed a version without it: the Core runtime now lives in a small wrapper whose Drop releases the GIL, so dropping the last Arc shuts Tokio down with the GIL released. No ManualDrop and no unsafe. 40 runs of test_runtime.py with no hang.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Dropping any client stops log forwarding for every other client of the same Runtime

2 participants