Repository navigation
Conversation
The Streams API moves into temporalio.common.streams, so the module becomes a package with the same contents.
The first-class API lives next to the converter it registers with. The paths of the generators and the generated imports follow.
A Stream's lifecycle is its owner's to manage, so a run end closes nothing. drain() still lets unawaited writes finish before the run ends.
Only the run-end close on Redis used them, along with workflow.Info.execution_expiration_time.
Backends go in stream_backends= of Client.connect, a Worker takes its client's, and the Replayer and WorkflowEnvironment take them too. temporalio.workflow, Client.stream and WorkflowHandle.stream replace StreamClient, and Redis moves to temporalio.contrib.stream.
A read is stream-level only, so Topic objects lose subscribe and pass to the Stream's subscribe as topics. The proto field names stay.
The crate reserves the auto_close fields, so the pb2 and the system bindings lose them. The generators write into the new package.
s23 drops auto_close from the Stores and the protos, and c8 drops the cache-off eviction reason that only the run-end close used.
A Local Activity close outlived a Workflow Task that failed after it, so the retry's create failed the run. The block end now leaves the Stream open in a cron run and on a failure that a retry run follows.
A ClientStream kept a failed or cancelled Describe forever, and the handle map kept a failed one with its handle. A found id now stays as text, and a failed call goes.
A Replayer serves one backend as a Worker does, a Worker configures its backends after its plugins, and a generator of backends stays. A backend in plugins= fails before the connect, and the per-client calls are private.
A Worker serves one backend, so the backend-move recipe can't run. A continued run reads its Streams' backends after @workflow.init.
Redis keeps its regular Activity close, and a backend that closes another way overrides _close_in_workflow. Workers that share a task queue all register the Redis backend, since it polls Activities there.
A Local Activity writes before its Workflow Task commits, so a close there stayed when the task failed, and the task's retry could not create the Stream again.
A run's cancel reaches the block as the error of an Activity, a Local Activity or a child, and no retry run follows a cancelled run.
The block end decided from the exception and the retry policy alone, so it left Streams open when no retry run followed.
A Store scans a limited number of entries per call, so a filtered read ended early after 10,000 entries of other Topics.
The open of a new attempt withdraws earlier Records, and a Stream that takes no write failed the attempt for good before user code ran.
The read path converts Record bodies later, so the failure kept RawValue details and an encoded message.
The decode of a dataclass inside a JSON value reads its fields, so the encoder's transfer type broke the round trip.
An entry left only on success, a rerun under a new epoch added another, and the count bound let up to 2 GiB stay.
The close of a Workflow's Stream is a regular Activity, so such a Worker alone left every close waiting. Another Worker on the queue may run it, so it is a warning.
The create took the expired status as a success, and the error came only at the first publish.
A plain TypeError ended the read with next_offset before the Record, so every resume failed on it again.
The close at the block end always raised StreamNotOwnerError and hid the block's own error.
An empty id failed only at the first call, with a bare AssertionError.
Nothing withdraws a client's Records, so a long UI read grew its withdrawal map with no end.
subscribe takes offset: int | End | None, and nothing in Brook or Jetty calls from_rpc_error.
The key is in History, so a run that never creates, binds or receives a Stream adds nothing there. A Local Activity keeps its key always, since its headers stay out of History.
Retention is a Record's age and trims open Streams, a trimmed offset raises StreamExpiredError, and aclose from another task raises.
The Brook crate and the Jetty server both answer a Record or a batch over the Store's limit with it, so both stacks raise one class.
The crate refuses a Local Activity's close of a Workflow's Stream, so a close by id from one is refused too. The vendored proto and WIT differ in comments only.
The code is in the private temporalio.common._streams, and temporalio.common loads its names on first use, since the Stream modules import temporalio.client and temporalio.workflow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR moves the Streams API into
temporalio.common, with the API of the SDK review and the fixes of an independent review.The SDK review asked for Streams as a first-class SDK API, with native as the default backend. It also asked for backends registered on the client, and an owner that closes each Stream. An earlier commit of this PR put the API in
temporalio.common.streams. It's now intemporalio.commonitself, with the same public names.What changed?
Modules.
temporalio/common.pyis now a package. The Stream code is in the privatetemporalio.common._streams.temporalio.commonloads the Stream names on first use, soimport temporalio.commonloads no Stream code.temporalio.contrib.stream(RedisStreamBackend,create(client, ...)andworkflow.*). The memory backend istemporalio.contrib.stream.testing.MemoryStreamBackend.temporalio.contrib.streamsis gone.Registration and entry points.
Client.connect(..., stream_backends=[...]). A Worker uses the backends of its client, andReplayerandWorkflowEnvironmenttakestream_backends=. With more than one backend, exactly one hasdefault=True. A backend inplugins=is refused.native. Brook has no native backend (Jetty adds it), so a create with no backend raisesStreamConfigError.workflow.create_stream,create_stream_nowaitandstream_handle,Client.stream, andWorkflowHandle.stream.StreamClientis gone.offsetandtopic_prefixes. Only a Stream subscribes, not a Topic.Lifecycle.
auto_closeand noclose_with_run. On Redis, a run end closes nothing.CANCELED.Plugin.on_workflow_evicted,workflow.Info.execution_expiration_timeand the plugin shutdown wait of the Worker are removed.Other review fixes.
StreamExpiredError. A Record or batch over the Store limit raises the newStreamTooLargeError.next_offset. ACloseditem's failure is decoded with the reader's converter. A Record that fails to decode raisesStreamErrorwith its offset, andnext_offsetis the offset after it. The reader stores no offsets of client Records.async withon a client handle of a Workflow's Stream raisesStreamNotOwnerErrorbefore the block runs.Client.streamrefuses an empty id.Endis public. A run adds the Stream key header to its Activities only when it uses Streams.Subscription.aclose, the errors and the eviction match the code.Pins and CI.
ee6d1e51and Core c8f7826366, with regenerated protos and bindings.interceptors. The generated Stream Nexus clients are hidden from the API docs. Upstream's pydantic 2.14 annotation (fix: annotate pydantic TypeAdapter cache for pydantic 2.14 temporalio/sdk-python#2011) is cherry-picked fortest-latest-deps. Changelog fragments inadded/andchanged/describe the new API.Known limits.
no_remote_activities=Truelogs a warning, since another Worker must run the close.Why?
A close must not apply if its Workflow Task fails, so it runs as an Activity. The independent review of Brook and Jetty found that a Local Activity could still close early. It also found that a cancel closed a Stream as failed, that filtered reads could stall, and that an Activity retry failed on a closed Stream.
How did you test it?
Each review fix has a test that fails without it, apart from the docstring fixes.
pytest tests/contrib/streamspasses on memory and Redis. A test checks thatimport temporalio.commonloads notemporalio.common._streamsin a fresh interpreter. The worker, client, plugin, replayer, testing, pydantic and converter suites pass. The upstreamtest_schedule_basicsfailed once on a shared dev server with schedules of other tests, and it passes alone.poe lint(ruff, pyright, mypy, basedpyright and pydocstyle) andpoe gen-docsare clean.On the fork CI, every job passes except
check-protosand the arm64 jobs.check-protoswaits on a nexgen release with temporalio/nexgen#260, which merged after v0.2.10. The arm64 jobs have no runner on the fork.The superproject branch
temporalio/internal-ai-prototypes2026/10/ai198-streamspins the head,8f71f3ee. sdk-python j9 and the harness use the newtemporalio.commonnames.