Skip to content

feat: add temporalio-streams, the Stream runtime that SDK bridges link - #1799

Draft
moetemp wants to merge 6 commits into
moe/streams-crate-protosfrom
moe/streams-crate
Draft

moetemp wants to merge 6 commits into
moe/streams-crate-protosfrom
moe/streams-crate

Conversation

@moetemp

@moetemp moetemp commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

What was changed

This PR adds the temporalio-streams crate. It runs the five operations of the Stream service (create, append, read, close and describe) on a Stream Store, for the SDK bridges. It is experimental, and Core's Worker gets no Stream code.

How a call flows

An SDK generates its Streams API from the Stream WIT, and hands each call to its bridge as an operation name and an encoded request. The bridge calls StreamRuntime::call(operation, request) and gets the encoded response. That call goes to one typed method for each operation. The method applies the rules that every Store shares, and then calls the Store.

What is in the crate

Files What they have
store.rs, store/lazy.rs The Store trait and its capabilities. LazyStore connects a Store on its first call, so a bridge can build the runtime with no async runtime.
runtime.rs, read.rs, dispatch.rs StreamRuntime with its typed methods, the long-poll read, and the byte form of the calls.
epoch.rs, ids.rs, offset.rs, record.rs, digest.rs The parts that every Store shares. They are the order of epochs, the id checks, the opaque offsets, the stored form of a Record and the digest of a batch.
memory.rs A memory Store for tests. It has the key layout and the rule order of the Redis Store.
error.rs, metrics.rs StreamError with its reason and gRPC code, and the metrics of the runtime.
testing/ The shared suites, behind the test-support feature. Each Store crate runs them on its own Store.
testdata/, examples/gen_vectors.rs The shared vectors. A second implementation writes them, and the tests read them.

Contract

  • A higher epoch of a Producer fences a lower one.
  • A batch with a sequence is written once. The same sequence with other content fails.
  • A Workflow Producer can't append, since a Workflow appends through its Local Activity.
  • Stream ids and Producer ids are opaque text. The runtime refuses only an empty id and one over 4096 bytes.

Why?

Without this crate, each SDK would write these rules again in its own language. With it, the Python, TypeScript and .NET bridges link the same code, and every Store passes the same suite.

The base is the branch of #1733, which adds the Stream protos and the WIT. The Redis Store is a crate of its own, in temporal-community/sdk-streams#8.

Checklist

  1. How was this tested:

    • cargo test -p temporalio-streams runs the unit tests, the shared suites on the memory Store, and the tests of the shared vectors.
  2. Any docs updates needed?

    • No. The crate has its own README.md.

A Stream id and a Producer id are opaque bytes, since the caller builds both. The Store requires a
non-empty Producer id of at most 4096 bytes, and a kind.
A Store crate runs the suite on its own Store through `testing::TestStore`. The suite checks that a
Producer id is stored as bytes, also with a field separator in it.
A bridge calls `StreamRuntime::call` with an operation name and the encoded request. The read path,
the metrics and the shared service suite come with it, and the Store name is on each ref it builds.
The fence of a Producer is kept by kind and id, so the id of a run-based kind must hold its run.
The shared suites now cover two runs, two kinds with one id, any text in an id and a close of each
kind, so every Store keeps these rules.
The repo has no Python, so the script is now an example of the Stream crate. It is still a second
implementation, and it shares no code with the modules that the vector tests check.
The crate is for the maintainers of the language SDKs, so its entry is lang-internal.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant