Skip to content

[Brook] Made ContribStream the default event stream and kept Workflow Streams available. - #6

Closed
moetemp wants to merge 66 commits into
mainfrom
moe/brook/v3/h1-streams
Closed

moetemp wants to merge 66 commits into
mainfrom
moe/brook/v3/h1-streams

Conversation

@moetemp

@moetemp moetemp commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

What changed?

This PR makes ContribStream (temporalio.contrib.streams) the default event stream of AgentWorkflowRunner and keeps Workflow Streams available with stream=WorkflowStream().

  • Upstream main (dbc89c93) is merged in, so the code is on the python/ and typescript/ layout with Bazel and @agent.init.
  • With no stream=, an agent tree shares one Stream, each agent on a Topic of its own. stream=WorkflowStream() works as upstream does, and the examples pass it.
  • The default needs the temporalio build that just install-streams-sdk installs (604ee9f3, [Brook] Built the Stream runtime per client and closed after the run. sdk-python#361). An agent on stream=WorkflowStream() runs on a released SDK.
  • A subagent publishes where its parent does, so one tree never mixes the two backends.
  • A subagent's answer comes from new send_subagent_message and await_message updates on the child, on both backends, so a lost event can't lose it.
  • The README says how to choose, with the trade-offs and the Temporal Cloud Actions of each backend.

The wire format changes in three places.

  • SSE frames and /api/attach use an opaque resume string. from_offset= still works, and a point the store lacks is a 400 with invalid_resume_point.
  • AgentEvent.seq, AgentStatus.stream, AgentStatus.last_event_seq, AgentConfig.stream and the attempt_superseded event are new. The schema, protocol.ts and the goldens are regenerated.
  • The Nexus IDL adds streamCursor and next_cursor for a Stream agent. A Workflow Streams poll is unchanged.

Why?

Jason expected Workflow Streams as the default, and this branch uses Brook instead. Both backends stay supported behind one stream= choice, and the clients read either one from the place the agent reports. The PR also fixes the failure paths that the reviews of the earlier heads found, and an upstream bug where a failed subagent turn never closed its subagent_message_sent bracket.

Known limits

  • The Slack and Teams connectors show both attempts of a retried model call on a Stream, since the router protocol has no event that takes sent text back.
  • A session from this PR's earlier heads doesn't replay on this code, so drain or terminate it first. Open Workflow Streams sessions from upstream main do replay.
  • Upgrade the workers before the clients, since a client of this version reads seq fields that an older worker doesn't send.

How did you test it?

  • Unit Tests: the Python suite on the default ContribStream with the memory and Redis stores, and with stream=WorkflowStream(), on the just install-streams-sdk build. A replay of a parent and subagent history recorded on upstream main. bazel test //... and Gazelle on the released temporalio. The TypeScript and Go tests.
  • Staging
  • End to End Tests

The tree of PR #4 at ee24991, so the v3 changes below read as their own diff.
The Brook v3 streams keep every Stream in Redis, so the suite needs one.
Every agent of a tree publishes on its root's Stream, on a Topic of its own, so one read
gives the tree in order and the client-side merge is gone. The child writes the parent's
subagent_message_sent marker as it admits the message, so the brackets hold by construction.
The suite defaults to the memory Stream plugin, so it needs no Redis.
The subagent memo carries the ref in its proto JSON form, so it no longer needs the root
workflow id beside it.
The SDK StreamRef dumps its proto JSON names inside a pydantic model, so the harness no
longer needs a model of its own for it.
@moetemp moetemp changed the title [Brook] Moved the harness to the Brook v3 streams API. [Brook] Moved the harness to the Brook v4 streams API. Oct 22, 2026
Sync Workflow code uses publish_nowait now. A root whose run never completes, such as a terminated one, closes its Stream after a day with no event, so its readers end. Handles take [Any] again and the client uses workflow_stream.
The runner's constructor can't await, and p26 now has a sync create, so the run-to-end helper and its test are gone.
A Stream offset passes what a JavaScript number holds, so the console and the client package key a withdrawn attempt's records by their string offsets and drop exactly those. The console reads a session again from the beginning when the server refuses its resume point.
The console starts over on invalid_resume_point, and a send's reply carries the point to attach from.
The token rode the activity input as Any, so it came back as a dict and the Stream plugin found no Stream to withdraw on when a retry started.
The SDK's error names no harness setup, and the dependency comment now says which SDK build ContribStream needs.
The README compares them, Cloud Actions included, and the console's API doc has the opaque resume points.
The wheel test imports every module on a released SDK, where temporalio.contrib.streams does not exist.
A Stream agent's poll item has no integer offset, which the IDL now says.
An open session on the default backend keeps running after a deploy, so the test keeps one parent and subagent history that upstream main recorded.
A web CLI test sets TEMPORAL_ADDRESS in the test process, and later time-skipping starts then connected to that host.
A git pin in the lock makes Bazel build temporalio from source, which it can't, so the lock keeps the release.
A from_offset or a send's resume carries no seq, so the seq check alone sent such an attach to wait for the next turn, where upstream compared the stream head.
@moetemp moetemp changed the title [Brook] Moved the harness to the Brook v4 streams API. [Brook] Kept Workflow Streams as the default and added ContribStream as a second event stream. Oct 23, 2026
An agent that passes no `stream=` now uses `temporalio.contrib.streams`, and
`stream=WorkflowStream()` runs as upstream does. A subagent of a Workflow Streams
parent stays on Workflow Streams, so one tree never mixes the two.
A close still drains the open turn, so `await_message` waits for that turn. Every waiter
of a message gets its result, a closed agent waits a short time for an unread one, and
continue-as-new passes unread results on. `send_subagent_message` checks the exact Topics.
A Nexus poll or a reconnect is a new read, which saw none of the withdrawn records, so
it reads the Topic before its cursor at a `Superseded` item. Only a workflow that is
gone ends a read, a read gives the newest records before it ends, and a Nexus batch
stays inside its timeout.
@moetemp moetemp changed the title [Brook] Kept Workflow Streams as the default and added ContribStream as a second event stream. [Brook] Made ContribStream the default event stream and kept Workflow Streams available. Oct 23, 2026
Another waiter can take the result between the validator and the handler. A subagent
starts with no unread results of its parent's run.
A cursor says where the read agent's turn started, so a read looks back no further than
that for what a retry withdraws. A cursor past a marker skips its `Superseded` item. A
Nexus poll of a closed agent whose query fails goes to the Workflow Streams path.
Its `Superseded` item shares an offset with the retry's first record, and both clients
tell frames apart by offset, so they dropped that record as a repeat of the marker.
A cursor starts the look-back at the first record of the oldest turn still open. A
Nexus poll with a stream cursor of a closed agent whose query fails reads the root's
Stream, which the workflow id and first run give.
@moetemp

moetemp commented Oct 25, 2026

Copy link
Copy Markdown
Owner Author

Closing as stale. This layer is folded into the v5 tip, #7, and the upstream review uses one fresh PR instead. Plan: https://www.notion.so/4048fc567738814990b7c796ac28fe3f

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant