Skip to content

[Brook K15] Added buffered publishers with a codec pump. - #225

Closed
moetemp wants to merge 1 commit into
moe/brook/review/streams-k14-readersfrom
moe/brook/review/streams-k15-buffers
Closed

moetemp wants to merge 1 commit into
moe/brook/review/streams-k14-readersfrom
moe/brook/review/streams-k15-buffers

Conversation

@moetemp

@moetemp moetemp commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

What changed?

This PR adds buffered publishers. The runtime keeps the batching rules, and lang keeps only its codec, which runs once per batch.

  • A batch is due max_delay after its first value, at once when it has max_batch values, or at once for a flush. One batch is in flight at a time. After a failed batch, the buffer writes nothing more, so a reader sees no gap.
  • With no codec, lang opens the buffer with encoded, and the runtime writes each batch itself on a Tokio task.
  • With a codec, lang runs a small pump. It takes each batch with BufferNextBatch, runs the codec, and gives the batch back with BufferWrite, or the codec's error.
  • BufferPush returns at once, and fails with the buffer's error after a failed batch. BufferFlush waits for every value pushed before it, and answers with the last offset, the unwritten count and the failure.
  • CloseScope and Shutdown flush and free the scope's buffers first. They report a buffer failure that lang didn't get yet. During Shutdown, lang's pump calls still run. A push during the scope's close fails, so the close drops no value.
  • A BufferWrite writes on a task of its own, so a dropped call doesn't leave its batch in flight. A batch that a dropped pump took but never wrote comes again on the next BufferNextBatch. Records of another count than the batch fail the call and the buffer. A freed buffer gives its pump done, and counts a taken batch as unwritten.
  • A batch that the writer splits and whose later part fails counts only that part as unwritten, and keeps the first part's offset.
  • A write task that ends early, such as by a panic, fails its batch, so a flush still ends.
  • The crate's tokio dependency takes the macros feature, for select!.

Depends on #224 (K14). Review lane is the Rust reviewers.
Part of the Brook review series.

Why?

BufferedPublisher has subtle timer, batch and failure rules, and every SDK needs the same ones. A remote codec costs a round trip per call, so the codec stays in lang and runs once per batch.

How did you test it?

cargo fmt, cargo clippy -- -D warnings (with and without features), cargo test -p temporalio-streams --all-features and cargo doc pass at this commit. The buffer suite runs on the memory Store here, and on Redis in R4 (temporal-community/sdk-streams#4).

@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 28, 2026
@moetemp
moetemp force-pushed the moe/brook/review/streams-k14-readers branch from 45029f7 to 91b73d5 Compare October 28, 2026 18:45
@moetemp
moetemp force-pushed the moe/brook/review/streams-k15-buffers branch 2 times, most recently from 6e9fe8f to 2aa1859 Compare October 28, 2026 18:59
@moetemp
moetemp marked this pull request as draft October 28, 2026 21:55
@moetemp
moetemp force-pushed the moe/brook/review/streams-k15-buffers branch from 2aa1859 to 8bf74f3 Compare October 28, 2026 22:42
@moetemp
moetemp force-pushed the moe/brook/review/streams-k14-readers branch from 91b73d5 to b4a1e0d Compare October 28, 2026 22:42
@moetemp
moetemp force-pushed the moe/brook/review/streams-k15-buffers branch from 8bf74f3 to 515ba9c Compare October 29, 2026 04:06
@moetemp
moetemp force-pushed the moe/brook/review/streams-k14-readers branch from b4a1e0d to 1965e18 Compare October 29, 2026 04:06
@moetemp
moetemp force-pushed the moe/brook/review/streams-k15-buffers branch from 515ba9c to aac18ed Compare October 29, 2026 08:37
@moetemp
moetemp force-pushed the moe/brook/review/streams-k14-readers branch 2 times, most recently from 4bade9b to b457a88 Compare October 29, 2026 09:17
@moetemp
moetemp force-pushed the moe/brook/review/streams-k15-buffers branch from aac18ed to 07f6bec Compare October 29, 2026 09:17
The batching rules are the same in every SDK, so the runtime keeps them. A codec runs once per batch, so lang runs a small pump around `BufferNextBatch` and `BufferWrite` when it has one.
@moetemp
moetemp force-pushed the moe/brook/review/streams-k15-buffers branch from 07f6bec to ebb26f8 Compare October 29, 2026 10:03
@moetemp

moetemp commented Oct 30, 2026

Copy link
Copy Markdown
Owner Author

Parked. Nothing calls these runtime methods yet, since sdk-python does this work in Python. The branch stays, and this layer comes back with its first caller, for example a TypeScript or .NET port or Rust read handles.

@moetemp moetemp closed this Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant