Repository navigation
Conversation
moetemp
force-pushed
the
moe/brook/review/streams-k14-readers
branch
from
October 28, 2026 18:45
45029f7 to
91b73d5
Compare
moetemp
force-pushed
the
moe/brook/review/streams-k15-buffers
branch
2 times, most recently
from
October 28, 2026 18:59
6e9fe8f to
2aa1859
Compare
moetemp
marked this pull request as draft
October 28, 2026 21:55
moetemp
force-pushed
the
moe/brook/review/streams-k15-buffers
branch
from
October 28, 2026 22:42
2aa1859 to
8bf74f3
Compare
moetemp
force-pushed
the
moe/brook/review/streams-k14-readers
branch
from
October 28, 2026 22:42
91b73d5 to
b4a1e0d
Compare
moetemp
force-pushed
the
moe/brook/review/streams-k15-buffers
branch
from
October 29, 2026 04:06
8bf74f3 to
515ba9c
Compare
moetemp
force-pushed
the
moe/brook/review/streams-k14-readers
branch
from
October 29, 2026 04:06
b4a1e0d to
1965e18
Compare
moetemp
force-pushed
the
moe/brook/review/streams-k15-buffers
branch
from
October 29, 2026 08:37
515ba9c to
aac18ed
Compare
moetemp
force-pushed
the
moe/brook/review/streams-k14-readers
branch
2 times, most recently
from
October 29, 2026 09:17
4bade9b to
b457a88
Compare
moetemp
force-pushed
the
moe/brook/review/streams-k15-buffers
branch
from
October 29, 2026 09:17
aac18ed to
07f6bec
Compare
The batching rules are the same in every SDK, so the runtime keeps them. A codec runs once per batch, so lang runs a small pump around `BufferNextBatch` and `BufferWrite` when it has one.
moetemp
force-pushed
the
moe/brook/review/streams-k15-buffers
branch
from
October 29, 2026 10:03
07f6bec to
ebb26f8
Compare
Owner
Author
|
Parked. Nothing calls these runtime methods yet, since sdk-python does this work in Python. The branch stays, and this layer comes back with its first caller, for example a TypeScript or .NET port or Rust read handles. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
This PR adds buffered publishers. The runtime keeps the batching rules, and lang keeps only its codec, which runs once per batch.
max_delayafter its first value, at once when it hasmax_batchvalues, or at once for a flush. One batch is in flight at a time. After a failed batch, the buffer writes nothing more, so a reader sees no gap.encoded, and the runtime writes each batch itself on a Tokio task.BufferNextBatch, runs the codec, and gives the batch back withBufferWrite, or the codec's error.BufferPushreturns at once, and fails with the buffer's error after a failed batch.BufferFlushwaits for every value pushed before it, and answers with the last offset, the unwritten count and the failure.CloseScopeandShutdownflush and free the scope's buffers first. They report a buffer failure that lang didn't get yet. DuringShutdown, lang's pump calls still run. A push during the scope's close fails, so the close drops no value.BufferWritewrites on a task of its own, so a dropped call doesn't leave its batch in flight. A batch that a dropped pump took but never wrote comes again on the nextBufferNextBatch. Records of another count than the batch fail the call and the buffer. A freed buffer gives its pumpdone, and counts a taken batch as unwritten.tokiodependency takes themacrosfeature, forselect!.Depends on #224 (K14). Review lane is the Rust reviewers.
Part of the Brook review series.
Why?
BufferedPublisherhas subtle timer, batch and failure rules, and every SDK needs the same ones. A remote codec costs a round trip per call, so the codec stays in lang and runs once per batch.How did you test it?
cargo fmt,cargo clippy -- -D warnings(with and without features),cargo test -p temporalio-streams --all-featuresandcargo docpass at this commit. The buffer suite runs on the memory Store here, and on Redis in R4 (temporal-community/sdk-streams#4).