Skip to content

[Brook] Added buffered writers for Activities and clients. - #21

Closed
moetemp wants to merge 8 commits into
moe/brook/v3/s4-workflow-flushfrom
moe/brook/v3/s5-writers
Closed

moetemp wants to merge 8 commits into
moe/brook/v3/s4-workflow-flushfrom
moe/brook/v3/s5-writers

Conversation

@moetemp

@moetemp moetemp commented Oct 21, 2026

Copy link
Copy Markdown
Owner

What changed?

This PR adds buffered writers for Activities and clients: OpenWriter, Publish (synchronous), FlushWriter and CloseWriter. A writer appends every batch_interval (100 ms), at once on force_flush, in batches of at most max_batch_size Records and the Store's batch bytes. A Record over the Record limit fails its publish. A reopen of a closed writer goes on from its last sequence. It retries a Store failure with the same sequence for up to max_retry_duration. A full buffer fails a publish with stream_buffer_full, and a flush that failed for good fails every later call. A writer of an Activity attempt above 1 first sends an empty batch.

Why?

A publish must return at once, and one Store call per Record costs too much. The empty batch lets a retried attempt withdraw the earlier attempt's Records, also when it publishes nothing.

How did you test it?

STREAMS_REQUIRE=redis cargo test -p temporalio-streams --test writer, on standalone and cluster Redis.

  • Unit Tests
  • Staging
  • End to End Tests

A publish from an Activity or a client must return at once, so the crate buffers it and appends one batch per interval. The writer keeps the batch sequence of its epoch, which keeps a resend after a Store failure from going in twice.
A writer batch over the Store's limit failed for good and lost the buffer, and a reopen of a closed writer sent a used sequence again, which dropped its Records as resends.
A writer that lang never closed kept its flusher task until the process ended, so a shutdown now closes them all.
@moetemp

moetemp commented Oct 25, 2026

Copy link
Copy Markdown
Owner Author

Closing as stale. This layer is folded into the v5 tip, #41, and the upstream review uses a fresh series instead. Plan: https://www.notion.so/4048fc567738814990b7c796ac28fe3f

@moetemp moetemp closed this Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant