Skip to content

[Brook] Added the Stream WIT and removed the buffered Publish path. - #31

Closed
moetemp wants to merge 2 commits into
moe/brook/v4/s14-awaited-publishfrom
moe/brook/v4/s15-wit-docs-and-no-interval
Closed

moetemp wants to merge 2 commits into
moe/brook/v4/s14-awaited-publishfrom
moe/brook/v4/s15-wit-docs-and-no-interval

Conversation

@moetemp

@moetemp moetemp commented Oct 22, 2026

Copy link
Copy Markdown
Owner

What changed?

This PR adds the Stream service WIT to the crate with docs that fit 100 columns, and removes the buffered Publish path of writers.
crates/streams/nexus/stream-service.wit is now the source copy, taken from the sdk-python copy at 9f9e3f017. Each SDK copies it. The eleven @nexus.doc lines over 100 columns are shorter, with the same meaning. Two use the continuation form that nexgen documents (/// @nexus.doc then /// "<text>"). The header points at the proto. A new test keeps every line at most 100 columns and checks the five operations.
Removed from the extension:

  • the sync call Publish with PublishRequest and PublishResponse;
  • the async call FlushWriter with FlushWriterRequest and FlushWriterResponse;
  • the interval flush task and the writer buffer;
  • OpenWriterRequest.batch_interval (4) and max_buffered_records (7), now reserved by number and name;
  • the stream_buffer_full reason (Reason::BufferFull) and the stream_buffer_full metric.
    OpenWriterRequest.max_batch_size (5) and max_retry_duration (6) stay, since WriteRecords splits by the first and retries by the second. CloseWriter waits for a write in flight, then drops the writer and answers its last offset, or the error it failed with. CloseWriters drops every writer.

Why?

Lang no longer uses the interval batching path. Python publishes with WriteRecords, and BufferedPublisher buffers in Python. The WIT docs had lines over the 100-column limit.

How did you test it?

cargo fmt --all -- --check, cargo clippy --workspace --tests -- -D warnings (also with --no-default-features --features tls-aws-lc), and STREAMS_REQUIRE=redis cargo test --workspace on standalone Redis, a 3-primary Redis cluster and the memory Store.
The writer tests now use WriteRecords only, for order, batch split, a failed write, an Activity retry, close, reopen and CloseWriters. A new test checks that Publish and FlushWriter are unknown methods.

@moetemp

moetemp commented Oct 25, 2026

Copy link
Copy Markdown
Owner Author

Closing as stale. This layer is folded into the v5 tip, #41, and the upstream review uses a fresh series instead. Plan: https://www.notion.so/4048fc567738814990b7c796ac28fe3f

@moetemp moetemp closed this Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant