Skip to content

[Brook K8] Served reads with long polls, last-N reads and byte caps. - #217

Closed
moetemp wants to merge 29 commits into
moe/brook/review/streams-k6-service-writesfrom
moe/brook/review/streams-k8-reads
Closed

moetemp wants to merge 29 commits into
moe/brook/review/streams-k6-service-writesfrom
moe/brook/review/streams-k8-reads

Conversation

@moetemp

@moetemp moetemp commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

What changed?

This PR serves ReadRecords, with long polls, last-N reads and byte caps.

  • A read long-polls up to 60 s. A Stream carries one payload type, so a read has no filter.
  • A gap that retention or expiry left fails the read, so a Reader can't miss it. That includes an open Stream's expired log, before the next write and after it.
  • to_offset and wake_only fail with stream_unsupported, since only the native server serves them.
  • An offset past the newest + 1 fails with stream_offset_too_high (OUT_OF_RANGE), as after a failover, and the message says what to do. A negative offset is stream_invalid.
  • With a byte cap, a page reads its entries from the Store in chunks that the bytes left size, so a small cap doesn't read a full page of large Records. A last-N read stops at 10,000 entries or 64 MiB.
  • max_records counts items. A Superseded item and the Record after it stay in one page.
  • A read of a closed Stream is done when it gives the last item. With no item left, it answers at once with done, and a waiting read ends with done at a close.

Depends on #216 (K6), and #218 (K7) is based on it. Review lane is the Rust reviewers.
Part of the Brook review series.

Why?

Reads are about half of the service. The long poll needs the most review time. The Redis read script came with R1, so this PR has no Redis part.

How did you test it?

cargo fmt, cargo clippy -- -D warnings (with and without features), cargo test -p temporalio-streams --all-features and cargo doc pass at this commit. The read cases also run on Redis, through the Redis crate.

@moetemp
moetemp force-pushed the moe/brook/review/streams-k8-reads branch from bbe1e40 to bd7cdf3 Compare October 27, 2026 07:35
@moetemp
moetemp force-pushed the moe/brook/review/streams-k6-service-writes branch 2 times, most recently from 593a974 to a39dd25 Compare October 27, 2026 08:17
@moetemp
moetemp force-pushed the moe/brook/review/streams-k8-reads branch from bd7cdf3 to a428ee2 Compare October 27, 2026 08:17
@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 27, 2026
@moetemp
moetemp force-pushed the moe/brook/review/streams-k6-service-writes branch from a39dd25 to 550c147 Compare October 27, 2026 16:55
@moetemp
moetemp force-pushed the moe/brook/review/streams-k8-reads branch 2 times, most recently from 08eb55b to 913292f Compare October 27, 2026 23:46
@moetemp
moetemp force-pushed the moe/brook/review/streams-k6-service-writes branch from 550c147 to 6f26a43 Compare October 27, 2026 23:46
@moetemp moetemp changed the title [Brook K8] Served reads with long polls, Topic filters, last-N reads and byte caps. [Brook K8] Served reads with long polls, last-N reads and byte caps. Oct 27, 2026
@moetemp
moetemp force-pushed the moe/brook/review/streams-k6-service-writes branch from 6f26a43 to 6b804be Compare October 28, 2026 08:55
@moetemp
moetemp force-pushed the moe/brook/review/streams-k8-reads branch from 913292f to 04b1a06 Compare October 28, 2026 08:55
@moetemp
moetemp force-pushed the moe/brook/review/streams-k6-service-writes branch from 6b804be to 5ccca2b Compare October 29, 2026 04:06
@moetemp
moetemp force-pushed the moe/brook/review/streams-k8-reads branch from 04b1a06 to 1970b1f Compare October 29, 2026 04:06
@moetemp
moetemp force-pushed the moe/brook/review/streams-k6-service-writes branch from 5ccca2b to f4950ec Compare October 29, 2026 08:37
@moetemp
moetemp force-pushed the moe/brook/review/streams-k8-reads branch 2 times, most recently from c454268 to a42f66d Compare October 29, 2026 09:17
@moetemp
moetemp force-pushed the moe/brook/review/streams-k6-service-writes branch from f4950ec to 7a22520 Compare October 29, 2026 09:17
A read waits for the create and for new Records, and fails on a gap that the
Retention left. The service suite that reads back what the writes stored comes
here too.
@moetemp
moetemp force-pushed the moe/brook/review/streams-k8-reads branch from a42f66d to 84fb6ca Compare October 29, 2026 10:03
The scans test drops default_retention, which K3 removed from the Store trait.
MAX_WAIT stays public, since the native backend reads it.
The service suite keeps K6's case of an append with no sequence before the read cases.
The test closes once with a plain failure and once with an Activity failure whose cause is a
cancel. The expiry case sends a negative sequence, since zero takes the next one.
# Conflicts:
#	crates/streams/src/service.rs
#	crates/streams/src/testing/service.rs
# Conflicts:
#	crates/streams/src/service.rs
The read past the end fails as stream_offset_too_high, and each create names its Stream by its handle. The suite keeps one close helper and one Workflow Producer helper.
A read is done when it gives the last item of a closed Stream, and a read with no item left answers at once. A stored item that doesn't decode is corrupt Store data.
A read and a describe carry their StreamRef in ref, as the other requests do.
The review asked to stop representing invalid states. The Producer kind is now a oneof with run_id only in the run-based variants, so a client can't carry a run. A write's task facts and withdraw_run move into RunWithdrawal, which only a run writer sets.
@moetemp

moetemp commented Nov 10, 2026

Copy link
Copy Markdown
Owner Author

Closed in favor of #228, the combined branch, which has this change.

@moetemp moetemp closed this Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant