Skip to content

[Brook K2] Added the Stream ids, offsets, epochs and batch digest, with vectors. - #213

Closed
moetemp wants to merge 24 commits into
moe/brook/review/streams-k1-protosfrom
moe/brook/review/streams-k2-ids-vectors
Closed

moetemp wants to merge 24 commits into
moe/brook/review/streams-k1-protosfrom
moe/brook/review/streams-k2-ids-vectors

Conversation

@moetemp

@moetemp moetemp commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

What changed?

This PR adds the Stream ids, offsets, epoch keys and batch digest, with shared vectors.

  • An offset packs the Redis entry id <ms>-<seq> as ms << 20 | seq.
  • Epoch keys are fixed-width big-endian bytes, so a byte compare orders the epochs. The vectors give them in hex.
  • A Producer's kind is the case of its epoch. epoch::ProducerKind gives it, with a fixed number for the stored field names, and refuses a Producer whose epoch has no case.
  • testdata/gen_vectors.py is a second implementation. The crate's tests read every vector file, and sdk-python also reads ids.json and withdrawals.json.
  • testing::fields, behind the test-support feature, has the stored field names that the store suite checks.
  • At the completed task's started id, the withdrawal rule withdraws a Record whose task key is lower than the completed task's. C1 sends that task's attempt and started time, and the vectors use them. A completed task with no attempt withdraws nothing at its id.
  • The batch digest also hashes whether a Record has a body, and the sizes of its external payloads.
  • A Stream has a third key, :producers, the sorted set of each Producer's last append.
  • check_producer takes only ids in the form of encode_part.

Depends on #212 (K1). Review lane is the Rust reviewers.
Part of the Brook review series.

Why?

Every SDK must compute these the same way. They have no I/O, so reviewers can check them against the vectors before any Store exists.

How did you test it?

cargo fmt, cargo clippy -- -D warnings (with and without features), cargo test -p temporalio-streams --all-features and cargo doc pass at this commit.

@moetemp
moetemp force-pushed the moe/brook/review/streams-k1-protos branch from adac538 to 4078d83 Compare October 27, 2026 07:35
@moetemp
moetemp force-pushed the moe/brook/review/streams-k2-ids-vectors branch 2 times, most recently from 3c800ad to f5d807e Compare October 27, 2026 08:17
@moetemp
moetemp force-pushed the moe/brook/review/streams-k1-protos branch from 4078d83 to 5cad429 Compare October 27, 2026 08:17
@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 27, 2026
@moetemp
moetemp force-pushed the moe/brook/review/streams-k1-protos branch from 5cad429 to 345aa03 Compare October 27, 2026 16:55
@moetemp
moetemp force-pushed the moe/brook/review/streams-k2-ids-vectors branch 2 times, most recently from 16de1da to bd35e0c Compare October 27, 2026 23:46
@moetemp
moetemp force-pushed the moe/brook/review/streams-k1-protos branch from 345aa03 to 6b5105d Compare October 27, 2026 23:46
@moetemp moetemp closed this Oct 28, 2026
@moetemp moetemp reopened this Oct 28, 2026
@moetemp
moetemp force-pushed the moe/brook/review/streams-k1-protos branch from 6b5105d to b906e6f Compare October 28, 2026 08:55
@moetemp
moetemp force-pushed the moe/brook/review/streams-k2-ids-vectors branch from bd35e0c to fe3f05a Compare October 28, 2026 08:55
These are the rules that every SDK, every Store and the server must compute
the same way. `testdata/gen_vectors.py` writes the shared vectors from a
second implementation, and `tests/vectors.rs` checks the crate on them.
@moetemp
moetemp force-pushed the moe/brook/review/streams-k2-ids-vectors branch from fe3f05a to 98ba416 Compare October 29, 2026 04:06
Nothing outside the tests built a default Stream id or parsed a Workflow Producer's run. Only the shared Store suite reads the meta field names, so they sit behind test-support. The README says which side reads each vectors file, and the digest docs no longer claim that the server computes the digest.
The server keeps its own copies of some vectors, and two of them differ from these files.
A task key is 24 big-endian bytes, an epoch key 40 and a digest the first 16 bytes of the SHA-256,
which saves Store bytes. Both Stores compare the keys as bytes, and the vectors give them in hex.
Each kind keeps its number, so the stored names and the epoch keys keep their bytes. The vector loaders build the case from the flat fields of the JSON.
A retry reads the same entry id, so the error isn't retryable.
A close writes no log entry, so a Store that waits on its log needs another key that a close wakes.
The facts and the withdrawing Producer moved into StreamWriter. The vector loader builds the writer from the same JSON fields.
The review asked to stop representing invalid states. The Producer kind is now a oneof with run_id only in the run-based variants, so a client can't carry a run. A write's task facts and withdraw_run move into RunWithdrawal, which only a run writer sets.
Matches the proto change, so the ids and epoch vectors stay the same.
@moetemp

moetemp commented Nov 10, 2026

Copy link
Copy Markdown
Owner Author

Closed in favor of #228, the combined branch, which has this change.

@moetemp moetemp closed this Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant