Skip to content

feat: count a Stream's Retention from its close - #8

Draft
moetemp wants to merge 18 commits into
mainfrom
moe/retention-from-close
Draft

moetemp wants to merge 18 commits into
mainfrom
moe/retention-from-close

Conversation

@moetemp

@moetemp moetemp commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

This PR counts a Stream's Retention from its close in the Redis Store, as a namespace's Retention
does. An open Stream also keeps every Producer's dedupe and fence state.

A Stream that stayed open longer than its Retention lost its first Records. Each append ran XTRIM
on the Records older than the Retention, and the log expired one Retention after the last write.
The log now expires at exp, so an open Stream keeps every Record. It expires only if the Store config sets
last_write_ttl, that long after its last write. A closed Stream still expires one Retention after the close.

No append drops a Producer's p| field by its age either. So a resend of a silent Producer's last
batch gets the first answer, a read never gets a second copy, and a lower epoch of the Producer
stays fenced. The state goes only with the Stream, at the cost of one 88-byte p| field for each
Producer that wrote to it.

  • trim is now mark_log_gone. It only raises trimmed when a write finds the log gone, as after
    an eviction.
  • prune and the :producers key are gone, since nothing else read the key. The write scripts take
    two keys, and a Producer no longer needs +xtrim, +zadd, +zrangebyscore or +zrem.
  • scripts/patch-sdk-rust.sh builds against 5a5f856f of Streams crate: combined branch for review moetemp/sdk-rust#228, which makes the same
    changes in the memory Store and the shared suites.
  • The meta hash has no closer and cstat, since a read gives only Records and tells of a close
    with done.
  • The test runtime sets a 300 s last_write_ttl, since an open Stream keeps the suites' Records
    until it expires.

Two later commits follow the Stream crate's changes of Nov. 10.

  • last_write_ttl replaces the safety TTL and has no default. A write with no TTL clears an expiry
    that an earlier write set, so a Producer needs +hdel and +persist.
  • The Store's read and wait calls take a StreamRef, as the other calls do.
  • The keys have no Temporal namespace. They are <prefix>:{<stream id>}:meta, :log and :closed,
    so a Stream id is unique in one Redis and key prefix. An ACL key pattern now scopes a user to a
    prefix, and docs/redis.md says so.

Two more commits follow the Stream crate's changes of Nov. 11. A read has no create_wait, a write names
its Producer directly, and each read item carries its own offset. The Redis Store needed no change for
them beyond its pin and its tests.

The last commits follow the crate's new shape of Nov. 11.

  • The runtime builds on the crate's LazyStore, which connects on the first call that needs Redis. So
    a bridge can build the runtime with Redis down, and the first call then fails with
    stream_storage_unavailable.
  • The API has no UNKNOWN_WRITER reason, so the three reasons after it have new numbers.

The last three commits follow the crate's restart of Nov. 12. The crate now keeps only what answers
the five Stream operations.

  • The crate has no writer objects, so the writer suite and tests/writer.rs are gone. A bridge calls
    StreamRuntime::call with an operation name and the encoded request.
  • store_name replaces backend in RedisStreamConfig and in RuntimeOptions, as StreamRef.store
    does in the API.
  • A Producer id is opaque text of at most 4096 bytes, and the p| field takes its bytes as they are.
    The Store keeps a Producer's fence by kind and id, so the id of a run-based kind must hold its run
    id. docs/redis.md says so.
  • The shared suites have new cases, and Redis passes them: two runs with their own ids, two kinds
    with one id, an id of any text, and a close by each Producer kind.

A Stream that stayed open longer than its Retention lost its first Records. The memory Store of `temporalio-streams` makes the same change, so both Stores keep one rule.
A resend of a silent Producer's last batch wrote a second readable copy, and a lower epoch of it wasn't fenced. Without the pruning, the `:producers` key had no reader, so the scripts drop it.
The crate dropped the newest-N read and the safety TTL default, and its Store calls take a StreamRef. The patch script pins the sdk-rust commit with those changes.
The Stream crate's Store calls take no namespace, and its keys hold only the Stream id in the hash tag. The tests that used a namespace to keep runs apart take Stream ids of their own.
A read of a Stream that nobody created now fails at once with NOT_FOUND, so the changelog says so for this runtime.
The API dropped StreamWriter, moved each read item's offset into the item, and marks the fields a Producer must send. The patch script pins the sdk-rust commit with those changes.
The Stream runtime takes a Store now, so the Redis runtime wraps its Store in a LazyStore that connects on the first call. The patch script pins the sdk-rust commit with the typed runtime.
The patch script pins the sdk-rust commit where a bridge holds each writer and drops it.
The API dropped the unknown writer reason, so three reasons after it have new numbers. The patch script pins the sdk-rust commit that has them.
The crate had no test target for the Stream crate's writer suite, so no test covered the writers on Redis.
A comment and the docs used an internal name. An open with no batch no longer needs Redis, and two runtimes in one process don't order the batches of one epoch.
The patch script pins the sdk-rust commit where a retry epoch keeps its fence state in its lane and a write on a closed writer fails as closed.
The patch script pins the sdk-rust commit that has the Workflow Producer comments of the API.
A read gives only Records, so the Redis meta hash no longer keeps `closer` and `cstat`. The pin
moves to the sdk-rust commit that has the same API.
The Store name replaces the backend name in the config, the runtime and the tests. The writer
suite is gone, and the tests build Producer ids as plain text. The pin moves to the new crate.
The README, the changelog and `docs/redis.md` no longer name a writer. They say that a Producer id
is opaque and that a bridge calls `StreamRuntime::call`.
The `p|` field and an epoch key have no run id, so the docs give the caller's rule for the id of a
run-based kind. The patch script points at the crate commit with the new suite cases.
The crate's code is the same. Its base now has the API files of temporalio/api#895 with an optional
sequence on append.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant