Repository navigation
Conversation
…s13-local-activity-flush # Conflicts: # crates/streams/tests/store.rs
|
Review fixes in 78650ea: a deferred batch that withdraws a retry run's previous run writes a |
|
Review item 4 fixed in the new s13 head: a Workflow batch with facts and no Records goes in also after a close, and a read that reaches the close Record gives the |
|
Per the lead's decision, the close rule stays. The post-close withdrawal change (d220473) is reverted on this layer. |
|
Final state, as the lead decided: the post-close withdrawal fix is restored. 7682982 reverts the revert. It now also covers a retry batch with Records: when a close refuses it, the Records are dropped and a batch with only the facts goes in. |
|
Closing as stale. This layer is folded into the v5 tip, #41, and the upstream review uses a fresh series instead. Plan: https://www.notion.so/4048fc567738814990b7c796ac28fe3f |
What changed?
This PR adds the extension call
FlushRunBuffer, which writes a run's buffered Workflow calls before a Local Activity of that run starts.FlushRunBufferRequest { namespace, run_id, workflow_task, timeout }takes the dispatching Workflow Task fromActivityTask.Startfield 20.FlushRunBufferResponse { records }answers how many Records it wrote. The crate writes the run's buffered creates and appends in that task's epoch, with the same Producer, epoch fence, dedupe and retry-run rules as thebefore_completeflush. Closes still wait for an accepted completion.Core gives a task's withdrawal facts (previous completed task, failed tasks) only with the completion. A batch with no previous completed task would withdraw every earlier task, so the early batch is deferred:
Withdrawal.defer, sent to the append script as the completed task key*. It withdraws nothing, flags nothing and keeps nolffacts, but it still fences, dedupes, enters the task index and applieswithdraw_run. The memory Store does the same.The task's completion writes only what was buffered after the early write. Its batches go on from the early batch numbers of the same epoch, so nothing goes in twice and nothing is dropped as a resend. It also sends a batch with the facts, an empty one if needed, to each Stream that the early write reached, so the withdrawal reaches those Streams.
A Local Activity of a task that already completed (for example a retry after a heartbeat completion), or of a task older than an earlier early write, writes nothing. When the early write's task never completes, the next task writes those calls again in its own epoch, and its facts withdraw the old ones. A failed early write fails the run's next completion with the same error, so the retry withdraws what went in. A failed task keeps the batch numbers of its epoch, so a late write in that epoch never reuses a number. One lock per run orders its writes, so a Local Activity that starts while another one's flush runs waits until the run's Records are in. A run with nothing buffered answers at once and needs no Redis.
Why?
Decision S70. Without this, a Local Activity could publish before the Workflow Records that came before it in Workflow code, since the Workflow buffer goes out only when the task completes. The epoch of the dispatching task keeps the existing withdrawal rules, so a Workflow Task that fails after the early write has those Records withdrawn.
How did you test it?
cargo fmt --all -- --check,cargo clippy --workspace --tests -- -D warnings, andSTREAMS_REQUIRE=redis cargo test --workspaceon standalone Redis, a 3-primary Redis cluster and the memory Store.New Workflow tests check the order (Workflow Records, then the Local Activity's, then the rest) with no duplicate and with two Local Activities in one task, that the completion takes its facts to a Stream that only the early write reached, that a task that fails after an early write has its Records and its Local Activity's withdrawn by the retry, also with a late flush of the failed attempt, that a Local Activity of a completed task writes nothing, that two Local Activities that start together both see the run's Records before they go on, and that a run with nothing buffered needs no Redis. A capability test checks that a failed early write fails the task's completion with nothing written. A Store test checks the deferred batch on Redis and memory.