Skip to content

[Brook] Withdrew a run's Local Activity Records of tasks that never completed. - #37

Closed
moetemp wants to merge 5 commits into
moe/brook/v4/s19-no-workflow-bufferfrom
moe/brook/v4/s20-orphan-local-activities
Closed

moetemp wants to merge 5 commits into
moe/brook/v4/s19-no-workflow-bufferfrom
moe/brook/v4/s20-orphan-local-activities

Conversation

@moetemp

@moetemp moetemp commented Oct 22, 2026

Copy link
Copy Markdown
Owner

What changed?

This PR withdraws a run's Local Activity Records of Workflow Tasks that never completed (S110).
A Local Activity batch that gives previous_completed_task (the run's last completed task, 0 before the first) and failed_tasks now applies the orphan rule in the same append script. Its dispatching task is in its epoch. The Store keeps the run's open Local Activity task keys per Stream (oa|<run>). It withdraws those after the completed task and before the batch's task, the failed ones, and the lower attempts, and drops the tasks at or before the completed one. The withdrawal is one s-only entry with a Workflow-kind Supersession (the run's Workflow Producer, the batch's task, the completed and failed tasks), so the existing withdraws() reader rule removes exactly those Records. The withdrawn tasks stay in ow|<run>, which fences a late write of such a task. A batch with no completed task skips the rule, so an older lang behaves as before. The rerun-by-epoch rule is unchanged. The memory Store does the same, and the Redis docs describe the rule.

Why?

Decision S110. A Local Activity writes in task 3, task 3 fails, and the retry never runs that call again, so no newer epoch supersedes it. Before, its Records stayed visible.

How did you test it?

cargo fmt --all -- --check, cargo clippy --workspace --tests -- -D warnings (also with --no-default-features --features tls-aws-lc), and STREAMS_REQUIRE=redis cargo test --workspace on standalone Redis, cluster Redis and memory. New service tests: a task 3 Local Activity orphaned by task 4's write on another path, a late write of it fenced, and a later write that withdraws nothing more. Also a failed task before the last completed one, withdrawn by its id. The rerun test still passes.
The main missing element is the Core field for the last completed task in ActivityTask.Start field 20, which waits for the lead's go-ahead.

@moetemp

moetemp commented Oct 25, 2026

Copy link
Copy Markdown
Owner Author

Closing as stale. This layer is folded into the v5 tip, #41, and the upstream review uses a fresh series instead. Plan: https://www.notion.so/4048fc567738814990b7c796ac28fe3f

@moetemp moetemp closed this Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant