[feature] Rerun a failed run from the step that failed, reusing results already recorded

[bilyiv]

Primary owning public component

Workflow engine

User problem

A run that has failed terminally can only be replaced by a new run started from the top. Every step the failed run already completed is executed again, even though its result is recorded in that run's history and will not be different this time.

Where those steps are cheap and internal that is merely wasteful. Where they are paid, rate-limited or externally visible — credit checks, payment attempts, notifications, anything a third party bills for or a customer sees — re-running them is itself a business event. The practical consequence is that operators stop using the retry: they read the history, work out what actually needs redoing, and do it by hand. That is precisely the position durable execution is meant to remove them from.

Nothing in the public control plane continues a closed run. start makes a new one, signal, update and query talk to a live one, cancel and terminate end one, repair recovers a stuck one, archive and describe are about records. A failed run is finished, and the only forward path is to start again from nothing.

Child runs come closer: a child can be retried under a retry policy, and the history contract records retry_of_child_workflow_run_id, so the engine already models "this run is another attempt at that one". But a child retry begins the child again from its first step, and there is no equivalent for a top-level run at all.

So the gap gets filled outside the engine: read the failed run's history, work out how far it got, start a fresh run, and feed the earlier results back so the completed steps are not executed a second time. That works, but it depends on the recorded shape of history, which is engine-internal and not something an application should be reasoning about — and it is quiet when it is wrong. The failure mode is re-executing a step that should have been skipped, which looks like success and surfaces as a duplicate charge or a second notification.

Proposed public contract

A control-plane operation that continues a failed run from where it stopped, rather than from the beginning. It produces a new run; the failed one stays as it is, closed and readable.

Behaviour

• Activities that completed successfully in the failed run are not executed again; their recorded results are returned to the workflow body as it replays.
• The step that failed is attempted again, and execution carries on from there.
• Called on a run that did not fail, it is refused with a reason rather than quietly starting a fresh run from the top. A run that is still going, one that completed, one that was cancelled or terminated — none of these has a failure to continue from.
• The new run records its provenance: which run it continues, and which step it resumed at.

Operator experience

That provenance is readable through describe(), and steps served from the earlier run are distinguishable from steps executed afresh, in history and in any activity listing. Without that distinction an operator cannot answer "which external calls did this attempt actually make?" — which is the question the whole feature exists to make answerable, and the one that decides whether a customer was charged twice.

Worth deciding explicitly

• Whether the resume point can be chosen rather than always being the failure. When the failing step is a symptom of a bad value produced two steps earlier, resuming at the failure just fails again.
• Whether a resumed run is itself resumable, and what its history looks like to a second resume.
• What happens to a failed run's children — both those that had already completed, and those still in flight when the parent failed.

Acceptance criteria

1. A failed run can be continued through the public control plane, producing a new run that starts at the step which failed.
2. Activities that completed in the original run are not executed again; their results are served from that run's recorded history.
3. The step that failed is executed again, and the run proceeds from there.
4. The new run names the run it continues and the step it resumed at, both through describe() and in its own recorded history.
5. The original run is left as it was — closed, with its history intact and readable.
6. Continuing a run that is running, completed, cancelled or terminated is refused with a stated reason, never silently converted into a fresh run from the top.
7. Steps served from the earlier run are distinguishable from steps executed afresh, in history and in activity listings.
8. Additive — callers that never invoke it see no change in behaviour, and existing runs are unaffected.

Supporting public evidence

Comparable durable-execution and orchestration products all provide some form of this, which suggests it is expected rather than exotic:

• AWS Step Functions — Redrive. Restarts a failed execution from the point of failure; states that already succeeded are not run again. The closest match to what is asked for here, in both name and semantics.
• Temporal — Reset. Starts a new run from a chosen point in a previous run's history and re-executes only from there.
• Netflix Conductor — retry from last failed task. Resumes a failed workflow at the task that failed, keeping the work already done.

People arriving from any of these expect a retry-from-failure to exist, and tend to discover it does not the first time an expensive workflow fails in production.

patched(), with its VersionMarkerRecorded history entry, already gives a replayed prefix a defined answer when code has changed since it ran — so the harder half of this is shipped.

Dependencies and affected repositories

None.

Public intake checks

• I searched open and closed GitHub issues for an existing request.
• This request contains only public product contracts and public-safe rationale.

[rmcdaniel]

Intake approved. The operator need is clear: recover a terminally failed run without automatically repeating previously completed external work. I am triaging this as a user-reported recovery capability, not as a generic retry helper.

The safe contract needs an immutable original run, an explicitly linked new run, a validated replay prefix, and visible provenance for reused results. We also need to decide how this interacts with changed workflow code, signals/updates, timers, side effects, child runs, concurrency, and failure after an external effect but before its acknowledgement. Reusing a recorded success cannot make an ambiguous failed payment or other external effect exactly-once; activity idempotency still matters.

I will check existing reset/recovery APIs and related requests before fixing the scope. This also needs a consistent service-mode contract and operator visibility, so the eventual affected repositories may extend beyond the embedded engine. No release commitment or implementation is being claimed from intake alone.

[rmcdaniel]

Triage is complete: accepted as a new recovery capability, not an existing repair() option. WorkflowControlPlane only repairs an open current run; WorkflowStub::attemptStart() rejects a previously started instance. The replayer and lineage projection are useful foundations, but they do not already implement redrive.

The implementation should start with an explicit, idempotent resume command targeting a particular failed run:

• Keep the failed run and its history immutable; create a linked new run and record the retry boundary.
• Replay the validated durable prefix, including recorded side effects/version markers, without scheduling its completed activities again. Show reused results separately from newly executed attempts.
• Reject stale/foreign run targets, non-failed runs, missing history or payloads, and incompatible replay. A repeated request must return the same successor, not create another attempt.
• Qualify cancellation/timeout, parallel branches, child runs, signals/updates, and compensation/finally effects explicitly. Until a case is supported, reject it with a useful reason rather than silently rerunning successful work.
• Preserve recovery provenance through another resume, retention and export. Cover cold-process replay and acknowledgement loss, not just in-memory results.

One important limit: a failed activity may already have performed an external effect before losing its acknowledgement. Reusing recorded successes prevents those known successes from being rescheduled; it cannot make an unknown failed effect exactly-once. Provider idempotency/reconciliation remains necessary.

This belongs in the shared engine contract, followed by Server/CLI, PHP/Python/Rust clients and Waterline visibility, rather than a PHP-only history-editing workaround. Keeping this issue as the owning feature record; implementation follows the current fixes and recovery commitments. No release number is promised yet.