feat(workflows): per-attempt task timeout and continue-on-error by TsuyoshiUshio · Pull Request #186 · Azure/azure-functions-agents-runtime

Tsuyoshi Ushio (TsuyoshiUshio) · 2026-08-29T18:30:04Z

PR 2 of 3 — stacked on #185

This is the second slice of the task execution policy work decomposed from #170.

Order	PR	Scope
1	#185 (`tsuyoshiushio-workflow-native-retry`)	Durable native Activity retry
2	this PR (base: `tsuyoshiushio-workflow-native-retry`)	Per-attempt `execution.timeout` + `execution.continue_on_error`
3	#187	Retry/timeout observability and the structured status contract

The base branch is tsuyoshiushio-workflow-native-retry, not main, so the diff
here contains only PR-2 changes. #170 is untouched and was used as source
material only.

Rebased 2026-08-29 onto PR 1's Durable 2.x E2E fixes (1777c85, f7bc03e7,
bd56857f). The six PR-2 commits are preserved. PR 1's
_await_wave / _first_wave_failure primitives now hand back per-node wave
outcomes directly, which removed the extra per-child read this slice
originally needed — the engine diff got smaller and no nested task_all
is reintroduced.

What this adds

execution.timeout — an optional per-attempt deadline, ISO-8601 PT1S–PT10M,
authored on a task or declared authoritatively on the tool with
@workflow_tool(timeout=...). It is applied at the Activity invocation boundary
for both synchronous workflow tools and asynchronous Sub Agent deliveries. An
expired deadline is a retryable workflow_task_timeout failure, so it feeds
straight into PR 1's Durable native retry: the next attempt is scheduled under
the already-frozen policy, and with the default single attempt the task simply
fails with that error code. A Sub Agent's own resolved agent timeout is mapped
onto the same classification, so the tighter of the two bounds is retried instead
of degrading to a non-retryable execution_unknown.

The bounded-execution guard now covers deadlines too: max_attempts × timeout
plus the retry delays must stay inside the internal one-hour ceiling, and a plan
that exceeds it is rejected at submission.

execution.continue_on_error — a task-local opt-in that lets the DAG proceed
past a failure. It applies only after the attempt budget is spent: a retryable
failure is retried by Durable first, and only a terminal or exhausted outcome can
be continued, which is what keeps downstream dependency handling deterministic.
A continued node commits a sanitized {"failed": true, "error_code", "error", "kind"}
result — the same shape the orchestrator already returns for controlled failures —
so dependents run and a when predicate can branch on ${node.result.failed}.

Continuation cannot cross the authorization boundary. Only application-level
failures are continuable:

`kind`	Continuable
`timeout`, `handler_transient`	✅ after retries are exhausted
`handler_terminal`, `execution_unknown`	✅
`authorization`	❌
`handler_contract`	❌

An opaque Durable failure carries no classification and is never continued.

Replay compatibility

Both new persisted keys are NotRequired and are written only when they were
asked for. A task that uses neither freezes a payload byte-identical to the one
PR 1's runtime wrote and replays through exactly the same path.
Absent timeout_ms reproduces PR-1 behavior exactly (asyncio.timeout(None)),
and absent continue_on_error defaults to false.
The Activity outcome envelope is not extended. PR feat: add Dynamic Workflow task execution policies #170 added a continuable
flag to the persisted failure; that would break replay here, because
validate_activity_result checks an exact key set and a failure written by PR 1
would stop validating. Continuability is derived from the already-persisted
kind instead.
Everything is read from the persisted orchestration input alone, so deploying a
new decorator declaration never changes how an in-flight workflow replays.
On a failed wave, _first_wave_failure still fails the whole wave unless a task
in that wave declared continue_on_error; only then are the per-node outcomes
_await_wave already returned applied node by node. Existing plans keep the
previous failure path unchanged.
Rollback: persisted keys are additive and forward-tolerant so an upgrade
replays cleanly, but an older orchestrator cannot honor semantics it does not
implement. A runtime rollback across this policy version must drain or terminate
workflows started with the new fields (FRD 0004, Decision 81).

Known limitation (documented, not hidden)

The deadline bounds the wait, not the worker. Workflow tool handlers are
synchronous and run on a worker thread, and a thread cannot be cancelled from the
outside, so a timed-out attempt is reported while the handler may still be
running. This is the same at-least-once exposure Durable already has for a
redelivered Activity, and the mitigation is the same: key side effects on
current_workflow_task_context().idempotency_key. Documented in
docs/workflows.md, in the activity.py module docstring, logged at warning
level, and asserted directly by a test. Recorded as FRD 0004 Decision 77.

Explicitly out of scope

Retry/timeout telemetry expansion, the structured status schema and retry-wait
metadata, new status states, UI cards, read-only Skill approval, the
extension-bundle sweep, and any SSE change beyond what PR 1 already carries.
A continued node therefore reuses the existing completed state rather than
introducing failed_continued.

Architecture review

An independent review of the PR-2-only surface ran before implementation. It
confirmed that per-node wave outcomes are deterministic, that continuation can
only follow an exhausted or terminal attempt, and that rejecting authorization /
handler_contract preserves the authorization boundary. It raised three blocking
findings — uncancellable synchronous handlers, runtime rollback across a policy
version, and a Sub Agent's own timeout bypassing the retryable classification.
The third is fixed in code; the other two are bounded and documented. Recorded in
FRD 0004 as Decisions 74–81 plus a review checkpoint.

Validation (after the rebase onto `bd56857f`)

ruff check src tests — clean
mypy src (strict) — clean
Targeted pytest over the timeout/continuation, native-retry, engine, registry,
schema, and retry-sample suites — 282 passed
Full pytest tests — 1171 passed. The 7 failures in
tests/test_trigger_serialization.py are pre-existing on the parent branch
(that file is not in this diff) and are unrelated to workflows.
mkdocs build --strict — clean
uv sync --locked and eng/scripts/generate_config_reference.py — no lock or
generated-reference drift (only workflows/schema.py changed, not config/schema.py)
Full real Functions-host E2E for this slice is deferred until all three drafts exist.

Retarget / rebase after PR 1 merges

Once #185 merges into main:

git fetch origin
git rebase --onto origin/main origin/tsuyoshiushio-workflow-native-retry \
  tsuyoshiushio-workflow-timeout-continuation
git push --force-with-lease
gh pr edit 186 --base main

The commits touch only PR-2 concerns, so they replay onto main without needing
PR 1's commits reordered.

Durable native Activity retry needs the Python 2.x RetryPolicy (backoff coefficient, maximum interval, and retry timeout); the 1.x RetryOptions can only express a constant first interval and an attempt count. Carry only the collateral the bump requires: the DurableFunctionsClient binding type, Task.result now raising instead of returning its failure, the 2.x trigger decorator typing, and a Durable client whose lifetime spans SSE stream consumption instead of the invocation that created it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add the smallest public surface a workflow task needs to opt into Durable native retry: a bounded exponential `retry` declaration, the WorkflowRetryableError / WorkflowTerminalError signals a handler uses to classify its own failure, and a task context exposing a per-instance idempotency key that is stable across attempts. Durable's RetryPolicy has no exception predicate, so the Activity side splits the two outcomes explicitly: retryable failures raise a private versioned marker so Durable schedules the next attempt, terminal ones return a structured outcome so it does not. Models read back out of Durable history ignore unknown keys so a history written by a later runtime still validates. Per-attempt timeout is deliberately absent: Durable native retry does not need it. The only bound is an internal one-hour retry_timeout ceiling. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Schedule a task through call_activity_with_retry when, and only when, the persisted orchestration input carries a frozen retry policy. The decision reads nothing from the currently deployed tool registration, so an orchestration started before a tool declared retry keeps replaying through call_activity and keeps the legacy result envelope. On exhaustion Durable raises one failure for the whole wave; match the sanitized payload back to its node by id so the workflow reports the application's own error code rather than an opaque Durable wrapper. The tool Activity becomes async so a policy-aware delivery can be awaited; synchronous handlers still run off the event loop, as they did when the Functions worker ran the Activity in a thread. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Let a tool author declare its own retry with @workflow_tool(retry=...) and carry it through discovery, the handler catalog, and the frozen per-agent plan policy. A tool declaration is authoritative at submission time and overrides a plan-authored policy, so retryability is a property of the tool rather than of what the model happened to write. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

One module covers the whole path rather than mirroring it per source file: the declaration bounds, the frozen wire shape, the raise-versus-return split, the exhaustion round-trip and everything it must refuse to decode, the orchestrator's driver selection, and submission through start_workflow. The replay case is asserted explicitly: a history with no persisted policy keeps using call_activity even after its tool starts declaring retry. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

An operations agent recovers a delayed order whose inventory reservation is temporarily unavailable. Retry is declared on the tool rather than in the prompt, so the model authors an ordinary three-task DAG and the runtime applies the policy at submission. The Blob-backed incident counter is only a deterministic stand-in for a flaky dependency; it is not part of using retry. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Describe the declaration, the retryable-versus-terminal contract, the idempotency key handlers get instead of an attempt number, and why an in-flight workflow is unaffected by a new retry declaration. Record Decisions 66-73 and both review checkpoints in FRD 0004. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A terminal policy-aware outcome now raises out of result application, so the wave's timer cleanup has to cover that path too rather than only the branch where Durable itself reported the failure. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The docs workflow syncs with --locked, so the pin change has to be reflected in uv.lock. Resolution adds durabletask and its grpcio/protobuf/asyncio dependencies, drops furl and orderedmultidict, and moves azure-functions from 2.1.0 to 2.3.0 because azure-functions-durable 2.0.0b2 requires >=2.3.0b2. Artifact URLs for packages that did not change are kept as committed: the Microsoft feed proxy hands out a different mirror host on every resolution, which would otherwise rewrite every URL in the file for no change in content. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Durable Functions Python 2.x pulls in durabletask, which depends on grpcio. The Python worker bundles its own grpcio, and with no customer dependency path the worker resolved a mixed grpcio/protobuf set and segfaulted (exit 139) while importing function_app.py — so every E2E app failed to start, including ones that never touch workflows. Install the runtime once into a shared tree, link it in as .python_packages for every app with a host.json, and put that path ahead of the worker's own packages. This is what a deployed app already looks like. The extension bundle is unchanged: the host's LATEST channel already selects 4.37.1 for the existing [4.*, 5.0.0) and [4.32.0, 5.0.0) ranges. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Durable's composite tasks only learn that a child completed when the child is a leaf: CompositeTask subclasses never notify their own parent. Racing the cancel event against task_all(wave) nests one composite inside another, so the inner completion never reached the outer selection and the orchestrator hung after its first wave — no workflow could make progress under Durable 2.x. Select over the individual wave tasks instead, so every child of the awaited task is a leaf. Wave semantics are unchanged: results stay in wave order, cancellation still restores the wave and cancels pending timers, and the first failure in wave order is surfaced exactly as task_all did. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Drive the retry sample's workflow through Durable's orchestration HTTP API so both cases are deterministic and model-free; what is under test is the runtime's retry behaviour, not an agent's ability to author a plan. Covers the two behaviours that matter: a transiently failing tool is retried and the workflow still completes with the expected result, and a tool that keeps failing exhausts its attempt budget and fails with the application's own sanitized error_code rather than an opaque Durable message. The attempt count is asserted from the sample's own incident counter. Both cases share one host because a second host contends for the task-hub lease. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Extend the bounded execution policy a task freezes at submission time with the two remaining authored controls: an optional per-attempt `execution.timeout` (PT1S-PT10M), declarable on the task or authoritatively on the tool, and a task-local `execution.continue_on_error`. Both persisted keys are optional and written only when they were asked for, so a task that uses neither freezes a payload identical to the one the native-retry runtime wrote and replays through exactly the same path. `continue_on_error` is deliberately never decorator metadata: whether a workflow may proceed past a failed node is a property of the plan, not of the tool. The bounded-execution guard now covers attempt deadlines too, so the worst case of `max_attempts * timeout` plus the retry delays cannot escape the internal one-hour ceiling. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Apply the persisted attempt deadline around the handler invocation for both synchronous tools and asynchronous Sub Agent deliveries, and classify an expired deadline as a retryable `workflow_task_timeout` so Durable schedules the next attempt under the already-frozen policy. A history without a persisted deadline uses `asyncio.timeout(None)`, so it takes the same code path and keeps its unbounded attempt. A handler that raises `TimeoutError` itself is still an unknown execution failure: only the scope that actually expired reports a timeout. A nested execution bound raises `WorkflowTaskTimeoutError` so it lands on the same classification instead of degrading to a non-retryable failure. Continuability is derived from the persisted failure `kind` rather than added to the outcome envelope, because that envelope is validated by an exact key set and must keep validating failures written by the previous runtime. `authorization` and `handler_contract` are never continuable. The deadline bounds the wait, not the worker: a synchronous handler already running on a worker thread cannot be cancelled, so a timed-out attempt is reported while the handler may still be running. That is the at-least-once exposure the idempotency key already exists for, and it is now documented in the module docstring and logged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A task whose persisted policy set `continue_on_error` now commits a sanitized failure result instead of failing the workflow, so its dependents run. It only applies once the attempt budget is spent: a retryable failure is retried by Durable first, and only a terminal or exhausted outcome can be continued, which is what keeps downstream dependency handling deterministic. An authorization or contract failure, and any Durable failure that carries no classification, still fails the workflow. Reading per-node outcomes on a failed wave is safe because Durable's `WhenAllTask` completes only once every child has completed and then surfaces the first failure; the previous whole-wave failure path is kept byte-for-byte for waves where no task declared continuation, so existing plans are unaffected. A continued node reuses the existing `completed` state with the `failed` result envelope the orchestrator already returns for controlled failures, so a downstream reference or `when` predicate has one stable shape to read and the structured status contract is untouched. A Workflow Sub Agent's own resolved timeout is now mapped onto the same retryable timeout classification as the attempt deadline, so the tighter of the two bounds is retried rather than reported as an unknown execution failure. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Exercise the new slice end to end: the authored and tool-declared timeout bounds and their precedence, the combined attempt-deadline plus retry-delay ceiling, and the guarantee that a task using neither new field still freezes the previous payload. On the Activity side, cover an expired deadline for synchronous and asynchronous handlers, a nested Sub Agent timeout, a handler-raised `TimeoutError` staying an unknown execution failure, host cancellation, and a tampered persisted deadline. The synchronous case asserts the documented limitation directly: the handler is still running when the attempt is reported as timed out. On the orchestrator side, cover continuation of a terminal failure and of an exhausted native retry, a downstream `when` predicate branching on the continued result, `for_each` aggregation of a continued instance, a wait timer beside a continued task, and the four cases continuation must never absorb: an authorization failure, a contract failure, an unclassified Durable failure, and a legacy sibling that never declared continuation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Restore the `timeout="PT5S"` declaration on `reserve_inventory` so the sample shows the whole execution policy a customer writes on one tool, and describe what the attempt deadline does alongside the retry policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Replace the retry-only section with a task execution policy section covering the per-attempt deadline, its precedence rules, the combined bounded-execution ceiling, and the honest statement that the deadline bounds the wait rather than the worker thread a synchronous handler runs on. Document continuing past a failed task: that it applies only after the attempt budget is spent, the sanitized result a continued node commits, and the table of which failure kinds are continuable — `authorization` and `handler_contract` never are. Record Decisions 74-81 and the architecture-review checkpoint in FRD 0004, mark Decision 70 superseded, and update the `activity.py` / `engine.py` module map rows plus the `README.md` workflow tool example. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Tsuyoshi Ushio and others added 10 commits August 29, 2026 10:14

Tsuyoshi Ushio (TsuyoshiUshio) mentioned this pull request Aug 29, 2026

feat(workflows): task execution telemetry and structured status #187

Draft

Tsuyoshi Ushio and others added 2 commits August 29, 2026 13:19

Tsuyoshi Ushio (TsuyoshiUshio) mentioned this pull request Aug 29, 2026

feat(workflows): add Durable native Activity retry #185

Closed

Tsuyoshi Ushio and others added 6 commits August 29, 2026 14:05

Tsuyoshi Ushio (TsuyoshiUshio) force-pushed the tsuyoshiushio-workflow-timeout-continuation branch from 2fce92a to f78d313 Compare August 29, 2026 21:10

Tsuyoshi Ushio (TsuyoshiUshio) force-pushed the tsuyoshiushio-workflow-native-retry branch from 32ae3d9 to a8b15c8 Compare September 2, 2026 17:19

Navigation Menu

Uh oh!

feat(workflows): per-attempt task timeout and continue-on-error - #186

feat(workflows): per-attempt task timeout and continue-on-error#186
Tsuyoshi Ushio (TsuyoshiUshio) wants to merge 18 commits into
tsuyoshiushio-workflow-native-retryfrom
tsuyoshiushio-workflow-timeout-continuation

Tsuyoshi Ushio (TsuyoshiUshio) commented Aug 29, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Tsuyoshi Ushio (TsuyoshiUshio) commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR 2 of 3 — stacked on #185

What this adds

Replay compatibility

Known limitation (documented, not hidden)

Explicitly out of scope

Architecture review

Validation (after the rebase onto bd56857f)

Retarget / rebase after PR 1 merges

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Tsuyoshi Ushio (TsuyoshiUshio) commented Aug 29, 2026 •

edited

Loading

Validation (after the rebase onto `bd56857f`)