feat(workflows): add Durable native Activity retry

[Tsuyoshi Ushio (TsuyoshiUshio)]

Draft — PR 1 of 3. This is the minimal, independently mergeable slice of #170.
Per-attempt timeout and continue_on_error are PR 2 (#186). Retry/timeout observability and the structured status contract + UI cards are PR 3 (#187). Neither is in this PR.

What this adds

A workflow task can opt into Durable Functions native Activity retry. Retry is declared where the knowledge of what is safe to retry actually lives — on the tool:

@workflow_tool(
    description="Reserve inventory for an order.",
    retry=WorkflowRetryPolicy(
        max_attempts=3,
        backoff=WorkflowRetryBackoff(initial="PT1S", multiplier=2.0, max="PT4S"),
    ),
)
def reserve_inventory(args: dict[str, Any]) -> dict[str, Any]:
    if _inventory_unavailable():
        raise WorkflowRetryableError(
            "inventory_temporarily_unavailable",
            "Inventory reservation is temporarily unavailable.",
        )
    ...

A plan may also author execution.retry on a task, but a tool declaration wins, so retryability is not left to what the model happened to write.

Design notes

Durable owns retry. There is no orchestrator-managed retry loop, no retry timers, and no attempt state in the scheduler — that would duplicate scheduling Durable already does.

RetryPolicy has no exception predicate, so the Activity splits the two outcomes itself: a retryable failure raises a private versioned marker carrying a sanitized payload, and a terminal failure returns a structured outcome so Durable treats the attempt as complete. On exhaustion the sanitized failure is decoded back out of TaskFailedError, so a workflow reports the application's own error_code rather than an opaque Durable message. No raw exception text reaches Durable history.

Per-attempt timeout is not required by native retry, so it is not here. The only bound is an internal one-hour retry_timeout ceiling; the authored execution.timeout surface is PR 2.

Replay safety. The retry driver and the result envelope are selected from the persisted orchestration input alone, never from the currently deployed tool registration. A workflow started before its tool declared retry keeps replaying through call_activity with the legacy envelope — asserted explicitly by a test.

Forward compatibility. Models that read the policy back out of Durable history ignore unknown keys, so PR 2 can add fields without breaking replay of workflows started on this version.

Durable Python 2.x adoption

1.x RetryOptions can only express a first interval and an attempt count, so an authored exponential backoff cannot be honored. The bump to azure-functions-durable==2.0.0b2 is therefore intrinsic, and it carries four pieces of forced collateral — each one required, and each proven necessary by an observed failure rather than assumed:

1. DurableFunctionsClient binding type and Task.result now raising instead of returning its failure; 2.x trigger decorator typing.
2. SSE Durable client lifetime. The client injected by durable_client_input is closed when the invocation returns, but an SSE response generator is consumed after that — a workflow-enabled agent could not start a workflow over the streaming chat route at all. The route now takes the raw durableClient binding config, owns one client per stream, and closes it when the stream ends.
3. E2E customer dependency layout. durabletask depends on grpcio and the Functions Python worker bundles its own. With no customer dependency path the worker resolved a mixed grpcio/protobuf set and segfaulted (exit 139 on Linux, 0xC0000005 reproduced locally on Windows) while importing function_app.py — every E2E app failed to start, including minimal-http and builtin-endpoints, which never touch workflows. Each app with a host.json now gets a .python_packages tree pointing at one shared install, which is what a deployed app already looks like.
4. Wave selection. Durable's CompositeTask subclasses never notify their own parent — only leaf CompletableTask.complete() propagates. The existing task_any([cancel_event, task_all(wave)]) nests one composite inside another, so the inner completion never reached the outer selection and the orchestrator hung after its first wave: no workflow could make progress at all under 2.x. The engine now selects over the individual wave tasks so every child of the awaited task is a leaf. Wave semantics are unchanged — results stay in wave order, cancellation still restores the wave and cancels pending timers, and the first failure in wave order is surfaced exactly as task_all did.

Extension bundle deliberately unchanged. The worker log confirms the host's LATEST release channel already selects bundle 4.37.1 for the existing [4.*, 5.0.0) and [4.32.0, 5.0.0) ranges, so raising the floor is cosmetic and stays out.

Real-host E2E evidence

tests/endtoend/test_workflow_native_retry_e2e.py boots the workflow-retry-policy sample under func start and drives its workflow through Durable's orchestration HTTP API. Going straight to Durable keeps both cases deterministic and model-free: what is under test is the runtime's retry behaviour, not an agent's ability to author a plan. The customer sample itself is unchanged.

1 — transient failure is retried, workflow completes

runtimeStatus: Completed          customStatus: 3/3 tasks done
reserve_inventory.reserved                     = True
reserve_inventory.transient_failures_observed  = 2
confirm_order.status                           = confirmed

The sample's inventory tool raised WorkflowRetryableError on the first two deliveries; Durable re-delivered the Activity and the third attempt succeeded.

2 — exhausted budget fails with the application's error code

runtimeStatus: Failed
output: builtins.RuntimeError: task 'reserve_inventory':
        Inventory reservation is temporarily unavailable. (inventory_temporarily_unavailable)
incident after run: transient_failures_observed = 3, failures_remaining = 96 (seeded 99)

The sanitized application error_code survives instead of degrading to an opaque TaskFailedError, and the incident counter shows exactly max_attempts = 3 deliveries — no more, no fewer.

Both cases pass in CI on Python 3.13 and 3.14 (agent-runtime.e2e-tests build 301106, 55 passed per leg).

Scope

Included: the retry declaration and its bounds; WorkflowRetryableError / WorkflowTerminalError; the per-delivery task context with an attempt-stable idempotency key; the Durable mapping and failure bridge; policy-aware Activity execution; orchestrator dispatch and replay compatibility; @workflow_tool(retry=...) plumbing; the workflow-retry-policy sample; unit + real-host E2E tests; docs and FRD 0004 Decisions 66-73.

Deliberately excluded: execution.timeout, continue_on_error, workflow-task telemetry, status schema_version 3/4 and the UI status cards, the read-only Skill approval change, and the extension-bundle floor sweep.

Dependency lock

uv.lock is relocked because the docs workflow syncs with --locked. Resolution adds durabletask and its grpcio / protobuf / asyncio dependencies, drops furl and orderedmultidict, and moves azure-functions from 2.1.0 to 2.3.0 — forced, because azure-functions-durable==2.0.0b2 requires azure-functions>=2.3.0b2. Nothing else is upgraded. Artifact URLs for unchanged packages are kept as already committed, because the Microsoft feed proxy hands out a different mirror host on every resolution and would otherwise rewrite all ~900 URLs for no change in content.

Validation

• ruff check src tests — clean
• mypy src (strict) — clean
• pytest --cache-clear --cov=./src/azure_functions_agents --cov-report=xml --cov-branch tests — 1134 passed; new modules at 85%+
• pytest -m e2e tests/endtoend/test_workflow_native_retry_e2e.py — 2 passed, evidence above
• pytest -m e2e tests/endtoend/test_apps_start.py — all curated E2E apps start
• uv lock --check clean; uv sync --extra docs --no-install-project --locked and uv run --no-sync mkdocs build --strict both pass
• agent-runtime.e2e-tests — green on Python 3.13 and 3.14

About the red agent-runtime.public-build unit job

Those 7 tests/test_trigger_serialization.py failures are a pre-existing broken baseline on main, not a regression from this branch:

Build	Ref	Result
300539 (Aug 26)	main @ 50ae6c1b	succeeded
300966 (Aug 28)	main @ 94f92130 (current HEAD)	failed — same 7 tests, 1095 passed
301107 (Aug 29)	this PR	failed — same 7 tests, 1134 passed

The failing set is identical; this branch only raises the passing count. The trigger is upstream: azure-functions 2.3.0 was published on 2026-08-28 16:14 UTC, and the repo's unconstrained >=2.1.0,<3 range picked it up on the next build. 2.3.0 changed the SDK-type trigger serialization surface (body_encoding removed, Event Hub / Service Bus metadata reshaped). Both builds install the same azure_functions-2.3.0-py3-none-any.whl, so this PR's Durable pin is not involved.

Fixed separately in #184, which this PR deliberately does not absorb; once it lands, a rebase clears these.

Related: #170 (not modified), #184 (independent baseline fix), #186 / #187 (stacked follow-ups).

[Tsuyoshi Ushio (TsuyoshiUshio)]

So many changed. So that we replace this PR to https://github.com/Azure/azure-functions-agents-runtime/pull/207/changes#diff-7e4b0d797cf1d925b351ba9c8a0cdb95ec9210f3d5f7828b1336a4dfa6ac49da