feat(workflows): task execution telemetry and structured status

[Tsuyoshi Ushio (TsuyoshiUshio)]

Stack

This is PR 3 of 3 in the decomposition of #170.

Order	PR	Branch	Base
1	#185 — Durable native Activity retry	tsuyoshiushio-workflow-native-retry	main
2	#186 — per-attempt timeout + continue_on_error	tsuyoshiushio-workflow-timeout-continuation	tsuyoshiushio-workflow-native-retry
3	this PR — execution observability	tsuyoshiushio-workflow-execution-telemetry	tsuyoshiushio-workflow-timeout-continuation

The base is deliberately #186's branch, not main, so the GitHub diff shows
only this slice. Review #185 → #186 → this PR in order.

Rebased 2026-08-29 onto #186's new head f78d3135 (which now includes
#185's _await_wave / _first_wave_failure fix). The nine PR-3 commits are
preserved. See Rebase notes.

What this adds

The task execution policy is only useful if an operator can see what it did.
Two surfaces carry that, split along the Durable replay boundary.

1. Activity-side telemetry

Each delivery of a policy-aware task records one workflow.task.activity span
from the Activity — never from the orchestrator, which re-executes from
history on every replay and would re-emit the span for work that already
happened. Deliveries a policy denies before the handler runs (authorization,
unregistered target, malformed persisted input) record the same span, so the
deliveries an operator most needs to see are not the ones without telemetry.

Attributes are af.workflow_task.*: workflow_id, task_id,
node_instance_id, target_type, target_name, max_attempts, timeout_ms,
continue_on_error, retry_driver, plus the terminal outcome_kind,
error_code, and disposition.

disposition reports what the Activity did — return_result,
request_durable_retry, return_failure, abort — never what Durable will
decide, which the Activity cannot see. The attempt number stays absent for the
same reason (FRD 0004 Decision 73).

Two counters (azure_functions_agents.workflow_task.attempts / .outcomes)
carry a deliberately narrow dimension set. Workflow ids, node instance ids,
target names, and handler-authored error codes would create one time series per
task attempt, so they stay on the span.

_ensure_metrics now creates each instrument in its own guard: a meter that
rejected one instrument previously nulled every counter in the runtime.

2. Structured custom_status schema version 3

A dynamically controlled plan whose persisted input froze at least one
execution policy publishes schema_version: 3. It adds
retry_driver: "durable", the counts pending / failed / failed_continued,
the per-node max_attempts, and the sanitized last_failure_kind /
last_error_code of a continued failure.

The node whose failure ends a workflow is now marked and published before the
exception propagates, so the last status names it instead of freezing on the
pre-failure snapshot.

3. UI

formatWorkflowStatus accepts version 3 and renders the new counts, the frozen
budget, and a continued failure's code. Two small CSS additions let a longer
status line wrap inside the card instead of overflowing it. Nothing else in the
UI changed.

Compatibility

• Version 2 is byte-identical for a plan that froze no execution policy —
  same keys, same values, same materialized_total semantics. Version 3 is
  gated on the persisted orchestration input alone, so an older history replays
  unchanged and the version cannot differ between replays.
• A version 2-only client is unaffected: it sees an unknown
  schema_version and degrades to compact JSON, exactly as it already must for
  any future version.
• A task with no frozen policy is untouched: no span, legacy
  {"id", "result"} Activity envelope, string custom_status for static plans.
• No new scheduler state. failed_continued is a reporting projection.
  _aggregate_dynamic_node copies the instance state into the persisted
  {index, status, result} aggregate, so a continued instance stays
  completed and carries private metadata only the status reads. Every
  terminal-state predicate in the scheduler is unchanged.
• Rollback still follows FRD 0004 Decision 81: drain or terminate in-flight
  workflows started by the newer runtime.

Rebase notes

#186 was rebased onto #185's f7bc03e7 ("select over wave tasks instead of
nesting composites"), which replaced task_all(wave) with _await_wave /
_first_wave_failure. This branch was replayed onto the new head with
git rebase --onto + --force-with-lease; all nine commits are intact and no
nested task_all was reintroduced.

Two conflicts, both resolved toward the new flow:

• engine.py, static scheduler — _decode_wave_failure now receives the
  single failure _first_wave_failure selected rather than the whole wave
  result.
• docs/frds/0004-dynamic-workflows.md — the two slices' review entries are
  both kept.

The new flow also improved one behavior. _await_wave reports each node's own
outcome in wave order, so the instance that ends a workflow is now identified
positionally instead of by decoding a sanitized marker. An opaque Durable
failure is therefore attributed just as precisely as a classified one (only the
reason is still left to the Durable error), and _decode_wave_failure keeps
its original single-value signature — so this PR no longer touches it at all.
Decision 88 and its test were updated accordingly; the engine diff shrank by 15
lines.

The PR-3 orchestrator test harness was ported to the new
context.task_any([cancel, *wave_tasks]) selection loop, matching the harness
#186 already uses.

Explicitly out of scope

Read-only Skill approval, extension bundle sweeps, unrelated runner behavior,
unrelated UI cleanup, further SSE/streaming client changes, sample rewrites,
plans/ ExecPlan files, uv.lock churn, and all other #170 collateral. No
change to config/schema.py, so front-matter-reference.md is unaffected.
status_envelope() already passes custom_status through unchanged, so no
public serialization change was needed.

Reviews

• Architecture review (independent, pre-implementation) raised 5 blocking
  findings: a new failed_continued scheduler state would have changed the
  shipped aggregate contract; the slice claimed exhaustion observability the
  runtime did not have; counters reused high-cardinality span attributes;
  failure spans would have closed as successful (failures are returned) while
  the private retry marker would have been recorded as an exception on every
  attempt; and the FRD did not yet record the contract. All resolved —
  Decisions 82–89.
• Code review (independent, post-implementation) raised one medium finding:
  the early-failure path fed unvalidated history values into metric
  dimensions. Fixed (Decision 90), plus a corrected truncation-order claim and a
  for_each instance error code the card was dropping.

Validation (re-run after the rebase)

• ruff check src tests — clean
• mypy src (strict) — clean
• Targeted: telemetry/status/UI + workflow engine + native retry + task timeout
  — 167 passed
• Full unit gate: pytest --cov — 1203 passed, 89% total coverage
• mkdocs build --strict — clean
• Focused UI tests run under Node 24 (not skipped)

Real Functions-host E2E for PR 1 now lives in #185 (bd56857f) and is run
across the stack separately.

Retarget / rebase plan

1. feat(workflows): add Durable native Activity retry #185 merges to main.
2. feat(workflows): per-attempt task timeout and continue-on-error #186 is retargeted to main and rebased; its diff reduces to slice 2.
3. This PR is retargeted to main and rebased onto the merged feat(workflows): per-attempt task timeout and continue-on-error #186.

Every commit here touches only PR-3 files and depends on PR-2 semantics but not
PR-2 commit ids, so the series stays rebaseable and cherry-pickable.