samples: benchmark Dynamic Workflow token reduction

[Tsuyoshi Ushio (TsuyoshiUshio)]

Summary

• add a reproducible E2E sample comparing ordinary tool calling with Dynamic Workflow
• run both modes against identical deterministic evidence and require byte-equivalent canonical reports
• preserve the existing compact token event and add a default-off, versioned detail event containing all numeric MAF usage dimensions
• capture provider-reported input/output/cache usage and report paired reductions across increasing workload sizes
• measure report-completion latency and output quality with a deterministic oracle plus a blinded Vally LLM judge

Initial E2E results (retained)

Using Foundry gpt-5.4-mini with 40 evidence lines per service:

Services	Baseline tokens	Dynamic Workflow tokens	Reduction
1	12,723	4,522	64.5%
3	31,990	4,816	84.9%
5	51,398	5,045	90.2%
10	98,854	5,805	94.1%

The canonical reports matched for every valid pair. The 20-service baseline processed only 2 of 20 services, so that pair is invalid and intentionally excluded from the reduction calculation.

This initial run used the machine's ambient editable runtime install. It remains here as the original observation, but the harness now prepends this checkout's src/ to the worker PYTHONPATH so subsequent runs cannot resolve another worktree.

Detailed opt-in E2E results

After enabling AZURE_FUNCTIONS_AGENTS_DETAILED_TOKEN_USAGE=true and pinning the worker to this checkout, the same workload produced:

Services	Baseline input	Baseline output	Baseline total	Workflow input	Workflow output	Workflow total	Reduction
1	10,320	2,415	12,735	5,436	179	5,615	55.9%
3	24,833	7,163	31,996	5,595	304	5,899	81.6%
5	39,481	11,941	51,422	5,758	418	6,176	88.0%
10	75,242	23,621	98,863	6,117	655	6,772	93.2%

All four canonical report pairs matched exactly. Input-token reduction increased from 47.3% to 91.9%, while output-token reduction ranged from 92.6% to 97.2%.

Foundry also reported openai.cached_input_tokens:

Services	Baseline cached	Workflow cached	Baseline uncached input	Workflow uncached input
1	5,632	4,608	4,688	828
3	15,872	4,608	8,961	987
5	25,600	2,560	13,881	3,198
10	49,152	5,120	26,090	997

The baseline's cached input, uncached input, and output all grow with service count because evidence returns through the model's tool-calling loop. Dynamic Workflow keeps intermediate evidence in Activities, so model usage stays comparatively flat; its growth is primarily the workflow plan itself. The provider emitted no reasoning-token dimension in these trials, so none is inferred.

Repeated latency and quality results

A second run used the same Foundry gpt-5.4-mini deployment, 40 evidence lines per service, alternating execution order, and three paired repeats per workload size. report_latency_ms measures queue submission through canonical report Blob availability; it excludes host/emulator startup and token-log validation.

Services	Valid pairs	Baseline tokens (median)	Workflow tokens (median)	Token reduction (paired median)	Baseline latency ms (median)	Workflow latency ms (median)	Latency reduction (paired median)
1	3/3	12,744	5,609	56.0%	38,635	7,132	77.2%
3	3/3	31,996	5,867	81.7%	84,272	8,629	88.5%
5	3/3	51,295	6,174	88.0%	134,446	8,112	94.3%
10	3/3	99,097	6,781	93.1%	247,052	12,695	95.1%

Measured accuracy was equivalent: Baseline and Dynamic Workflow received identical deterministic-oracle and Vally overall scores.

The independent field-level oracle scored all 24 mode outputs 1.0 with exact pass. Missing, wrong, reordered, or fabricated fields reduce this score. Vally 0.14.0 then graded blinded ATIF trajectories for repeat 1 with a fixed rubric and gpt-5.4-mini: all eight Baseline/Workflow outputs scored 1.0 overall and passed every criterion. Three of 32 individual rubric scores were 4/5 for raw-JSON presentation; every factual score was 5/5. No quality loss was observed for this structured task, but this does not establish equivalence for open-ended tasks.

The machine-readable run is committed under samples/workflow-token-benchmark/results/2026-08-28-foundry/, including paired benchmark JSON, 24 ATIF trajectories, eight Vally JSONL verdicts, commands, environment, and limitations. Python 3.14.0 was used because the installed Python 3.13.15 Functions worker exited with Windows access violation 0xC0000005 during indexing; those failed startups produced no observations.

Runtime telemetry

The existing Agent token usage: {json} event is unchanged. When explicitly enabled, the runtime emits a separate Agent token usage detail: {json} event with schema_version: 1 and the provider/MAF numeric dimensions. Detail normalization and logging are separately guarded so they cannot suppress the stable event. Invalid or missing provider dimensions are not estimated.

The runtime change and benchmark capture are separate commits so the opt-in feature can be reverted without removing the sample. Latency/quality support and the recorded result set are also separate commits.

Validation

• python -m ruff check src tests
• python -m mypy src
• python -m pytest --cache-clear --cov=./src/azure_functions_agents --cov-report=xml --cov-branch tests — 1,084 passed
• benchmark sample tests — 20 passed
• Vally 0.14.0 eval spec lint
• repeated local E2E workload curve at 1, 3, 5, and 10 services
• blinded Vally prompt grading for eight representative outputs

Temporary dependency

This PR temporarily includes the ExecPlan guidance commits from #174, which were used while developing the sample. Those commits will be removed from this PR before final readiness.