Experimental hybrid ACA sandbox tool execution through APIM

[Laveesh Rohra (larohra)]

Purpose

• Add a private experimental hybrid execution spike: MAF orchestration and remote MCP stay in Azure Functions through APIM, while customer and generic local tools execute in one fresh customer-owned ACA Sandbox per top-level invocation.
• Discover/import customer tools only inside the sandbox, expose inert worker stubs, and route exact stub identities through MAF FunctionMiddleware over a strict file-journal protocol.
• Keep active runs safe from auto-suspend, then arm terminal lifecycle and attempt the existing five-second invocation-handle delete initiation without awaiting its LRO. If initiation fails before acceptance, invoke one separately bounded exact-ID provider deletion; completed output remains authoritative and the 300/600 lifecycle plus scoped reaper remain backstops.
• Upload the deterministic app artifact once, verify its SHA-256 inside the sandbox before extraction/readiness, and support an explicit private bundle-root boundary without implicit include globs.
• Add bounded content-free progress/latency telemetry, a deployable sample, benchmark helpers, FRD/architecture documentation, machine-readable live results, and a live-first leadership demo package with redacted Azure evidence.
• This PR is stacked on larohra/aca-qualification-sweep and is intentionally not production-ready or a public API commitment.

Does this introduce a breaking change?

[ ] Yes
[x] No

Pull Request Type

[x] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[x] Documentation content changes
[x] Other... Experimental deployed spike

How to Test

git clone https://github.com/Azure/azure-functions-agents-runtime.git
cd azure-functions-agents-runtime
git checkout larohra/hybrid-sandbox-apim-spike
pip install -e .[dev]
python -m ruff check src tests
python -m mypy src
python -m pytest --cache-clear --cov=./src/azure_functions_agents --cov-report=xml --cov-branch tests

• Cleanup-fallback focused hybrid/transport/observability suite: 163 passed.
• Full gate for fd6589e: Ruff passed; mypy checked 106 source files; pytest passed 2,574 tests with 71 skipped and 85 deselected; coverage XML generated.
• Final evidence checks parsed all JSON, compiled/linted the recording helpers, rendered the changed deck slides, and verified six slides, six notes parts, and zero external relationships.

What to Check

• Worker hybrid mode never imports or executes customer tools; discovery and execution occur only in the invocation sandbox.
• The optional AZURE_FUNCTIONS_AGENTS_EXPERIMENTAL_HYBRID_TOOL_BUNDLE_ROOT is contained and fail-closed; absence preserves full app-root capture, while the sample owns its tools under sandbox_bundle/.
• Package delivery never reads app.zip back. The executor validates SHA-256 and publishes only bounded startup failure metadata before readiness/manifest.
• Acquisition awaits auto_suspend=None before package/model/tool work. Normal close drains, marks shutdown, and applies Disk 300/600 lifecycle before delete initiation.
• The normal cleanup path still bounds handle.request_delete() at five seconds and never awaits its LRO. A pre-acceptance timeout/exception goes directly to one exact-ID provider.delete_sandbox() attempt under the existing 24-second completed-run budget, without retrying the stalled handle seam.
• SandboxNotFoundError is successful cleanup. Dual failure preserves completed output and leaves terminal lifecycle/reaper backstops armed.
• sandbox_delete_fallbacks records fallback use; overall terminal cleanup failure remains sandbox_delete_failures. Confirmed deletion/duration metrics are emitted when the provider fallback completes.
• Exact local stubs route to the sandbox while lazy MCP tools call next and remain in Functions.
• Progress events expose only fixed phase/status and optional duration; public chat SSE and non-hybrid behavior are unchanged.

Independent Review Resolution

• Fixed completed-run cleanup, app-scoped reaping, production model fallback, exception-path tool failure metrics, and structured SSE benchmark validation in 4bc61dc.
• Replaced full archive readback with in-sandbox SHA-256 and added an explicit bundle-root contract that preserves arbitrary imports, helpers, package data, and vendored dependencies.
• Added an early executor failure marker, separate upload/verify telemetry, bounded progress enums, and regression coverage for lifecycle ordering, cancellation/disconnect, policy/initiation failures, idempotency, bundle validation, and digest mismatch.
• Added the approved exact-ID provider fallback in fd6589e; an independent review found no blocking correctness, lifecycle, timeout, cancellation, telemetry, or output-preservation issue.
• Compared fd6589e with bcdd366 without cherry-picking. Both implement the same bounded safety seam; this PR retains fd6589e because sandbox_delete_fallbacks has unambiguous pre-acceptance semantics while sandbox_delete_failures remains the overall failure signal.
• Accepted for this spike: one broken customer module rejects the whole invocation before exact-manifest admission.

Latest authoritative live qualification

• Deployed exact fix commit fd6589eb0df88be39c8777caa807946fc513d8d2 through the proven Flex remote-build ZIP path with build ID pr200-fd6589e-final. Wheel SHA-256: b3cc3b2089759cd43a9ce530b5e274e2179b17230158156517cd24cd9ac43a13; upload ZIP SHA-256: bc820818ed771f0fe68c735e0ace8841290042fd29c5727f9b2f58c1f686607a.
• Downloaded released package SHA-256 038a89a9be84c3940cb40fa224db0728cb85fd6c242c778ef2079c99da63fb4a matched the changed fd6589e runtime source after Windows CRLF normalization. The fd fallback metric was present and the bcdd-only metric absent.
• The one authorized restart completed at 2026-09-05T01:22:21.3350812Z. A 300.775-second attribution boundary then observed zero agent-chat requests and zero fd/bcdd fallback rows before load.
• Exact nonstream sequence: cold N=1, c1 N=10, c10 N=10 using customer-probe-alpha-repeat-3; exactly 21 requests, all HTTP 200, zero client errors, no extra request/c25/c30/duplicate stage.
• Cold: 22,021.121 ms, 0.045410 req/s.
• c1: p50/p95/p99 12,862.314/17,524.462/17,524.462 ms, 0.077131 req/s.
• c10: p50/p95/p99 22,490.808/25,900.302/25,900.302 ms, 0.385972 req/s.
• Canonical 77ef399 baseline: cold 27,705.7 ms; c1 p50 19,720.69 ms at 0.047903 req/s; c10 p50 27,559.73 ms at 0.307089 req/s.
• Runtime-average latency was 13,218.470 ms: 7,274.530 ms (35.50%) below baseline. Cold/c1/c10 p50 improved 20.52%/34.78%/18.39%; c1/c10 throughput improved 61.01%/25.69%.
• Versus the historical 2026-09-03 optimized run, runtime average was 2,569.112 ms (24.12%) slower, consistent with 17 requests paying the five-second handle bound before successful fallback. c10 throughput was effectively unchanged (-0.25%). Remaining differences are natural model/platform variance across days.

Runtime, cleanup, APIM, and token evidence

• Function clean window: 21 requests, zero errors, p50/p95/p99 14,052.128/20,320.389/20,893.798 ms.
• Runtime counts: 21 requests, creates, lifecycle handoffs, and tool calls; 42 model calls; four normal delete acceptances; 17 exact-ID provider fallbacks; 17 confirmed provider deletions; no hybrid failure or bcdd-only metric.
• Provider fallback: 17/17 confirmed, 592.645 ms average, 464.601-1,098.475 ms range. Affected-path cleanup cost averaged about 5,592.645 ms including the handle bound; weighted across all requests it was about 4,527.379 ms/request. No cleanup replaced completed output.
• Component averages: create 3,148.713 ms; upload 103.177 ms; verify 0.160 ms; executor ready 464.447 ms; model 1,749.332 ms; tool 0.283 ms; queue 23.734 ms; transfer 159.013 ms; lifecycle handoff 77.439 ms.
• Client-observed APIM dependencies: 42 successful model POSTs, 44 successful MCP POSTs, and 22 expected MCP close 405s; zero unexpected errors. Model p50/p95/p99 was 1,461.384/2,006.946/2,633.163 ms; MCP POST p50/p95/p99 was 75.704/171.044/187.080 ms.
• APIM AzureDiagnostics contained no row for the clean window or preceding day after the final read-only poll at 2026-09-05T01:47:06.9293196Z. Backend/gateway-only percentiles and gateway cache count are therefore unavailable and are not inferred from client spans. APIM policies, diagnostics, identity, egress, and capacity were unchanged.
• Tokens across 42 model calls: 46,662 prompt, 39,936 cached prompt, 1,505 completion, 48,167 total.
• Progress traces retained 15 sampled core lifecycle series and two sampled tool start/completion pairs. Aggregate metrics, not sampled trace-event counts, are authoritative.
• Typed inventory was exactly zero initially, before/after cold, before/after c1, before/after c10, at completion, and again at 2026-09-05T01:42:17.603235Z. No operator cleanup occurred.

Diagnostic history preserved, excluded from current claims

• 2026-09-04 streaming attempt: stopped at the inventory gate after four of 21 planned requests; a shell-chaining defect admitted three c1 requests; c10 never started. All four delete initiations stalled; scoped reaping eventually returned inventory to zero. No percentile comparison is published.
• First corrected nonstream canary: one HTTP-200 cold request completed in 22,700.556 ms, but the handle delete stalled while awaiting response headers. Fail-closed orchestration stopped before c1/c10; scoped provider deletion later returned 200 in 244 ms and inventory reached zero.
• Mixed-deployment attempt: all 21 requests completed, but another session deployed bcdd366 during c1 and c10 ran after that boundary. The bcdd-only metric proved mixed provenance, so the complete artifact is diagnostic only and none of its latency values is used.

Other Information

• Existing live deployment also passed shared executor, shell/file/search, MCP-through-APIM, identity 200/403, egress 200/403, streaming/disconnect, timeout cleanup, same-sandbox queueing, and explicit orphan-reaper scenarios.
• Retained state: Function App Running/Normal, six functions, bundle root sandbox_bundle, no debug/worker OTel/always-ready, Sandbox Group max 100/Succeeded/timeout 1800, final typed inventory zero. No teardown or unrelated resource change occurred.
• The revised live-first demo shows an expanded three-tool-call chat, synchronized active ACA Sandbox inventory, returned ALPHA/BETA result, APIM policy evidence, same-run telemetry analysis, and final typed inventory zero.
• Leadership assets under docs/demo/hybrid-sandbox-leadership/ include the storyboard, narration, schema-v3 manifest, recording helpers, eleven redacted 1920x1080 stills, and the refreshed six-slide PowerPoint. The deck now carries the 2026-09-05 35.50% result, updated lifecycle/fallback data, speaker notes, and zero external relationships.
• A new local 1920x1080 result-scene clip is recorded in the manifest. The external v4 final video retains the historical 2026-09-03 qualification because regenerating its live-capture workflow would issue prohibited additional Function requests.
• Complete evidence: docs/decisions/0009-hybrid-sandbox-tool-execution-spike.md and docs/decisions/0009-hybrid-sandbox-tool-execution-results.json.
• Azure resources remain in larohra-test-adc-tools-hosted-skill; cleanup commands are documented. No teardown was performed.