Add deployed ACA qualification assets

docs/frds/0008-aca-sandbox-session-runtime.md

@@ -5,7 +5,7 @@ status: Finalized
 author: larohra
 created: 2026-07-20
 updated: 2026-09-02
-issues: []
+issues: [166]
 pull_requests: []
 branch: feature/aca-sandboxes
 ---
@@ -437,6 +437,8 @@ controlling amendments.
 | 189 | Required Sandbox Group region and endpoint (revises #99/#184/#188) | ARM discovery/equality check / authored direct endpoint | Require normalized `region` beside the group resource ID and construct its regional data-plane client directly. Use no ARM lookup or fallback, and permit Function App and Sandbox Group regions to differ. | Human | 2026-09-02 | Replacement stack layer 1 |
 | 190 | ACA provider error boundary (refines #159/#184/#187/#188) | Raw SDK propagation / typed redacted boundary | Translate all SDK failures: group 401/403 authorization, group 404 binding, 429/5xx/timeout/transport transient, sandbox 404 missing backing, and 409 invalid state unless typed state is `Running` or `Resuming`. | Human | 2026-09-02 | Replacement stack layer 1 |
 | 191 | Result retention and reclaim authority | Complete run scan / monotonic session bound | `session.expires_at` never decreases and upper-bounds every result hold. Bounded scans may evict observed expired results but never block due reclaim; persisted off-page operation backing is exact-read before absence handling. | Human | 2026-09-02 | Replacement stack layer 2 |
+| 192 | Deployed ACA qualification fixture and coverage | Reuse E2E / dedicated N=5 fixture / formal N=100 fixture | Add a dedicated fixture and live suites for fresh acceptance, authenticated turn, lifecycle, backing loss, and N=5 admission/idempotency/events/results/cleanup. Keep its 120-second reclaim policy and reject N=100 preflight; Decision #29 requires a future purpose-built workflow. | Human | 2026-09-02 | Replacement stack layer 3 |
+| 193 | Lightweight in-package build provenance | No provenance / marker file / content-addressed chain | Ship a `BUILD_INFO.json` inside the deployed package and check build ID, commit SHA, and live Python minor after cold-start timing; a mismatch fails and suppresses metrics. Explicitly excludes wheel digest, installed package version, deploy-input manifest, deployment-storage chain, and rollback. | Human | 2026-09-02 | Replacement stack layer 3 |
 
 *Terminology note.* "Signed package" / "signed content package" phrasing in
 earlier decision rows (e.g. #17, #43), and the historical
@@ -1051,10 +1053,12 @@ Invariants: no anonymous ingress; no ingress ports; one active run; free slot on
   credential-source isolation, identity boundaries, SSRF defenses, static
   single-level delegation, and whole-chain timeout behavior.
 * Real ACA acceptance requires create/submit/result, stop/resume readiness,
-  loss-to-`410`, egress audit, and 100-concurrent/large-payload validation.
-  Status/event visibility, cancellation/lifecycle repair, cost, and throttling
-  must meet the documented acceptance target; anonymous ingress is never a
-  fallback.
+  loss-to-`410`, egress audit, and large-payload validation. Current deployed
+  qualification uses N=5 as an orchestration diagnostic; 100-concurrent
+  acceptance remains human-only (Decision #192). Status/event visibility is
+  observed against the documented target but does not gate; cancellation,
+  lifecycle repair, cost, and throttling remain acceptance evidence. Anonymous
+  ingress is never a fallback.
 
 All changes run ruff, strict mypy, pytest, observability/redaction checks, and
 the relevant documentation and real-ACA validation slices.
@@ -1422,7 +1426,7 @@ an enabled v1 surface.
 * Security/egress: reject unsafe defaults/bypass, rule ordering lint, static and secret credential Transform sources, group-identity boundaries, redirect/DNS-rebind revalidation, block sandbox-to-control-plane SSRF, journal/Table redaction.
 * Harness: bootstrap ABI/protocol/digest failure, no anonymous ingress, workflows/code-interpreter fail-closed, semantic golden traces every CI, advertise capability only after exercised trace.
 * Delegation: static/single-level guard, cycle/depth guard, egress union, co-location/no second run, recoverable specialist failure, whole-chain sync timeout.
-* Real ACA E2E/full-system: create-submit-result, stop-resume-ensure-ready, egress deny/transform audit, and large-payload gates are evidenced by U3. The committed deployed destructive backing-loss proof has passed; only formal human sign-off remains deferred. The committed public Easy-Auth load runner uses N=5 as the sole agent/CI diagnostic validation for orchestration and cleanup; it is not capacity evidence. Decision #29 remains human-only N=100 formal acceptance and is not passed until human-supplied N=100 evidence is available. At default preview quota, assert <=2-second p95 status/event visibility at <=1 poll/s per active stream, reliable cancellation/lifecycle repair, and acceptable cost/throttling; failure is an explicit private-ingress/load-shaping review finding, never a reason to permit anonymous ingress. Every validation slice also requires `ruff`, strict `mypy`, and `pytest`, plus docs/observability/redaction gates.
+* Real ACA E2E/full-system: create-submit-result, stop-resume-ensure-ready, egress deny/transform audit, and large-payload gates are evidenced by U3. The committed deployed destructive backing-loss proof has passed; only formal human sign-off remains deferred. Under Decision #192, a deployed qualification run exercises fresh-session acceptance, public turn, lifecycle, backing loss, and N=5 in that order with provisioning concurrency 1. N=5 is diagnostic evidence for orchestration and cleanup, not capacity. This 120-second-reclaim fixture rejects N=100 before authentication or provider work; Decision #29 remains pending a future purpose-built, human-only formal workflow. Observe status/event latency against the two-second p95 target at <=1 poll/s per active stream, but never gate on it; report reliable cancellation/lifecycle repair and acceptable cost/throttling. Failure is an explicit private-ingress/load-shaping review finding, never a reason to permit anonymous ingress. Every validation slice also requires `ruff`, strict `mypy`, and `pytest`, plus docs/observability/redaction gates.
 
 ## 11. Sandbox harness integration
 
@@ -1805,5 +1809,63 @@ BuildId-derived labels isolate concurrent runs. Fixture cleanup plus an
 remains. The smoke deploys no Function and performs no artifact attestation.
 
 ADO 298692 passed the low-level entrypoint, journal acceptance, real model turn,
-and cleanup. Post-main deployment, external attestation, py313/py314
-lifecycle/loss/N=5 qualification, and rollback remain owned by issue #166.
+and cleanup. The deployed qualification fixture, suites, and packaging/deploy
+tooling described in §14 are now committed and runnable by hand. Pipeline
+wiring for them, external attestation, and rollback remain owned by issue #166;
+there is intentionally no rollback machinery.
+
+## 14. Deployed ACA qualification assets — issue #166
+
+**Status: Finalized for the committed assets.** Decisions #192 and #193 are the
+qualification contract implemented here. This section describes committed test,
+fixture, and tooling assets only. **No pipeline wiring exists in this
+repository for them**; scheduling, gating, promotion criteria, and group sweep
+remain open under issue #166.
+
+`tests/live/apps/aca-qualification/` is the deployable qualification fixture: an
+agent app that selects the ACA Sandbox backend, authors its Sandbox Group region,
+reads every environment-specific value from app settings, and adds one
+fixture-only `/__buildinfo` route. The route lives in the fixture and imports
+nothing from product endpoint registration, so it cannot collide with product
+surface work. The fixture is deliberately outside `tests/endtoend/apps/`, whose
+suite auto-parameterizes a `func start` test over every app it finds and would
+fail on an app that requires real Azure.
+
+`eng/scripts/aca_qualification_pipeline.py` packages, deploys, and verifies that
+fixture by hand: it installs tooling, stamps `BUILD_INFO.json`, assembles an
+upload from exactly one runtime wheel plus a pinned, Oryx-compatible
+`eng/constraints/aca-fixture-requirements.txt` export of `uv.lock`, preflights
+deployment rights, configures the authored region, deploys through the normal
+Flex Consumption ZIP remote-build path, adds best-effort redacted portal metadata,
+and fetches and compares the deployed build info. Ambiguous wheel selection is a
+hard error rather than a silent "newest wins".
+
+`eng/scripts/aca_deployed_qualification.py` runs one ordered suite: the
+cold/fresh-session module first, then public turn, lifecycle, backing loss, and
+N=5 load. It invokes the cold/fresh-session module as a separate gate and does
+not start the remaining suites unless that process succeeds. Within that first
+module, fresh-session acceptance, first-event, and
+terminal timing complete *before* the same test reads the deployed marker and
+compares build ID, commit SHA, and live Python minor version. A missing or
+mismatched marker fails the test and suppresses latency metrics, so evidence
+from the wrong deployment is never reported as trustworthy.
+
+The canonical qualification uses N=5 with provisioning concurrency 1. Manual
+diagnostics retain load values 1–99 and provisioning values 1, 2, or 4 under
+operator-owned quota and cost. Both the operator command and direct live-test
+entry point reject N=100 before authentication or provider work. Formal N=100
+remains future human-only acceptance requiring a purpose-built workflow; these
+assets do not discharge Decision #29.
+
+This is deliberately lightweight in-package provenance, not a detached
+content-addressed attestation chain. Per Decision #193 it does not prove the
+wheel digest, the installed package version, a deploy-input manifest, or the
+deployment-storage version, and there is no retained-package or automatic
+rollback. A later deployment corrects a bad deployment.
+
+The qualification client is hardened for a real App Service frontend: bounded
+502/503 retries applied only to safe or explicitly idempotent requests, bounded
+result-materialization retry, SSE throttling retry that honors `Retry-After`,
+bounded event-batch visibility metrics, stable journal-acceptance assertions,
+and cleanup/recovery safeguards. All of that lives in test and live assets; no
+product runtime code participates.

eng/constraints/README.md

@@ -0,0 +1,23 @@
+# ACA fixture constraints
+
+This directory pins the dependency closure used by the ACA qualification fixture at
+`tests/live/apps/aca-qualification/`. The fixture is deployed with Azure
+Functions Flex remote build, so Oryx resolves `requirements.txt` on the server.
+Fully pinned constraints prevent those remote builds from drifting when PyPI
+publishes new compatible releases.
+
+`aca-fixture-requirements.txt` is a pip/Oryx-compatible export of `uv.lock`.
+It includes the runtime's core dependencies plus the `aca_sandbox` and
+`monitor` extras, and deliberately omits the local project because deployment
+prepends the exact wheel built by the pipeline. Platform markers keep the same
+file valid for Linux Python 3.13 and 3.14.
+
+Regenerate from the repository root with:
+
+```powershell
+uv export --frozen --no-dev --extra aca_sandbox --extra monitor --no-emit-project --no-hashes --no-header --no-annotate --format requirements.txt --output-file .\eng\constraints\aca-fixture-requirements.txt
+```
+
+The qualification-pipeline unit tests verify the checked-in export's package
+closure and exact versions against `uv.lock`, reject pip directives and source
+references, and evaluate its markers for both supported Python minors.

eng/constraints/aca-fixture-requirements.txt

@@ -0,0 +1,103 @@
+agent-framework-core==1.3.0
+agent-framework-foundry==1.3.0
+agent-framework-openai==1.3.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.14.2
+aiosignal==1.4.0
+annotated-doc==0.0.4
+annotated-types==0.7.0
+anyio==4.13.0
+asgiref==3.11.1
+attrs==26.1.0
+azure-ai-inference==1.0.0b9
+azure-ai-projects==2.1.0
+azure-containerapps-sandbox==0.1.0b4
+azure-core==1.41.0
+azure-core-tracing-opentelemetry==1.0.0b13
+azure-data-tables==12.7.0
+azure-functions==2.1.0
+azure-functions-durable==1.6.0
+azure-identity==1.25.3
+azure-mgmt-core==1.6.0
+azure-monitor-opentelemetry==1.8.8
+azure-monitor-opentelemetry-exporter==1.0.0b53
+azure-storage-blob==12.28.0
+azurefunctions-extensions-base==1.2.0
+azurefunctions-extensions-http-fastapi==1.0.1
+certifi==2026.4.22
+cffi==2.0.0 ; platform_python_implementation != 'PyPy'
+charset-normalizer==3.4.7
+click==8.3.3
+colorama==0.4.6 ; sys_platform == 'win32'
+cryptography==48.0.0
+distro==1.9.0
+fastapi==0.136.1
+frozenlist==1.8.0
+furl==2.1.4
+h11==0.16.0
+httpcore==1.0.9
+httpx==0.28.1
+httpx-sse==0.4.3
+idna==3.15
+importlib-metadata==8.7.1
+isodate==0.7.2
+jiter==0.14.0
+jsonschema==4.26.0
+jsonschema-specifications==2025.9.1
+markupsafe==3.0.3
+mcp==1.28.1
+msal==1.36.0
+msal-extensions==1.3.1
+msrest==0.7.1
+multidict==6.7.1
+oauthlib==3.3.1
+openai==2.36.0
+opentelemetry-api==1.40.0
+opentelemetry-instrumentation==0.61b0
+opentelemetry-instrumentation-asgi==0.61b0
+opentelemetry-instrumentation-dbapi==0.61b0
+opentelemetry-instrumentation-django==0.61b0
+opentelemetry-instrumentation-fastapi==0.61b0
+opentelemetry-instrumentation-flask==0.61b0
+opentelemetry-instrumentation-logging==0.61b0
+opentelemetry-instrumentation-psycopg2==0.61b0
+opentelemetry-instrumentation-requests==0.61b0
+opentelemetry-instrumentation-urllib==0.61b0
+opentelemetry-instrumentation-urllib3==0.61b0
+opentelemetry-instrumentation-wsgi==0.61b0
+opentelemetry-resource-detector-azure==0.1.5
+opentelemetry-sdk==1.40.0
+opentelemetry-semantic-conventions==0.61b0
+opentelemetry-util-http==0.61b0
+orderedmultidict==1.0.2
+packaging==26.2
+propcache==0.5.2
+psutil==7.2.2
+pycparser==3.0 ; implementation_name != 'PyPy' and platform_python_implementation != 'PyPy'
+pydantic==2.13.4
+pydantic-core==2.46.4
+pydantic-settings==2.14.1
+pyjwt==2.12.1
+python-dateutil==2.9.0.post0
+python-dotenv==1.2.2
+python-frontmatter==1.1.0
+python-multipart==0.0.28
+pywin32==311 ; sys_platform == 'win32'
+pyyaml==6.0.3
+referencing==0.37.0
+requests==2.34.2
+requests-oauthlib==2.0.0
+rpds-py==0.30.0
+six==1.17.0
+sniffio==1.3.1
+sse-starlette==3.4.4
+starlette==1.3.1
+tqdm==4.67.3
+typing-extensions==4.15.0
+typing-inspection==0.4.2
+urllib3==2.7.0
+uvicorn==0.47.0
+werkzeug==3.1.8
+wrapt==1.17.3
+yarl==1.23.0
+zipp==3.23.1

eng/scripts/README.md

@@ -68,3 +68,40 @@ role-assignment attestation.
 
 The retained `aca_deployed_qualification.py` and deployed suite helpers are
 manual/local assets only pending the separate post-main qualification work.
+
+### `aca_qualification_pipeline.py`
+
+Packages, deploys, and verifies the deployed ACA qualification fixture
+(`tests/live/apps/aca-qualification/`). Every command is run by hand; this
+repository contains no pipeline wiring for it.
+
+| Command | Purpose |
+| --- | --- |
+| `install-tooling` | Install the shared Python dependencies used by qualification runs |
+| `stamp` | Write `BUILD_INFO.json` into the fixture app before packaging |
+| `assemble` | Build the deployable upload: fixture source, the runtime wheel, the marker, and pinned requirements |
+| `deploy` | Preflight deployment rights, configure the authored region, package and deploy the staged fixture, and add best-effort portal metadata |
+| `check-build` | Verify lightweight in-package build ID, commit SHA, and Python-minor provenance |
+
+`assemble` requires exactly one runtime wheel in the build output; ambiguity is
+a hard error rather than a silent "newest wins", because deploying the wrong
+wheel is precisely the failure `check-build` exists to catch. Fixture
+dependencies come from the single Oryx-compatible
+`eng/constraints/aca-fixture-requirements.txt` export of `uv.lock`, valid for
+both supported interpreter minors.
+
+`check-build` is meaningful only because the marker is a *file inside the
+deployed package*: a file can be served only if that package is genuinely on
+disk, so a stale app cannot claim a build it is not running. An app setting or
+resource tag could be changed without deploying anything.
+
+The provenance is deliberately narrow. It does not cover the wheel digest, the
+installed package version, a deploy-input manifest, the deployment-storage
+chain, or rollback; those remain open under issue #166.
+
+The canonical qualification uses N=5 with provisioning concurrency 1 and the
+fixture's 120-second reclaim policy. Manual diagnostics retain load values 1–99
+and provisioning values 1, 2, or 4; their operator owns shared-group quota and
+cost. `aca_deployed_qualification.py` rejects N=100 before authentication or
+provider work with `formal_n100_unsupported_by_qualification_fixture`. Formal
+N=100 remains future human-only acceptance requiring a purpose-built workflow.

eng/scripts/aca_deployed_qualification.py

@@ -22,11 +22,15 @@
     "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_TABLE_NAME",
     "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_APP_SUBSCRIPTION_ID",
     "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_APP_SITE_NAME",
+    "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_BUILD_ID",
+    "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_COMMIT_SHA",
+    "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_PYTHON_VERSION",
     "AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_GROUP_RESOURCE_ID",
     "AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_REGION",
 )
 _PROVISION_CONCURRENCIES = frozenset({1, 2, 4})
 _SMOKE_RUN_ID = "AZURE_FUNCTIONS_AGENTS_ACA_SMOKE_RUN_ID"
+FORMAL_N100_UNSUPPORTED_ERROR = "formal_n100_unsupported_by_qualification_fixture"
 
 
 class QualificationError(Exception):
@@ -66,29 +70,28 @@ def validate_deployed_environment(
     provision_concurrency: str | None = None,
 ) -> tuple[int | None, int | None]:
     """Validate redacted deployed-suite inputs and shared-group limits."""
-    for name in _DEPLOYED_ENVIRONMENT:
-        _required(environment, name)
-    if runtime_target not in {"both", "python313", "python314"}:
-        raise QualificationError("invalid_runtime_target")
-
     load = (
         _integer(load_concurrency, name="acaLoadConcurrency", minimum=1, maximum=100)
         if load_concurrency is not None
         else None
     )
+    if load == 100:
+        raise QualificationError(FORMAL_N100_UNSUPPORTED_ERROR)
     provision = (
         _integer(provision_concurrency, name="acaProvisionConcurrency", minimum=1, maximum=4)
         if provision_concurrency is not None
         else None
     )
+    for name in _DEPLOYED_ENVIRONMENT:
+        _required(environment, name)
+    if runtime_target not in {"both", "python313", "python314"}:
+        raise QualificationError("invalid_runtime_target")
     if provision is not None and provision not in _PROVISION_CONCURRENCIES:
         raise QualificationError("invalid_provision_concurrency")
     if runtime_target == "both" and load is not None and load > 5:
         raise QualificationError("dual_runtime_load_concurrency_requires_single_runtime")
     if runtime_target == "both" and provision is not None and provision > 1:
         raise QualificationError("provision_concurrency_requires_single_runtime")
-    if load == 100 and environment.get("BUILD_REASON", "").strip() != "Manual":
-        raise QualificationError("formal_n100_requires_manual_build")
     return load, provision
 
 
@@ -154,7 +157,7 @@ def run_deployed_suite(
     load_concurrency: str,
     provision_concurrency: str,
 ) -> int:
-    """Run the protected deployed turn, lifecycle, loss, and N=5 smoke suite."""
+    """Gate turn, lifecycle, loss, and load on cold-start provenance."""
     load, provision = validate_deployed_environment(
         environment,
         runtime_target=runtime_target,
@@ -165,7 +168,16 @@ def run_deployed_suite(
     inherited = dict(environment)
     inherited["AZURE_FUNCTIONS_AGENTS_ACA_LOAD_CONCURRENCY"] = str(load)
     inherited["AZURE_FUNCTIONS_AGENTS_ACA_PROVISION_CONCURRENCY"] = str(provision)
+    samples = validate_cold_start_samples(inherited)
+    if samples is not None:
+        inherited["AZURE_FUNCTIONS_AGENTS_ACA_COLD_START_SAMPLES"] = str(samples)
     preflight_auth(inherited)
+    cold_start_result = _run_pytest(
+        ("tests/live/test_aca_deployed_cold_start.py",),
+        inherited,
+    )
+    if cold_start_result != 0:
+        return cold_start_result
     return _run_pytest(
         (
             "tests/live/test_aca_deployed_agent_turn.py",