diff --git a/docs/frds/0008-aca-sandbox-session-runtime.md b/docs/frds/0008-aca-sandbox-session-runtime.md index 9197829f..2864ec19 100644 --- a/docs/frds/0008-aca-sandbox-session-runtime.md +++ b/docs/frds/0008-aca-sandbox-session-runtime.md @@ -5,7 +5,7 @@ status: Finalized author: larohra created: 2026-07-20 updated: 2026-09-02 -issues: [] +issues: [166] pull_requests: [] branch: feature/aca-sandboxes --- @@ -437,6 +437,8 @@ controlling amendments. | 189 | Required Sandbox Group region and endpoint (revises #99/#184/#188) | ARM discovery/equality check / authored direct endpoint | Require normalized `region` beside the group resource ID and construct its regional data-plane client directly. Use no ARM lookup or fallback, and permit Function App and Sandbox Group regions to differ. | Human | 2026-09-02 | Replacement stack layer 1 | | 190 | ACA provider error boundary (refines #159/#184/#187/#188) | Raw SDK propagation / typed redacted boundary | Translate all SDK failures: group 401/403 authorization, group 404 binding, 429/5xx/timeout/transport transient, sandbox 404 missing backing, and 409 invalid state unless typed state is `Running` or `Resuming`. | Human | 2026-09-02 | Replacement stack layer 1 | | 191 | Result retention and reclaim authority | Complete run scan / monotonic session bound | `session.expires_at` never decreases and upper-bounds every result hold. Bounded scans may evict observed expired results but never block due reclaim; persisted off-page operation backing is exact-read before absence handling. | Human | 2026-09-02 | Replacement stack layer 2 | +| 192 | Deployed ACA qualification fixture and coverage | Reuse E2E / dedicated N=5 fixture / formal N=100 fixture | Add a dedicated fixture and live suites for fresh acceptance, authenticated turn, lifecycle, backing loss, and N=5 admission/idempotency/events/results/cleanup. Keep its 120-second reclaim policy and reject N=100 preflight; Decision #29 requires a future purpose-built workflow. | Human | 2026-09-02 | Replacement stack layer 3 | +| 193 | Lightweight in-package build provenance | No provenance / marker file / content-addressed chain | Ship a `BUILD_INFO.json` inside the deployed package and check build ID, commit SHA, and live Python minor after cold-start timing; a mismatch fails and suppresses metrics. Explicitly excludes wheel digest, installed package version, deploy-input manifest, deployment-storage chain, and rollback. | Human | 2026-09-02 | Replacement stack layer 3 | *Terminology note.* "Signed package" / "signed content package" phrasing in earlier decision rows (e.g. #17, #43), and the historical @@ -1051,10 +1053,12 @@ Invariants: no anonymous ingress; no ingress ports; one active run; free slot on credential-source isolation, identity boundaries, SSRF defenses, static single-level delegation, and whole-chain timeout behavior. * Real ACA acceptance requires create/submit/result, stop/resume readiness, - loss-to-`410`, egress audit, and 100-concurrent/large-payload validation. - Status/event visibility, cancellation/lifecycle repair, cost, and throttling - must meet the documented acceptance target; anonymous ingress is never a - fallback. + loss-to-`410`, egress audit, and large-payload validation. Current deployed + qualification uses N=5 as an orchestration diagnostic; 100-concurrent + acceptance remains human-only (Decision #192). Status/event visibility is + observed against the documented target but does not gate; cancellation, + lifecycle repair, cost, and throttling remain acceptance evidence. Anonymous + ingress is never a fallback. All changes run ruff, strict mypy, pytest, observability/redaction checks, and the relevant documentation and real-ACA validation slices. @@ -1422,7 +1426,7 @@ an enabled v1 surface. * Security/egress: reject unsafe defaults/bypass, rule ordering lint, static and secret credential Transform sources, group-identity boundaries, redirect/DNS-rebind revalidation, block sandbox-to-control-plane SSRF, journal/Table redaction. * Harness: bootstrap ABI/protocol/digest failure, no anonymous ingress, workflows/code-interpreter fail-closed, semantic golden traces every CI, advertise capability only after exercised trace. * Delegation: static/single-level guard, cycle/depth guard, egress union, co-location/no second run, recoverable specialist failure, whole-chain sync timeout. -* Real ACA E2E/full-system: create-submit-result, stop-resume-ensure-ready, egress deny/transform audit, and large-payload gates are evidenced by U3. The committed deployed destructive backing-loss proof has passed; only formal human sign-off remains deferred. The committed public Easy-Auth load runner uses N=5 as the sole agent/CI diagnostic validation for orchestration and cleanup; it is not capacity evidence. Decision #29 remains human-only N=100 formal acceptance and is not passed until human-supplied N=100 evidence is available. At default preview quota, assert <=2-second p95 status/event visibility at <=1 poll/s per active stream, reliable cancellation/lifecycle repair, and acceptable cost/throttling; failure is an explicit private-ingress/load-shaping review finding, never a reason to permit anonymous ingress. Every validation slice also requires `ruff`, strict `mypy`, and `pytest`, plus docs/observability/redaction gates. +* Real ACA E2E/full-system: create-submit-result, stop-resume-ensure-ready, egress deny/transform audit, and large-payload gates are evidenced by U3. The committed deployed destructive backing-loss proof has passed; only formal human sign-off remains deferred. Under Decision #192, a deployed qualification run exercises fresh-session acceptance, public turn, lifecycle, backing loss, and N=5 in that order with provisioning concurrency 1. N=5 is diagnostic evidence for orchestration and cleanup, not capacity. This 120-second-reclaim fixture rejects N=100 before authentication or provider work; Decision #29 remains pending a future purpose-built, human-only formal workflow. Observe status/event latency against the two-second p95 target at <=1 poll/s per active stream, but never gate on it; report reliable cancellation/lifecycle repair and acceptable cost/throttling. Failure is an explicit private-ingress/load-shaping review finding, never a reason to permit anonymous ingress. Every validation slice also requires `ruff`, strict `mypy`, and `pytest`, plus docs/observability/redaction gates. ## 11. Sandbox harness integration @@ -1805,5 +1809,63 @@ BuildId-derived labels isolate concurrent runs. Fixture cleanup plus an remains. The smoke deploys no Function and performs no artifact attestation. ADO 298692 passed the low-level entrypoint, journal acceptance, real model turn, -and cleanup. Post-main deployment, external attestation, py313/py314 -lifecycle/loss/N=5 qualification, and rollback remain owned by issue #166. +and cleanup. The deployed qualification fixture, suites, and packaging/deploy +tooling described in §14 are now committed and runnable by hand. Pipeline +wiring for them, external attestation, and rollback remain owned by issue #166; +there is intentionally no rollback machinery. + +## 14. Deployed ACA qualification assets — issue #166 + +**Status: Finalized for the committed assets.** Decisions #192 and #193 are the +qualification contract implemented here. This section describes committed test, +fixture, and tooling assets only. **No pipeline wiring exists in this +repository for them**; scheduling, gating, promotion criteria, and group sweep +remain open under issue #166. + +`tests/live/apps/aca-qualification/` is the deployable qualification fixture: an +agent app that selects the ACA Sandbox backend, authors its Sandbox Group region, +reads every environment-specific value from app settings, and adds one +fixture-only `/__buildinfo` route. The route lives in the fixture and imports +nothing from product endpoint registration, so it cannot collide with product +surface work. The fixture is deliberately outside `tests/endtoend/apps/`, whose +suite auto-parameterizes a `func start` test over every app it finds and would +fail on an app that requires real Azure. + +`eng/scripts/aca_qualification_pipeline.py` packages, deploys, and verifies that +fixture by hand: it installs tooling, stamps `BUILD_INFO.json`, assembles an +upload from exactly one runtime wheel plus a pinned, Oryx-compatible +`eng/constraints/aca-fixture-requirements.txt` export of `uv.lock`, preflights +deployment rights, configures the authored region, deploys through the normal +Flex Consumption ZIP remote-build path, adds best-effort redacted portal metadata, +and fetches and compares the deployed build info. Ambiguous wheel selection is a +hard error rather than a silent "newest wins". + +`eng/scripts/aca_deployed_qualification.py` runs one ordered suite: the +cold/fresh-session module first, then public turn, lifecycle, backing loss, and +N=5 load. It invokes the cold/fresh-session module as a separate gate and does +not start the remaining suites unless that process succeeds. Within that first +module, fresh-session acceptance, first-event, and +terminal timing complete *before* the same test reads the deployed marker and +compares build ID, commit SHA, and live Python minor version. A missing or +mismatched marker fails the test and suppresses latency metrics, so evidence +from the wrong deployment is never reported as trustworthy. + +The canonical qualification uses N=5 with provisioning concurrency 1. Manual +diagnostics retain load values 1–99 and provisioning values 1, 2, or 4 under +operator-owned quota and cost. Both the operator command and direct live-test +entry point reject N=100 before authentication or provider work. Formal N=100 +remains future human-only acceptance requiring a purpose-built workflow; these +assets do not discharge Decision #29. + +This is deliberately lightweight in-package provenance, not a detached +content-addressed attestation chain. Per Decision #193 it does not prove the +wheel digest, the installed package version, a deploy-input manifest, or the +deployment-storage version, and there is no retained-package or automatic +rollback. A later deployment corrects a bad deployment. + +The qualification client is hardened for a real App Service frontend: bounded +502/503 retries applied only to safe or explicitly idempotent requests, bounded +result-materialization retry, SSE throttling retry that honors `Retry-After`, +bounded event-batch visibility metrics, stable journal-acceptance assertions, +and cleanup/recovery safeguards. All of that lives in test and live assets; no +product runtime code participates. diff --git a/eng/constraints/README.md b/eng/constraints/README.md new file mode 100644 index 00000000..ceea815c --- /dev/null +++ b/eng/constraints/README.md @@ -0,0 +1,23 @@ +# ACA fixture constraints + +This directory pins the dependency closure used by the ACA qualification fixture at +`tests/live/apps/aca-qualification/`. The fixture is deployed with Azure +Functions Flex remote build, so Oryx resolves `requirements.txt` on the server. +Fully pinned constraints prevent those remote builds from drifting when PyPI +publishes new compatible releases. + +`aca-fixture-requirements.txt` is a pip/Oryx-compatible export of `uv.lock`. +It includes the runtime's core dependencies plus the `aca_sandbox` and +`monitor` extras, and deliberately omits the local project because deployment +prepends the exact wheel built by the pipeline. Platform markers keep the same +file valid for Linux Python 3.13 and 3.14. + +Regenerate from the repository root with: + +```powershell +uv export --frozen --no-dev --extra aca_sandbox --extra monitor --no-emit-project --no-hashes --no-header --no-annotate --format requirements.txt --output-file .\eng\constraints\aca-fixture-requirements.txt +``` + +The qualification-pipeline unit tests verify the checked-in export's package +closure and exact versions against `uv.lock`, reject pip directives and source +references, and evaluate its markers for both supported Python minors. diff --git a/eng/constraints/aca-fixture-requirements.txt b/eng/constraints/aca-fixture-requirements.txt new file mode 100644 index 00000000..17d05a5c --- /dev/null +++ b/eng/constraints/aca-fixture-requirements.txt @@ -0,0 +1,103 @@ +agent-framework-core==1.3.0 +agent-framework-foundry==1.3.0 +agent-framework-openai==1.3.0 +aiohappyeyeballs==2.6.1 +aiohttp==3.14.2 +aiosignal==1.4.0 +annotated-doc==0.0.4 +annotated-types==0.7.0 +anyio==4.13.0 +asgiref==3.11.1 +attrs==26.1.0 +azure-ai-inference==1.0.0b9 +azure-ai-projects==2.1.0 +azure-containerapps-sandbox==0.1.0b4 +azure-core==1.41.0 +azure-core-tracing-opentelemetry==1.0.0b13 +azure-data-tables==12.7.0 +azure-functions==2.1.0 +azure-functions-durable==1.6.0 +azure-identity==1.25.3 +azure-mgmt-core==1.6.0 +azure-monitor-opentelemetry==1.8.8 +azure-monitor-opentelemetry-exporter==1.0.0b53 +azure-storage-blob==12.28.0 +azurefunctions-extensions-base==1.2.0 +azurefunctions-extensions-http-fastapi==1.0.1 +certifi==2026.4.22 +cffi==2.0.0 ; platform_python_implementation != 'PyPy' +charset-normalizer==3.4.7 +click==8.3.3 +colorama==0.4.6 ; sys_platform == 'win32' +cryptography==48.0.0 +distro==1.9.0 +fastapi==0.136.1 +frozenlist==1.8.0 +furl==2.1.4 +h11==0.16.0 +httpcore==1.0.9 +httpx==0.28.1 +httpx-sse==0.4.3 +idna==3.15 +importlib-metadata==8.7.1 +isodate==0.7.2 +jiter==0.14.0 +jsonschema==4.26.0 +jsonschema-specifications==2025.9.1 +markupsafe==3.0.3 +mcp==1.28.1 +msal==1.36.0 +msal-extensions==1.3.1 +msrest==0.7.1 +multidict==6.7.1 +oauthlib==3.3.1 +openai==2.36.0 +opentelemetry-api==1.40.0 +opentelemetry-instrumentation==0.61b0 +opentelemetry-instrumentation-asgi==0.61b0 +opentelemetry-instrumentation-dbapi==0.61b0 +opentelemetry-instrumentation-django==0.61b0 +opentelemetry-instrumentation-fastapi==0.61b0 +opentelemetry-instrumentation-flask==0.61b0 +opentelemetry-instrumentation-logging==0.61b0 +opentelemetry-instrumentation-psycopg2==0.61b0 +opentelemetry-instrumentation-requests==0.61b0 +opentelemetry-instrumentation-urllib==0.61b0 +opentelemetry-instrumentation-urllib3==0.61b0 +opentelemetry-instrumentation-wsgi==0.61b0 +opentelemetry-resource-detector-azure==0.1.5 +opentelemetry-sdk==1.40.0 +opentelemetry-semantic-conventions==0.61b0 +opentelemetry-util-http==0.61b0 +orderedmultidict==1.0.2 +packaging==26.2 +propcache==0.5.2 +psutil==7.2.2 +pycparser==3.0 ; implementation_name != 'PyPy' and platform_python_implementation != 'PyPy' +pydantic==2.13.4 +pydantic-core==2.46.4 +pydantic-settings==2.14.1 +pyjwt==2.12.1 +python-dateutil==2.9.0.post0 +python-dotenv==1.2.2 +python-frontmatter==1.1.0 +python-multipart==0.0.28 +pywin32==311 ; sys_platform == 'win32' +pyyaml==6.0.3 +referencing==0.37.0 +requests==2.34.2 +requests-oauthlib==2.0.0 +rpds-py==0.30.0 +six==1.17.0 +sniffio==1.3.1 +sse-starlette==3.4.4 +starlette==1.3.1 +tqdm==4.67.3 +typing-extensions==4.15.0 +typing-inspection==0.4.2 +urllib3==2.7.0 +uvicorn==0.47.0 +werkzeug==3.1.8 +wrapt==1.17.3 +yarl==1.23.0 +zipp==3.23.1 diff --git a/eng/scripts/README.md b/eng/scripts/README.md index fa285e2b..8f1b8670 100644 --- a/eng/scripts/README.md +++ b/eng/scripts/README.md @@ -68,3 +68,40 @@ role-assignment attestation. The retained `aca_deployed_qualification.py` and deployed suite helpers are manual/local assets only pending the separate post-main qualification work. + +### `aca_qualification_pipeline.py` + +Packages, deploys, and verifies the deployed ACA qualification fixture +(`tests/live/apps/aca-qualification/`). Every command is run by hand; this +repository contains no pipeline wiring for it. + +| Command | Purpose | +| --- | --- | +| `install-tooling` | Install the shared Python dependencies used by qualification runs | +| `stamp` | Write `BUILD_INFO.json` into the fixture app before packaging | +| `assemble` | Build the deployable upload: fixture source, the runtime wheel, the marker, and pinned requirements | +| `deploy` | Preflight deployment rights, configure the authored region, package and deploy the staged fixture, and add best-effort portal metadata | +| `check-build` | Verify lightweight in-package build ID, commit SHA, and Python-minor provenance | + +`assemble` requires exactly one runtime wheel in the build output; ambiguity is +a hard error rather than a silent "newest wins", because deploying the wrong +wheel is precisely the failure `check-build` exists to catch. Fixture +dependencies come from the single Oryx-compatible +`eng/constraints/aca-fixture-requirements.txt` export of `uv.lock`, valid for +both supported interpreter minors. + +`check-build` is meaningful only because the marker is a *file inside the +deployed package*: a file can be served only if that package is genuinely on +disk, so a stale app cannot claim a build it is not running. An app setting or +resource tag could be changed without deploying anything. + +The provenance is deliberately narrow. It does not cover the wheel digest, the +installed package version, a deploy-input manifest, the deployment-storage +chain, or rollback; those remain open under issue #166. + +The canonical qualification uses N=5 with provisioning concurrency 1 and the +fixture's 120-second reclaim policy. Manual diagnostics retain load values 1–99 +and provisioning values 1, 2, or 4; their operator owns shared-group quota and +cost. `aca_deployed_qualification.py` rejects N=100 before authentication or +provider work with `formal_n100_unsupported_by_qualification_fixture`. Formal +N=100 remains future human-only acceptance requiring a purpose-built workflow. diff --git a/eng/scripts/aca_deployed_qualification.py b/eng/scripts/aca_deployed_qualification.py index 5d3e7d2c..742e346d 100644 --- a/eng/scripts/aca_deployed_qualification.py +++ b/eng/scripts/aca_deployed_qualification.py @@ -22,11 +22,15 @@ "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_TABLE_NAME", "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_APP_SUBSCRIPTION_ID", "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_APP_SITE_NAME", + "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_BUILD_ID", + "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_COMMIT_SHA", + "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_PYTHON_VERSION", "AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_GROUP_RESOURCE_ID", "AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_REGION", ) _PROVISION_CONCURRENCIES = frozenset({1, 2, 4}) _SMOKE_RUN_ID = "AZURE_FUNCTIONS_AGENTS_ACA_SMOKE_RUN_ID" +FORMAL_N100_UNSUPPORTED_ERROR = "formal_n100_unsupported_by_qualification_fixture" class QualificationError(Exception): @@ -66,29 +70,28 @@ def validate_deployed_environment( provision_concurrency: str | None = None, ) -> tuple[int | None, int | None]: """Validate redacted deployed-suite inputs and shared-group limits.""" - for name in _DEPLOYED_ENVIRONMENT: - _required(environment, name) - if runtime_target not in {"both", "python313", "python314"}: - raise QualificationError("invalid_runtime_target") - load = ( _integer(load_concurrency, name="acaLoadConcurrency", minimum=1, maximum=100) if load_concurrency is not None else None ) + if load == 100: + raise QualificationError(FORMAL_N100_UNSUPPORTED_ERROR) provision = ( _integer(provision_concurrency, name="acaProvisionConcurrency", minimum=1, maximum=4) if provision_concurrency is not None else None ) + for name in _DEPLOYED_ENVIRONMENT: + _required(environment, name) + if runtime_target not in {"both", "python313", "python314"}: + raise QualificationError("invalid_runtime_target") if provision is not None and provision not in _PROVISION_CONCURRENCIES: raise QualificationError("invalid_provision_concurrency") if runtime_target == "both" and load is not None and load > 5: raise QualificationError("dual_runtime_load_concurrency_requires_single_runtime") if runtime_target == "both" and provision is not None and provision > 1: raise QualificationError("provision_concurrency_requires_single_runtime") - if load == 100 and environment.get("BUILD_REASON", "").strip() != "Manual": - raise QualificationError("formal_n100_requires_manual_build") return load, provision @@ -154,7 +157,7 @@ def run_deployed_suite( load_concurrency: str, provision_concurrency: str, ) -> int: - """Run the protected deployed turn, lifecycle, loss, and N=5 smoke suite.""" + """Gate turn, lifecycle, loss, and load on cold-start provenance.""" load, provision = validate_deployed_environment( environment, runtime_target=runtime_target, @@ -165,7 +168,16 @@ def run_deployed_suite( inherited = dict(environment) inherited["AZURE_FUNCTIONS_AGENTS_ACA_LOAD_CONCURRENCY"] = str(load) inherited["AZURE_FUNCTIONS_AGENTS_ACA_PROVISION_CONCURRENCY"] = str(provision) + samples = validate_cold_start_samples(inherited) + if samples is not None: + inherited["AZURE_FUNCTIONS_AGENTS_ACA_COLD_START_SAMPLES"] = str(samples) preflight_auth(inherited) + cold_start_result = _run_pytest( + ("tests/live/test_aca_deployed_cold_start.py",), + inherited, + ) + if cold_start_result != 0: + return cold_start_result return _run_pytest( ( "tests/live/test_aca_deployed_agent_turn.py", diff --git a/eng/scripts/aca_qualification_pipeline.py b/eng/scripts/aca_qualification_pipeline.py new file mode 100644 index 00000000..baaf42f3 --- /dev/null +++ b/eng/scripts/aca_qualification_pipeline.py @@ -0,0 +1,612 @@ +#!/usr/bin/env python3 +"""Deployed ACA qualification tooling (FRD 0008 §14, issue #166). + +Five commands, each doing one thing a deployed qualification run needs. They are +runnable by hand and carry no pipeline wiring of their own: + +``install-tooling`` + Install the shared Python tooling a qualification run needs. + +``stamp`` + Write ``BUILD_INFO.json`` into the fixture app before it is packaged, so the + deployed bytes carry the identity of the build that produced them. + +``assemble`` + Copy the live fixture app and exactly one built runtime wheel into an upload + directory, then write the marker and final remote-build ``requirements.txt``. + +``deploy`` + Verify deployment rights, configure the authored region, package the staged + fixture, deploy it, and add best-effort portal metadata. + +``check-build`` + Ask the deployed app what it is running and compare it with this build. The + marker is only meaningful because it is a *file inside the package*: a file + can be served only if the package containing it is on disk, so a stale app + cannot claim a build it is not running. An app setting or resource tag could + be changed without deploying anything, which is where a service reporting + its own version stops being evidence. + +Each command fails closed on its own contract. ``check-build`` specifically +prevents qualification evidence from being attributed to the wrong deployment. +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import re +import shutil +import subprocess +import sys +import time +import zipfile +from collections.abc import Iterable, Mapping, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +_MARKER_FILENAME = "BUILD_INFO.json" +_MARKER_SCHEMA = 1 + +_DEPLOY_PREFLIGHT_TIMEOUT_SECONDS = 30.0 +_DEPLOY_CONFIGURATION_TIMEOUT_SECONDS = 300.0 +_DEPLOY_TIMEOUT_SECONDS = 1_200.0 +_DEFAULT_FIXTURE_ROOT = Path(__file__).resolve().parents[2] / "tests" / "live" / "apps" / "aca-qualification" + + +class QualificationPipelineError(Exception): + """A redacted pipeline configuration or verification failure.""" + + +def build_marker( + *, + commit_sha: str, + build_id: str, + branch: str, + runtime_version: str, +) -> dict[str, Any]: + """Build the marker object stamped into the deployed package.""" + for name, value in ( + ("commit_sha", commit_sha), + ("build_id", build_id), + ("branch", branch), + ("runtime_version", runtime_version), + ): + if not value or not value.strip(): + raise QualificationPipelineError(f"marker_field_empty:{name}") + return { + "schema": _MARKER_SCHEMA, + "commit_sha": commit_sha.strip(), + "build_id": build_id.strip(), + "branch": branch.strip(), + "runtime_version": runtime_version.strip(), + } + + +def stamp_marker(app_root: Path, marker: Mapping[str, Any]) -> Path: + """Write the marker into the fixture app root.""" + if not app_root.is_dir(): + raise QualificationPipelineError(f"app_root_missing:{app_root.name}") + target = app_root / _MARKER_FILENAME + target.write_text(json.dumps(dict(marker), indent=2, sort_keys=True) + "\n", encoding="utf-8") + return target + + +def select_runtime_wheel(candidate_filenames: Iterable[str]) -> str: + """Select the only runtime wheel eligible for deployment.""" + names = sorted(Path(name).name for name in candidate_filenames if Path(name).suffix == ".whl") + if not names: + raise QualificationPipelineError("runtime_wheel_missing") + if len(names) > 1: + raise QualificationPipelineError(f"runtime_wheel_ambiguous:{','.join(names)}") + return names[0] + + +def _render_requirements_text( + *, + wheel_filename: str, + template_text: str, + requirements_export_text: str, +) -> str: + """Render remote-build requirements from the wheel name and pinned deps.""" + wheel_requirement = f"./{wheel_filename}" + template_lines = template_text.splitlines() + rendered_template: list[str] = [] + replaced_placeholder = False + for line in template_lines: + if ( + "{{RUNTIME_WHEEL}}" in line + or "{runtime_wheel}" in line + or "BUILD_WHEEL_PLACEHOLDER" in line + ): + rendered_template.append(wheel_requirement) + replaced_placeholder = True + continue + if replaced_placeholder and line.strip() and not line.strip().startswith("#"): + break + rendered_template.append(line) + + sections: list[str] = [] + if not replaced_placeholder: + sections.append(wheel_requirement) + template = template_text.strip() + if template: + sections.append(template) + elif rendered_template: + sections.append("\n".join(rendered_template).strip()) + pinned = requirements_export_text.strip() + if pinned: + sections.append(pinned) + return "\n\n".join(sections) + "\n" + + +def render_requirements( + *, + wheel_filename: str, + requirements_export_path: Path, + template_path: Path | None = None, +) -> str: + """Render the final fixture requirements from checked-in dependency inputs.""" + if not requirements_export_path.is_file(): + raise QualificationPipelineError( + f"requirements_export_missing:{requirements_export_path}" + ) + template_text = "" + if template_path is not None: + if not template_path.is_file(): + raise QualificationPipelineError(f"requirements_template_missing:{template_path}") + template_text = template_path.read_text(encoding="utf-8") + return _render_requirements_text( + wheel_filename=wheel_filename, + template_text=template_text, + requirements_export_text=requirements_export_path.read_text(encoding="utf-8"), + ) + + +def _copy_fixture_app(fixture_root: Path, staging_root: Path) -> None: + if not fixture_root.is_dir(): + raise QualificationPipelineError(f"fixture_root_missing:{fixture_root}") + if staging_root.exists(): + shutil.rmtree(staging_root) + shutil.copytree( + fixture_root, + staging_root, + ignore=shutil.ignore_patterns("__pycache__", ".venv", ".pytest_cache", ".ruff_cache", "*.pyc"), + ) + + +def assemble_upload_directory(args: argparse.Namespace) -> Path: + """Create the deployable remote-build upload directory.""" + artifact_root = Path(args.artifact_root) + dist_root = artifact_root / "dist" + wheel_name = select_runtime_wheel(path.name for path in dist_root.glob("*.whl")) + fixture_root = Path(args.fixture_root) + staging_root = Path(args.staging_root) + _copy_fixture_app(fixture_root, staging_root) + shutil.copy2(dist_root / wheel_name, staging_root / wheel_name) + + marker = build_marker( + commit_sha=args.commit_sha, + build_id=args.build_id, + branch=args.branch, + runtime_version=args.runtime_version, + ) + stamp_marker(staging_root, marker) + template_path = Path(args.requirements_template) if args.requirements_template else fixture_root / "requirements.txt" + requirements = render_requirements( + wheel_filename=wheel_name, + requirements_export_path=Path(args.requirements_export), + template_path=template_path, + ) + (staging_root / "requirements.txt").write_text(requirements, encoding="utf-8") + return staging_root + + +@dataclass(frozen=True, slots=True) +class MarkerComparison: + """Result of comparing a deployed app's report against this build.""" + + mismatches: tuple[str, ...] + + @property + def matches(self) -> bool: + return not self.mismatches + + +def compare_marker( + reported: Mapping[str, Any], + *, + expected_build_id: str, + expected_commit_sha: str, + expected_python: str, +) -> MarkerComparison: + """Compare a ``/__buildinfo`` response with the build that should be live. + + Returns field names only. Values are deliberately omitted: this runs in a + pipeline log, and the comparison is a yes/no question. + """ + mismatches: list[str] = [] + + build = reported.get("build") + if not isinstance(build, Mapping): + return MarkerComparison(("build_section_missing",)) + + marker_state = build.get("marker") + if marker_state != "present": + return MarkerComparison((f"marker_{marker_state or 'unknown'}",)) + + if build.get("schema") != _MARKER_SCHEMA: + mismatches.append("schema") + if str(build.get("build_id", "")) != expected_build_id: + mismatches.append("build_id") + if str(build.get("commit_sha", "")) != expected_commit_sha: + mismatches.append("commit_sha") + + runtime = reported.get("runtime") + if not isinstance(runtime, Mapping): + mismatches.append("runtime_section_missing") + elif str(runtime.get("python_version", "")) != expected_python: + mismatches.append("python_version") + + return MarkerComparison(tuple(mismatches)) + + +def content_report(reported: Mapping[str, Any]) -> str: + """Render deployed-content size against the platform package limits.""" + content = reported.get("content") + if not isinstance(content, Mapping): + return "content=unavailable" + entries = content.get("entry_count") + total = content.get("total_bytes") + if not isinstance(entries, int) or not isinstance(total, int): + return "content=unavailable" + mib = total / (1024 * 1024) + return ( + f"content_entries={entries} (cap 65535, {entries / 65535:.1%}) " + f"content_size={mib:.1f}MiB (cap 256MiB, {mib / 256:.1%}) " + f"truncated={str(content.get('truncated', False)).lower()}" + ) + + +_MAX_REASON_CHARS = 400 +_SECRETISH = re.compile( + r"(?i)(bearer\s+\S+|[?&](sig|sv|se|st|skoid|sig)=[^&\s]+|eyJ[A-Za-z0-9_\-.]{20,})" +) + + +def _redacted_reason(error: BaseException) -> str: + """Render an operational cause without leaking credential material.""" + text = str(error).strip() or type(error).__name__ + text = _SECRETISH.sub("", text) + text = " ".join(text.split()) + if len(text) > _MAX_REASON_CHARS: + text = text[:_MAX_REASON_CHARS] + "…" + return text + + +def deploy_preflight_failure_message( + *, + app_name: str, + resource_group: str, + check_name: str, + slow: bool = False, +) -> str: + """Name the Function App deployment role needed before ZIP deployment.""" + slow_sentence = ( + f" The check hit its {_DEPLOY_PREFLIGHT_TIMEOUT_SECONDS:.0f}s deadline; fail-fast " + "authorization probes treat that as a deployment-readiness failure." + if slow + else "" + ) + return ( + f"functionapp_deploy_preflight_failed:{check_name}: grant Website Contributor on " + f"{app_name} in resource group {resource_group} to the deployment identity " + "so it can read the site before ZIP deployment." + f"{slow_sentence}" + ) + + +def _run_az(args: Sequence[str], *, timeout_seconds: float = _DEPLOY_PREFLIGHT_TIMEOUT_SECONDS) -> None: + command = ["az", *args, "--only-show-errors", "--output", "json"] + operation = ":".join(args[:2]) + try: + completed = subprocess.run( + command, + check=False, + capture_output=True, + text=True, + timeout=timeout_seconds, + ) + except subprocess.TimeoutExpired: + raise QualificationPipelineError( + f"az_timeout:{operation}:{int(timeout_seconds)}s" + ) from None + except OSError as error: + raise QualificationPipelineError( + f"az_unavailable:{operation}:{type(error).__name__}" + ) from None + if completed.returncode != 0: + raise QualificationPipelineError( + f"az_failed:{operation}:exit_{completed.returncode}" + ) from None + + +def run_install_tooling() -> int: + """Install the common dependencies for manual qualification runs.""" + commands = ( + [sys.executable, "-m", "pip", "install", "--upgrade", "pip"], + [sys.executable, "-m", "pip", "install", "-U", "-e", ".[dev,aca_sandbox]"], + ) + for command in commands: + try: + subprocess.run(command, check=True) + except subprocess.CalledProcessError as error: + raise QualificationPipelineError("pipeline_tooling_install_failed") from error + return 0 + + +def run_preflight_deploy(args: argparse.Namespace) -> int: + """Verify the deployment identity can read the app and publishing config.""" + app_name = args.app_name + resource_group = args.resource_group + checks = ( + ( + "site_read", + ["functionapp", "show", "--name", app_name, "--resource-group", resource_group], + ), + # Flex disables SCM basic auth, so publishing-profile checks are invalid. + ) + for check_name, command in checks: + try: + _run_az(command) + except QualificationPipelineError as error: + raise QualificationPipelineError( + deploy_preflight_failure_message( + app_name=app_name, + resource_group=resource_group, + check_name=check_name, + slow=str(error).startswith("az_timeout:"), + ) + ) from error + print("Function App deployment preflight succeeded.") + return 0 + + +def _write_deployment_archive(staging_root: Path, archive_path: Path) -> None: + if not staging_root.is_dir(): + raise QualificationPipelineError(f"staging_root_missing:{staging_root.name}") + archive_path.parent.mkdir(parents=True, exist_ok=True) + archive_path.unlink(missing_ok=True) + with zipfile.ZipFile(archive_path, "w", compression=zipfile.ZIP_DEFLATED) as archive: + for path in sorted(staging_root.rglob("*")): + if path.is_file(): + archive.write(path, path.relative_to(staging_root).as_posix()) + + +def run_deploy(args: argparse.Namespace) -> int: + """Preflight and deploy one staged fixture through the customer-equivalent path.""" + run_preflight_deploy(args) + region = args.region.strip() + if not region: + raise QualificationPipelineError("deployment_region_empty") + + _run_az( + [ + "functionapp", + "config", + "appsettings", + "set", + "--name", + args.app_name, + "--resource-group", + args.resource_group, + "--settings", + f"AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_REGION={region}", + ], + timeout_seconds=_DEPLOY_CONFIGURATION_TIMEOUT_SECONDS, + ) + + archive_path = Path(args.archive_path) + _write_deployment_archive(Path(args.staging_root), archive_path) + _run_az( + [ + "functionapp", + "deployment", + "source", + "config-zip", + "--name", + args.app_name, + "--resource-group", + args.resource_group, + "--src", + str(archive_path), + ], + timeout_seconds=_DEPLOY_TIMEOUT_SECONDS, + ) + + try: + _run_az( + [ + "tag", + "update", + "--operation", + "merge", + "--resource-group", + args.resource_group, + "--name", + args.app_name, + "--resource-type", + "Microsoft.Web/sites", + "--tags", + f"build_id={args.build_id}", + f"commit_sha={args.commit_sha}", + ] + ) + except QualificationPipelineError as error: + print( + "warning: Function App metadata tag failed: " + f"{_redacted_reason(error)}" + ) + print("Function App deployment completed.") + return 0 + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + subcommands = parser.add_subparsers(dest="command", required=True) + + subcommands.add_parser( + "install-tooling", + help="install Python dependencies shared by qualification runs", + ) + + stamp = subcommands.add_parser("stamp", help="write BUILD_INFO.json into the fixture app") + stamp.add_argument("--app-root", required=True) + stamp.add_argument("--commit-sha", required=True) + stamp.add_argument("--build-id", required=True) + stamp.add_argument("--branch", required=True) + stamp.add_argument("--runtime-version", required=True) + + assemble = subcommands.add_parser("assemble", help="build the Flex remote-build upload directory") + assemble.add_argument("--artifact-root", required=True) + assemble.add_argument("--staging-root", required=True) + assemble.add_argument("--fixture-root", default=str(_DEFAULT_FIXTURE_ROOT)) + assemble.add_argument("--requirements-export", required=True) + assemble.add_argument("--requirements-template") + assemble.add_argument("--commit-sha", required=True) + assemble.add_argument("--build-id", required=True) + assemble.add_argument("--branch", required=True) + assemble.add_argument("--runtime-version", required=True) + + deploy = subcommands.add_parser( + "deploy", + help="preflight, configure, package, and deploy one staged fixture", + ) + deploy.add_argument("--staging-root", required=True) + deploy.add_argument("--archive-path", required=True) + deploy.add_argument("--app-name", required=True) + deploy.add_argument("--resource-group", required=True) + deploy.add_argument("--region", required=True) + deploy.add_argument("--build-id", required=True) + deploy.add_argument("--commit-sha", required=True) + + check = subcommands.add_parser("check-build", help="verify the deployed app is this build") + check.add_argument("--base-url", required=True) + check.add_argument("--token-scope", required=True) + check.add_argument("--build-id", required=True) + check.add_argument("--commit-sha", required=True) + check.add_argument("--python-version", required=True) + + return parser + + +_READINESS_DEADLINE_SECONDS = 300.0 +_READINESS_POLL_SECONDS = 10.0 +_NOT_READY_STATUSES = frozenset({408, 429, *range(500, 600)}) + + +async def fetch_build_info(base_url: str, token_scope: str) -> dict[str, Any]: + """Read and parse the embedded build marker without logging its values.""" + import aiohttp + from azure.identity.aio import DefaultAzureCredential + + credential = DefaultAzureCredential() + try: + token = await credential.get_token(token_scope) + finally: + await credential.close() + url = f"{base_url.rstrip('/')}/__buildinfo" + timeout = aiohttp.ClientTimeout(total=60) + deadline = time.monotonic() + _READINESS_DEADLINE_SECONDS + last_status: int | None = None + + async with aiohttp.ClientSession(timeout=timeout) as session: + while True: + try: + async with session.get( + url, headers={"Authorization": f"Bearer {token.token}"} + ) as response: + if response.status == 200: + payload = await response.json(content_type=None) + break + last_status = response.status + if response.status not in _NOT_READY_STATUSES: + raise QualificationPipelineError(f"buildinfo_http_{response.status}") + except (TimeoutError, aiohttp.ClientError) as error: + last_status = last_status or -1 + if time.monotonic() >= deadline: + raise QualificationPipelineError( + f"buildinfo_unreachable:{type(error).__name__}" + ) from None + if time.monotonic() >= deadline: + raise QualificationPipelineError( + f"buildinfo_not_ready_after_{int(_READINESS_DEADLINE_SECONDS)}s" + f":last_status_{last_status}" + ) + print(f"App not ready yet (status {last_status}); waiting for restart to settle.") + await asyncio.sleep(_READINESS_POLL_SECONDS) + + if not isinstance(payload, dict): + raise QualificationPipelineError("buildinfo_malformed") + return payload + + +def run_check_build(args: argparse.Namespace) -> int: + """Fail fast when the deployed app is not the build under qualification.""" + try: + reported = asyncio.run(fetch_build_info(args.base_url, args.token_scope)) + except QualificationPipelineError: + raise + except Exception as error: + raise QualificationPipelineError(f"buildinfo_unreachable:{type(error).__name__}") from None + + comparison = compare_marker( + reported, + expected_build_id=args.build_id, + expected_commit_sha=args.commit_sha, + expected_python=args.python_version, + ) + print(content_report(reported)) + if not comparison.matches: + print( + "Deployed app does not match this build: " + f"{','.join(comparison.mismatches)}", + file=sys.stderr, + ) + return 1 + print(f"Deployed build verified (python {args.python_version}).") + return 0 + + +def main(arguments: Sequence[str] | None = None) -> int: + """Dispatch one qualification command.""" + args = _parser().parse_args(arguments) + try: + if args.command == "install-tooling": + return run_install_tooling() + if args.command == "stamp": + marker = build_marker( + commit_sha=args.commit_sha, + build_id=args.build_id, + branch=args.branch, + runtime_version=args.runtime_version, + ) + target = stamp_marker(Path(args.app_root), marker) + print(f"Stamped {target.name} for build {marker['build_id']}.") + return 0 + if args.command == "assemble": + staging_root = assemble_upload_directory(args) + print(f"Assembled upload directory {staging_root}.") + return 0 + if args.command == "deploy": + return run_deploy(args) + return run_check_build(args) + except QualificationPipelineError as error: + print(f"ACA qualification pipeline failed: {error}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/live/README.md b/tests/live/README.md index d5f2dd8e..5f3a9de6 100644 --- a/tests/live/README.md +++ b/tests/live/README.md @@ -22,7 +22,14 @@ backend. Deployed cold-start, lifecycle, loss, load, and one-shot recovery suites remain direct/manual test assets pending issue #166. They have no pipeline wiring, -target metadata, or queue-time parameters in this repository. +target metadata, or queue-time parameters in this repository. Drive them with +`eng/scripts/aca_deployed_qualification.py`, after packaging and deploying +`tests/live/apps/aca-qualification/` with +`eng/scripts/aca_qualification_pipeline.py`. They still skip unless +`AZURE_FUNCTIONS_AGENTS_RUN_DEPLOYED_ACA_SMOKE=1` is set explicitly. +`AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_FUNCTION_BASE_URL` may be the HTTPS site +origin or include `/api`; a pathless origin is normalized to the fixture's +default `/api` route root. ## Controlled deployed one-shot recovery @@ -44,7 +51,7 @@ deployment target. ```bash export AZURE_FUNCTIONS_AGENTS_RUN_DEPLOYED_ACA_SMOKE=1 -export AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_FUNCTION_BASE_URL="https://.azurewebsites.net" +export AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_FUNCTION_BASE_URL="https://.azurewebsites.net/api" export AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_AGENT_SLUG="deployed_setup_timeout" export AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_TOKEN_SCOPE="api:///.default" export AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_AUDIENCE="" @@ -62,3 +69,30 @@ cancel and poll for a terminal outcome. A cancellation `202` honors `Retry-After` before status polling. The terminal polling window is five minutes: it covers the 120-second operation lease plus a 60-second dedicated fixture reconciler cadence and scheduling jitter. + +## Deployed ACA qualification fixture + +`tests/live/apps/aca-qualification/` is the deployable fixture app that the +deployed cold-start, agent-turn, lifecycle, loss, and load suites target. It is +packaged and deployed by `eng/scripts/aca_qualification_pipeline.py`, which +stamps an in-package `BUILD_INFO.json` marker. The cold-start module runs first +in `eng/scripts/aca_deployed_qualification.py` and, once its timing assertions +complete, checks that marker's build ID and commit SHA plus the live Python +minor version against the expected values in the environment. A mismatch fails +the run and suppresses the cold-start metrics, so timings from a stale build are +never reported as if they described the build under test. + +The lifecycle suite verifies terminal session expiry on both authoritative +clocks: at least 120 seconds after session activity for reclaim, and at least +300 seconds after the matching terminal run update for successful-result +availability. It does not compare the result hold to the later rearm activity +timestamp. + +The canonical qualification uses N=5 with provisioning concurrency 1. Manual +diagnostics retain load values 1–99 and provisioning values 1, 2, or 4; their +operator owns shared-group quota and cost. The operator wrapper and direct +live-test entry point reject N=100 before authentication or provider work. +Formal N=100 remains future human-only acceptance and requires a purpose-built +workflow; this fixture does not discharge Decision #29. + +There is no pipeline wiring for any of this; every step is run by hand. diff --git a/tests/live/aca_deployed_agent_support.py b/tests/live/aca_deployed_agent_support.py index 11a8b1b9..0eb704d6 100644 --- a/tests/live/aca_deployed_agent_support.py +++ b/tests/live/aca_deployed_agent_support.py @@ -33,6 +33,16 @@ _CANCEL_RETRY_AFTER_SECONDS = 2.0 _TIMEOUT_RECOVERY_AGENT_SLUG = "deployed_setup_timeout" _TIMEOUT_RECOVERY_MINIMUM_TIMEOUT_SECONDS = 120.0 +_SSE_THROTTLE_MAX_ATTEMPTS = 4 +_SSE_THROTTLE_STATUSES = frozenset({429, 503}) +_SSE_THROTTLE_RETRY_AFTER_MAXIMUM_SECONDS = 10.0 + +_JSON_THROTTLE_MAX_ATTEMPTS = 4 +_JSON_THROTTLE_STATUSES = frozenset({429, 503}) +_FRONTEND_UNAVAILABLE_STATUSES = frozenset({502, 503}) +_FRONTEND_UNAVAILABLE_RETRY_SECONDS = 2.0 +# A 504 is an asserted setup outcome, not a retryable frontend response. +_JSON_THROTTLE_RETRY_AFTER_MAXIMUM_SECONDS = 10.0 class _TokenCredential(Protocol): @@ -57,6 +67,49 @@ def cancel_retry_after_seconds(headers: Mapping[str, str]) -> float: ) +def sse_throttle_retry_delay( + status: int, + headers: Mapping[str, str], + *, + is_final_attempt: bool, +) -> float | None: + """Return the delay before retrying a throttled SSE response, or None to give up.""" + if is_final_attempt: + return None + if status in _SSE_THROTTLE_STATUSES: + delay = optional_retry_after_seconds( + headers, maximum_seconds=_SSE_THROTTLE_RETRY_AFTER_MAXIMUM_SECONDS + ) + if delay is not None: + return delay + if _is_frontend_unavailable_response(status, headers): + return _FRONTEND_UNAVAILABLE_RETRY_SECONDS + return None + + +def optional_retry_after_seconds( + headers: Mapping[str, str], + *, + maximum_seconds: float, +) -> float | None: + """Return a bounded ``Retry-After``, or None when the server did not ask for one. + + Distinct from the fallback variant below: a caller deciding *whether* to + retry must not invent a delay the server never sent, or it would retry + responses that carry no backpressure signal at all. + """ + value = response_header(headers, "retry-after") + if value is None: + return None + try: + seconds = int(value.strip()) + except ValueError: + return None + if not 1 <= seconds <= int(maximum_seconds): + return None + return float(seconds) + + def response_header(headers: Mapping[str, str], name: str) -> str | None: """Return one case-insensitive response header value.""" return next( @@ -69,6 +122,28 @@ def response_header(headers: Mapping[str, str], name: str) -> str | None: ) +def _is_frontend_unavailable_response( + status: int, + headers: Mapping[str, str], +) -> bool: + if status not in _FRONTEND_UNAVAILABLE_STATUSES: + return False + content_type = response_header(headers, "content-type") + if content_type is None: + return False + media_type = content_type.partition(";")[0].strip().casefold() + return media_type in {"text/html", "text/plain"} + + +def _request_can_retry_frontend_unavailable( + method: str, + headers: Mapping[str, str] | None, +) -> bool: + if method.upper() in {"GET", "HEAD"}: + return True + return headers is not None and response_header(headers, "idempotency-key") is not None + + def timeout_recovery_submission_headers( authorization: str, idempotency_key: str, @@ -339,17 +414,63 @@ async def json_request( *, headers: Mapping[str, str] | None = None, payload: dict[str, object] | None = None, + retry_throttled: bool = True, ) -> tuple[int, dict[str, object], Mapping[str, str]]: """Make one public JSON request without logging prompt, result, or credentials.""" - try: - async with session.request(method, url, headers=headers, json=payload) as response: - return response.status, await _json_body(response), dict(response.headers) - except (TimeoutError, OSError) as exc: - raise AcaSmokeEnvironmentError( - f"Function App was unavailable at {redact_deployed_aca_evidence(url)}: " - f"{type(exc).__name__}" - ) from exc + for attempt in range(1, _JSON_THROTTLE_MAX_ATTEMPTS + 1): + try: + async with session.request(method, url, headers=headers, json=payload) as response: + status = response.status + resp_headers: Mapping[str, str] = dict(response.headers) + frontend_unavailable = _is_frontend_unavailable_response( + status, + resp_headers, + ) + if frontend_unavailable: + await response.read() + body: dict[str, object] = {} + else: + body = await _json_body(response) + except (TimeoutError, OSError) as exc: + raise AcaSmokeEnvironmentError( + f"Function App was unavailable at {redact_deployed_aca_evidence(url)}: " + f"{type(exc).__name__}" + ) from exc + + if not retry_throttled: + if frontend_unavailable: + raise AcaSmokeEnvironmentError(f"Function App returned HTTP {status}.") + return status, body, resp_headers + is_final = attempt == _JSON_THROTTLE_MAX_ATTEMPTS + if frontend_unavailable: + if ( + is_final + or not _request_can_retry_frontend_unavailable(method, headers) + ): + raise AcaSmokeEnvironmentError(f"Function App returned HTTP {status}.") + delay = optional_retry_after_seconds( + resp_headers, + maximum_seconds=_JSON_THROTTLE_RETRY_AFTER_MAXIMUM_SECONDS, + ) + await asyncio.sleep( + delay + if delay is not None + else _FRONTEND_UNAVAILABLE_RETRY_SECONDS + ) + continue + if status not in _JSON_THROTTLE_STATUSES: + return status, body, resp_headers + if is_final: + return status, body, resp_headers + delay = optional_retry_after_seconds( + resp_headers, maximum_seconds=_JSON_THROTTLE_RETRY_AFTER_MAXIMUM_SECONDS + ) + if delay is None: + return status, body, resp_headers + await asyncio.sleep(delay) + + raise AssertionError("unreachable") async def read_sse_events( @@ -378,21 +499,30 @@ async def read_sse_until_matching_event( """Read one public SSE stream until one strictly ordered matching event arrives.""" try: async with asyncio.timeout(overall_timeout_seconds): - async with session.get(url, headers=headers) as response: - status = response.status - response_headers = dict(response.headers) - if status != 200: - raise SseResponseStatusError(url, status) - events: list[SseEvent] = [] - pending = "" - async for chunk in response.content: - pending += chunk.decode("utf-8").replace("\r\n", "\n") - frames = pending.split("\n\n") - pending = frames.pop() - for event in parse_sse_frames(frames): - events = append_contiguous_sse_events(events, [event]) - if matches(event): - return status, event, response_headers + for attempt in range(1, _SSE_THROTTLE_MAX_ATTEMPTS + 1): + is_final = attempt == _SSE_THROTTLE_MAX_ATTEMPTS + async with session.get(url, headers=headers) as response: + status = response.status + response_headers = dict(response.headers) + if status != 200: + delay = sse_throttle_retry_delay( + status, response_headers, is_final_attempt=is_final + ) + if delay is None: + raise SseResponseStatusError(url, status) + await asyncio.sleep(delay) + continue + events: list[SseEvent] = [] + pending = "" + async for chunk in response.content: + pending += chunk.decode("utf-8").replace("\r\n", "\n") + frames = pending.split("\n\n") + pending = frames.pop() + for event in parse_sse_frames(frames): + events = append_contiguous_sse_events(events, [event]) + if matches(event): + return status, event, response_headers + raise SseResponseStatusError(url, status) except TimeoutError as exc: raise AcaSmokeEnvironmentError( "Function App SSE stream did not emit the required event before the overall deadline." @@ -413,8 +543,28 @@ async def read_sse_events_with_first_event_time( overall_timeout_seconds: float = 240.0, ) -> tuple[int, list[SseEvent], Mapping[str, str], float | None]: """Reconnect public SSE with ``Last-Event-ID`` until the terminal ``done`` event.""" + status, events, response_headers, first_event_at, _ = ( + await read_sse_events_with_observation_times( + session, + url, + headers=headers, + overall_timeout_seconds=overall_timeout_seconds, + ) + ) + return status, events, response_headers, first_event_at + + +async def read_sse_events_with_observation_times( + session: ClientSession, + url: str, + *, + headers: Mapping[str, str], + overall_timeout_seconds: float = 240.0, +) -> tuple[int, list[SseEvent], Mapping[str, str], float | None, tuple[float, ...]]: + """Reconnect public SSE and record client-side observation times for each event.""" deadline = time.perf_counter() + overall_timeout_seconds events: list[SseEvent] = [] + observed_at: list[float] = [] first_event_at: float | None = None response_headers: Mapping[str, str] = {} while True: @@ -428,7 +578,13 @@ async def read_sse_events_with_first_event_time( request_headers["Last-Event-ID"] = str(events[-1].sequence) try: async with asyncio.timeout(remaining): - status, segment, response_headers, segment_first_event_at = await _read_sse_response( + ( + status, + segment, + response_headers, + segment_first_event_at, + segment_observed_at, + ) = await _read_sse_response( session, url, headers=request_headers, @@ -438,10 +594,11 @@ async def read_sse_events_with_first_event_time( "Function App SSE stream did not reach done before the overall deadline." ) from exc events = append_contiguous_sse_events(events, segment) + observed_at.extend(segment_observed_at) if first_event_at is None and segment_first_event_at is not None: first_event_at = segment_first_event_at if events and events[-1].payload.get("type") == "done": - return status, events, response_headers, first_event_at + return status, events, response_headers, first_event_at, tuple(observed_at) if time.perf_counter() >= deadline: raise AcaSmokeEnvironmentError( "Function App SSE stream did not reach done before the overall deadline." @@ -454,34 +611,55 @@ async def _read_sse_response( url: str, *, headers: Mapping[str, str], -) -> tuple[int, list[SseEvent], Mapping[str, str], float | None]: +) -> tuple[int, list[SseEvent], Mapping[str, str], float | None, tuple[float, ...]]: """Read one public SSE response, which may end at the server lease boundary.""" - try: - async with session.get(url, headers=headers) as response: - status = response.status - response_headers = dict(response.headers) - if status != 200: - raise SseResponseStatusError(url, status) - chunks: list[str] = [] - first_event_at: float | None = None - pending = "" - async for chunk in response.content: - decoded = chunk.decode("utf-8") - chunks.append(decoded) - pending += decoded.replace("\r\n", "\n") - frames = pending.split("\n\n") - pending = frames.pop() - if first_event_at is None and parse_sse_frames(frames): - first_event_at = time.perf_counter() - body = "".join(chunks).replace("\r\n", "\n") - return status, parse_sse_frames(body.split("\n\n")), response_headers, first_event_at - except AcaSmokeEnvironmentError: - raise - except (TimeoutError, OSError, UnicodeDecodeError) as exc: - raise AcaSmokeEnvironmentError( - f"Function App SSE endpoint was unavailable at {redact_deployed_aca_evidence(url)}: " - f"{type(exc).__name__}" - ) from exc + for attempt in range(1, _SSE_THROTTLE_MAX_ATTEMPTS + 1): + is_final = attempt == _SSE_THROTTLE_MAX_ATTEMPTS + try: + async with session.get(url, headers=headers) as response: + status = response.status + response_headers = dict(response.headers) + if status != 200: + delay = sse_throttle_retry_delay( + status, response_headers, is_final_attempt=is_final + ) + if delay is None: + raise SseResponseStatusError(url, status) + await asyncio.sleep(delay) + continue + events: list[SseEvent] = [] + observed_at: list[float] = [] + first_event_at: float | None = None + pending = "" + async for chunk in response.content: + decoded = chunk.decode("utf-8") + pending += decoded.replace("\r\n", "\n") + frames = pending.split("\n\n") + pending = frames.pop() + parsed = parse_sse_frames(frames) + if parsed: + timestamp = time.perf_counter() + if first_event_at is None: + first_event_at = timestamp + events.extend(parsed) + observed_at.extend([timestamp] * len(parsed)) + if pending.strip(): + parsed = parse_sse_frames([pending]) + if parsed: + timestamp = time.perf_counter() + if first_event_at is None: + first_event_at = timestamp + events.extend(parsed) + observed_at.extend([timestamp] * len(parsed)) + return status, events, response_headers, first_event_at, tuple(observed_at) + except AcaSmokeEnvironmentError: + raise + except (TimeoutError, OSError, UnicodeDecodeError) as exc: + raise AcaSmokeEnvironmentError( + f"Function App SSE endpoint was unavailable at " + f"{redact_deployed_aca_evidence(url)}: {type(exc).__name__}" + ) from exc + raise SseResponseStatusError(url, status) def append_contiguous_sse_events( @@ -547,7 +725,7 @@ def client_timeout(config: DeployedAcaSmokeConfig) -> ClientTimeout: def _required_function_base_url(name: str) -> str: value = _required_value(name) parsed = urlsplit(value) - normalized_path = parsed.path.rstrip("/") + normalized_path = parsed.path.rstrip("/") or "/api" if ( parsed.scheme != "https" or not parsed.netloc @@ -555,11 +733,11 @@ def _required_function_base_url(name: str) -> str: or parsed.password is not None or parsed.query or parsed.fragment - or normalized_path not in ("", "/api") + or normalized_path != "/api" ): raise AcaSmokeEnvironmentError( - f"{name} must be an HTTPS Function base URL without a path, credentials, " - "query, or fragment, except for the /api route root." + f"{name} must be an HTTPS Function base URL at the site origin or /api " + "route root, without credentials, query, or fragment." ) return urlunsplit(("https", parsed.netloc, normalized_path, "", "")) diff --git a/tests/live/aca_deployed_cold_start_support.py b/tests/live/aca_deployed_cold_start_support.py index 2410b7de..737b0566 100644 --- a/tests/live/aca_deployed_cold_start_support.py +++ b/tests/live/aca_deployed_cold_start_support.py @@ -10,6 +10,9 @@ from tests.aca_smoke_diagnostics import AcaSmokeEnvironmentError _SAMPLES_ENV = "AZURE_FUNCTIONS_AGENTS_ACA_COLD_START_SAMPLES" +_EXPECTED_BUILD_ID_ENV = "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_BUILD_ID" +_EXPECTED_COMMIT_SHA_ENV = "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_COMMIT_SHA" +_EXPECTED_PYTHON_ENV = "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_PYTHON_VERSION" _DEFAULT_SAMPLES = 3 _MIN_SAMPLES = 1 _MAX_SAMPLES = 5 @@ -38,6 +41,31 @@ class ColdStartMetrics: terminal_ms: tuple[float, float, float] +@dataclass(frozen=True, slots=True) +class ExpectedBuildIdentity: + """Expected deployed identity supplied by the qualification job.""" + + build_id: str + commit_sha: str + python_version: str + + +def expected_build_identity_from_environment() -> ExpectedBuildIdentity: + """Read expected identity without including its values in failures.""" + + def required(name: str) -> str: + value = os.environ.get(name, "").strip() + if not value or "$(" in value: + raise AcaSmokeEnvironmentError(f"{name} is required.") + return value + + return ExpectedBuildIdentity( + build_id=required(_EXPECTED_BUILD_ID_ENV), + commit_sha=required(_EXPECTED_COMMIT_SHA_ENV), + python_version=required(_EXPECTED_PYTHON_ENV), + ) + + def cold_start_samples_from_option_or_environment(config: _PytestConfig) -> int: """Resolve an explicitly configured safe sample count, defaulting to three.""" option_value = config.getoption("aca_cold_start_samples") @@ -98,10 +126,11 @@ def render_cold_start_report( retries: int, metrics: ColdStartMetrics | None, cleanup_complete: bool, + provenance_verified: bool = True, ) -> str: """Render aggregate-only evidence; never include IDs, prompts, or model output.""" metric_text = "not-available" - if metrics is not None: + if metrics is not None and provenance_verified: metric_text = ( f"first_attempt_acceptance_ms={_format(metrics.first_attempt_acceptance_ms)} " f"total_acceptance_ms={_format(metrics.total_acceptance_ms)} " @@ -111,6 +140,7 @@ def render_cold_start_report( return ( "ACA deployed cold-start qualification: " f"samples={sample_count} retries={retries} {metric_text} " + f"provenance={'verified' if provenance_verified else 'unverified'} " f"cleanup={'complete' if cleanup_complete else 'incomplete'}" ) diff --git a/tests/live/aca_deployed_load_support.py b/tests/live/aca_deployed_load_support.py index 90b4e4de..67a30607 100644 --- a/tests/live/aca_deployed_load_support.py +++ b/tests/live/aca_deployed_load_support.py @@ -4,12 +4,18 @@ import math import os +from collections.abc import Mapping, Sequence from dataclasses import dataclass from datetime import UTC, datetime +from itertools import pairwise from typing import Protocol import pytest +from eng.scripts.aca_deployed_qualification import FORMAL_N100_UNSUPPORTED_ERROR from tests.aca_smoke_diagnostics import AcaSmokeEnvironmentError +from tests.live.aca_deployed_agent_support import ( + optional_retry_after_seconds, +) _LOAD_CONCURRENCY_ENV = "AZURE_FUNCTIONS_AGENTS_ACA_LOAD_CONCURRENCY" _MIN_CONCURRENCY = 1 @@ -18,6 +24,40 @@ _DEFAULT_PROVISION_CONCURRENCY = 4 _MIN_PROVISION_CONCURRENCY = 1 _MAX_PROVISION_CONCURRENCY = 4 +THROTTLE_RETRY_AFTER_MAXIMUM_SECONDS = 10.0 + + +def throttle_retry_after_seconds(headers: Mapping[str, str]) -> float | None: + """Return Retry-After if present and within bounds, else None.""" + return optional_retry_after_seconds( + headers, maximum_seconds=THROTTLE_RETRY_AFTER_MAXIMUM_SECONDS + ) + + +THROTTLED_ADMISSION_STATUSES = frozenset({429, 503}) + + +def throttled_admission_retry_delay( + status: int, + headers: Mapping[str, str], + *, + is_final_attempt: bool, +) -> float | None: + """Return the delay before retrying a throttled admission, or None to give up. + + Honoring the server's backpressure is what a correct client does, but the + caller must still fail when admissions never recover, so the final attempt + always declines to retry. + """ + if status not in THROTTLED_ADMISSION_STATUSES: + return None + if is_final_attempt: + return None + return throttle_retry_after_seconds(headers) + + +# Bound one poll burst without allowing adjacent events to chain indefinitely. +_BATCH_WINDOW_SECONDS = 0.05 class _PytestConfig(Protocol): @@ -34,11 +74,28 @@ class CommonActiveInterval: @dataclass(frozen=True, slots=True) class LoadLatencyMetrics: - """Redacted latency quantiles in milliseconds.""" + """Redacted latency and client-observed streaming quantiles in milliseconds.""" submission_ms: tuple[float, float, float] first_event_ms: tuple[float, float, float] terminal_ms: tuple[float, float, float] + visibility_gap_ms: tuple[float, float, float] | None + visibility_gap_all_ms: tuple[float, float, float] | None + observed_poll_cadence_ms: tuple[float, float, float] | None + events_per_batch: tuple[tuple[int, int], ...] + visibility_gap_sample_count: int + visibility_gap_all_sample_count: int + observed_poll_cadence_sample_count: int + event_batch_count: int + observed_event_count: int + + +@dataclass(frozen=True, slots=True) +class ObservedEventBatch: + """Client-side arrival burst summary.""" + + observed_at: float + event_count: int def load_concurrency_from_option_or_environment(config: _PytestConfig) -> int | None: @@ -61,6 +118,8 @@ def load_concurrency_from_option_or_environment(config: _PytestConfig) -> int | raise AcaSmokeEnvironmentError( f"{source} must be between {_MIN_CONCURRENCY} and {_MAX_CONCURRENCY}." ) + if value == _MAX_CONCURRENCY: + raise AcaSmokeEnvironmentError(FORMAL_N100_UNSUPPORTED_ERROR) return value @@ -105,15 +164,113 @@ def latency_metrics( submission_seconds: list[float], first_event_seconds: list[float], terminal_seconds: list[float], + observed_event_timestamp_sequences: Sequence[Sequence[float]] = (), ) -> LoadLatencyMetrics: """Calculate nearest-rank p50/p95/p99 latencies without retaining request data.""" + primary_visibility_gaps = [ + gap + for timestamps in observed_event_timestamp_sequences + for gap in visibility_gap_seconds(timestamps, waiting_only=True) + ] + all_visibility_gaps = [ + gap + for timestamps in observed_event_timestamp_sequences + for gap in visibility_gap_seconds(timestamps, waiting_only=False) + ] + observed_poll_cadences = [ + cadence + for timestamps in observed_event_timestamp_sequences + for cadence in observed_poll_cadence_seconds(timestamps) + ] + batch_counts: dict[int, int] = {} + event_batch_count = 0 + observed_event_count = 0 + for timestamps in observed_event_timestamp_sequences: + batches = observed_event_batches(timestamps) + event_batch_count += len(batches) + observed_event_count += sum(batch.event_count for batch in batches) + for batch in batches: + batch_counts[batch.event_count] = batch_counts.get(batch.event_count, 0) + 1 return LoadLatencyMetrics( submission_ms=_percentiles(submission_seconds), first_event_ms=_percentiles(first_event_seconds), terminal_ms=_percentiles(terminal_seconds), + visibility_gap_ms=_optional_percentiles(primary_visibility_gaps), + visibility_gap_all_ms=_optional_percentiles(all_visibility_gaps), + observed_poll_cadence_ms=_optional_percentiles(observed_poll_cadences), + events_per_batch=tuple(sorted(batch_counts.items())), + visibility_gap_sample_count=len(primary_visibility_gaps), + visibility_gap_all_sample_count=len(all_visibility_gaps), + observed_poll_cadence_sample_count=len(observed_poll_cadences), + event_batch_count=event_batch_count, + observed_event_count=observed_event_count, ) +def observed_event_batches( + observed_event_timestamps: Sequence[float], + *, + batch_window_seconds: float = _BATCH_WINDOW_SECONDS, +) -> tuple[ObservedEventBatch, ...]: + """Group client-side event observation timestamps into arrival bursts. + + Events delivered by one poll are read from the stream microseconds apart, so + they share an arrival *window* rather than an identical timestamp. Grouping + on exact equality would put every event in its own batch, which silently + empties the waiting-only series and collapses the measured cadence to the + cost of parsing two adjacent events -- numbers that look plausible and mean + nothing. The window is what makes a burst detectable in real data. + """ + ordered = sorted(observed_event_timestamps) + if not ordered: + return () + batches: list[ObservedEventBatch] = [] + batch_start = ordered[0] + current_count = 1 + for timestamp in ordered[1:]: + if timestamp - batch_start <= batch_window_seconds: + current_count += 1 + continue + batches.append(ObservedEventBatch(batch_start, current_count)) + batch_start = timestamp + current_count = 1 + batches.append(ObservedEventBatch(batch_start, current_count)) + return tuple(batches) + + +def visibility_gap_seconds( + observed_event_timestamps: Sequence[float], + *, + waiting_only: bool, +) -> tuple[float, ...]: + """Return client-observed streaming gaps, optionally requiring waiting-event evidence.""" + batches = observed_event_batches(observed_event_timestamps) + if waiting_only: + return tuple( + current.observed_at - previous.observed_at + for previous, current in pairwise(batches) + if current.event_count >= 2 + ) + ordered = sorted(observed_event_timestamps) + return tuple(current - previous for previous, current in pairwise(ordered)) + + +def observed_poll_cadence_seconds( + observed_event_timestamps: Sequence[float], +) -> tuple[float, ...]: + """Return inter-batch client arrival spacing measured from observed bursts.""" + batches = observed_event_batches(observed_event_timestamps) + return tuple( + current.observed_at - previous.observed_at + for previous, current in pairwise(batches) + ) + + +def events_per_batch(observed_event_timestamps: Sequence[float]) -> tuple[int, ...]: + """Return the number of events in each client-observed arrival burst.""" + return tuple(batch.event_count for batch in observed_event_batches(observed_event_timestamps)) + + def _percentiles(values: list[float]) -> tuple[float, float, float]: if not values: raise AssertionError("Latency percentiles require at least one value.") @@ -124,6 +281,10 @@ def _percentiles(values: list[float]) -> tuple[float, float, float]: return p50, p95, p99 +def _optional_percentiles(values: list[float]) -> tuple[float, float, float] | None: + return _percentiles(values) if values else None + + def render_load_report( *, concurrency: int, @@ -156,7 +317,18 @@ def render_load_report( metric_text = ( f"submission_ms={_format_quantiles(metrics.submission_ms)} " f"first_event_ms={_format_quantiles(metrics.first_event_ms)} " - f"terminal_ms={_format_quantiles(metrics.terminal_ms)}" + f"terminal_ms={_format_quantiles(metrics.terminal_ms)} " + f"visibility_gap_ms={_format_optional_quantiles(metrics.visibility_gap_ms)} " + f"visibility_gap_samples={metrics.visibility_gap_sample_count} " + f"visibility_gap_all_ms={_format_optional_quantiles(metrics.visibility_gap_all_ms)} " + f"visibility_gap_all_samples={metrics.visibility_gap_all_sample_count} " + f"observed_poll_cadence_ms=" + f"{_format_optional_quantiles(metrics.observed_poll_cadence_ms)} " + f"observed_poll_cadence_samples={metrics.observed_poll_cadence_sample_count} " + f"events_per_batch={_format_batch_distribution(metrics.events_per_batch)} " + f"event_batches={metrics.event_batch_count} observed_events={metrics.observed_event_count} " + f"visibility_attribution={_visibility_attribution(metrics)} " + f"{_visibility_warning(metrics)}" ) failure_categories = ( ",".join(f"{category}={count}" for category, count in admission_failure_categories) @@ -184,6 +356,9 @@ def render_load_report( f"unresolved_idempotencies={unresolved_idempotency_count} " f"admission_failure_categories={failure_categories} " f"cleanup={'complete' if cleanup_complete else 'incomplete'}" + " visibility_proxy_note=client-only observed arrival gaps; does not capture true " + "sandbox-write-to-client-observe delta because sandbox and CI clocks are not on a " + "common basis and clock-skew correction would add error comparable to the 2s budget." ) @@ -194,3 +369,34 @@ def utc_now() -> datetime: def _format_quantiles(values: tuple[float, float, float]) -> str: return f"p50={values[0]:.1f},p95={values[1]:.1f},p99={values[2]:.1f}" + + +def _format_optional_quantiles(values: tuple[float, float, float] | None) -> str: + return _format_quantiles(values) if values is not None else "not-available" + + +def _format_batch_distribution(events_per_batch_distribution: tuple[tuple[int, int], ...]) -> str: + if not events_per_batch_distribution: + return "not-available" + return ",".join( + f"{event_count}x{batch_count}" + for event_count, batch_count in events_per_batch_distribution + ) + + +def _visibility_warning(metrics: LoadLatencyMetrics) -> str: + if metrics.visibility_gap_ms is not None and metrics.visibility_gap_ms[1] > 2000: + return "visibility_warning=p95_exceeds_2s" + return "visibility_warning=none" + + +def _visibility_attribution(metrics: LoadLatencyMetrics) -> str: + if metrics.visibility_gap_ms is None or metrics.observed_poll_cadence_ms is None: + return "not-available" + gap_p95 = metrics.visibility_gap_ms[1] + cadence_p95 = metrics.observed_poll_cadence_ms[1] + if abs(gap_p95 - cadence_p95) <= max(250.0, cadence_p95 * 0.25): + return "poll_timing_dominates" + if gap_p95 > cadence_p95: + return "transport_exceeds_cadence" + return "below_observed_cadence" diff --git a/tests/live/aca_deployed_loss_support.py b/tests/live/aca_deployed_loss_support.py index ba309d95..f5c9796b 100644 --- a/tests/live/aca_deployed_loss_support.py +++ b/tests/live/aca_deployed_loss_support.py @@ -99,7 +99,10 @@ def assert_public_backing_loss_contract( ) -> None: """Verify public terminal status stays readable while the unavailable result is gone.""" assert status_code == 200 - assert status.get("state") == "abandoned" + assert status.get("state") == "abandoned", ( + f"public state was {status.get('state')!r}; the durable record is abandoned, " + "so the public projection disagrees with the authoritative store." + ) error = status.get("error") if error is not None: assert isinstance(error, dict) diff --git a/tests/live/apps/aca-qualification/.funcignore b/tests/live/apps/aca-qualification/.funcignore new file mode 100644 index 00000000..4480f18b --- /dev/null +++ b/tests/live/apps/aca-qualification/.funcignore @@ -0,0 +1,4 @@ +.venv/ +local.settings.json +__pycache__/ +*.py[cod] diff --git a/tests/live/apps/aca-qualification/.gitignore b/tests/live/apps/aca-qualification/.gitignore new file mode 100644 index 00000000..9807694c --- /dev/null +++ b/tests/live/apps/aca-qualification/.gitignore @@ -0,0 +1 @@ +BUILD_INFO.json diff --git a/tests/live/apps/aca-qualification/README.md b/tests/live/apps/aca-qualification/README.md new file mode 100644 index 00000000..7b359c3f --- /dev/null +++ b/tests/live/apps/aca-qualification/README.md @@ -0,0 +1,69 @@ +> **Deployed qualification fixture — not an E2E app.** +> +> This app is deployed to a protected Flex Consumption test app by the deployed +> ACA qualification tooling in `eng/scripts/` (FRD 0008 §14, issue #166). It is +> deliberately **not** under `tests/endtoend/apps/`, because +> `tests/endtoend/test_apps_start.py` globs `*/host.json` there and +> auto-parameterizes a `func start` test over every match. This app selects the +> ACA Sandbox backend, so it cannot start without real Azure and would fail that +> suite. + +# ACA qualification fixture app + +A minimal agent app whose only job is to be **deployed** and then exercised by +the deployed qualification suites (`tests/live/test_aca_deployed_*.py`). + +## Contents + +| File | Purpose | +| --- | --- | +| `agents.config.yaml` | Selects the ACA Sandbox backend, Entra HTTP auth, model, timeout, and 120-second reclaim retention; successful results preserve expiry for 300 seconds from terminal run update | +| `deployed_turn.agent.md` | No-tools built-in-endpoint agent, slug `deployed_turn` | +| `deployed_load.agent.md` | Load/loss built-in-endpoint agent, slug `deployed_load` | +| `tools/qualification_hold.py` | Fixture-only tool that holds an active run for load and backing-loss suites | +| `function_app.py` | `create_function_app()` plus the fixture-only `/__buildinfo` route | +| `host.json` | Default `/api` route prefix, so agent routes are `/api/agents//...` | +| `requirements.txt` | Generated, fully pinned; installs the runtime wheel by filename | + +## No configuration is committed + +Every environment-specific value is read from app settings at load time using +`$VAR` substitution — the sandbox group resource ID, the model endpoint and +deployment, and storage. Nothing identifying a subscription, resource group, +site, or endpoint appears in this directory. + +The operator supplies those settings as app settings on the target Function +App before deployment. The deploy command sets only the required Sandbox Group +region; the group resource ID, model deployment, storage, and Entra values must +already be configured. This layer adds no pipeline wiring — the deploy is +driven by `eng/scripts/aca_qualification_pipeline.py` by hand. + +Configure `AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_FUNCTION_BASE_URL` as either the +HTTPS site origin or that origin plus `/api`; the client normalizes a pathless +origin to the fixture's default `/api` route root. + +The canonical qualification uses N=5 with provisioning concurrency 1. Manual +diagnostics retain load values 1–99 and provisioning values 1, 2, or 4; their +operator owns shared-group quota and cost. N=100 is rejected before +authentication or provider work and remains future human-only formal acceptance +requiring a purpose-built workflow. + +## `/__buildinfo` + +Returns the `BUILD_INFO.json` that `aca_qualification_pipeline.py assemble` +stamps into this directory before deployment, plus values read live from the +running host. + +It is **corroborating evidence, not self-attestation**: it is trustworthy only +because the marker is a *file inside the deployed package*. A file can be served +only if the package containing it is genuinely on disk, so the app cannot report +a build it is not running. An app setting or resource tag could be changed +without deploying anything, which is exactly where "the service reports its own +version" stops being evidence. + +This is intentionally not a detached content-addressed chain: it does not prove +the wheel digest, installed package version, deploy-input manifest, or deployment +storage version. FRD 0008 Decision 193 records that narrowed scope. + +The route is defined here, in the fixture, and touches no product module — in +particular not `registration/endpoints.py` (FRD 0008 Decision 172). diff --git a/tests/live/apps/aca-qualification/agents.config.yaml b/tests/live/apps/aca-qualification/agents.config.yaml new file mode 100644 index 00000000..a40d9081 --- /dev/null +++ b/tests/live/apps/aca-qualification/agents.config.yaml @@ -0,0 +1,18 @@ +session_runtime: + aca_sandbox: + sandbox_group_resource_id: $AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_GROUP_RESOURCE_ID + region: $AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_REGION + retention: + auto_suspend_idle: 60 + reclaim_idle: 120 + +model: $AZURE_OPENAI_DEPLOYMENT +timeout: 300 +http_auth: + mode: entra + entra: + tenant_id: $AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_ENTRA_TENANT_ID + allowed_audiences: + - $AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_AUDIENCE + allowed_client_ids: + - $AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_TEST_INVOKER_CLIENT_ID diff --git a/tests/live/apps/aca-qualification/deployed_load.agent.md b/tests/live/apps/aca-qualification/deployed_load.agent.md new file mode 100644 index 00000000..2aacf044 --- /dev/null +++ b/tests/live/apps/aca-qualification/deployed_load.agent.md @@ -0,0 +1,16 @@ +--- +name: deployed_load +description: Deployed ACA load and backing-loss qualification agent. +builtin_endpoints: + chat_api: true + mcp: false +timeout: 900 +mcp: false +skills: false +tools: true +system_tools: + web_request: false +--- + +For a load qualification request, call `qualification_hold` exactly once, then return a brief +acknowledgement. Do not access web, MCP, or external resources. diff --git a/tests/live/apps/aca-qualification/deployed_turn.agent.md b/tests/live/apps/aca-qualification/deployed_turn.agent.md new file mode 100644 index 00000000..1b315c38 --- /dev/null +++ b/tests/live/apps/aca-qualification/deployed_turn.agent.md @@ -0,0 +1,17 @@ +--- +name: deployed_turn +description: Deployed ACA qualification agent exercised by the deployed qualification suite. +builtin_endpoints: + chat_api: true + mcp: false +timeout: 120 +mcp: false +skills: false +tools: false +system_tools: + web_request: false +--- + +You are a qualification probe for the ACA Sandbox session runtime. + +Answer the user's question directly and briefly. Do not use tools. diff --git a/tests/live/apps/aca-qualification/function_app.py b/tests/live/apps/aca-qualification/function_app.py new file mode 100644 index 00000000..5fbe83af --- /dev/null +++ b/tests/live/apps/aca-qualification/function_app.py @@ -0,0 +1,113 @@ +"""Deployed ACA qualification fixture app. + +Adds one fixture-only ``/__buildinfo`` route to the standard agent app so a +qualification run can confirm that the app it is about to qualify is running +the build that was just deployed. + +The route lives here, in the fixture, and imports nothing from the runtime's +registration layer, so it cannot collide with product endpoint work. + +**Why this is evidence rather than self-report.** The marker is a file inside +the deployed package. A file can be served only if the package containing it is +genuinely on disk, so a stale app cannot claim a build it is not running. An app +setting or resource tag could be changed without deploying anything, which is +precisely where a service reporting its own version stops proving anything. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +import azure.functions as func +from azurefunctions.extensions.http.fastapi import JSONResponse, Request + +from azure_functions_agents import create_function_app + +app = create_function_app() + +_APP_ROOT = Path(__file__).resolve().parent +_BUILD_INFO_PATH = _APP_ROOT / "BUILD_INFO.json" + +_MAX_SCANNED_ENTRIES = 20_000 + + +def _load_marker() -> dict[str, Any]: + """Read the stamped marker, or report its absence explicitly.""" + try: + raw = _BUILD_INFO_PATH.read_text(encoding="utf-8") + except FileNotFoundError: + return {"marker": "absent"} + except OSError as error: + return {"marker": "unreadable", "error": type(error).__name__} + try: + parsed = json.loads(raw) + except json.JSONDecodeError: + return {"marker": "invalid"} + if not isinstance(parsed, dict): + return {"marker": "invalid"} + parsed["marker"] = "present" + return parsed + + +def _content_size() -> dict[str, Any]: + """Measure deployed content against the platform's package limits. + + Reported so a qualification run can trend closure growth toward the 256 MiB + / 65,535-entry caps while there is still headroom to act, rather than + discovering the ceiling as a deployment failure. + """ + total_bytes = 0 + entry_count = 0 + truncated = False + for path in _APP_ROOT.rglob("*"): + if entry_count >= _MAX_SCANNED_ENTRIES: + truncated = True + break + try: + if not path.is_file(): + continue + total_bytes += path.stat().st_size + except OSError: + pass + entry_count += 1 + return { + "entry_count": entry_count, + "total_bytes": total_bytes, + "truncated": truncated, + } + + +@app.route( + route="__buildinfo", + methods=["GET"], + auth_level=func.AuthLevel.ANONYMOUS, +) +def build_info(req: Request) -> JSONResponse: + """Return the deployed build marker plus live host facts. + + The parameter and return types must be the **FastAPI** ``Request`` and a + FastAPI response, matching how the runtime registers every other route. + Using ``azure.functions.HttpRequest`` here makes the worker reject the + binding, and because indexing is all-or-nothing that single bad function + takes down the entire app -- every agent route included, reported only as + "No job functions found". + + ``auth_level`` is anonymous because Easy Auth is the gate: the platform is + configured with ``requireAuthentication`` and ``Return401``, so an + unauthenticated request never reaches this function. Adding a function key + would impose a second, different credential that the qualification job's + Entra token could not satisfy. + """ + del req + payload = { + "build": _load_marker(), + "runtime": { + "python_version": f"{sys.version_info.major}.{sys.version_info.minor}", + "python_micro": sys.version_info.micro, + }, + "content": _content_size(), + } + return JSONResponse(content=payload, status_code=200) diff --git a/tests/live/apps/aca-qualification/host.json b/tests/live/apps/aca-qualification/host.json new file mode 100644 index 00000000..7d551012 --- /dev/null +++ b/tests/live/apps/aca-qualification/host.json @@ -0,0 +1,12 @@ +{ + "version": "2.0", + "logging": { + "logLevel": { + "default": "Information" + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle", + "version": "[4.*, 5.0.0)" + } +} diff --git a/tests/live/apps/aca-qualification/requirements.txt b/tests/live/apps/aca-qualification/requirements.txt new file mode 100644 index 00000000..8f83b1ac --- /dev/null +++ b/tests/live/apps/aca-qualification/requirements.txt @@ -0,0 +1,26 @@ +# GENERATED AT DEPLOY TIME — do not add pins here. +# +# The qualification pipeline's `assemble` step overwrites this file in its +# staging copy with: +# +# 1. the locally built runtime wheel, referenced by its exact filename, and +# 2. the pinned dependency set exported from uv.lock at +# eng/constraints/aca-fixture-requirements.txt. +# +# The wheel must ship in the upload because the merged commit is not on PyPI, so +# Azure's remote build could never fetch it; shipping it is what guarantees the +# deployed app is the merge under test rather than the last release. +# +# This file deliberately does NOT duplicate the pinned set. An inlined copy +# would be dead weight that drifts out of step with the real export the moment +# it is regenerated, and a stale duplicate here would be invisible because +# nothing reads it. +# +# To deploy this fixture by hand, generate the same content the pipeline would: +# +# python eng/scripts/aca_qualification_pipeline.py assemble \ +# --artifact-root \ +# --staging-root \ +# --requirements-export eng/constraints/aca-fixture-requirements.txt \ +# --commit-sha --build-id local --branch local \ +# --runtime-version diff --git a/tests/live/apps/aca-qualification/tools/qualification_hold.py b/tests/live/apps/aca-qualification/tools/qualification_hold.py new file mode 100644 index 00000000..da71d069 --- /dev/null +++ b/tests/live/apps/aca-qualification/tools/qualification_hold.py @@ -0,0 +1,13 @@ +"""Fixture-only active-run hold for deployed ACA load qualification.""" + +from __future__ import annotations + +import asyncio + +QUALIFICATION_HOLD_SECONDS = 300 + + +async def qualification_hold() -> str: + """Hold long enough for the live suites to observe and disrupt an active run.""" + await asyncio.sleep(QUALIFICATION_HOLD_SECONDS) + return "hold complete" diff --git a/tests/live/test_aca_deployed_cold_start.py b/tests/live/test_aca_deployed_cold_start.py index cfdbf261..b8b276b0 100644 --- a/tests/live/test_aca_deployed_cold_start.py +++ b/tests/live/test_aca_deployed_cold_start.py @@ -10,6 +10,10 @@ import pytest from aiohttp import ClientSession, ClientTimeout +from eng.scripts.aca_qualification_pipeline import ( + compare_marker, + fetch_build_info, +) from tests.aca_smoke_diagnostics import AcaSmokeEnvironmentError from tests.live.aca_deployed_agent_support import ( AcceptedRun, @@ -29,6 +33,7 @@ SSE_TERMINAL_WINDOW_SECONDS, cold_start_metrics, cold_start_samples_from_option_or_environment, + expected_build_identity_from_environment, first_attempt_slo_failure, render_cold_start_report, ) @@ -125,6 +130,7 @@ async def test_deployed_aca_cold_start_acceptance_is_bounded_and_cleaned( attempted_keys: list[str] = [] progress = _Progress(samples=[]) cleanup_complete = False + provenance_verified = False primary_error: BaseException | None = None try: resources = await open_deployed_aca_lifecycle_resources(config) @@ -146,6 +152,11 @@ async def test_deployed_aca_cold_start_acceptance_is_bounded_and_cleaned( attempted_keys, progress, ) + await _verify_deployed_build( + config.deployed.base_url, + config.deployed.token_scope, + ) + provenance_verified = True _require( len({sample.candidate.accepted.session_id for sample in progress.samples}) == sample_count, "distinct_fresh_sessions_required", @@ -174,16 +185,35 @@ async def test_deployed_aca_cold_start_acceptance_is_bounded_and_cleaned( [sample.first_event_seconds for sample in progress.samples], [sample.terminal_seconds for sample in progress.samples], ) - if progress.samples + if progress.samples and provenance_verified else None ), cleanup_complete=cleanup_complete, + provenance_verified=provenance_verified, ) ) if primary_error is not None: raise primary_error +async def _verify_deployed_build(base_url: str, token_scope: str) -> None: + expected = expected_build_identity_from_environment() + try: + reported = await fetch_build_info(base_url, token_scope) + except Exception: + raise AssertionError("cold_start_provenance_unavailable") from None + comparison = compare_marker( + reported, + expected_build_id=expected.build_id, + expected_commit_sha=expected.commit_sha, + expected_python=expected.python_version, + ) + if not comparison.matches: + raise AssertionError( + f"cold_start_provenance_mismatch:{','.join(comparison.mismatches)}" + ) + + def _cold_start_config(config: DeployedAcaLifecycleConfig) -> DeployedAcaLifecycleConfig: """Force the customer-facing no-tools fixture agent, never the load hold agent.""" return replace(config, deployed=replace(config.deployed, agent_slug=_COLD_START_AGENT_SLUG)) diff --git a/tests/live/test_aca_deployed_lifecycle.py b/tests/live/test_aca_deployed_lifecycle.py index 79f27932..d2a077a5 100644 --- a/tests/live/test_aca_deployed_lifecycle.py +++ b/tests/live/test_aca_deployed_lifecycle.py @@ -22,6 +22,7 @@ submission_payload, ) from tests.live.aca_deployed_lifecycle_support import ( + LIFECYCLE_RECLAIM_IDLE_SECONDS, DeployedAcaLifecycleConfig, DeployedAcaLifecycleResources, assert_session_belongs_to_deployment, @@ -30,6 +31,7 @@ open_deployed_aca_lifecycle_resources, owned_sandbox, owned_snapshots, + read_authoritative_run, read_authoritative_session, wait_for_idle_session, wait_for_reclaimed_session, @@ -37,7 +39,8 @@ wait_until_reclaim_due, ) -from azure_functions_agents.session_state import DurableSessionRecord +from azure_functions_agents.sandbox_runtime_limits import RESULT_HOLD_SECONDS +from azure_functions_agents.session_state import DurableRunRecord, DurableSessionRecord _SETUP_RETRY_ATTEMPTS = 4 @@ -107,7 +110,11 @@ async def _qualify_first_run_and_suspend( assert_session_belongs_to_deployment(first_session, config) assert first_session.generation >= 1 assert first_session.sandbox_id is not None - assert first_session.expires_at - first_session.last_activity_at == timedelta(seconds=120) + await _assert_terminal_retention( + resources, + session=first_session, + accepted=first_run, + ) first_sandbox_id = first_session.sandbox_id first_generation = first_session.generation @@ -136,7 +143,11 @@ async def _qualify_resume_and_reclaim( progress.cleanup_session = resumed_session assert resumed_session.sandbox_id == first.sandbox_id assert resumed_session.generation == first.generation - assert resumed_session.expires_at - resumed_session.last_activity_at == timedelta(seconds=120) + await _assert_terminal_retention( + resources, + session=resumed_session, + accepted=resumed_run, + ) resumed_suspended = await wait_for_suspended_sandbox( resources, resumed_session, timeout_seconds=105.0 @@ -155,6 +166,36 @@ async def _qualify_resume_and_reclaim( return resumed_run +async def _assert_terminal_retention( + resources: DeployedAcaLifecycleResources, + *, + session: DurableSessionRecord, + accepted: AcceptedRun, +) -> None: + terminal_run = await read_authoritative_run( + resources, + session_id=accepted.session_id, + run_id=accepted.run_id, + ) + _assert_matching_successful_run(terminal_run, accepted) + assert session.expires_at >= session.last_activity_at + timedelta( + seconds=LIFECYCLE_RECLAIM_IDLE_SECONDS + ) + assert session.expires_at >= terminal_run.updated_at + timedelta( + seconds=RESULT_HOLD_SECONDS + ) + + +def _assert_matching_successful_run( + terminal_run: DurableRunRecord, + accepted: AcceptedRun, +) -> None: + assert terminal_run.session_id == accepted.session_id + assert terminal_run.run_id == accepted.run_id + assert terminal_run.status == "succeeded" + assert terminal_run.result_available + + async def _assert_public_terminal_after_reclaim( client: ClientSession, resumed_run: AcceptedRun, diff --git a/tests/live/test_aca_deployed_load.py b/tests/live/test_aca_deployed_load.py index 92e3e822..27c12a56 100644 --- a/tests/live/test_aca_deployed_load.py +++ b/tests/live/test_aca_deployed_load.py @@ -23,8 +23,11 @@ acquire_default_authorization_evidence, deployed_aca_smoke_enabled, json_request, + optional_retry_after_seconds, parse_accepted_run, read_sse_events_with_first_event_time, + read_sse_events_with_observation_times, + response_header, setup_retry_after_seconds, submission_payload, ) @@ -43,12 +46,15 @@ read_session_operations, ) from tests.live.aca_deployed_load_support import ( + THROTTLED_ADMISSION_STATUSES, CommonActiveInterval, LoadLatencyMetrics, latency_metrics, provision_concurrency_from_option_or_environment, render_load_report, require_load_concurrency, + throttle_retry_after_seconds, + throttled_admission_retry_delay, utc_now, ) @@ -68,16 +74,18 @@ _COMMON_ACTIVE_WAIT_SECONDS = 1.0 _ACTIVE_PROOF_TIMEOUT_SECONDS = 120.0 _EVENT_STREAM_GRACE_SECONDS = 360.0 +_RESULT_RETRY_AFTER_MAXIMUM_SECONDS = 10.0 _HOLD_SECONDS = 300.0 +_RESULT_MATERIALIZATION_TIMEOUT_SECONDS = _HOLD_SECONDS _MINIMUM_HOLD_TERMINAL_SECONDS = _HOLD_SECONDS - 1.0 _SETUP_DEADLINE_ATTEMPTS = 2 +_MAX_ADMISSION_ATTEMPTS = 5 _SETUP_HTTP_ATTEMPT_TIMEOUT_SECONDS = 105.0 _PROVISION_BATCH_TIMEOUT_SECONDS = 330.0 _PHASE_B_ADMISSION_TIMEOUT_SECONDS = 330.0 _RECOVERY_ATTEMPTS = 5 _RECOVERY_POLL_SECONDS = 1.0 _FINAL_RECOVERY_TIMEOUT_SECONDS = 60.0 -_OVERLAP_BUDGET_MARGIN_SECONDS = 15.0 _SETTLEMENT_TIMEOUT_SECONDS = 900.0 _RACE_SAMPLE_LIMIT = 5 _CONNECTION_HEADROOM = 10 @@ -117,6 +125,7 @@ class _SubmittedRun: class _EventEvidence: first_event_at: float terminal_at: float + observed_event_timestamps: tuple[float, ...] @dataclass(frozen=True, slots=True) @@ -200,6 +209,16 @@ class _SetupRetry: retry_after_seconds: float +@dataclass(frozen=True, slots=True) +class _SetupDeadlineRetry: + retry_after_seconds: float + + +def _throttle_retry_after_seconds(headers: Mapping[str, str]) -> float | None: + """Delegate to the support module's testable implementation.""" + return throttle_retry_after_seconds(headers) + + class _AdmissionFailureError(AcaSmokeEnvironmentError): """Sanitized aggregate admission failure that retains all prior admissions for cleanup.""" @@ -410,7 +429,6 @@ async def _admit_held_load_sessions( raise _record_held_admission_summary(state, admission) _assert_distinct_admissions(state.held, context.concurrency) - _assert_remaining_hold_budget(state.held) def _record_held_admission_failure( @@ -476,6 +494,7 @@ async def _verify_held_load_sessions( event.terminal_at - item.submitted_at for item, event in zip(state.held, event_evidence, strict=True) ], + [event.observed_event_timestamps for event in event_evidence], ) @@ -990,6 +1009,7 @@ async def _post_admission_request( config.deployed.chat_url, headers=_admission_request_headers(headers, request), payload=submission_payload(prompt), + retry_throttled=False, ) return _AdmissionResponse( status=status, @@ -1079,9 +1099,9 @@ async def _setup_deadline_response_outcome( retries: int, *, is_final_attempt: bool, -) -> _AdmissionOutcome | _SetupRetry: +) -> _AdmissionOutcome | _SetupDeadlineRetry: if not is_final_attempt: - return _SetupRetry(setup_retry_after_seconds(response.response_headers)) + return _SetupDeadlineRetry(setup_retry_after_seconds(response.response_headers)) return await _recover_admission_outcome( resources, config, @@ -1102,7 +1122,8 @@ async def _admission_response_outcome( retries: int, *, is_final_attempt: bool, -) -> _AdmissionOutcome | _SetupRetry: + is_final_setup_deadline_attempt: bool, +) -> _AdmissionOutcome | _SetupRetry | _SetupDeadlineRetry: if response.status == 202: return await _accepted_admission_outcome( resources, @@ -1112,7 +1133,14 @@ async def _admission_response_outcome( response, retries, ) - if response.status in {429, 503}: + if response.status in THROTTLED_ADMISSION_STATUSES: + retry_after = throttled_admission_retry_delay( + response.status, + response.response_headers, + is_final_attempt=is_final_attempt, + ) + if retry_after is not None: + return _SetupRetry(retry_after) return await _recover_ambiguous_http_outcome( resources, config, @@ -1132,7 +1160,7 @@ async def _admission_response_outcome( request, response, retries, - is_final_attempt=is_final_attempt, + is_final_attempt=is_final_setup_deadline_attempt, ) if response.status in {400, 401, 403, 404}: return _AdmissionOutcome( @@ -1176,7 +1204,8 @@ async def _submit_one( deadline: float | None = None, ) -> _AdmissionOutcome: request = _new_admission_request(attempted_idempotency_keys, session_id) - for retries in range(_SETUP_DEADLINE_ATTEMPTS): + setup_deadline_attempts = 0 + for retries in range(_MAX_ADMISSION_ATTEMPTS): if _admission_deadline_elapsed(deadline): return _setup_deadline_outcome(request.idempotency_key, retries) try: @@ -1205,15 +1234,21 @@ async def _submit_one( request, response, retries, - is_final_attempt=retries + 1 == _SETUP_DEADLINE_ATTEMPTS, + is_final_attempt=retries + 1 == _MAX_ADMISSION_ATTEMPTS, + is_final_setup_deadline_attempt=( + setup_deadline_attempts + 1 >= _SETUP_DEADLINE_ATTEMPTS + or retries + 1 == _MAX_ADMISSION_ATTEMPTS + ), ) if isinstance(outcome, _AdmissionOutcome): return outcome + if isinstance(outcome, _SetupDeadlineRetry): + setup_deadline_attempts += 1 retry_count = retries + 1 if _retry_would_exceed_setup_deadline(deadline, outcome.retry_after_seconds): return _setup_deadline_outcome(request.idempotency_key, retry_count) await asyncio.sleep(outcome.retry_after_seconds) - raise AssertionError("setup-deadline admission loop must return an outcome") + raise AssertionError("admission loop must return an outcome") def _setup_deadline_outcome(idempotency_key: str, retries: int) -> _AdmissionOutcome: @@ -1366,17 +1401,6 @@ async def _recover_ambiguous_http_outcome( ) -def _assert_remaining_hold_budget(submitted: list[_SubmittedRun]) -> None: - """Reject runs that cannot leave the formal proof enough shared hold time.""" - accepted_times = [item.accepted_at for item in submitted] - admission_spread = max(accepted_times) - min(accepted_times) - required_budget = _ACTIVE_PROOF_TIMEOUT_SECONDS + _OVERLAP_BUDGET_MARGIN_SECONDS - remaining_budget = _HOLD_SECONDS - admission_spread - assert remaining_budget > required_budget, ( - "Admission spread leaves insufficient remaining qualification hold: " - f"remaining={remaining_budget:.1f}s required>{required_budget:.1f}s." - ) - def _assert_distinct_admissions(submitted: list[_SubmittedRun], concurrency: int) -> None: assert len(submitted) == concurrency @@ -1406,7 +1430,12 @@ async def _establish_common_active_interval( first = await _active_observations(resources, config, partition_key, submitted) if first is not None: replay_count, conflict_count = await _exercise_active_races( - client, config, headers, submitted[:_RACE_SAMPLE_LIMIT] + resources, + client, + config, + partition_key, + headers, + submitted[:_RACE_SAMPLE_LIMIT], ) await asyncio.sleep(_COMMON_ACTIVE_WAIT_SECONDS) second = await _active_observations(resources, config, partition_key, submitted) @@ -1502,26 +1531,40 @@ def _assert_active_operation_consistency( async def _exercise_active_races( + resources: DeployedAcaLifecycleResources, client: ClientSession, config: DeployedAcaLifecycleConfig, + partition_key: str, headers: dict[str, str], sample: list[_SubmittedRun], ) -> tuple[int, int]: outcomes = await asyncio.gather( - *(_exercise_one_active_race(client, config, headers, item) for item in sample) + *( + _exercise_one_active_race( + resources, + client, + config, + partition_key, + headers, + item, + ) + for item in sample + ) ) return sum(replays for replays, _ in outcomes), sum(conflicts for _, conflicts in outcomes) async def _exercise_one_active_race( + resources: DeployedAcaLifecycleResources, client: ClientSession, config: DeployedAcaLifecycleConfig, + partition_key: str, headers: dict[str, str], submitted: _SubmittedRun, ) -> tuple[int, int]: accepted = submitted.accepted assert submitted.session_id_header == accepted.session_id - replay_status, replay_payload, _ = await json_request( + replay_status, replay_payload, replay_headers = await json_request( client, "POST", config.deployed.chat_url, @@ -1532,8 +1575,23 @@ async def _exercise_one_active_race( }, payload=submission_payload(_LOAD_PROMPT), ) - assert replay_status == 202 - replay = parse_accepted_run(replay_payload, config.deployed) + if replay_status == 202: + replay = parse_accepted_run(replay_payload, config.deployed) + else: + assert replay_status == 504 + assert replay_payload.get("error") == "setup_deadline_exceeded" + assert replay_payload.get("retry_with") == "respond-async" + assert response_header(replay_headers, "x-ms-retry-with") == "respond-async" + recovered = await _recover_submitted_run( + resources, + config, + partition_key, + submitted.idempotency_key, + submitted.submitted_at, + session_id_header=submitted.session_id_header, + ) + assert recovered is not None + replay = recovered.accepted assert replay.session_id == accepted.session_id assert replay.run_id == accepted.run_id conflict_status, conflict_payload, _ = await json_request( @@ -1557,20 +1615,31 @@ async def _read_events( submitted: _SubmittedRun, authorization: str, ) -> _EventEvidence: - status, events, _, first_event_at = await read_sse_events_with_first_event_time( - client, - submitted.accepted.management_urls["events_url"], - headers={"Authorization": authorization}, - overall_timeout_seconds=_HOLD_SECONDS + _EVENT_STREAM_GRACE_SECONDS, + status, events, _, first_event_at, observed_event_timestamps = ( + await read_sse_events_with_observation_times( + client, + submitted.accepted.management_urls["events_url"], + headers={"Authorization": authorization}, + overall_timeout_seconds=_HOLD_SECONDS + _EVENT_STREAM_GRACE_SECONDS, + ) ) terminal_at = time.perf_counter() assert status == 200 assert first_event_at is not None assert events + assert len(observed_event_timestamps) == len(events), ( + f"observed {len(observed_event_timestamps)} arrival timestamps for " + f"{len(events)} events; the visibility series would be computed from an " + "incomplete sample." + ) assert [event.sequence for event in events] == list(range(1, len(events) + 1)) assert events[-1].payload.get("type") == "done" _assert_public_hold_events(events) - return _EventEvidence(first_event_at=first_event_at, terminal_at=terminal_at) + return _EventEvidence( + first_event_at=first_event_at, + terminal_at=terminal_at, + observed_event_timestamps=observed_event_timestamps, + ) def _assert_public_hold_events(events: list[SseEvent]) -> None: @@ -1628,12 +1697,27 @@ async def _read_terminal_result( ) assert status_code == 200 assert status.get("state") == "succeeded" - result_code, result, _ = await json_request( - client, - "GET", - accepted.management_urls["result_url"], - headers={"Authorization": authorization}, - ) + deadline = time.monotonic() + _RESULT_MATERIALIZATION_TIMEOUT_SECONDS + while True: + result_code, result, response_headers = await json_request( + client, + "GET", + accepted.management_urls["result_url"], + headers={"Authorization": authorization}, + retry_throttled=False, + ) + if result_code == 200: + break + assert result_code == 503 + assert result.get("error") == "result_temporarily_unavailable" + retry_after = optional_retry_after_seconds( + response_headers, + maximum_seconds=_RESULT_RETRY_AFTER_MAXIMUM_SECONDS, + ) + assert retry_after is not None + if time.monotonic() + retry_after > deadline: + raise AssertionError("Terminal result did not materialize within the retry window.") + await asyncio.sleep(retry_after) assert result_code == 200 assert isinstance(result.get("result"), dict) return True diff --git a/tests/live/test_aca_deployed_loss.py b/tests/live/test_aca_deployed_loss.py index 4b154e6b..e48fd03b 100644 --- a/tests/live/test_aca_deployed_loss.py +++ b/tests/live/test_aca_deployed_loss.py @@ -496,7 +496,15 @@ async def _wait_for_loss_projection( expected_session_id=accepted.session_id, expected_run_id=accepted.run_id, ) and status_code == 200: - return _LossProjection(status_code, status, run, operations) + # Re-read after durable terminal evidence to avoid a stale projection. + status_code, status, _ = await json_request( + client, + "GET", + accepted.management_urls["status_url"], + headers={"Authorization": authorization}, + ) + if status_code == 200: + return _LossProjection(status_code, status, run, operations) if asyncio.get_running_loop().time() >= deadline: raise AssertionError( "The deployed controller did not terminalize the lost backing within the " diff --git a/tests/test_aca_deployed_agent_support.py b/tests/test_aca_deployed_agent_support.py new file mode 100644 index 00000000..a479f6c0 --- /dev/null +++ b/tests/test_aca_deployed_agent_support.py @@ -0,0 +1,585 @@ +"""Unit tests for SSE retry behavior in aca_deployed_agent_support.""" + +from __future__ import annotations + +import json +from collections.abc import AsyncIterator, Mapping + +import pytest + +from tests.aca_smoke_diagnostics import AcaSmokeEnvironmentError +from tests.live import aca_deployed_agent_support as support + + +class TestSseThrottleRetryDelay: + """Cover the pure decision function that gates SSE stream retries on 503/429.""" + + def test_throttled_status_with_valid_header_retries(self) -> None: + for status in (429, 503): + assert support.sse_throttle_retry_delay( + status, {"Retry-After": "2"}, is_final_attempt=False + ) == pytest.approx(2.0) + + def test_final_attempt_never_retries(self) -> None: + """Bounding retries preserves the reader's ability to fail.""" + assert ( + support.sse_throttle_retry_delay( + 503, {"Retry-After": "2"}, is_final_attempt=True + ) + is None + ) + + def test_missing_header_declines_to_retry(self) -> None: + assert support.sse_throttle_retry_delay(503, {}, is_final_attempt=False) is None + + def test_malformed_header_declines_to_retry(self) -> None: + assert ( + support.sse_throttle_retry_delay(503, {"Retry-After": "abc"}, is_final_attempt=False) + is None + ) + + def test_out_of_range_header_declines_to_retry(self) -> None: + """A server asking for an implausible wait is not honored blindly.""" + assert ( + support.sse_throttle_retry_delay( + 503, {"Retry-After": "600"}, is_final_attempt=False + ) + is None + ) + + def test_zero_retry_after_declines(self) -> None: + assert ( + support.sse_throttle_retry_delay( + 503, {"Retry-After": "0"}, is_final_attempt=False + ) + is None + ) + + def test_non_throttled_statuses_are_not_retried(self) -> None: + for status in (200, 400, 401, 403, 404, 500, 502, 504): + assert ( + support.sse_throttle_retry_delay( + status, {"Retry-After": "2"}, is_final_attempt=False + ) + is None + ), f"status {status} must not be treated as SSE throttling" + + def test_429_is_retried_same_as_503(self) -> None: + delay_429 = support.sse_throttle_retry_delay( + 429, {"Retry-After": "3"}, is_final_attempt=False + ) + delay_503 = support.sse_throttle_retry_delay( + 503, {"Retry-After": "3"}, is_final_attempt=False + ) + assert delay_429 == delay_503 == pytest.approx(3.0) + + @pytest.mark.parametrize("status", [502, 503]) + def test_frontend_unavailable_without_retry_after_uses_fixed_delay( + self, + status: int, + ) -> None: + assert support.sse_throttle_retry_delay( + status, + {"Content-Type": "text/html; charset=utf-8", "Connection": "close"}, + is_final_attempt=False, + ) == pytest.approx(support._FRONTEND_UNAVAILABLE_RETRY_SECONDS) + + def test_max_attempts_constant_is_positive_and_bounded(self) -> None: + """Guard: the attempt limit is positive and not accidentally huge.""" + assert 2 <= support._SSE_THROTTLE_MAX_ATTEMPTS <= 10 + + +class _FakeSseResponse: + """One aiohttp-shaped response: status, headers, and a byte-chunk body.""" + + def __init__(self, status: int, headers: dict[str, str], body: bytes) -> None: + self.status = status + self.headers = headers + self._body = body + + async def __aenter__(self) -> _FakeSseResponse: + return self + + async def __aexit__(self, *exc_info: object) -> None: + return None + + @property + def content(self) -> AsyncIterator[bytes]: + async def _chunks() -> AsyncIterator[bytes]: + yield self._body + + return _chunks() + + +class _FakeSseSession: + """Serve a scripted sequence of responses and record the requests made.""" + + def __init__(self, responses: list[_FakeSseResponse]) -> None: + self._responses = list(responses) + self.request_headers: list[Mapping[str, str]] = [] + + def get(self, url: str, headers: Mapping[str, str]) -> _FakeSseResponse: + del url + self.request_headers.append(dict(headers)) + return self._responses.pop(0) + + +def _sse_body(sequence: int) -> bytes: + return f"id: {sequence}\ndata: {{\"type\": \"token\"}}\n\n".encode() + + +class TestSseReaderRetryLoop: + """Cover the reader loop, not just the decision it consults. + + The decision function alone was covered, which is the same layer seam that + let earlier defects reach a live run: the loop could stop sleeping, stop + raising, or ignore the helper entirely and every test stayed green. + """ + + @pytest.fixture(autouse=True) + def _no_real_sleeping(self, monkeypatch: pytest.MonkeyPatch) -> list[float]: + slept: list[float] = [] + + async def _record(seconds: float) -> None: + slept.append(seconds) + + monkeypatch.setattr(support.asyncio, "sleep", _record) + self.slept = slept + return slept + + @pytest.mark.asyncio + async def test_a_throttled_response_is_retried_then_succeeds(self) -> None: + session = _FakeSseSession( + [ + _FakeSseResponse(503, {"Retry-After": "2"}, b""), + _FakeSseResponse(200, {}, _sse_body(1)), + ] + ) + status, events, _, _, _ = await support._read_sse_response( + session, # type: ignore[arg-type] + "https://example.invalid/events", + headers={"Authorization": "Bearer x"}, + ) + assert status == 200 + assert len(events) == 1 + assert self.slept == [2.0], "the server's delay must be honored exactly once" + + @pytest.mark.asyncio + async def test_persistent_throttling_still_raises(self) -> None: + """Bounding the retry is what preserves the suite's ability to fail.""" + responses = [ + _FakeSseResponse(503, {"Retry-After": "1"}, b"") + for _ in range(support._SSE_THROTTLE_MAX_ATTEMPTS) + ] + session = _FakeSseSession(responses) + with pytest.raises(support.SseResponseStatusError): + await support._read_sse_response( + session, # type: ignore[arg-type] + "https://example.invalid/events", + headers={"Authorization": "Bearer x"}, + ) + assert len(session.request_headers) == support._SSE_THROTTLE_MAX_ATTEMPTS + assert len(self.slept) == support._SSE_THROTTLE_MAX_ATTEMPTS - 1 + + @pytest.mark.asyncio + async def test_a_non_throttled_status_is_not_retried(self) -> None: + session = _FakeSseSession([_FakeSseResponse(500, {"Retry-After": "2"}, b"")]) + with pytest.raises(support.SseResponseStatusError): + await support._read_sse_response( + session, # type: ignore[arg-type] + "https://example.invalid/events", + headers={"Authorization": "Bearer x"}, + ) + assert len(session.request_headers) == 1 + assert self.slept == [] + + @pytest.mark.asyncio + async def test_a_throttle_without_a_header_is_not_retried(self) -> None: + session = _FakeSseSession([_FakeSseResponse(503, {}, b"")]) + with pytest.raises(support.SseResponseStatusError): + await support._read_sse_response( + session, # type: ignore[arg-type] + "https://example.invalid/events", + headers={"Authorization": "Bearer x"}, + ) + assert len(session.request_headers) == 1 + + @pytest.mark.asyncio + async def test_frontend_unavailable_without_header_retries_then_succeeds(self) -> None: + session = _FakeSseSession( + [ + _FakeSseResponse(503, {"Content-Type": "text/html"}, b"Site Unavailable"), + _FakeSseResponse(200, {}, _sse_body(1)), + ] + ) + + status, events, _, _, _ = await support._read_sse_response( + session, # type: ignore[arg-type] + "https://example.invalid/events", + headers={"Authorization": "******", "Last-Event-ID": "0"}, + ) + + assert status == 200 + assert [event.sequence for event in events] == [1] + assert self.slept == [support._FRONTEND_UNAVAILABLE_RETRY_SECONDS] + assert [headers["Last-Event-ID"] for headers in session.request_headers] == ["0", "0"] + + @pytest.mark.asyncio + async def test_persistent_frontend_unavailability_still_raises(self) -> None: + session = _FakeSseSession( + [ + _FakeSseResponse(503, {"Content-Type": "text/html"}, b"Site Unavailable") + for _ in range(support._SSE_THROTTLE_MAX_ATTEMPTS) + ] + ) + + with pytest.raises(support.SseResponseStatusError): + await support._read_sse_response( + session, # type: ignore[arg-type] + "https://example.invalid/events", + headers={"Authorization": "******"}, + ) + + assert len(session.request_headers) == support._SSE_THROTTLE_MAX_ATTEMPTS + assert self.slept == [ + support._FRONTEND_UNAVAILABLE_RETRY_SECONDS + ] * (support._SSE_THROTTLE_MAX_ATTEMPTS - 1) + + @pytest.mark.asyncio + async def test_resume_header_survives_a_throttled_attempt(self) -> None: + """A retry must not drop the caller's resume position.""" + session = _FakeSseSession( + [ + _FakeSseResponse(503, {"Retry-After": "1"}, b""), + _FakeSseResponse(200, {}, _sse_body(4)), + ] + ) + await support._read_sse_response( + session, # type: ignore[arg-type] + "https://example.invalid/events", + headers={"Authorization": "Bearer x", "Last-Event-ID": "3"}, + ) + assert [h.get("Last-Event-ID") for h in session.request_headers] == ["3", "3"] + + +class _FakeJsonResponse: + """One aiohttp-shaped response for json_request faking.""" + + def __init__(self, status: int, headers: dict[str, str], body: dict[str, object]) -> None: + self.status = status + self.headers = headers + self._body = body + + async def __aenter__(self) -> _FakeJsonResponse: + return self + + async def __aexit__(self, *exc_info: object) -> None: + return None + + async def json(self, content_type: object = None) -> dict[str, object]: + return self._body + + async def read(self) -> bytes: + import json as _json + + return _json.dumps(self._body).encode() + + +class _FakeFrontendResponse(_FakeJsonResponse): + """An App Service front-end response that never reached the JSON app.""" + + def __init__(self, status: int = 503) -> None: + super().__init__( + status, + {"Content-Type": "text/html; charset=utf-8", "Connection": "close"}, + {}, + ) + self.reads = 0 + + async def json(self, content_type: object = None) -> dict[str, object]: + del content_type + raise json.JSONDecodeError("not JSON", "Site Unavailable", 0) + + async def read(self) -> bytes: + self.reads += 1 + return b"Site Unavailable" + + +class _FakeJsonSession: + """Serve scripted JSON responses and record requests made.""" + + def __init__(self, responses: list[_FakeJsonResponse]) -> None: + self._responses = list(responses) + self.requests: list[tuple[str, str]] = [] + + def request( + self, + method: str, + url: str, + *, + headers: object = None, + json: object = None, + ) -> _FakeJsonResponse: + self.requests.append((method, url)) + return self._responses.pop(0) + + +class TestJsonRequestThrottleRetry: + """Cover the json_request retry loop for transient backpressure.""" + + @pytest.fixture(autouse=True) + def _no_real_sleeping(self, monkeypatch: pytest.MonkeyPatch) -> list[float]: + slept: list[float] = [] + + async def _record(seconds: float) -> None: + slept.append(seconds) + + monkeypatch.setattr(support.asyncio, "sleep", _record) + self.slept = slept + return slept + + @pytest.mark.asyncio + async def test_503_with_valid_retry_after_then_200_succeeds(self) -> None: + """A single transient 503 with Retry-After is retried and the success returned.""" + session = _FakeJsonSession( + [ + _FakeJsonResponse(503, {"Retry-After": "5"}, {}), + _FakeJsonResponse(200, {}, {"result": "ok"}), + ] + ) + status, body, _headers = await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + ) + assert status == 200 + assert body == {"result": "ok"} + assert self.slept == [5.0], "must sleep exactly the server-requested delay" + assert len(session.requests) == 2 + + @pytest.mark.asyncio + async def test_503_exhausts_all_attempts_returns_last_response(self) -> None: + """When all attempts are throttled, the last 503 is returned (not raised).""" + responses = [ + _FakeJsonResponse(503, {"Retry-After": "1"}, {"error": "busy"}) + for _ in range(support._JSON_THROTTLE_MAX_ATTEMPTS) + ] + session = _FakeJsonSession(responses) + status, body, _ = await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + ) + assert status == 503 + assert body == {"error": "busy"} + assert len(session.requests) == support._JSON_THROTTLE_MAX_ATTEMPTS + assert len(self.slept) == support._JSON_THROTTLE_MAX_ATTEMPTS - 1 + + @pytest.mark.asyncio + async def test_429_handled_same_as_503(self) -> None: + session = _FakeJsonSession( + [ + _FakeJsonResponse(429, {"Retry-After": "3"}, {}), + _FakeJsonResponse(200, {}, {"ok": True}), + ] + ) + status, _body, _ = await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + ) + assert status == 200 + assert self.slept == [3.0] + + @pytest.mark.asyncio + async def test_504_setup_deadline_exceeded_is_not_retried(self) -> None: + """504 with setup_deadline_exceeded is a meaningful assertion target.""" + session = _FakeJsonSession( + [ + _FakeJsonResponse( + 504, + {"Retry-After": "120"}, + {"error": "setup_deadline_exceeded"}, + ), + ] + ) + status, body, _ = await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + ) + assert status == 504 + assert body["error"] == "setup_deadline_exceeded" + assert len(session.requests) == 1 + assert self.slept == [] + + @pytest.mark.asyncio + async def test_500_is_not_retried(self) -> None: + session = _FakeJsonSession([_FakeJsonResponse(500, {"Retry-After": "2"}, {})]) + status, _, _ = await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + ) + assert status == 500 + assert len(session.requests) == 1 + assert self.slept == [] + + @pytest.mark.asyncio + async def test_404_is_not_retried(self) -> None: + session = _FakeJsonSession([_FakeJsonResponse(404, {}, {})]) + status, _, _ = await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + ) + assert status == 404 + assert len(session.requests) == 1 + + @pytest.mark.asyncio + async def test_409_is_not_retried(self) -> None: + session = _FakeJsonSession([_FakeJsonResponse(409, {}, {})]) + status, _, _ = await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + ) + assert status == 409 + assert len(session.requests) == 1 + + @pytest.mark.asyncio + async def test_missing_retry_after_on_503_is_not_retried(self) -> None: + """No Retry-After means no server backpressure signal — do not invent one.""" + session = _FakeJsonSession([_FakeJsonResponse(503, {}, {"error": "busy"})]) + status, _, _ = await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + ) + assert status == 503 + assert len(session.requests) == 1 + assert self.slept == [] + + @pytest.mark.asyncio + async def test_idempotent_post_retries_frontend_unavailable_then_succeeds(self) -> None: + unavailable = _FakeFrontendResponse() + session = _FakeJsonSession( + [ + unavailable, + _FakeJsonResponse(202, {}, {"run_id": "run-1"}), + ] + ) + + status, body, _ = await support.json_request( + session, # type: ignore[arg-type] + "POST", + "https://example.invalid/chat", + headers={"Idempotency-Key": "same-attempt"}, + ) + + assert status == 202 + assert body == {"run_id": "run-1"} + assert unavailable.reads == 1 + assert len(session.requests) == 2 + assert self.slept == [support._FRONTEND_UNAVAILABLE_RETRY_SECONDS] + + @pytest.mark.asyncio + async def test_unsafe_post_does_not_retry_frontend_unavailable(self) -> None: + unavailable = _FakeFrontendResponse() + session = _FakeJsonSession([unavailable]) + + with pytest.raises(AcaSmokeEnvironmentError, match="HTTP 503"): + await support.json_request( + session, # type: ignore[arg-type] + "POST", + "https://example.invalid/chat", + ) + + assert unavailable.reads == 1 + assert len(session.requests) == 1 + assert self.slept == [] + + @pytest.mark.asyncio + async def test_persistent_frontend_unavailability_still_fails(self) -> None: + responses = [ + _FakeFrontendResponse() + for _ in range(support._JSON_THROTTLE_MAX_ATTEMPTS) + ] + session = _FakeJsonSession(responses) + + with pytest.raises(AcaSmokeEnvironmentError, match="HTTP 503"): + await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + ) + + assert len(session.requests) == support._JSON_THROTTLE_MAX_ATTEMPTS + assert self.slept == [ + support._FRONTEND_UNAVAILABLE_RETRY_SECONDS + ] * (support._JSON_THROTTLE_MAX_ATTEMPTS - 1) + + @pytest.mark.asyncio + async def test_malformed_retry_after_on_503_is_not_retried(self) -> None: + session = _FakeJsonSession( + [_FakeJsonResponse(503, {"Retry-After": "abc"}, {})] + ) + status, _, _ = await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + ) + assert status == 503 + assert len(session.requests) == 1 + assert self.slept == [] + + @pytest.mark.asyncio + async def test_out_of_range_retry_after_on_503_is_not_retried(self) -> None: + session = _FakeJsonSession( + [_FakeJsonResponse(503, {"Retry-After": "999"}, {})] + ) + status, _, _ = await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + ) + assert status == 503 + assert len(session.requests) == 1 + assert self.slept == [] + + @pytest.mark.asyncio + async def test_retry_disabled_returns_503_immediately(self) -> None: + """Callers with their own retry logic can opt out.""" + session = _FakeJsonSession( + [_FakeJsonResponse(503, {"Retry-After": "5"}, {"error": "busy"})] + ) + status, _body, _ = await support.json_request( + session, # type: ignore[arg-type] + "GET", + "https://example.invalid/status", + retry_throttled=False, + ) + assert status == 503 + assert len(session.requests) == 1 + assert self.slept == [] + + def test_max_attempts_constant_is_bounded(self) -> None: + """Guard: attempt limit is positive and not accidentally huge.""" + assert 2 <= support._JSON_THROTTLE_MAX_ATTEMPTS <= 10 + + +def test_json_retry_budget_cannot_exhaust_a_caller_deadline() -> None: + """Bound the retry budget itself, not just the loop that spends it. + + The sleep is patched out in every other retry test, so nothing else notices + if the ceiling grows. Every 503 this service emits asks for two seconds, so + a ten second cap is generous; admitting the 120 second setup-timeout value + here would let one request sleep for minutes and exhaust the very budgets + these suites stopped asserting on. + """ + worst_case = ( + support._JSON_THROTTLE_MAX_ATTEMPTS - 1 + ) * support._JSON_THROTTLE_RETRY_AFTER_MAXIMUM_SECONDS + assert worst_case <= 30.0, ( + f"json_request can now sleep {worst_case}s across its retries." + ) diff --git a/tests/test_aca_deployed_agent_turn.py b/tests/test_aca_deployed_agent_turn.py index 7a553cd3..23042caf 100644 --- a/tests/test_aca_deployed_agent_turn.py +++ b/tests/test_aca_deployed_agent_turn.py @@ -40,7 +40,9 @@ from tests.live import aca_deployed_agent_support as support from tests.live import aca_deployed_lifecycle_support as lifecycle_support -_DEPLOYABLE_FIXTURE = Path(__file__).resolve().parent / "fixtures" / "live_aca_deployed_agent_turn" +_DEPLOYABLE_FIXTURE = ( + Path(__file__).resolve().parent / "live" / "apps" / "aca-qualification" +) _TIMEOUT_RECOVERY_FIXTURE = ( Path(__file__).resolve().parent / "fixtures" / "live_aca_setup_timeout_recovery" ) @@ -150,6 +152,87 @@ async def record_sleep(delay: float) -> None: assert sleeps == [120.0] +def _load_lifecycle_test_module( + monkeypatch: pytest.MonkeyPatch, + module_name: str, +) -> Any: + monkeypatch.setenv("AZURE_FUNCTIONS_AGENTS_RUN_DEPLOYED_ACA_SMOKE", "1") + module_path = Path(__file__).parent / "live" / "test_aca_deployed_lifecycle.py" + spec = importlib.util.spec_from_file_location(module_name, module_path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +@pytest.mark.asyncio +async def test_lifecycle_retention_uses_session_and_terminal_run_clocks( + monkeypatch: pytest.MonkeyPatch, +) -> None: + module = _load_lifecycle_test_module(monkeypatch, "_lifecycle_retention_regression") + accepted = SimpleNamespace(session_id="session-1", run_id="run-1") + terminal_updated_at = _NOW + session = SimpleNamespace( + last_activity_at=terminal_updated_at + timedelta(milliseconds=660), + expires_at=terminal_updated_at + timedelta(seconds=300), + ) + terminal_run = SimpleNamespace( + session_id="session-1", + run_id="run-1", + status="succeeded", + result_available=True, + updated_at=terminal_updated_at, + ) + + async def read_run(*_args: object, **_kwargs: object) -> object: + return terminal_run + + monkeypatch.setattr(module, "read_authoritative_run", read_run) + + await module._assert_terminal_retention( + object(), + session=session, + accepted=accepted, + ) + + +@pytest.mark.parametrize( + "terminal_run", + [ + SimpleNamespace( + session_id="different-session", + run_id="run-1", + status="succeeded", + result_available=True, + ), + SimpleNamespace( + session_id="session-1", + run_id="different-run", + status="succeeded", + result_available=True, + ), + SimpleNamespace( + session_id="session-1", + run_id="run-1", + status="failed", + result_available=False, + ), + ], +) +def test_lifecycle_retention_rejects_wrong_or_unsuccessful_run( + monkeypatch: pytest.MonkeyPatch, + terminal_run: object, +) -> None: + module = _load_lifecycle_test_module(monkeypatch, "_lifecycle_retention_fail_closed") + + with pytest.raises(AssertionError): + module._assert_matching_successful_run( + terminal_run, + SimpleNamespace(session_id="session-1", run_id="run-1"), + ) + + @pytest.mark.asyncio async def test_timeout_recovery_cancel_honors_retry_after_then_polls_terminal_status( monkeypatch: pytest.MonkeyPatch, @@ -529,6 +612,23 @@ def test_deployed_config_reads_only_safe_url_and_route_contract( } +def test_deployed_config_normalizes_a_pathless_origin_to_the_api_route_root( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _set_deployed_environment(monkeypatch) + monkeypatch.setenv( + "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_FUNCTION_BASE_URL", + "https://deployed-aca.azurewebsites.net/", + ) + + config = support.deployed_aca_smoke_config_from_environment() + + assert config.base_url == "https://deployed-aca.azurewebsites.net/api" + assert config.chat_url == ( + "https://deployed-aca.azurewebsites.net/api/agents/deployed_turn/chat" + ) + + def test_timeout_recovery_config_requires_the_controlled_fixture_route( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -866,12 +966,12 @@ def _lifecycle_session() -> DurableSessionRecord: ( "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_FUNCTION_BASE_URL", "https://user:password@deployed-aca.azurewebsites.net", - "without a path, credentials, query, or fragment", + "without credentials, query, or fragment", ), ( "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_FUNCTION_BASE_URL", "https://deployed-aca.azurewebsites.net?code=secret", - "without a path, credentials, query, or fragment", + "without credentials, query, or fragment", ), ( "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_TOKEN_SCOPE", diff --git a/tests/test_aca_deployed_cold_start.py b/tests/test_aca_deployed_cold_start.py index 064a2ea2..c0788c86 100644 --- a/tests/test_aca_deployed_cold_start.py +++ b/tests/test_aca_deployed_cold_start.py @@ -2,6 +2,7 @@ import importlib import sys +from pathlib import Path from types import SimpleNamespace import pytest @@ -43,9 +44,26 @@ def test_cold_start_report_is_aggregate_and_redacted() -> None: assert "samples=3" in report assert "p50=2000.0,p95=3000.0,max=3000.0" in report assert "cleanup=complete" in report + assert "provenance=verified" in report assert all(forbidden not in report for forbidden in ("session", "run_id", "prompt", "result")) +def test_unverified_provenance_suppresses_latency_metrics() -> None: + metrics = support.cold_start_metrics([1], [2], [3], [4]) + + report = support.render_cold_start_report( + sample_count=1, + retries=0, + metrics=metrics, + cleanup_complete=True, + provenance_verified=False, + ) + + assert "not-available" in report + assert "p50=" not in report + assert "provenance=unverified" in report + + @pytest.mark.parametrize( ("status", "elapsed", "typed_deadline", "expected"), [ @@ -154,6 +172,56 @@ async def no_sleep(_: float) -> None: assert len(candidates) == 1 +@pytest.mark.asyncio +async def test_provenance_is_checked_after_measured_turn_without_value_leakage( + monkeypatch: pytest.MonkeyPatch, + cold_start_module: object, +) -> None: + module = cold_start_module + monkeypatch.setenv("AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_BUILD_ID", "expected-build") + monkeypatch.setenv( + "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_COMMIT_SHA", "expected-commit" + ) + monkeypatch.setenv("AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_PYTHON_VERSION", "3.13") + + async def mismatched(*_: object) -> dict[str, object]: + return { + "build": { + "marker": "present", + "schema": 1, + "build_id": "wrong-build", + "commit_sha": "wrong-commit", + }, + "runtime": {"python_version": "3.14"}, + } + + monkeypatch.setattr(module, "fetch_build_info", mismatched) + with pytest.raises(AssertionError, match="cold_start_provenance_mismatch") as error: + await module._verify_deployed_build("https://example.invalid", "scope/.default") + + rendered = str(error.value) + assert "build_id,commit_sha,python_version" in rendered + assert all( + value not in rendered + for value in ("expected-build", "expected-commit", "wrong-build", "wrong-commit", "3.14") + ) + + +def test_cold_test_calls_attestation_only_after_timing_collection() -> None: + source = ( + Path(__file__).resolve().parent / "live" / "test_aca_deployed_cold_start.py" + ).read_text(encoding="utf-8") + test_body = source[ + source.index("async def test_deployed_aca_cold_start_acceptance_is_bounded_and_cleaned") + : source.index("\n\nasync def _verify_deployed_build") + ] + + assert test_body.index("await _run_samples_sequentially(") < test_body.index( + "await _verify_deployed_build(" + ) + assert "fetch_build_info(" not in test_body + + @pytest.mark.asyncio async def test_samples_are_orchestrated_strictly_sequentially( monkeypatch: pytest.MonkeyPatch, diff --git a/tests/test_aca_deployed_load.py b/tests/test_aca_deployed_load.py index 6eafb331..b21e07ac 100644 --- a/tests/test_aca_deployed_load.py +++ b/tests/test_aca_deployed_load.py @@ -32,13 +32,32 @@ def test_load_concurrency_rejects_invalid_explicit_values(value: str) -> None: def test_load_concurrency_uses_environment_and_omission_skips(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("AZURE_FUNCTIONS_AGENTS_ACA_LOAD_CONCURRENCY", "100") - assert support.load_concurrency_from_option_or_environment(_config(None)) == 100 + monkeypatch.setenv("AZURE_FUNCTIONS_AGENTS_ACA_LOAD_CONCURRENCY", "5") + assert support.load_concurrency_from_option_or_environment(_config(None)) == 5 monkeypatch.delenv("AZURE_FUNCTIONS_AGENTS_ACA_LOAD_CONCURRENCY") with pytest.raises(pytest.skip.Exception): support.require_load_concurrency(_config(None)) +@pytest.mark.parametrize( + ("option_value", "environment_value"), + [("100", None), (None, "100")], +) +def test_formal_n100_is_rejected_by_direct_live_entrypoints( + monkeypatch: pytest.MonkeyPatch, + option_value: str | None, + environment_value: str | None, +) -> None: + if environment_value is not None: + monkeypatch.setenv("AZURE_FUNCTIONS_AGENTS_ACA_LOAD_CONCURRENCY", environment_value) + + with pytest.raises( + AcaSmokeEnvironmentError, + match=support.FORMAL_N100_UNSUPPORTED_ERROR, + ): + support.load_concurrency_from_option_or_environment(_config(option_value)) + + def test_load_concurrency_rejects_invalid_environment(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setenv("AZURE_FUNCTIONS_AGENTS_ACA_LOAD_CONCURRENCY", "101") @@ -160,6 +179,107 @@ def load_module(monkeypatch: pytest.MonkeyPatch) -> object: return importlib.import_module("tests.live.test_aca_deployed_load") +def test_result_materialization_window_covers_the_result_hold( + load_module: object, +) -> None: + module = load_module + assert module._RESULT_MATERIALIZATION_TIMEOUT_SECONDS == module._HOLD_SECONDS # type: ignore[attr-defined] + + +@pytest.mark.asyncio +async def test_terminal_result_honors_temporary_unavailability_retry( + monkeypatch: pytest.MonkeyPatch, + load_module: object, +) -> None: + module = load_module + responses = iter( + [ + (200, {"state": "succeeded"}, {}), + ( + 503, + {"error": "result_temporarily_unavailable"}, + {"Retry-After": "2"}, + ), + (200, {"result": {"text": "done"}}, {}), + ] + ) + calls: list[bool] = [] + sleeps: list[float] = [] + + async def request(*_: object, **kwargs: object) -> tuple[int, dict[str, object], dict[str, str]]: + calls.append(bool(kwargs.get("retry_throttled", True))) + return next(responses) + + async def sleep(seconds: float) -> None: + sleeps.append(seconds) + + monkeypatch.setattr(module, "json_request", request) + monkeypatch.setattr(module.asyncio, "sleep", sleep) # type: ignore[attr-defined] + submitted = SimpleNamespace( + accepted=SimpleNamespace( + management_urls={ + "status_url": "https://example.test/status", + "result_url": "https://example.test/result", + } + ) + ) + + assert await module._read_terminal_result( # type: ignore[attr-defined] + object(), + submitted, + "Bearer redacted", + ) + assert calls == [True, False, False] + assert sleeps == [2.0] + + +@pytest.mark.asyncio +async def test_active_replay_recovers_durable_identity_after_setup_timeout( + monkeypatch: pytest.MonkeyPatch, + load_module: object, +) -> None: + module = load_module + accepted = SimpleNamespace(session_id="session-a", run_id="run-a") + submitted = module._SubmittedRun( # type: ignore[attr-defined] + accepted=accepted, + idempotency_key="original-key", + submitted_at=1.0, + accepted_at=2.0, + session_id_header="session-a", + ) + responses = iter( + [ + ( + 504, + { + "error": "setup_deadline_exceeded", + "retry_with": "respond-async", + }, + {"x-ms-retry-with": "respond-async"}, + ), + (409, {"error": "active_run_exists"}, {}), + ] + ) + + async def request(*_: object, **__: object) -> tuple[int, dict[str, object], dict[str, str]]: + return next(responses) + + async def recover(*_: object, **__: object) -> object: + return submitted + + monkeypatch.setattr(module, "json_request", request) + monkeypatch.setattr(module, "_recover_submitted_run", recover) + + assert await module._exercise_one_active_race( # type: ignore[attr-defined] + object(), + object(), + SimpleNamespace(deployed=SimpleNamespace(chat_url="https://example.test/chat")), + "partition", + {"Authorization": "redacted"}, + submitted, + ) == (1, 1) + + @pytest.mark.asyncio async def test_phase_a_batch_preserves_prepared_candidates_before_aggregate_failure( monkeypatch: pytest.MonkeyPatch, @@ -946,6 +1066,54 @@ async def no_sleep(delay: float) -> None: assert retry_delays == [120.0] +@pytest.mark.asyncio +async def test_final_total_attempt_recovers_after_throttles_then_setup_deadline( + monkeypatch: pytest.MonkeyPatch, + load_module: object, +) -> None: + module = load_module + deployed = SimpleNamespace(chat_url="https://example.test/chat") + responses = iter( + [ + *[(503, {}, {"Retry-After": "1"}) for _ in range(4)], + (504, {"error": "setup_deadline_exceeded"}, {}), + ] + ) + headers_seen: list[dict[str, str]] = [] + retry_delays: list[float] = [] + + async def request(*_: object, **kwargs: object) -> tuple[int, dict[str, object], dict[str, str]]: + headers_seen.append(dict(kwargs["headers"])) # type: ignore[arg-type,index] + return next(responses) + + async def no_record(*_: object, **__: object) -> None: + return None + + async def no_sleep(delay: float) -> None: + retry_delays.append(delay) + + monkeypatch.setattr(module, "json_request", request) + monkeypatch.setattr(module, "_recover_submitted_run", no_record) + monkeypatch.setattr(module.asyncio, "sleep", no_sleep) + + keys: list[str] = [] + outcome = await module._submit_one( # type: ignore[attr-defined] + object(), + SimpleNamespace(deployed=deployed), + {}, + object(), + "partition", + keys, + ) + + assert outcome.failure == "setup_deadline_exceeded" + assert outcome.retries == 4 + assert outcome.unresolved_idempotency + assert len(headers_seen) == 5 + assert {headers["Idempotency-Key"] for headers in headers_seen} == {keys[0]} + assert retry_delays == [1.0] * 4 + + @pytest.mark.asyncio async def test_provisioning_deadline_before_retry_preserves_attempt_metrics( monkeypatch: pytest.MonkeyPatch, @@ -1311,8 +1479,10 @@ async def request(*_: object, **kwargs: object) -> tuple[int, dict[str, str], di monkeypatch.setattr(module, "parse_accepted_run", lambda *_: accepted) assert await module._exercise_one_active_race( # type: ignore[attr-defined] + object(), object(), SimpleNamespace(deployed=SimpleNamespace(chat_url="https://example.test/chat")), + "partition", {"Authorization": "redacted"}, submitted, ) == (1, 1) @@ -1611,22 +1781,15 @@ def test_overlap_math_and_hold_budget_margin(load_module: object) -> None: assert module._overlapping_interval(first, second) is not None # type: ignore[attr-defined] assert module._overlapping_interval(second, first) is None # type: ignore[attr-defined] - module._assert_remaining_hold_budget( # type: ignore[attr-defined] - [SimpleNamespace(accepted_at=0.0), SimpleNamespace(accepted_at=100.0)] - ) - with pytest.raises(AssertionError, match="insufficient remaining"): - module._assert_remaining_hold_budget( # type: ignore[attr-defined] - [SimpleNamespace(accepted_at=0.0), SimpleNamespace(accepted_at=200.0)] - ) - def test_hold_constant_and_sse_continuation_guard(load_module: object) -> None: module = load_module fixture = runpy.run_path( str( Path(__file__).parent - / "fixtures" - / "live_aca_deployed_agent_turn" + / "live" + / "apps" + / "aca-qualification" / "tools" / "qualification_hold.py" ) diff --git a/tests/test_aca_deployed_load_support.py b/tests/test_aca_deployed_load_support.py new file mode 100644 index 00000000..146ddf02 --- /dev/null +++ b/tests/test_aca_deployed_load_support.py @@ -0,0 +1,183 @@ +from __future__ import annotations + +import pytest + +from tests.live import aca_deployed_load_support as support + + +def test_realistic_burst_timestamps_are_grouped_into_one_batch() -> None: + """Regression: events in one poll arrive microseconds apart, not identically. + + Grouping on exact float equality passed every synthetic test while producing + all-size-1 batches on real data -- which silently empties the waiting-only + series that answers the streaming-visibility question, and collapses the + measured cadence to the cost of parsing two adjacent events. Both numbers + still look plausible, which is what makes the failure dangerous. + """ + timestamps: list[float] = [] + moment = 100.0 + for _ in range(4): + timestamps.extend(moment + offset * 0.0004 for offset in range(3)) + moment += 1.0 + + assert support.events_per_batch(timestamps) == (3, 3, 3, 3) + + primary = support.visibility_gap_seconds(timestamps, waiting_only=True) + assert len(primary) == 3 + assert all(abs(gap - 1.0) < 0.01 for gap in primary) + + cadence = support.observed_poll_cadence_seconds(timestamps) + assert all(abs(spacing - 1.0) < 0.01 for spacing in cadence) + + +def test_slow_trickle_is_not_merged_into_one_batch() -> None: + """The window must separate distinct polls, not swallow a steady trickle.""" + assert support.events_per_batch([0.0, 0.2, 0.4, 0.6]) == (1, 1, 1, 1) + + +def test_subwindow_trickle_cannot_chain_into_an_unbounded_batch() -> None: + """Adjacent events inside the window must not extend a burst transitively.""" + assert support.events_per_batch([0.0, 0.03, 0.06, 0.09]) == (2, 2) + + +def test_observed_event_visibility_requires_at_least_two_events() -> None: + assert support.observed_event_batches([]) == () + assert support.visibility_gap_seconds([1.0], waiting_only=True) == () + assert support.visibility_gap_seconds([1.0], waiting_only=False) == () + assert support.observed_poll_cadence_seconds([1.0]) == () + assert support.events_per_batch([1.0]) == ((1,)) + + +def test_waiting_visibility_gap_counts_only_gaps_ending_in_multi_event_batches() -> None: + timestamps = [10.0, 11.0, 11.0, 12.5] + + assert support.visibility_gap_seconds(timestamps, waiting_only=True) == (1.0,) + assert support.visibility_gap_seconds(timestamps, waiting_only=False) == (1.0, 0.0, 1.5) + assert support.observed_poll_cadence_seconds(timestamps) == (1.0, 1.5) + assert support.events_per_batch(timestamps) == (1, 2, 1) + + +def test_single_batch_has_no_primary_gap_or_cadence() -> None: + timestamps = [4.0, 4.0, 4.0] + + assert support.visibility_gap_seconds(timestamps, waiting_only=True) == () + assert support.visibility_gap_seconds(timestamps, waiting_only=False) == (0.0, 0.0) + assert support.observed_poll_cadence_seconds(timestamps) == () + assert support.events_per_batch(timestamps) == (3,) + + +def test_observed_event_helpers_sort_timestamps_before_grouping() -> None: + timestamps = [5.0, 1.0, 3.0, 3.0] + + assert support.observed_event_batches(timestamps) == ( + support.ObservedEventBatch(observed_at=1.0, event_count=1), + support.ObservedEventBatch(observed_at=3.0, event_count=2), + support.ObservedEventBatch(observed_at=5.0, event_count=1), + ) + assert support.visibility_gap_seconds(timestamps, waiting_only=True) == (2.0,) + assert support.observed_poll_cadence_seconds(timestamps) == (2.0, 2.0) + + +def test_latency_metrics_reports_visibility_attribution_and_warning() -> None: + metrics = support.latency_metrics( + [0.1], + [0.2], + [4.0], + [ + [1.0, 2.0, 2.0, 4.5, 4.5], + [10.0, 11.0, 11.0], + ], + ) + report = support.render_load_report( + concurrency=5, + prepared_count=5, + provision_concurrency=4, + provisioning_duration_seconds=None, + provisioning_attempt_count=5, + provisioning_retry_count=0, + suspended_prepared_count=0, + common_interval=None, + admitted_count=5, + succeeded_count=5, + metrics=metrics, + replay_count=0, + active_run_conflict_count=0, + retry_count=0, + unclassified_service_throttle_count=0, + unresolved_idempotency_count=0, + cleanup_complete=True, + ) + + assert metrics.visibility_gap_ms == (1000.0, 2500.0, 2500.0) + assert metrics.visibility_gap_all_ms == (0.0, 2500.0, 2500.0) + assert metrics.observed_poll_cadence_ms == (1000.0, 2500.0, 2500.0) + assert metrics.events_per_batch == ((1, 2), (2, 3)) + assert "visibility_warning=p95_exceeds_2s" in report + assert "visibility_attribution=poll_timing_dominates" in report + assert "does not capture true sandbox-write-to-client-observe delta" in report + assert "clock-skew correction would add error comparable to the 2s budget" in report + + +def test_throttle_retry_returns_seconds_for_valid_retry_after() -> None: + """A 503/429 with Retry-After: 2 yields 2.0 seconds.""" + assert support.throttle_retry_after_seconds({"Retry-After": "2"}) == 2.0 + + +def test_throttle_retry_returns_none_when_header_missing() -> None: + """No Retry-After header means no throttle delay.""" + assert support.throttle_retry_after_seconds({}) is None + + +@pytest.mark.parametrize( + "value", + ["0", "-1", "11", "abc", "1.5"], + ids=["zero", "negative", "exceeds_max", "non_numeric", "float"], +) +def test_throttle_retry_returns_none_for_out_of_bounds(value: str) -> None: + """Out-of-range or non-integer Retry-After is ignored.""" + assert support.throttle_retry_after_seconds({"Retry-After": value}) is None + + +def test_throttle_retry_is_case_insensitive() -> None: + """Header lookup is case-insensitive (delegated to response_header).""" + assert support.throttle_retry_after_seconds({"retry-after": "3"}) == 3.0 + + +class TestThrottledAdmissionRetryDelay: + """Cover the branch that decides whether a throttled admission is retried. + + Mutating that branch to ignore Retry-After previously left every test green, + so the client behavior honoring backpressure was unprotected. + """ + + def test_throttled_status_with_a_valid_header_retries(self) -> None: + for status in (429, 503): + assert support.throttled_admission_retry_delay( + status, {"retry-after": "2"}, is_final_attempt=False + ) == pytest.approx(2.0) + + def test_final_attempt_never_retries(self) -> None: + """Bounding the retry is what preserves the run's ability to fail.""" + assert ( + support.throttled_admission_retry_delay(503, {"retry-after": "2"}, is_final_attempt=True) + is None + ) + + def test_a_missing_header_declines_to_retry(self) -> None: + assert support.throttled_admission_retry_delay(503, {}, is_final_attempt=False) is None + + def test_an_out_of_range_header_declines_to_retry(self) -> None: + """A server asking for an implausible wait is not honored blindly.""" + assert ( + support.throttled_admission_retry_delay(503, {"retry-after": "600"}, is_final_attempt=False) + is None + ) + + def test_non_throttled_statuses_are_not_retried(self) -> None: + for status in (202, 400, 401, 403, 404, 500, 504): + assert ( + support.throttled_admission_retry_delay( + status, {"retry-after": "2"}, is_final_attempt=False + ) + is None + ), f"status {status} must not be treated as throttling" diff --git a/tests/test_aca_qualification_pipeline.py b/tests/test_aca_qualification_pipeline.py new file mode 100644 index 00000000..322dec92 --- /dev/null +++ b/tests/test_aca_qualification_pipeline.py @@ -0,0 +1,1043 @@ +"""Tests for the deployed ACA qualification tooling. + +``eng/`` is outside the CI lint and type-check paths (`ruff check src tests`, +`mypy src`), so these tests are the only automated coverage the qualification +tooling gets. They deliberately exercise the pure decision functions -- marker +construction, marker comparison, and package assembly -- rather than the network +commands, because those are where a wrong answer would silently attribute +qualification evidence to the wrong deployment. +""" + +from __future__ import annotations + +import ast +import inspect +import json +import re +import subprocess +import tomllib +import zipfile +from argparse import Namespace +from pathlib import Path + +import pytest +from azure.identity import aio as azure_identity_aio +from eng.scripts import aca_deployed_qualification, aca_qualification_pipeline +from eng.scripts.aca_qualification_pipeline import ( + _DEPLOY_CONFIGURATION_TIMEOUT_SECONDS, + _DEPLOY_TIMEOUT_SECONDS, + _NOT_READY_STATUSES, + QualificationPipelineError, + _redacted_reason, + _run_az, + _write_deployment_archive, + assemble_upload_directory, + build_marker, + compare_marker, + content_report, + deploy_preflight_failure_message, + fetch_build_info, + render_requirements, + run_deploy, + select_runtime_wheel, + stamp_marker, +) +from packaging.markers import default_environment +from packaging.requirements import Requirement +from packaging.utils import canonicalize_name + +_BUILD_ID = "12345" +_COMMIT = "2415708287d1ce719e8380208d0ba52a8df9c080" + + +def _marker_payload(**overrides: object) -> dict[str, object]: + build: dict[str, object] = { + "marker": "present", + "schema": 1, + "build_id": _BUILD_ID, + "commit_sha": _COMMIT, + "branch": "refs/heads/main", + "runtime_version": "1.2.3", + } + build.update(overrides) + return { + "build": build, + "runtime": {"python_version": "3.13", "python_micro": 2}, + "content": {"entry_count": 5968, "total_bytes": 79_600_000, "truncated": False}, + } + + +class TestBuildMarker: + def test_marker_carries_build_identity(self) -> None: + marker = build_marker( + commit_sha=_COMMIT, + build_id=_BUILD_ID, + branch="refs/heads/main", + runtime_version="1.2.3", + ) + assert marker["schema"] == 1 + assert marker["build_id"] == _BUILD_ID + assert marker["commit_sha"] == _COMMIT + + @pytest.mark.parametrize("field", ["commit_sha", "build_id", "branch", "runtime_version"]) + def test_empty_field_is_rejected(self, field: str) -> None: + kwargs = { + "commit_sha": _COMMIT, + "build_id": _BUILD_ID, + "branch": "refs/heads/main", + "runtime_version": "1.2.3", + } + kwargs[field] = " " + with pytest.raises(QualificationPipelineError, match=f"marker_field_empty:{field}"): + build_marker(**kwargs) # type: ignore[arg-type] + + def test_stamp_writes_readable_json(self, tmp_path: Path) -> None: + marker = build_marker( + commit_sha=_COMMIT, + build_id=_BUILD_ID, + branch="refs/heads/main", + runtime_version="1.2.3", + ) + target = stamp_marker(tmp_path, marker) + assert json.loads(target.read_text(encoding="utf-8")) == marker + + def test_stamp_rejects_missing_app_root(self, tmp_path: Path) -> None: + with pytest.raises(QualificationPipelineError, match="app_root_missing"): + stamp_marker(tmp_path / "absent", {"schema": 1}) + + +@pytest.mark.parametrize("status", [500, 501, 505, 599]) +def test_build_attestation_retries_every_server_error(status: int) -> None: + assert status in _NOT_READY_STATUSES + + +@pytest.mark.parametrize("status", [400, 401, 403, 404, 600]) +def test_build_attestation_does_not_retry_definitive_responses(status: int) -> None: + assert status not in _NOT_READY_STATUSES + + +class TestCompareMarker: + def test_matching_build_passes(self) -> None: + result = compare_marker( + _marker_payload(), + expected_build_id=_BUILD_ID, + expected_commit_sha=_COMMIT, + expected_python="3.13", + ) + assert result.matches + + def test_stale_build_is_detected(self) -> None: + result = compare_marker( + _marker_payload(build_id="99999"), + expected_build_id=_BUILD_ID, + expected_commit_sha=_COMMIT, + expected_python="3.13", + ) + assert not result.matches + assert "build_id" in result.mismatches + + def test_wrong_python_leg_is_detected(self) -> None: + """The 3.14 package deployed onto the 3.13 app must not pass.""" + result = compare_marker( + _marker_payload(), + expected_build_id=_BUILD_ID, + expected_commit_sha=_COMMIT, + expected_python="3.14", + ) + assert "python_version" in result.mismatches + + def test_absent_marker_never_passes(self) -> None: + """An app with no marker must fail, not be treated as unverified-but-fine.""" + payload = _marker_payload() + payload["build"] = {"marker": "absent"} + result = compare_marker( + payload, + expected_build_id=_BUILD_ID, + expected_commit_sha=_COMMIT, + expected_python="3.13", + ) + assert not result.matches + assert result.mismatches == ("marker_absent",) + + def test_missing_build_section_never_passes(self) -> None: + result = compare_marker( + {"runtime": {"python_version": "3.13"}}, + expected_build_id=_BUILD_ID, + expected_commit_sha=_COMMIT, + expected_python="3.13", + ) + assert result.mismatches == ("build_section_missing",) + + def test_mismatch_output_carries_no_values(self) -> None: + """Field names only: this lands in a pipeline log.""" + result = compare_marker( + _marker_payload(build_id="99999", commit_sha="deadbeef"), + expected_build_id=_BUILD_ID, + expected_commit_sha=_COMMIT, + expected_python="3.13", + ) + rendered = ",".join(result.mismatches) + assert "99999" not in rendered + assert "deadbeef" not in rendered + + +@pytest.mark.asyncio +async def test_build_info_retries_total_timeout_within_readiness_deadline( + monkeypatch: pytest.MonkeyPatch, +) -> None: + payload = _marker_payload() + attempts = 0 + + class FakeCredential: + async def get_token(self, _: str) -> object: + return type("Token", (), {"token": "credential-material"})() + + async def close(self) -> None: + return None + + class FakeResponse: + status = 200 + + async def json(self, *, content_type: object) -> dict[str, object]: + del content_type + return payload + + class FakeRequest: + async def __aenter__(self) -> FakeResponse: + nonlocal attempts + attempts += 1 + if attempts == 1: + raise TimeoutError + return FakeResponse() + + async def __aexit__(self, *_: object) -> None: + return None + + class FakeSession: + def __init__(self, *, timeout: object) -> None: + del timeout + + async def __aenter__(self) -> FakeSession: + return self + + async def __aexit__(self, *_: object) -> None: + return None + + def get(self, *_: object, **__: object) -> FakeRequest: + return FakeRequest() + + async def no_sleep(_: float) -> None: + return None + + monkeypatch.setattr(azure_identity_aio, "DefaultAzureCredential", FakeCredential) + monkeypatch.setattr("aiohttp.ClientSession", FakeSession) + monkeypatch.setattr(aca_qualification_pipeline.asyncio, "sleep", no_sleep) + + assert await fetch_build_info("https://example.test", "api://scope") == payload + assert attempts == 2 + + +class TestContentReport: + def test_reports_usage_against_caps(self) -> None: + rendered = content_report(_marker_payload()) + assert "content_entries=5968" in rendered + assert "75.9MiB" in rendered + + def test_missing_content_is_explicit(self) -> None: + assert content_report({}) == "content=unavailable" + + +class TestQualificationEnvironmentContract: + """Validate what the qualification tooling authors against the enforced bounds. + + The fixture and the deploy command must agree with the live-test support + module, which validates its environment at fixture time. Nothing else + compares the two, so a drift here would surface only as an + ``AcaSmokeEnvironmentError`` during a paid deployed run. + """ + + def test_fixture_authors_the_required_region(self) -> None: + root = Path(__file__).resolve().parents[1] + source = ( + root + / "tests" + / "live" + / "apps" + / "aca-qualification" + / "agents.config.yaml" + ).read_text(encoding="utf-8") + assert "region: $AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_REGION" in source + + def test_deploy_configures_the_region_app_setting(self) -> None: + root = Path(__file__).resolve().parents[1] + script = (root / "eng" / "scripts" / "aca_qualification_pipeline.py").read_text( + encoding="utf-8" + ) + assert "AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_REGION={region}" in script + + def test_the_deployed_timeout_bound_is_the_platform_ceiling(self) -> None: + """Fail if the support module's bound moves away from the platform limit. + + 230 is not arbitrary: it is the Azure Functions platform HTTP request + ceiling, so any larger authored timeout is unreachable regardless of + what the client asks for. + """ + source = ( + Path(__file__).resolve().parents[1] + / "tests" + / "live" + / "aca_deployed_agent_support.py" + ).read_text(encoding="utf-8") + assert "1 <= timeout <= 230" in source, ( + "The enforced timeout bound changed; re-check the deployed timeout " + "any qualification run passes." + ) + + +class TestCombinedDeployedSuite: + def test_n100_is_rejected_before_environment_or_auth_preflight( + self, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + auth_called = False + + def unexpected_auth(_: object) -> None: + nonlocal auth_called + auth_called = True + + monkeypatch.setattr(aca_deployed_qualification, "preflight_auth", unexpected_auth) + + with pytest.raises( + aca_deployed_qualification.QualificationError, + match=aca_deployed_qualification.FORMAL_N100_UNSUPPORTED_ERROR, + ): + aca_deployed_qualification.run_deployed_suite( + {}, + runtime_target="python313", + load_concurrency="100", + provision_concurrency="4", + ) + + assert not auth_called + + def test_n5_remains_an_accepted_diagnostic(self) -> None: + environment = { + name: "configured" + for name in aca_deployed_qualification._DEPLOYED_ENVIRONMENT + } + + assert aca_deployed_qualification.validate_deployed_environment( + environment, + runtime_target="python313", + load_concurrency="5", + provision_concurrency="1", + ) == (5, 1) + + def test_n100_command_failure_is_stable_and_redacted( + self, + capsys: pytest.CaptureFixture[str], + ) -> None: + result = aca_deployed_qualification.main( + [ + "deployed-suite", + "--runtime-target", + "python313", + "--load-concurrency", + "100", + "--provision-concurrency", + "4", + ], + environment={}, + ) + + assert result == 1 + assert capsys.readouterr().err.strip() == ( + "ACA qualification failed: " + f"{aca_deployed_qualification.FORMAL_N100_UNSUPPORTED_ERROR}" + ) + + def test_cold_start_gates_the_remaining_modules( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + captured: list[tuple[str, ...]] = [] + monkeypatch.setattr( + aca_deployed_qualification, + "validate_deployed_environment", + lambda *args, **kwargs: (5, 1), + ) + monkeypatch.setattr(aca_deployed_qualification, "preflight_auth", lambda _: None) + monkeypatch.setattr( + aca_deployed_qualification, + "_run_pytest", + lambda paths, _: captured.append(tuple(paths)) or 0, + ) + + result = aca_deployed_qualification.run_deployed_suite( + {}, + runtime_target="python313", + load_concurrency="5", + provision_concurrency="1", + ) + + assert result == 0 + assert captured == [ + ("tests/live/test_aca_deployed_cold_start.py",), + ( + "tests/live/test_aca_deployed_agent_turn.py", + "tests/live/test_aca_deployed_lifecycle.py", + "tests/live/test_aca_deployed_loss.py", + "tests/live/test_aca_deployed_load.py", + ), + ] + + def test_cold_start_failure_suppresses_later_suites( + self, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + captured: list[tuple[str, ...]] = [] + monkeypatch.setattr( + aca_deployed_qualification, + "validate_deployed_environment", + lambda *args, **kwargs: (5, 1), + ) + monkeypatch.setattr(aca_deployed_qualification, "preflight_auth", lambda _: None) + + def fail_cold_start( + paths: tuple[str, ...], + _: object, + ) -> int: + captured.append(paths) + return 1 + + monkeypatch.setattr(aca_deployed_qualification, "_run_pytest", fail_cold_start) + + assert ( + aca_deployed_qualification.run_deployed_suite( + {}, + runtime_target="python313", + load_concurrency="5", + provision_concurrency="1", + ) + == 1 + ) + assert captured == [("tests/live/test_aca_deployed_cold_start.py",)] + + def test_the_expected_identity_environment_is_required(self) -> None: + """Every provenance input must be required before a deployed run starts.""" + for name in ( + "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_BUILD_ID", + "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_COMMIT_SHA", + "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_PYTHON_VERSION", + ): + assert name in aca_deployed_qualification._DEPLOYED_ENVIRONMENT + + +class TestDeploymentCommand: + def test_azure_cli_failure_diagnostics_are_omitted( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + def failed(*args: object, **kwargs: object) -> subprocess.CompletedProcess[str]: + del args, kwargs + return subprocess.CompletedProcess( + args=["az"], + returncode=1, + stdout="", + stderr="request failed with Bearer secret-token-value", + ) + + monkeypatch.setattr(subprocess, "run", failed) + + with pytest.raises(QualificationPipelineError) as caught: + _run_az(["functionapp", "show"]) + + assert "secret-token-value" not in str(caught.value) + assert str(caught.value) == "az_failed:functionapp:show:exit_1" + + def test_azure_cli_timeout_is_a_typed_redacted_failure( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + def timed_out(*args: object, **kwargs: object) -> None: + del args, kwargs + raise subprocess.TimeoutExpired("az", 30) + + monkeypatch.setattr(subprocess, "run", timed_out) + + with pytest.raises( + QualificationPipelineError, + match=r"az_timeout:functionapp:show:30s", + ): + _run_az(["functionapp", "show"]) + + def test_archive_contains_staged_files_relative_to_its_root(self, tmp_path: Path) -> None: + staging = tmp_path / "staging" + (staging / "nested").mkdir(parents=True) + (staging / "host.json").write_text("{}", encoding="utf-8") + (staging / "nested" / "file.txt").write_text("content", encoding="utf-8") + archive = tmp_path / "fixture.zip" + + _write_deployment_archive(staging, archive) + + with zipfile.ZipFile(archive) as package: + assert package.namelist() == ["host.json", "nested/file.txt"] + + def test_deploy_combines_preflight_region_upload_and_best_effort_tag( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + staging = tmp_path / "staging" + staging.mkdir() + (staging / "host.json").write_text("{}", encoding="utf-8") + archive = tmp_path / "fixture.zip" + commands: list[tuple[list[str], float]] = [] + + def fake_run_az(args: list[str], *, timeout_seconds: float = 30.0) -> None: + commands.append((args, timeout_seconds)) + + monkeypatch.setattr( + "eng.scripts.aca_qualification_pipeline._run_az", + fake_run_az, + ) + + result = run_deploy( + Namespace( + staging_root=str(staging), + archive_path=str(archive), + app_name="qualification-app", + resource_group="qualification-rg", + region="westus3", + build_id="12345", + commit_sha=_COMMIT, + ) + ) + + assert result == 0 + assert [command[:3] for command, _ in commands] == [ + ["functionapp", "show", "--name"], + ["functionapp", "config", "appsettings"], + ["functionapp", "deployment", "source"], + ["tag", "update", "--operation"], + ] + assert [timeout for _, timeout in commands] == [ + 30.0, + _DEPLOY_CONFIGURATION_TIMEOUT_SECONDS, + _DEPLOY_TIMEOUT_SECONDS, + 30.0, + ] + assert archive.is_file() + + def test_deploy_continues_when_redacted_metadata_tag_fails( + self, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + ) -> None: + staging = tmp_path / "staging" + staging.mkdir() + (staging / "host.json").write_text("{}", encoding="utf-8") + + def fake_run_az(args: list[str], *, timeout_seconds: float = 30.0) -> None: + del timeout_seconds + if args[:2] == ["tag", "update"]: + raise QualificationPipelineError( + "tag_failed: Bearer credential-material" + ) + + monkeypatch.setattr( + "eng.scripts.aca_qualification_pipeline._run_az", + fake_run_az, + ) + + result = run_deploy( + Namespace( + staging_root=str(staging), + archive_path=str(tmp_path / "fixture.zip"), + app_name="qualification-app", + resource_group="qualification-rg", + region="westus3", + build_id="12345", + commit_sha=_COMMIT, + ) + ) + + output = capsys.readouterr().out + assert result == 0 + assert "warning: Function App metadata tag failed" in output + assert "credential-material" not in output + assert "" in output + + +class TestFixtureRouteBinding: + """Guard the fixture's HTTP types against the worker's binding validation. + + The route originally annotated ``azure.functions.HttpRequest``. The runtime + registers every route with the FastAPI ``Request``/response types instead, + and the worker rejects the mismatch at indexing time. Indexing is + all-or-nothing, so that single bad function took down the whole deployed + app -- every agent route included -- surfacing only as the generic + "No job functions found". + + The fixture cannot be imported here (it requires a Linux host, and fails + closed on Windows by design), so this reads the source instead. + """ + + def _build_info_signature(self) -> tuple[str, str]: + source = ( + Path(__file__).resolve().parents[1] + / "tests" + / "live" + / "apps" + / "aca-qualification" + / "function_app.py" + ) + if not source.is_file(): + source = ( + Path(__file__).resolve().parent + / "live" + / "apps" + / "aca-qualification" + / "function_app.py" + ) + tree = ast.parse(source.read_text(encoding="utf-8")) + for node in ast.walk(tree): + if isinstance(node, ast.FunctionDef) and node.name == "build_info": + annotation = node.args.args[0].annotation + returns = node.returns + return (ast.unparse(annotation), ast.unparse(returns)) + raise AssertionError("build_info not found in the fixture app.") + + def test_uses_the_fastapi_request_type(self) -> None: + parameter, _ = self._build_info_signature() + assert parameter == "Request", ( + "The worker rejects azure.functions.HttpRequest here and refuses to " + "index the entire app." + ) + + def test_returns_a_fastapi_response(self) -> None: + _, returns = self._build_info_signature() + assert "Response" in returns + assert "HttpResponse" not in returns + + def test_registers_the_expected_binding_shape(self) -> None: + """The corrected annotation must produce httpTrigger + http $return.""" + import azure.functions as func + from azurefunctions.extensions.http.fastapi import JSONResponse, Request + + probe = func.FunctionApp() + + @probe.route(route="__buildinfo", methods=["GET"], auth_level=func.AuthLevel.ANONYMOUS) + def build_info(req: Request) -> JSONResponse: # pragma: no cover - shape probe + return JSONResponse(content={}, status_code=200) + + bindings = { + (binding.type, binding.name) + for function in probe.get_functions() + for binding in function.get_bindings() + } + assert ("httpTrigger", "req") in bindings + assert ("http", "$return") in bindings + + +class TestDeployedAcaQualificationFixtureContract: + """Keep the deployed fixture aligned with the live suites that invoke it.""" + + _LIVE_SUITE_NAMES = ( + "test_aca_deployed_agent_turn.py", + "test_aca_deployed_cold_start.py", + "test_aca_deployed_lifecycle.py", + "test_aca_deployed_load.py", + "test_aca_deployed_loss.py", + ) + _CALL_EXACTLY_ONCE = re.compile(r"\bCall\s+([a-z][a-z0-9_]*)\s+exactly once\b") + + def _repo_root(self) -> Path: + return Path(__file__).resolve().parents[1] + + def _fixture_root(self) -> Path: + return self._repo_root() / "tests" / "live" / "apps" / "aca-qualification" + + def _live_suite_paths(self) -> tuple[Path, ...]: + live_root = self._repo_root() / "tests" / "live" + return tuple(live_root / name for name in self._LIVE_SUITE_NAMES) + + def _suite_agent_slugs(self) -> dict[str, set[str]]: + slugs: dict[str, set[str]] = {} + for path in self._live_suite_paths(): + tree = ast.parse(path.read_text(encoding="utf-8")) + for node in ast.walk(tree): + if not isinstance(node, ast.Assign): + continue + if not isinstance(node.value, ast.Constant) or not isinstance(node.value.value, str): + continue + names = [target.id for target in node.targets if isinstance(target, ast.Name)] + if any(name.endswith("_AGENT_SLUG") for name in names): + slugs.setdefault(path.name, set()).add(node.value.value) + return slugs + + def _suite_tool_names(self) -> dict[str, set[str]]: + tools: dict[str, set[str]] = {} + for path in self._live_suite_paths(): + source = path.read_text(encoding="utf-8") + names = set(self._CALL_EXACTLY_ONCE.findall(source)) + tree = ast.parse(source) + for node in ast.walk(tree): + if not isinstance(node, ast.Compare): + continue + compared = [node.left, *node.comparators] + constants = { + item.value + for item in compared + if isinstance(item, ast.Constant) and isinstance(item.value, str) + } + if not constants: + continue + calls = [item for item in compared if isinstance(item, ast.Call)] + if any(self._is_tool_name_get(call) for call in calls): + names.update(constants) + if names: + tools[path.name] = names + return tools + + @staticmethod + def _is_tool_name_get(call: ast.Call) -> bool: + return ( + isinstance(call.func, ast.Attribute) + and call.func.attr == "get" + and bool(call.args) + and isinstance(call.args[0], ast.Constant) + and call.args[0].value == "tool_name" + ) + + def _fixture_agent_slugs(self) -> set[str]: + return { + path.name.removesuffix(".agent.md") + for path in self._fixture_root().glob("*.agent.md") + if path.is_file() + } + + def _fixture_tool_names(self) -> set[str]: + tool_names: set[str] = set() + tools_root = self._fixture_root() / "tools" + for path in tools_root.glob("*.py"): + tree = ast.parse(path.read_text(encoding="utf-8")) + for node in ast.walk(tree): + if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef) and not node.name.startswith( + "_" + ): + tool_names.add(node.name) + return tool_names + + def test_live_suite_agent_slugs_exist_in_the_deployed_fixture(self) -> None: + required = { + slug + for slugs in self._suite_agent_slugs().values() + for slug in slugs + if slug.startswith("deployed_") + } + assert required, "No deployed agent slugs were derived from the live suites." + + missing = required - self._fixture_agent_slugs() + assert not missing, f"Missing deployed fixture agent(s): {sorted(missing)}" + + def test_live_suite_tool_names_exist_in_the_deployed_fixture(self) -> None: + required = { + tool + for tools in self._suite_tool_names().values() + for tool in tools + if tool.startswith("qualification_") + } + assert required, "No qualification tool names were derived from the live suites." + + missing = required - self._fixture_tool_names() + assert not missing, f"Missing deployed fixture tool(s): {sorted(missing)}" + + +class TestRedactedReason: + """A reason must be diagnosable without carrying credential material.""" + + def test_keeps_the_operational_cause(self) -> None: + reason = _redacted_reason(OSError("Cannot connect to host aca.example: [Errno -2]")) + assert "Cannot connect to host" in reason + + def test_strips_bearer_tokens_and_sas_signatures(self) -> None: + reason = _redacted_reason( + RuntimeError("failed https://x/y?sv=2024-01-01&sig=AbCdEf123 Bearer eyJhbGciOiJIUzI1NiJ9abcdefghijk") + ) + assert "AbCdEf123" not in reason + assert "eyJhbGciOiJIUzI1NiJ9" not in reason + assert "" in reason + + def test_falls_back_to_type_when_empty(self) -> None: + assert _redacted_reason(ValueError()) == "ValueError" + + def test_truncates_runaway_messages(self) -> None: + assert len(_redacted_reason(RuntimeError("x" * 5000))) < 500 + + +class TestAdapterRegionContract: + """Bind the qualification assets to the real adapter signature. + + The deployed suites and the qualification fixture both depend on the + authored Sandbox Group region being required, keyword-only, and free of ARM + discovery. Copying the real signature here makes a future drift fail during + the normal gate instead of in a paid deployed run. + """ + + def test_real_adapter_requires_an_authored_region(self) -> None: + from azure_functions_agents.transport.aca_sdk import AcaSandboxAdapter + + parameters = inspect.signature(AcaSandboxAdapter.open).parameters + region = parameters["region"] + assert region.kind is inspect.Parameter.KEYWORD_ONLY + assert region.default is inspect.Parameter.empty + + +class TestPackageAssemblyHelpers: + def test_select_runtime_wheel_returns_the_only_wheel_name(self) -> None: + selected = select_runtime_wheel(["azure_functions_agents-1.0.0-py3-none-any.whl"]) + assert selected == "azure_functions_agents-1.0.0-py3-none-any.whl" + + def test_select_runtime_wheel_rejects_no_candidates(self) -> None: + with pytest.raises(QualificationPipelineError, match="runtime_wheel_missing"): + select_runtime_wheel([]) + + def test_select_runtime_wheel_rejects_ambiguous_candidates(self) -> None: + with pytest.raises(QualificationPipelineError, match="runtime_wheel_ambiguous"): + select_runtime_wheel(["a.whl", "b.whl"]) + + def test_render_requirements_installs_actual_wheel_and_pinned_deps(self, tmp_path: Path) -> None: + requirements_export = tmp_path / "aca-fixture-requirements.txt" + requirements_export.write_text( + "azure-functions==1.23.0\npydantic==2.11.0\n", + encoding="utf-8", + ) + rendered = render_requirements( + wheel_filename="azure_functions_agents-1.2.3-py3-none-any.whl", + requirements_export_path=requirements_export, + ) + assert rendered.splitlines() == [ + "./azure_functions_agents-1.2.3-py3-none-any.whl", + "", + "azure-functions==1.23.0", + "pydantic==2.11.0", + ] + + def test_render_requirements_uses_template_placeholder_once(self, tmp_path: Path) -> None: + requirements_export = tmp_path / "aca-fixture-requirements.txt" + requirements_export.write_text("azure-functions==1.24.0\n", encoding="utf-8") + template = tmp_path / "requirements.txt" + template.write_text("{{RUNTIME_WHEEL}}\n", encoding="utf-8") + rendered = render_requirements( + wheel_filename="runtime-1.0.0-py3-none-any.whl", + requirements_export_path=requirements_export, + template_path=template, + ) + assert rendered.splitlines().count("./runtime-1.0.0-py3-none-any.whl") == 1 + assert "azure-functions==1.24.0" in rendered + + def test_render_requirements_replaces_generated_template_placeholder(self, tmp_path: Path) -> None: + requirements_export = tmp_path / "aca-fixture-requirements.txt" + requirements_export.write_text("azure-functions==1.23.0\n", encoding="utf-8") + template = tmp_path / "requirements.txt" + template.write_text( + "# generated template\n" + "./azurefunctions_agents_runtime-BUILD_WHEEL_PLACEHOLDER-py3-none-any.whl\n" + "# pins follow\n" + "stale-pin==0.0.1\n", + encoding="utf-8", + ) + rendered = render_requirements( + wheel_filename="azurefunctions_agents_runtime-1.0.0-py3-none-any.whl", + requirements_export_path=requirements_export, + template_path=template, + ) + assert "BUILD_WHEEL_PLACEHOLDER" not in rendered + assert "./azurefunctions_agents_runtime-1.0.0-py3-none-any.whl" in rendered + assert "stale-pin==0.0.1" not in rendered + assert "azure-functions==1.23.0" in rendered + + def test_render_requirements_rejects_missing_export(self, tmp_path: Path) -> None: + with pytest.raises(QualificationPipelineError, match="requirements_export_missing"): + render_requirements( + wheel_filename="runtime-1.0.0-py3-none-any.whl", + requirements_export_path=tmp_path / "missing.txt", + ) + + def test_assemble_upload_materializes_fixture_wheel_marker_and_requirements( + self, + tmp_path: Path, + ) -> None: + artifact_root = tmp_path / "artifact" + dist_root = artifact_root / "dist" + dist_root.mkdir(parents=True) + wheel = dist_root / "azurefunctions_agents_runtime-1.0.0-py3-none-any.whl" + wheel.write_bytes(b"wheel") + fixture_root = tmp_path / "fixture" + fixture_root.mkdir() + (fixture_root / "host.json").write_text("{}", encoding="utf-8") + (fixture_root / "requirements.txt").write_text( + "{{RUNTIME_WHEEL}}\n", + encoding="utf-8", + ) + ignored = fixture_root / "__pycache__" + ignored.mkdir() + (ignored / "fixture.pyc").write_bytes(b"ignored") + requirements_export = tmp_path / "aca-fixture-requirements.txt" + requirements_export.write_text("azure-functions==1.24.0\n", encoding="utf-8") + staging_root = tmp_path / "staging" + + result = assemble_upload_directory( + Namespace( + artifact_root=str(artifact_root), + fixture_root=str(fixture_root), + staging_root=str(staging_root), + requirements_template=None, + requirements_export=str(requirements_export), + commit_sha=_COMMIT, + build_id=_BUILD_ID, + branch="refs/heads/main", + runtime_version="1.0.0", + ) + ) + + assert result == staging_root + assert (staging_root / wheel.name).read_bytes() == b"wheel" + assert json.loads( + (staging_root / "BUILD_INFO.json").read_text(encoding="utf-8") + ) == build_marker( + commit_sha=_COMMIT, + build_id=_BUILD_ID, + branch="refs/heads/main", + runtime_version="1.0.0", + ) + assert (staging_root / "requirements.txt").read_text( + encoding="utf-8" + ).splitlines() == [ + f"./{wheel.name}", + "", + "azure-functions==1.24.0", + ] + assert not (staging_root / "__pycache__").exists() + + +class TestDeployPreflightHelpers: + def test_preflight_failure_names_actionable_role(self) -> None: + rendered = deploy_preflight_failure_message( + app_name="aca-app", + resource_group="rg", + check_name="publishing_config_read", + ) + assert "functionapp_deploy_preflight_failed:publishing_config_read" in rendered + assert "grant Website Contributor on aca-app" in rendered + + +class TestFixtureRequirementsExport: + """Keep the Oryx export synchronized with the selected uv.lock closure.""" + + def _repo_root(self) -> Path: + return Path(__file__).resolve().parents[1] + + def _export_path(self) -> Path: + return self._repo_root() / "eng" / "constraints" / "aca-fixture-requirements.txt" + + def _requirements(self) -> tuple[Requirement, ...]: + lines = self._export_path().read_text(encoding="utf-8").splitlines() + assert all( + line and not line.startswith(("-", ".", "git+")) and " @ " not in line + for line in lines + ) + requirements = tuple(Requirement(line) for line in lines) + assert all(requirement.url is None for requirement in requirements) + return requirements + + def _locked_closure(self) -> dict[str, str]: + lock = tomllib.loads((self._repo_root() / "uv.lock").read_text(encoding="utf-8")) + packages = { + canonicalize_name(package["name"]): package + for package in lock["package"] + } + project_name = canonicalize_name("azurefunctions-agents-runtime") + project = packages[project_name] + pending = [ + *project["dependencies"], + *project["optional-dependencies"]["aca-sandbox"], + *project["optional-dependencies"]["monitor"], + ] + closure: dict[str, str] = {} + processed_extras: dict[str, set[str]] = {} + while pending: + dependency = pending.pop() + name = canonicalize_name(dependency["name"]) + package = packages[name] + requested_extras = set(dependency.get("extra", ())) + new_extras = requested_extras - processed_extras.get(name, set()) + if name not in closure: + closure[name] = package["version"] + pending.extend(package.get("dependencies", ())) + for extra in new_extras: + pending.extend(package.get("optional-dependencies", {}).get(extra, ())) + processed_extras.setdefault(name, set()).update(requested_extras) + return closure + + def test_export_exactly_matches_the_selected_uv_lock_closure(self) -> None: + actual: dict[str, str] = {} + for requirement in self._requirements(): + specifiers = list(requirement.specifier) + assert len(specifiers) == 1 + specifier = specifiers[0] + assert specifier.operator == "==" + actual[canonicalize_name(requirement.name)] = specifier.version + + assert actual == self._locked_closure() + assert "azurefunctions-agents-runtime" not in actual + assert not { + "azure-storage-queue", + "jmespath", + "mypy", + "pytest", + "pytest-asyncio", + "pytest-cov", + "ruff", + "types-jsonschema", + } & actual.keys() + + def test_export_has_the_same_active_linux_closure_for_both_python_minors(self) -> None: + requirements = self._requirements() + + def active_names(python_version: str) -> set[str]: + environment = default_environment() + environment.update( + python_version=python_version, + python_full_version=f"{python_version}.0", + sys_platform="linux", + platform_system="Linux", + implementation_name="cpython", + platform_python_implementation="CPython", + ) + return { + canonicalize_name(requirement.name) + for requirement in requirements + if requirement.marker is None or requirement.marker.evaluate(environment) + } + + python313 = active_names("3.13") + python314 = active_names("3.14") + assert python313 == python314 + assert { + "azure-containerapps-sandbox", + "azure-data-tables", + "azure-monitor-opentelemetry", + } <= python313 + assert {"colorama", "pywin32"}.isdisjoint(python313) + + def test_assembled_requirements_include_the_monitor_distribution(self) -> None: + root = self._repo_root() + rendered = render_requirements( + wheel_filename="azurefunctions_agents_runtime-test.whl", + requirements_export_path=( + root / "eng" / "constraints" / "aca-fixture-requirements.txt" + ), + template_path=( + root / "tests" / "live" / "apps" / "aca-qualification" / "requirements.txt" + ), + ) + assert "./azurefunctions_agents_runtime-test.whl" in rendered + assert "azure-monitor-opentelemetry==1.8.8" in rendered diff --git a/tests/test_aca_smoke_support.py b/tests/test_aca_smoke_support.py index 2984f18d..773965cb 100644 --- a/tests/test_aca_smoke_support.py +++ b/tests/test_aca_smoke_support.py @@ -4,6 +4,7 @@ import hashlib import io import zipfile +from pathlib import Path from types import SimpleNamespace import pytest @@ -13,6 +14,7 @@ FUNCS_ZIP_DIGEST_KIND, CapturedContentPackage, ) +from azure_functions_agents.controller.sandbox_config import build_sandbox_environment from azure_functions_agents.harness.delegation import rebuild_agent_catalog from tests.live import aca_smoke_support @@ -268,6 +270,22 @@ def test_real_turn_fixture_rebuilds_a_no_tools_catalog( assert entry.capabilities.web_request_tools == [] +def test_deployed_fixture_rebuilds_with_the_forwarded_group_region( + monkeypatch: pytest.MonkeyPatch, +) -> None: + source_environment = { + "AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_REGION": "westus2", + "AZURE_OPENAI_DEPLOYMENT": "u3-gpt-5-6-luna-20260709", + } + for name, value in build_sandbox_environment(source_environment).items(): + monkeypatch.setenv(name, value) + fixture_root = Path(__file__).parent / "live" / "apps" / "aca-qualification" + + catalog = rebuild_agent_catalog(fixture_root) + + assert set(catalog) == {"deployed_load", "deployed_turn"} + + @pytest.mark.asyncio async def test_cleanup_requires_confirmation_when_a_create_may_have_started( monkeypatch: pytest.MonkeyPatch,