diff --git a/docs/frds/0008-aca-sandbox-session-runtime.md b/docs/frds/0008-aca-sandbox-session-runtime.md index 2864ec19..10480c8a 100644 --- a/docs/frds/0008-aca-sandbox-session-runtime.md +++ b/docs/frds/0008-aca-sandbox-session-runtime.md @@ -439,6 +439,7 @@ controlling amendments. | 191 | Result retention and reclaim authority | Complete run scan / monotonic session bound | `session.expires_at` never decreases and upper-bounds every result hold. Bounded scans may evict observed expired results but never block due reclaim; persisted off-page operation backing is exact-read before absence handling. | Human | 2026-09-02 | Replacement stack layer 2 | | 192 | Deployed ACA qualification fixture and coverage | Reuse E2E / dedicated N=5 fixture / formal N=100 fixture | Add a dedicated fixture and live suites for fresh acceptance, authenticated turn, lifecycle, backing loss, and N=5 admission/idempotency/events/results/cleanup. Keep its 120-second reclaim policy and reject N=100 preflight; Decision #29 requires a future purpose-built workflow. | Human | 2026-09-02 | Replacement stack layer 3 | | 193 | Lightweight in-package build provenance | No provenance / marker file / content-addressed chain | Ship a `BUILD_INFO.json` inside the deployed package and check build ID, commit SHA, and live Python minor after cold-start timing; a mismatch fails and suppresses metrics. Explicitly excludes wheel digest, installed package version, deploy-input manifest, deployment-storage chain, and rollback. | Human | 2026-09-02 | Replacement stack layer 3 | +| 194 | Qualification CI policy | Main CI only / trusted manual runs / PR or Schedule | Add one nonblocking post-Build stage using basic pipeline variables and parallel Python 3.13/3.14 jobs (provisioning 1 each, aggregate 2); restricted queue permissions authorize branch-controlled manual deployment, and the dedicated group retains headroom. | Human | 2026-09-02 | Replacement stack layer 4 | *Terminology note.* "Signed package" / "signed content package" phrasing in earlier decision rows (e.g. #17, #43), and the historical @@ -1814,13 +1815,31 @@ tooling described in §14 are now committed and runnable by hand. Pipeline wiring for them, external attestation, and rollback remain owned by issue #166; there is intentionally no rollback machinery. -## 14. Deployed ACA qualification assets — issue #166 - -**Status: Finalized for the committed assets.** Decisions #192 and #193 are the -qualification contract implemented here. This section describes committed test, -fixture, and tooling assets only. **No pipeline wiring exists in this -repository for them**; scheduling, gating, promotion criteria, and group sweep -remain open under issue #166. +## 14. Deployed ACA qualification — issue #166 + +**Status: Finalized for the committed assets and qualification stage.** +Decisions #192–#194 are the qualification contract implemented here. Group-wide +sweep and post-run cleanup remain outside this layer. + +The deployed qualification lives in `eng/ci/e2e-tests.yml`. One +`AcaQualification` stage depends only on `Build` and expands independent Python +3.13 and Python 3.14 jobs with `maxParallel: 2`. Each job assembles and deploys +its own fixture, then invokes one ordered suite with provisioning concurrency 1; +aggregate provisioning concurrency is therefore 2. The dedicated Sandbox Group +must retain quota and headroom for both jobs plus retained sessions. + +Automatic execution is limited to `IndividualCI` and `BatchedCI` builds of +`refs/heads/main`; manual runs are allowed from any branch. Pull request and +scheduled builds are excluded. Qualification remains `continueOnError` and +non-required until a separate promotion decision. + +ACA settings are ordinary/basic variables configured directly on Azure DevOps +pipeline 1777, not variable-group dependencies. Existing `- template:` entries +under `variables:` import unrelated build-infrastructure variable templates. +A manual run executes branch-controlled code under the deployment service +connection, so queue permission is restricted to trusted operators. Protected +branch or environment checks are intentionally not used because they would +prevent approved feature-branch validation. `tests/live/apps/aca-qualification/` is the deployable qualification fixture: an agent app that selects the ACA Sandbox backend, authors its Sandbox Group region, diff --git a/eng/ci/docs/aca-qualification.md b/eng/ci/docs/aca-qualification.md new file mode 100644 index 00000000..6d1de1c9 --- /dev/null +++ b/eng/ci/docs/aca-qualification.md @@ -0,0 +1,143 @@ +# ACA qualification CI guide + +Operational guide for the deployed ACA qualification stage in +[`eng/ci/e2e-tests.yml`](../e2e-tests.yml). + +## Execution conditions + +`AcaQualification` runs automatically only for `IndividualCI` and `BatchedCI` +builds of `refs/heads/main`. Trusted operators may also queue it manually from +any branch. Pull request and scheduled builds are excluded. The matrix job uses +`continueOnError: true` while qualification remains non-required; promotion to +a blocking gate requires a separate decision. + +## Current pipeline and job graph + +`AcaQualification` depends only on `Build`. Its matrix expands two independent +jobs that deploy and qualify Python 3.13 and Python 3.14 in parallel. + +| Stage | Depends on | Job | Current work | +| --- | --- | --- | --- | +| `AcaQualification` | `Build` | `AcaQualify_python313` | Deploy; cold start; public turn; lifecycle; backing loss; N=5 | +| `AcaQualification` | `Build` | `AcaQualify_python314` | Deploy; cold start; public turn; lifecycle; backing loss; N=5 | + +The cold-start module is first. Only after its acceptance, first-event, and +terminal timing completes does it compare the embedded marker with the expected +build ID, commit SHA, and Python runtime. A mismatch fails and suppresses the +cold-start metrics. The deployed-suite runner uses pytest fail-fast behavior and +does not start the turn, lifecycle, loss, or load modules after a cold/provenance +failure. This remains lightweight in-package provenance: it does not attest the +exact wheel digest, installed package version, deploy-input manifest, or +deployment-storage version. + +Each job passes provisioning concurrency 1; `maxParallel: 2` makes aggregate +provisioning concurrency 2. Do not serialize the matrix. The dedicated Sandbox +Group must have quota and operational headroom for both jobs plus sessions +retained by prior runs. Qualification does not rely on an unverifiable quota +API, and group-wide sweeping and post-run cleanup are outside this layer. + +## Required basic pipeline variables + +Configure these ordinary/basic variables directly on Azure DevOps pipeline +1777 without committing their values: + +- `ACA_DEPLOYED_APP_SUBSCRIPTION_ID` +- `ACA_DEPLOYED_RESOURCE_GROUP` +- `ACA_DEPLOYED_APP_SITE_NAME_PY313` +- `ACA_DEPLOYED_APP_SITE_NAME_PY314` +- `ACA_DEPLOYED_FUNCTION_BASE_URL_PY313` +- `ACA_DEPLOYED_FUNCTION_BASE_URL_PY314` +- `ACA_DEPLOYED_AGENT_SLUG` +- `ACA_DEPLOYED_EASY_AUTH_TOKEN_SCOPE` +- `ACA_DEPLOYED_EASY_AUTH_AUDIENCE` +- `ACA_DEPLOYED_TABLE_SERVICE_URI` +- `ACA_DEPLOYED_TABLE_NAME` +- `ACA_SANDBOX_GROUP_RESOURCE_ID` +- `ACA_SANDBOX_REGION` + +Do not place these values in or add a dependency on an Azure DevOps variable +group. The existing `- template:` entries under `variables:` import shared YAML +variable templates for build infrastructure; they are unrelated to variable +groups and do not provide the ACA settings. + +The service connection selected by the `acaServiceConnection` parameter must be +authorized for pipeline 1777. + +## Manual-run trust boundary + +A manual run executes YAML and scripts from the queued branch under the +deployment service connection. This is intentional so trusted operators can +validate a feature branch before merge. Restrict pipeline queue permission to +those operators. Do not substitute protected-branch or protected-environment +checks: either would defeat the approved feature-branch validation workflow. + +## Fixture-app prerequisites + +Each standing qualification Function App must have these app settings before +deployment: + +- ACA runtime: `AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_GROUP_RESOURCE_ID`. The + deployment job writes `AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_REGION`. +- Azure OpenAI: `AZURE_FUNCTIONS_AGENTS_PROVIDER=azure_openai`, + `AZURE_OPENAI_ENDPOINT`, and `AZURE_OPENAI_DEPLOYMENT`. + `AZURE_OPENAI_API_VERSION` is optional; when omitted, the Agent Framework + default is used. +- Auth allowlists: + `AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_ENTRA_TENANT_ID`, + `AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_AUDIENCE`, and + `AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_TEST_INVOKER_CLIENT_ID`. +- Session storage: either an `AzureWebJobsStorage` connection string or + identity-based `AzureWebJobsStorage__tableServiceUri`. For a user-assigned + storage identity, also set `AzureWebJobsStorage__clientId`; otherwise the + runtime follows its documented `AZURE_CLIENT_ID` or default-credential + resolution. + +The runtime table name is fixed as `AzureFunctionsAgentsSessions`; it is not an +additional app setting. Pipeline variable `ACA_DEPLOYED_TABLE_SERVICE_URI` must +identify the same Table service as the app's `AzureWebJobsStorage`, and +`ACA_DEPLOYED_TABLE_NAME` must be `AzureFunctionsAgentsSessions`. + +Platform Easy Auth must be enabled for the standing app. Its configured allowed +token audience must match `ACA_DEPLOYED_EASY_AUTH_AUDIENCE`, whose value is +passed into the fixture as +`AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_AUDIENCE`. App Service injects +`WEBSITE_AUTH_ENABLED` when Easy Auth is enforced; only environments where that +platform signal is unavailable need the explicit +`AZURE_FUNCTIONS_AGENTS_ENTRA_EASY_AUTH=true` assertion. + +Each `ACA_DEPLOYED_FUNCTION_BASE_URL_*` value must include the app's HTTP route +prefix. With the fixture's default `host.json`, the URL ends in `/api`. + +## Preview and manual execution + +Preview-compile a pushed branch without running jobs: + +```bash +az rest --method post \ + --resource 499b84ac-1321-427f-aa17-267ca6975798 \ + --headers "Content-Type=application/json" \ + --body '{"previewRun":true,"resources":{"repositories":{"self":{"refName":"refs/heads/"}}}}' \ + --uri "https://dev.azure.com///_apis/pipelines/1777/runs?api-version=7.1-preview.1" +``` + +Run pipeline 1777 manually from a pushed branch: + +```bash +az pipelines run \ + --id 1777 \ + --branch \ + --org https://dev.azure.com/ \ + --project +``` + +## Failure triage + +| Symptom | Check | +| --- | --- | +| Connection denied, DNS failure, or timeout to the regional ACA endpoint | Confirm the agent can resolve and reach the authored regional endpoint on port 443. | +| ACA data-plane `401` or `403` | Check the service-connection identity and Sandbox Group data-plane role. | +| Stage waits at an authorization checkpoint | Authorize the selected service connection for pipeline 1777. | +| Cold start reports unavailable or mismatched provenance | Check for a concurrent deployment or runtime-target wiring error. | +| Deployment fails during remote build | Inspect deployment logs and the generated `requirements.txt`. | +| Suite raises `ACA-SMOKE-ENV` | Correct basic pipeline variables, target configuration, identity, or capacity. | +| Suite assertion fails | Treat the environment as ready and investigate runtime behavior. | diff --git a/eng/ci/e2e-tests.yml b/eng/ci/e2e-tests.yml index 6ff0328c..2fa8a233 100644 --- a/eng/ci/e2e-tests.yml +++ b/eng/ci/e2e-tests.yml @@ -60,3 +60,14 @@ extends: artifactFeed: 'internal/PythonSDK_Internal_PublicPackages' PoolName: 1es-pool-azfunc acaServiceConnection: ${{ parameters.acaServiceConnection }} + + - stage: AcaQualification + displayName: 'ACA deployed qualification' + dependsOn: Build + condition: and(succeeded(), or(eq(variables['Build.Reason'], 'Manual'), and(in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI'), eq(variables['Build.SourceBranch'], 'refs/heads/main')))) + jobs: + - template: /eng/templates/official/jobs/aca-qualify.yml@self + parameters: + acaServiceConnection: ${{ parameters.acaServiceConnection }} + artifactFeed: 'internal/PythonSDK_Internal_PublicPackages' + poolName: 1es-pool-azfunc diff --git a/eng/scripts/README.md b/eng/scripts/README.md index 8f1b8670..15c6066b 100644 --- a/eng/scripts/README.md +++ b/eng/scripts/README.md @@ -66,14 +66,16 @@ inputs. Guest model-only, no-state/no-group RBAC is an IaC/operations prerequisite; the real model turn is positive access proof, not a negative role-assignment attestation. -The retained `aca_deployed_qualification.py` and deployed suite helpers are -manual/local assets only pending the separate post-main qualification work. +`aca_deployed_qualification.py` and the deployed suite helpers run in the +post-main `AcaQualification` stage of `eng/ci/e2e-tests.yml`; they also remain +available for trusted manual/local diagnostics. ### `aca_qualification_pipeline.py` Packages, deploys, and verifies the deployed ACA qualification fixture -(`tests/live/apps/aca-qualification/`). Every command is run by hand; this -repository contains no pipeline wiring for it. +(`tests/live/apps/aca-qualification/`). The post-main `AcaQualification` matrix +uses `install-tooling`, `assemble`, and `deploy` before running the combined +deployed suite. Operators can invoke the same commands by hand. | Command | Purpose | | --- | --- | diff --git a/eng/scripts/aca_deployed_qualification.py b/eng/scripts/aca_deployed_qualification.py index 742e346d..06b60def 100644 --- a/eng/scripts/aca_deployed_qualification.py +++ b/eng/scripts/aca_deployed_qualification.py @@ -129,7 +129,12 @@ def preflight_auth(environment: Mapping[str, str]) -> None: raise QualificationError("auth_preflight_failed") from None print("Azure service connection authenticated") -def _run_pytest(paths: Sequence[str], environment: Mapping[str, str]) -> int: +def _run_pytest( + paths: Sequence[str], + environment: Mapping[str, str], + *, + fail_fast: bool = False, +) -> int: result = subprocess.run( [ sys.executable, @@ -137,6 +142,7 @@ def _run_pytest(paths: Sequence[str], environment: Mapping[str, str]) -> int: "pytest", "-m", "live_aca", + *(("-x",) if fail_fast else ()), *paths, "-v", "-o", @@ -175,6 +181,7 @@ def run_deployed_suite( cold_start_result = _run_pytest( ("tests/live/test_aca_deployed_cold_start.py",), inherited, + fail_fast=True, ) if cold_start_result != 0: return cold_start_result @@ -186,6 +193,7 @@ def run_deployed_suite( "tests/live/test_aca_deployed_load.py", ), inherited, + fail_fast=True, ) diff --git a/eng/templates/official/jobs/aca-qualify.yml b/eng/templates/official/jobs/aca-qualify.yml new file mode 100644 index 00000000..7da0d5ef --- /dev/null +++ b/eng/templates/official/jobs/aca-qualify.yml @@ -0,0 +1,117 @@ +parameters: + - name: acaServiceConnection + type: string + - name: artifactFeed + type: string + - name: poolName + type: string + +jobs: + - job: AcaQualify + displayName: 'ACA deploy + qualification (Python $(pythonVersion))' + strategy: + maxParallel: 2 + matrix: + python313: + runtimeTarget: 'python313' + pythonVersion: '3.13' + appSiteName: $(ACA_DEPLOYED_APP_SITE_NAME_PY313) + baseUrl: $(ACA_DEPLOYED_FUNCTION_BASE_URL_PY313) + python314: + runtimeTarget: 'python314' + pythonVersion: '3.14' + appSiteName: $(ACA_DEPLOYED_APP_SITE_NAME_PY314) + baseUrl: $(ACA_DEPLOYED_FUNCTION_BASE_URL_PY314) + continueOnError: true + timeoutInMinutes: 135 + + pool: + name: ${{ parameters.poolName }} + image: 1es-ubuntu-22.04 + os: linux + + steps: + - checkout: self + + - download: current + artifact: azurefunctions-agents-runtime + displayName: 'Download the build artifact' + + - task: UsePythonVersion@0 + inputs: + versionSpec: '3.13' + displayName: 'Use Python for pipeline tooling' + + - task: PipAuthenticate@1 + displayName: 'Pip Authenticate' + inputs: + artifactFeeds: ${{ parameters.artifactFeed }} + + - bash: | + set -euo pipefail + python eng/scripts/aca_qualification_pipeline.py install-tooling + displayName: 'Install pipeline tooling' + + - bash: | + set -euo pipefail + RUNTIME_VERSION="$(python -c 'import azure_functions_agents as a; print(a.__version__)')" + echo "Runtime version: ${RUNTIME_VERSION}" + python eng/scripts/aca_qualification_pipeline.py assemble \ + --artifact-root "$(Pipeline.Workspace)/azurefunctions-agents-runtime" \ + --staging-root "$(Build.ArtifactStagingDirectory)/aca-fixture" \ + --requirements-export eng/constraints/aca-fixture-requirements.txt \ + --commit-sha "$(Build.SourceVersion)" \ + --build-id "$(Build.BuildId)" \ + --branch "$(Build.SourceBranch)" \ + --runtime-version "${RUNTIME_VERSION}" + displayName: 'Assemble the deployable fixture' + + - task: AzureCLI@2 + displayName: 'Preflight and deploy with remote build' + inputs: + azureSubscription: ${{ parameters.acaServiceConnection }} + scriptType: bash + scriptLocation: inlineScript + inlineScript: | + set -euo pipefail + python eng/scripts/aca_qualification_pipeline.py deploy \ + --staging-root "$(Build.ArtifactStagingDirectory)/aca-fixture" \ + --archive-path "$(Build.ArtifactStagingDirectory)/aca-fixture.zip" \ + --app-name "$(appSiteName)" \ + --resource-group "$(ACA_DEPLOYED_RESOURCE_GROUP)" \ + --region "$(ACA_SANDBOX_REGION)" \ + --build-id "$(Build.BuildId)" \ + --commit-sha "$(Build.SourceVersion)" + env: + AZURE_TOKEN_CREDENTIALS: 'dev' + + - task: AzureCLI@2 + displayName: 'Cold start, turn, lifecycle, loss, and N=5' + inputs: + azureSubscription: ${{ parameters.acaServiceConnection }} + scriptType: bash + scriptLocation: inlineScript + inlineScript: | + set -euo pipefail + python eng/scripts/aca_deployed_qualification.py deployed-suite \ + --runtime-target "$(runtimeTarget)" \ + --load-concurrency 5 \ + --provision-concurrency 1 + env: + AZURE_TOKEN_CREDENTIALS: 'dev' + AZURE_FUNCTIONS_AGENTS_RUN_DEPLOYED_ACA_SMOKE: '1' + ACA_DEPLOYED_COLD_START_SAMPLES: '3' + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_FUNCTION_BASE_URL: $(baseUrl) + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_APP_SITE_NAME: $(appSiteName) + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_AGENT_SLUG: $(ACA_DEPLOYED_AGENT_SLUG) + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_TOKEN_SCOPE: $(ACA_DEPLOYED_EASY_AUTH_TOKEN_SCOPE) + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_AUDIENCE: $(ACA_DEPLOYED_EASY_AUTH_AUDIENCE) + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_TIMEOUT_SECONDS: '230' + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_TABLE_SERVICE_URI: $(ACA_DEPLOYED_TABLE_SERVICE_URI) + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_TABLE_NAME: $(ACA_DEPLOYED_TABLE_NAME) + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_APP_SUBSCRIPTION_ID: $(ACA_DEPLOYED_APP_SUBSCRIPTION_ID) + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_BUILD_ID: $(Build.BuildId) + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_COMMIT_SHA: $(Build.SourceVersion) + AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_PYTHON_VERSION: $(pythonVersion) + AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_GROUP_RESOURCE_ID: $(ACA_SANDBOX_GROUP_RESOURCE_ID) + AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_REGION: $(ACA_SANDBOX_REGION) diff --git a/tests/live/README.md b/tests/live/README.md index 5f3a9de6..2df239e3 100644 --- a/tests/live/README.md +++ b/tests/live/README.md @@ -20,13 +20,12 @@ access. Its controller identity owns all ACA create, list, and cleanup actions. coverage. `test_aca_real_agent_turn.py` exercises only the production execution backend. -Deployed cold-start, lifecycle, loss, load, and one-shot recovery suites remain -direct/manual test assets pending issue #166. They have no pipeline wiring, -target metadata, or queue-time parameters in this repository. Drive them with -`eng/scripts/aca_deployed_qualification.py`, after packaging and deploying -`tests/live/apps/aca-qualification/` with -`eng/scripts/aca_qualification_pipeline.py`. They still skip unless -`AZURE_FUNCTIONS_AGENTS_RUN_DEPLOYED_ACA_SMOKE=1` is set explicitly. +Deployed cold-start, lifecycle, loss, and load suites are wired into the +post-main `AcaQualification` stage of `eng/ci/e2e-tests.yml`. See the +[ACA qualification CI guide](../../eng/ci/docs/aca-qualification.md) for +targets and triage. The one-shot recovery suite remains direct/manual. All +suites still skip unless `AZURE_FUNCTIONS_AGENTS_RUN_DEPLOYED_ACA_SMOKE=1` is +set explicitly. `AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_FUNCTION_BASE_URL` may be the HTTPS site origin or include `/api`; a pathless origin is normalized to the fixture's default `/api` route root. @@ -95,4 +94,5 @@ live-test entry point reject N=100 before authentication or provider work. Formal N=100 remains future human-only acceptance and requires a purpose-built workflow; this fixture does not discharge Decision #29. -There is no pipeline wiring for any of this; every step is run by hand. +The Python 3.13 and 3.14 deployed suites are wired into `AcaQualification`. +They remain runnable by hand. diff --git a/tests/live/apps/aca-qualification/README.md b/tests/live/apps/aca-qualification/README.md index 7b359c3f..b498f1c5 100644 --- a/tests/live/apps/aca-qualification/README.md +++ b/tests/live/apps/aca-qualification/README.md @@ -35,8 +35,9 @@ site, or endpoint appears in this directory. The operator supplies those settings as app settings on the target Function App before deployment. The deploy command sets only the required Sandbox Group region; the group resource ID, model deployment, storage, and Entra values must -already be configured. This layer adds no pipeline wiring — the deploy is -driven by `eng/scripts/aca_qualification_pipeline.py` by hand. +already be configured. The post-main `AcaQualification` stage drives deployment +with `eng/scripts/aca_qualification_pipeline.py` for independent Python 3.13 and +3.14 jobs. The same tooling remains runnable by hand. Configure `AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_FUNCTION_BASE_URL` as either the HTTPS site origin or that origin plus `/api`; the client normalizes a pathless diff --git a/tests/test_aca_qualification_pipeline.py b/tests/test_aca_qualification_pipeline.py index 322dec92..cb483c75 100644 --- a/tests/test_aca_qualification_pipeline.py +++ b/tests/test_aca_qualification_pipeline.py @@ -294,6 +294,106 @@ def test_the_deployed_timeout_bound_is_the_platform_ceiling(self) -> None: ) +class TestQualificationPipelineWiring: + def _root(self) -> Path: + return Path(__file__).resolve().parents[1] + + def _pipeline(self) -> str: + return (self._root() / "eng" / "ci" / "e2e-tests.yml").read_text(encoding="utf-8") + + def _template(self) -> str: + return ( + self._root() / "eng" / "templates" / "official" / "jobs" / "aca-qualify.yml" + ).read_text(encoding="utf-8") + + def _qualification_stage(self) -> str: + pipeline = self._pipeline() + marker = " - stage: AcaQualification" + assert pipeline.count(marker) == 1 + return pipeline.split(marker, 1)[1] + + def test_only_the_qualification_stage_is_added(self) -> None: + pipeline = self._pipeline() + assert pipeline.count("- stage: AcaQualification") == 1 + assert "AcaSweep" not in pipeline + assert not ( + self._root() / "eng" / "templates" / "official" / "jobs" / "aca-sweep.yml" + ).exists() + + def test_qualification_depends_only_on_build(self) -> None: + stage = self._qualification_stage() + assert "dependsOn: Build" in stage + assert "dependsOn:\n" not in stage + + def test_condition_allows_manual_or_main_ci_only(self) -> None: + condition = next( + line.strip() + for line in self._qualification_stage().splitlines() + if line.strip().startswith("condition:") + ) + assert "'Manual'" in condition + assert "'IndividualCI', 'BatchedCI'" in condition + assert "'refs/heads/main'" in condition + assert "PullRequest" not in condition + assert "Schedule" not in condition + + def test_matrix_runs_both_python_versions_in_parallel(self) -> None: + template = self._template() + assert "maxParallel: 2" in template + assert template.count("runtimeTarget: 'python313'") == 1 + assert template.count("runtimeTarget: 'python314'") == 1 + assert template.count("pythonVersion: '3.13'") == 1 + assert template.count("pythonVersion: '3.14'") == 1 + assert "constraintsFile:" not in template + assert "--constraints-file" not in template + assert ( + template.count( + "--requirements-export eng/constraints/aca-fixture-requirements.txt" + ) + == 1 + ) + + def test_each_leg_runs_the_combined_suite_with_provisioning_concurrency_one( + self, + ) -> None: + template = self._template() + assert template.count(" deployed-suite ") == 1 + assert template.count("--load-concurrency 5") == 1 + assert template.count("--provision-concurrency 1") == 1 + assert "continueOnError: true" in template + + def test_aca_settings_are_basic_variables_not_variable_groups(self) -> None: + source = self._pipeline() + "\n" + self._template() + assert not re.search(r"(?m)^\s*-\s*group\s*:", source) + for name in ( + "ACA_DEPLOYED_APP_SUBSCRIPTION_ID", + "ACA_DEPLOYED_RESOURCE_GROUP", + "ACA_DEPLOYED_APP_SITE_NAME_PY313", + "ACA_DEPLOYED_APP_SITE_NAME_PY314", + "ACA_DEPLOYED_FUNCTION_BASE_URL_PY313", + "ACA_DEPLOYED_FUNCTION_BASE_URL_PY314", + "ACA_DEPLOYED_AGENT_SLUG", + "ACA_DEPLOYED_EASY_AUTH_TOKEN_SCOPE", + "ACA_DEPLOYED_EASY_AUTH_AUDIENCE", + "ACA_DEPLOYED_TABLE_SERVICE_URI", + "ACA_DEPLOYED_TABLE_NAME", + "ACA_SANDBOX_GROUP_RESOURCE_ID", + "ACA_SANDBOX_REGION", + ): + assert f"$({name})" in source + + def test_template_keeps_lightweight_marker_attestation(self) -> None: + template = self._template() + for name in ( + "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_BUILD_ID", + "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_COMMIT_SHA", + "AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_PYTHON_VERSION", + ): + assert f"{name}:" in template + assert "wheel digest" not in template.lower() + assert "deployment storage" not in template.lower() + + class TestCombinedDeployedSuite: def test_n100_is_rejected_before_environment_or_auth_preflight( self, @@ -359,7 +459,7 @@ def test_n100_command_failure_is_stable_and_redacted( def test_cold_start_gates_the_remaining_modules( self, monkeypatch: pytest.MonkeyPatch ) -> None: - captured: list[tuple[str, ...]] = [] + captured: list[tuple[tuple[str, ...], bool]] = [] monkeypatch.setattr( aca_deployed_qualification, "validate_deployed_environment", @@ -369,7 +469,10 @@ def test_cold_start_gates_the_remaining_modules( monkeypatch.setattr( aca_deployed_qualification, "_run_pytest", - lambda paths, _: captured.append(tuple(paths)) or 0, + lambda paths, _, *, fail_fast=False: captured.append( + (tuple(paths), fail_fast) + ) + or 0, ) result = aca_deployed_qualification.run_deployed_suite( @@ -381,20 +484,49 @@ def test_cold_start_gates_the_remaining_modules( assert result == 0 assert captured == [ - ("tests/live/test_aca_deployed_cold_start.py",), ( - "tests/live/test_aca_deployed_agent_turn.py", - "tests/live/test_aca_deployed_lifecycle.py", - "tests/live/test_aca_deployed_loss.py", - "tests/live/test_aca_deployed_load.py", + ("tests/live/test_aca_deployed_cold_start.py",), + True, + ), + ( + ( + "tests/live/test_aca_deployed_agent_turn.py", + "tests/live/test_aca_deployed_lifecycle.py", + "tests/live/test_aca_deployed_loss.py", + "tests/live/test_aca_deployed_load.py", + ), + True, ), ] + def test_run_deployed_suite_pytest_command_enables_fail_fast( + self, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + captured: list[list[str]] = [] + + def capture_run(command: list[str], **_: object) -> subprocess.CompletedProcess[str]: + captured.append(command) + return subprocess.CompletedProcess(command, 0) + + monkeypatch.setattr(aca_deployed_qualification.subprocess, "run", capture_run) + + assert ( + aca_deployed_qualification._run_pytest( + ("cold.py", "later.py"), + {}, + fail_fast=True, + ) + == 0 + ) + assert captured[0].index("-x") < captured[0].index("cold.py") + assert captured[0].index("cold.py") < captured[0].index("later.py") + def test_cold_start_failure_suppresses_later_suites( self, monkeypatch: pytest.MonkeyPatch, ) -> None: - captured: list[tuple[str, ...]] = [] + captured: list[tuple[tuple[str, ...], bool]] = [] monkeypatch.setattr( aca_deployed_qualification, "validate_deployed_environment", @@ -405,8 +537,10 @@ def test_cold_start_failure_suppresses_later_suites( def fail_cold_start( paths: tuple[str, ...], _: object, + *, + fail_fast: bool = False, ) -> int: - captured.append(paths) + captured.append((paths, fail_fast)) return 1 monkeypatch.setattr(aca_deployed_qualification, "_run_pytest", fail_cold_start) @@ -420,7 +554,7 @@ def fail_cold_start( ) == 1 ) - assert captured == [("tests/live/test_aca_deployed_cold_start.py",)] + assert captured == [(("tests/live/test_aca_deployed_cold_start.py",), True)] def test_the_expected_identity_environment_is_required(self) -> None: """Every provenance input must be required before a deployed run starts."""