-
Notifications
You must be signed in to change notification settings - Fork 7
Wire deployed ACA qualification CI #197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Laveesh Rohra (larohra)
merged 4 commits into
feature/aca-sandboxes
from
larohra/aca-qualification-ci
Sep 11, 2026
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| # ACA qualification CI guide | ||
|
|
||
| Operational guide for the deployed ACA qualification stage in | ||
| [`eng/ci/e2e-tests.yml`](../e2e-tests.yml). | ||
|
|
||
| ## Execution conditions | ||
|
|
||
| `AcaQualification` runs automatically only for `IndividualCI` and `BatchedCI` | ||
| builds of `refs/heads/main`. Trusted operators may also queue it manually from | ||
| any branch. Pull request and scheduled builds are excluded. The matrix job uses | ||
| `continueOnError: true` while qualification remains non-required; promotion to | ||
| a blocking gate requires a separate decision. | ||
|
|
||
| ## Current pipeline and job graph | ||
|
|
||
| `AcaQualification` depends only on `Build`. Its matrix expands two independent | ||
| jobs that deploy and qualify Python 3.13 and Python 3.14 in parallel. | ||
|
|
||
| | Stage | Depends on | Job | Current work | | ||
| | --- | --- | --- | --- | | ||
| | `AcaQualification` | `Build` | `AcaQualify_python313` | Deploy; cold start; public turn; lifecycle; backing loss; N=5 | | ||
| | `AcaQualification` | `Build` | `AcaQualify_python314` | Deploy; cold start; public turn; lifecycle; backing loss; N=5 | | ||
|
|
||
| The cold-start module is first. Only after its acceptance, first-event, and | ||
| terminal timing completes does it compare the embedded marker with the expected | ||
| build ID, commit SHA, and Python runtime. A mismatch fails and suppresses the | ||
| cold-start metrics. The deployed-suite runner uses pytest fail-fast behavior and | ||
| does not start the turn, lifecycle, loss, or load modules after a cold/provenance | ||
| failure. This remains lightweight in-package provenance: it does not attest the | ||
| exact wheel digest, installed package version, deploy-input manifest, or | ||
| deployment-storage version. | ||
|
|
||
| Each job passes provisioning concurrency 1; `maxParallel: 2` makes aggregate | ||
| provisioning concurrency 2. Do not serialize the matrix. The dedicated Sandbox | ||
| Group must have quota and operational headroom for both jobs plus sessions | ||
| retained by prior runs. Qualification does not rely on an unverifiable quota | ||
| API, and group-wide sweeping and post-run cleanup are outside this layer. | ||
|
|
||
| ## Required basic pipeline variables | ||
|
|
||
| Configure these ordinary/basic variables directly on Azure DevOps pipeline | ||
| 1777 without committing their values: | ||
|
|
||
| - `ACA_DEPLOYED_APP_SUBSCRIPTION_ID` | ||
| - `ACA_DEPLOYED_RESOURCE_GROUP` | ||
| - `ACA_DEPLOYED_APP_SITE_NAME_PY313` | ||
| - `ACA_DEPLOYED_APP_SITE_NAME_PY314` | ||
| - `ACA_DEPLOYED_FUNCTION_BASE_URL_PY313` | ||
| - `ACA_DEPLOYED_FUNCTION_BASE_URL_PY314` | ||
| - `ACA_DEPLOYED_AGENT_SLUG` | ||
| - `ACA_DEPLOYED_EASY_AUTH_TOKEN_SCOPE` | ||
| - `ACA_DEPLOYED_EASY_AUTH_AUDIENCE` | ||
| - `ACA_DEPLOYED_TABLE_SERVICE_URI` | ||
| - `ACA_DEPLOYED_TABLE_NAME` | ||
| - `ACA_SANDBOX_GROUP_RESOURCE_ID` | ||
| - `ACA_SANDBOX_REGION` | ||
|
|
||
| Do not place these values in or add a dependency on an Azure DevOps variable | ||
| group. The existing `- template:` entries under `variables:` import shared YAML | ||
| variable templates for build infrastructure; they are unrelated to variable | ||
| groups and do not provide the ACA settings. | ||
|
|
||
| The service connection selected by the `acaServiceConnection` parameter must be | ||
| authorized for pipeline 1777. | ||
|
|
||
| ## Manual-run trust boundary | ||
|
|
||
| A manual run executes YAML and scripts from the queued branch under the | ||
| deployment service connection. This is intentional so trusted operators can | ||
| validate a feature branch before merge. Restrict pipeline queue permission to | ||
| those operators. Do not substitute protected-branch or protected-environment | ||
| checks: either would defeat the approved feature-branch validation workflow. | ||
|
|
||
| ## Fixture-app prerequisites | ||
|
|
||
| Each standing qualification Function App must have these app settings before | ||
| deployment: | ||
|
|
||
| - ACA runtime: `AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_GROUP_RESOURCE_ID`. The | ||
| deployment job writes `AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_REGION`. | ||
| - Azure OpenAI: `AZURE_FUNCTIONS_AGENTS_PROVIDER=azure_openai`, | ||
| `AZURE_OPENAI_ENDPOINT`, and `AZURE_OPENAI_DEPLOYMENT`. | ||
| `AZURE_OPENAI_API_VERSION` is optional; when omitted, the Agent Framework | ||
| default is used. | ||
| - Auth allowlists: | ||
| `AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_ENTRA_TENANT_ID`, | ||
| `AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_AUDIENCE`, and | ||
| `AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_TEST_INVOKER_CLIENT_ID`. | ||
| - Session storage: either an `AzureWebJobsStorage` connection string or | ||
| identity-based `AzureWebJobsStorage__tableServiceUri`. For a user-assigned | ||
| storage identity, also set `AzureWebJobsStorage__clientId`; otherwise the | ||
| runtime follows its documented `AZURE_CLIENT_ID` or default-credential | ||
| resolution. | ||
|
|
||
| The runtime table name is fixed as `AzureFunctionsAgentsSessions`; it is not an | ||
| additional app setting. Pipeline variable `ACA_DEPLOYED_TABLE_SERVICE_URI` must | ||
| identify the same Table service as the app's `AzureWebJobsStorage`, and | ||
| `ACA_DEPLOYED_TABLE_NAME` must be `AzureFunctionsAgentsSessions`. | ||
|
|
||
| Platform Easy Auth must be enabled for the standing app. Its configured allowed | ||
| token audience must match `ACA_DEPLOYED_EASY_AUTH_AUDIENCE`, whose value is | ||
| passed into the fixture as | ||
| `AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_AUDIENCE`. App Service injects | ||
| `WEBSITE_AUTH_ENABLED` when Easy Auth is enforced; only environments where that | ||
| platform signal is unavailable need the explicit | ||
| `AZURE_FUNCTIONS_AGENTS_ENTRA_EASY_AUTH=true` assertion. | ||
|
|
||
| Each `ACA_DEPLOYED_FUNCTION_BASE_URL_*` value must include the app's HTTP route | ||
| prefix. With the fixture's default `host.json`, the URL ends in `/api`. | ||
|
|
||
| ## Preview and manual execution | ||
|
|
||
| Preview-compile a pushed branch without running jobs: | ||
|
|
||
| ```bash | ||
| az rest --method post \ | ||
| --resource 499b84ac-1321-427f-aa17-267ca6975798 \ | ||
| --headers "Content-Type=application/json" \ | ||
| --body '{"previewRun":true,"resources":{"repositories":{"self":{"refName":"refs/heads/<branch>"}}}}' \ | ||
| --uri "https://dev.azure.com/<organization>/<project>/_apis/pipelines/1777/runs?api-version=7.1-preview.1" | ||
| ``` | ||
|
|
||
| Run pipeline 1777 manually from a pushed branch: | ||
|
|
||
| ```bash | ||
| az pipelines run \ | ||
| --id 1777 \ | ||
| --branch <branch> \ | ||
| --org https://dev.azure.com/<organization> \ | ||
| --project <project> | ||
| ``` | ||
|
|
||
| ## Failure triage | ||
|
|
||
| | Symptom | Check | | ||
| | --- | --- | | ||
| | Connection denied, DNS failure, or timeout to the regional ACA endpoint | Confirm the agent can resolve and reach the authored regional endpoint on port 443. | | ||
| | ACA data-plane `401` or `403` | Check the service-connection identity and Sandbox Group data-plane role. | | ||
| | Stage waits at an authorization checkpoint | Authorize the selected service connection for pipeline 1777. | | ||
| | Cold start reports unavailable or mismatched provenance | Check for a concurrent deployment or runtime-target wiring error. | | ||
| | Deployment fails during remote build | Inspect deployment logs and the generated `requirements.txt`. | | ||
| | Suite raises `ACA-SMOKE-ENV` | Correct basic pipeline variables, target configuration, identity, or capacity. | | ||
| | Suite assertion fails | Treat the environment as ready and investigate runtime behavior. | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| parameters: | ||
| - name: acaServiceConnection | ||
| type: string | ||
| - name: artifactFeed | ||
| type: string | ||
| - name: poolName | ||
| type: string | ||
|
|
||
| jobs: | ||
| - job: AcaQualify | ||
| displayName: 'ACA deploy + qualification (Python $(pythonVersion))' | ||
| strategy: | ||
| maxParallel: 2 | ||
| matrix: | ||
| python313: | ||
| runtimeTarget: 'python313' | ||
| pythonVersion: '3.13' | ||
| appSiteName: $(ACA_DEPLOYED_APP_SITE_NAME_PY313) | ||
| baseUrl: $(ACA_DEPLOYED_FUNCTION_BASE_URL_PY313) | ||
| python314: | ||
| runtimeTarget: 'python314' | ||
| pythonVersion: '3.14' | ||
| appSiteName: $(ACA_DEPLOYED_APP_SITE_NAME_PY314) | ||
| baseUrl: $(ACA_DEPLOYED_FUNCTION_BASE_URL_PY314) | ||
| continueOnError: true | ||
| timeoutInMinutes: 135 | ||
|
|
||
| pool: | ||
| name: ${{ parameters.poolName }} | ||
| image: 1es-ubuntu-22.04 | ||
| os: linux | ||
|
|
||
| steps: | ||
| - checkout: self | ||
|
|
||
| - download: current | ||
| artifact: azurefunctions-agents-runtime | ||
| displayName: 'Download the build artifact' | ||
|
|
||
| - task: UsePythonVersion@0 | ||
| inputs: | ||
| versionSpec: '3.13' | ||
| displayName: 'Use Python for pipeline tooling' | ||
|
|
||
| - task: PipAuthenticate@1 | ||
| displayName: 'Pip Authenticate' | ||
| inputs: | ||
| artifactFeeds: ${{ parameters.artifactFeed }} | ||
|
|
||
| - bash: | | ||
| set -euo pipefail | ||
| python eng/scripts/aca_qualification_pipeline.py install-tooling | ||
| displayName: 'Install pipeline tooling' | ||
|
|
||
| - bash: | | ||
| set -euo pipefail | ||
| RUNTIME_VERSION="$(python -c 'import azure_functions_agents as a; print(a.__version__)')" | ||
| echo "Runtime version: ${RUNTIME_VERSION}" | ||
| python eng/scripts/aca_qualification_pipeline.py assemble \ | ||
| --artifact-root "$(Pipeline.Workspace)/azurefunctions-agents-runtime" \ | ||
| --staging-root "$(Build.ArtifactStagingDirectory)/aca-fixture" \ | ||
| --requirements-export eng/constraints/aca-fixture-requirements.txt \ | ||
| --commit-sha "$(Build.SourceVersion)" \ | ||
| --build-id "$(Build.BuildId)" \ | ||
| --branch "$(Build.SourceBranch)" \ | ||
| --runtime-version "${RUNTIME_VERSION}" | ||
| displayName: 'Assemble the deployable fixture' | ||
|
|
||
| - task: AzureCLI@2 | ||
| displayName: 'Preflight and deploy with remote build' | ||
| inputs: | ||
| azureSubscription: ${{ parameters.acaServiceConnection }} | ||
| scriptType: bash | ||
| scriptLocation: inlineScript | ||
| inlineScript: | | ||
| set -euo pipefail | ||
| python eng/scripts/aca_qualification_pipeline.py deploy \ | ||
| --staging-root "$(Build.ArtifactStagingDirectory)/aca-fixture" \ | ||
| --archive-path "$(Build.ArtifactStagingDirectory)/aca-fixture.zip" \ | ||
| --app-name "$(appSiteName)" \ | ||
| --resource-group "$(ACA_DEPLOYED_RESOURCE_GROUP)" \ | ||
| --region "$(ACA_SANDBOX_REGION)" \ | ||
| --build-id "$(Build.BuildId)" \ | ||
| --commit-sha "$(Build.SourceVersion)" | ||
| env: | ||
| AZURE_TOKEN_CREDENTIALS: 'dev' | ||
|
|
||
| - task: AzureCLI@2 | ||
| displayName: 'Cold start, turn, lifecycle, loss, and N=5' | ||
| inputs: | ||
| azureSubscription: ${{ parameters.acaServiceConnection }} | ||
| scriptType: bash | ||
| scriptLocation: inlineScript | ||
| inlineScript: | | ||
| set -euo pipefail | ||
| python eng/scripts/aca_deployed_qualification.py deployed-suite \ | ||
| --runtime-target "$(runtimeTarget)" \ | ||
| --load-concurrency 5 \ | ||
| --provision-concurrency 1 | ||
| env: | ||
| AZURE_TOKEN_CREDENTIALS: 'dev' | ||
| AZURE_FUNCTIONS_AGENTS_RUN_DEPLOYED_ACA_SMOKE: '1' | ||
| ACA_DEPLOYED_COLD_START_SAMPLES: '3' | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_FUNCTION_BASE_URL: $(baseUrl) | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_APP_SITE_NAME: $(appSiteName) | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_AGENT_SLUG: $(ACA_DEPLOYED_AGENT_SLUG) | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_TOKEN_SCOPE: $(ACA_DEPLOYED_EASY_AUTH_TOKEN_SCOPE) | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EASY_AUTH_AUDIENCE: $(ACA_DEPLOYED_EASY_AUTH_AUDIENCE) | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_TIMEOUT_SECONDS: '230' | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_TABLE_SERVICE_URI: $(ACA_DEPLOYED_TABLE_SERVICE_URI) | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_TABLE_NAME: $(ACA_DEPLOYED_TABLE_NAME) | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_APP_SUBSCRIPTION_ID: $(ACA_DEPLOYED_APP_SUBSCRIPTION_ID) | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_BUILD_ID: $(Build.BuildId) | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_COMMIT_SHA: $(Build.SourceVersion) | ||
| AZURE_FUNCTIONS_AGENTS_DEPLOYED_ACA_EXPECTED_PYTHON_VERSION: $(pythonVersion) | ||
| AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_GROUP_RESOURCE_ID: $(ACA_SANDBOX_GROUP_RESOURCE_ID) | ||
| AZURE_FUNCTIONS_AGENTS_ACA_SANDBOX_REGION: $(ACA_SANDBOX_REGION) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.