Add post-main ACA deployment, attestation, and qualification pipeline - #169
Closed
Laveesh Rohra (larohra) wants to merge 41 commits into
Closed
Add post-main ACA deployment, attestation, and qualification pipeline#169Laveesh Rohra (larohra) wants to merge 41 commits into
Laveesh Rohra (larohra) wants to merge 41 commits into
Conversation
Laveesh Rohra (larohra)
changed the base branch from
main
to
feature/aca-sandboxes
August 19, 2026 22:10
Wires the deployed ACA qualification suites, which already existed as manual assets, into the official post-main pipeline so every merge to main/release proves a deployed build rather than only the current checkout. Two defects previously shipped through a fully green gate because a fake that implements the interface you wrote will always agree with you. This makes contact with real Azure routine rather than accidental. - Adds additive AcaSweep, AcaDeployCold, and AcaQualify stages to official-build.yml, preserving pr: none, the existing stages, and the current-checkout smoke. - Two-phase matrix: both runtimes finish cold start before either enters the expensive suite, so a cold-start regression fails fast for both. - Deploys via Flex remote build from pinned requirements plus the wheel the existing Build stage produces, exercising the customer deployment path. - Attests the deployment with a BUILD_INFO.json marker shipped inside the package, so a stale app cannot claim a build it is not running. - Sweeps leaked sandboxes before a run rather than after, so a leftover is a signal that automatic cleanup has stopped working instead of being quietly tidied away. - Measures streaming visibility latency to answer the conditional acceptance of the file transport, separating poll cadence from transport delay and stating what it cannot capture. - Records the design as an append-only FRD 0008 addendum. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A - group: reference that does not resolve fails pipeline compilation, so a missing or renamed variable group would take Build, RunTests, and RunE2ETests down with it -- turning a setup omission into an outage of the whole official build. Plain variables degrade to a failure inside the ACA stages instead, and those are continueOnError. None of these values are credentials; they are configuration that must not be committed. This also matches how the existing ACA smoke sources ACA_SANDBOX_GROUP_RESOURCE_ID. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The e2e pipeline already carries a proven ACA_DEPLOYED_* variable set from the manual deployed runs. Reusing that vocabulary, rather than inventing a parallel ACA_QUAL_* one, keeps both pipelines on a single set of names and lets values be copied between them. Per-runtime suffixes are the only addition, since this pipeline drives two apps. Two fixture mismatches surfaced while comparing against those values: - the deployed base URL ends in /api, so the app uses the default route prefix, but the fixture host.json overrode it to "". That would have moved every agent route and /__buildinfo, failing the build check against a healthy app. - the deployed agent slug is deployed_turn, not qualification. Both now match the deployed configuration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The first live run failed with a bare TypeError: the sweep called list_sandboxes() while the adapter declares list_sandboxes(*, labels) -- keyword-only and required. Every unit test still passed, because the selection logic is pure and was exercised without an adapter at all, so the seam that actually broke was never touched. Pass an empty selector, which means no label filter and preserves age as the scoping property. Also separate "the sweep ran and found nothing" from "the sweep never ran". A crash previously logged the same reassuring shape as a clean group, which is the failure mode this stage exists to expose. sweep_with_adapter is split out so the adapter interaction is testable without Azure, and the new test binds a stub to the real keyword-only signature so this drift fails locally rather than in a paid run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The live run reported only 'ServiceRequestError', which cannot distinguish blocked egress from denied authorization from a wrong endpoint, leaving an operator with nothing to act on. Include the cause, redacted. Redaction exists to keep prompts, tokens, and customer content out of logs, not to make an infrastructure failure undiagnosable. Bearer tokens, SAS signature parameters, and JWT-shaped strings are stripped and the reason is length-capped. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1ES network isolation on the production-classified official pipeline blocks outbound to the ACA data-plane host, which the e2e pipeline reaches successfully. Documents the evidence and the two ways out, since the choice between an egress exemption and a separate pipeline is a policy decision. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Both found by deploying to a real Flex app rather than reasoning about it. Flex rejects SCM_DO_BUILD_DURING_DEPLOYMENT outright -- 'not supported with this SKU' -- and the deploy step was setting it, which failed the deployment before any code was uploaded. Flex always performs a remote Oryx build for a zip carrying requirements.txt, so there is no toggle to set; removing the setting makes the deployment succeed, verified against the live app. The preflight also required list-publishing-profiles, which fails on a healthy Flex app because SCM basic auth is disabled by default, and which One Deploy never needs since it authenticates through Entra. That check turned a correct configuration into a preflight failure, blocking a deployment that would have worked. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The verification ran immediately after deployment and failed with buildinfo_http_503, because a Flex app restarts after a deploy. Both apps answered 401 minutes later, so they were healthy the whole time -- the check was simply racing the restart, and would have failed on every run. Poll until ready with a bounded deadline, retrying only restart-shaped statuses. An auth or not-found answer is a real answer and still fails immediately, so a genuinely wrong build is never mistaken for a slow one. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Deployment reports success but the app registers zero functions, so every route 404s while the host stays healthy and Easy Auth answers normally. Documents what has been ruled out, the leading hypothesis that the relative wheel path is not resolved by the remote build, and the alternatives. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Laveesh Rohra (larohra)
force-pushed
the
larohra/aca-qualification-pipeline
branch
from
August 20, 2026 18:22
335a64b to
72035c1
Compare
Laveesh Rohra (larohra)
marked this pull request as draft
August 20, 2026 18:57
Three defects, all of which passed the unit gate because the tests only exercised pipeline helper functions and never resolved the fixture against its real consumers. 1. The build-info route annotated azure.functions.HttpRequest, but the runtime registers routes with the FastAPI Request/Response types. The worker rejected the binding, and because indexing is all-or-nothing that one bad route took the whole app down -- every agent route included -- reported only as "No job functions found". Deployed: 0 functions before, 16 after. 2. The pipeline passed a 230-second-bounded timeout as 300, so every deployed suite would have raised AcaSmokeEnvironmentError during fixture setup and errored before a single assertion ran. 230 is the Azure Functions platform HTTP ceiling, so the larger value was never reachable anyway. 3. The fixture lacked the deployed_load agent and qualification_hold tool the load and loss suites invoke, and left retention at defaults while lifecycle asserts 120s. Load and loss would have targeted a route that does not exist. Adds guards for each: a binding-shape test, a template-vs-enforced-bound test, and a contract test deriving required agent slugs and tool names from the live suite sources. Each was mutation-tested -- broken deliberately, observed to fail, then restored -- because a guard never seen to fail is not evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
official-build.yml exists to build and release artifacts; deploying a package to real Function Apps and qualifying it against live Azure is end-to-end testing, so it belongs in e2e-tests.yml. That pipeline already builds the same `azurefunctions-agents-runtime` artifact the deploy consumes and already passes the ACA service connection, so no new build work is required. This also moves the stages onto a pipeline definition that is not currently enrolled in the 1ES DefaultDeny network isolation policy, which blocks the ACA sandbox data plane (management.<region>.azuredevcompute.io) that the lifecycle, loss, and load suites call directly. That is a reprieve rather than a fix -- enrollment is a progressive migration keyed by pipeline definition ID, and the endpoint still needs allow-listing on its own merits. Unlike official-build, this pipeline has PR triggers, so every stage is gated: CI runs only on main, manual runs are permitted on any branch so the stages stay testable pre-merge, and PR and scheduled builds are excluded. The condition is repeated per stage rather than hoisted into a variable, because a wrong single expression would silently skip or silently run all five and a skipped stage reports the same green as one that never applied. Adds guards asserting each stage is gated, that no condition names PullRequest or Schedule, that manual runs remain possible, and that official-build kept its Build/RunTests/RunE2ETests contracts. The first version of the exclusion guard passed when PullRequest was appended to the allowed reasons -- it only checked that the CI reasons were present -- so it now asserts the exclusions directly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
Decisions 191-193, append-only. 191 revises 182 (pipeline home) rather than rewriting it. 193 records that 1ES DefaultDeny is an allow-list keyed by pipeline definition ID rather than by pool, which is why the same pool and image enforce differently on 1733 and 1777, and why moving pipelines is a reprieve rather than a fix for the blocked ACA data-plane endpoint. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
The status, result, and events routes reconcile without a setup deadline. `SessionRuntimeBinding.reconcile_session` handled that by calling the registered reconciler with two arguments, but the reconciler the app registers takes the deadline as a third positional parameter, so every such call raised `TypeError` and the route returned 500. Deployed qualification hit this on the SSE events endpoint, failing agent-turn, cold-start, lifecycle, loss, and load against a real Function App. A `# type: ignore[misc, call-arg]` on that call suppressed exactly the mypy error that describes the defect. Removing it and passing the deadline unconditionally restores the check: reintroducing the two-argument call now fails mypy with "Too few arguments [call-arg]". The reconciler already accepted an optional deadline downstream, and the timer pass already reconciles without one, so widening the callable type matches existing behaviour rather than introducing a new state. Two existing tests declared two-parameter doubles and so agreed with the broken call instead of the registered reconciler; both now mirror the real arity, and the deadline assertion the first one implies is made explicit. The new regression test uses a three-parameter double for the same reason -- a `*args` double accepts any arity and cannot detect a mismatch. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
The wait loop read the public status first, then read the durable session, run, and operations, then returned the earlier status once the durable projection showed the fenced terminal write. Reconciliation can land between those two reads, so the returned status could predate the terminal write the gate had just observed, and the contract assertion failed against a pre-loss state while the table was already abandoned. The public projection derives its state from the run record, so re-reading the status after the durable gate is deterministic rather than another race. The contract assertion was bare, so a real deployed failure reported only "AssertionError" with no observed value. It now reports the state it saw. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
…D rows A rebase conflict was resolved on the upper half only, leaving the closing `=======` and `>>>>>>>` markers committed in tests/live/README.md. A repo-wide scan now finds no remaining markers. The ACA stages moved from official-build.yml to e2e-tests.yml late, so the runbook and both READMEs still described pipeline 1733 as their home. The runbook additionally presented the network isolation question as unresolved with two options, when option two had already been chosen and executed, and its operational commands still targeted 1733. Retained the 1733/1777 contrast only where it explains why the policy is keyed by definition rather than pool. FRD rows 191-194 ran 504-613 characters against a ~500 ceiling and carried mechanics that belong in the design section. Shortened to 330-385. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
…rtion The `_runtime` helper still declared `targeted_reconciler` as a two-parameter callable while the registered reconciler takes three. mypy does not check tests/, so nothing flagged it, and the signature invited exactly the two-parameter double that produced the production 500 this branch fixes. The new observed-timestamp assertion reported only two integers. A deployed qualification run costs roughly 35 minutes, so an assertion on that path that cannot say what it saw wastes the whole run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
The runbook still carried the zero-functions failure as open. Its cause was the fixture route's azure.functions.HttpRequest annotation rather than the remote build, it is fixed, and a unit guard now covers it. Decision 182 named official-build.yml as the stage home; 191 moved them. Per the append-only convention the original row is annotated rather than rewritten. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
Resolving the Sandbox Group under the controller credential treated every non-200 ARM response as a permanent binding failure. That error is not caught on the request path, so a transient ARM condition escaped as an untyped 500 from the chat endpoint. Deployed load qualification hit this on three runs; the caller cannot tell whether the run was admitted, and the count of unresolved idempotency records matched the 500 count exactly. The status was also conflated: a real authorization rejection, a missing group, and a throttle all produced the same message and the same response, so an RBAC misconfiguration was indistinguishable from a retry-worthy blip. Classification now follows the convention already used in this module for authorization and capacity rejection. 401/403 raise the existing authorization error; 429 and 5xx raise a new transient error that surfaces as 503 with Retry-After; other statuses keep the permanent binding error with a message that no longer implies a credential problem. Connection failures and timeouts are transient, while a decode error remains permanent -- previously all three shared the permanent path. The authorization mapping was verified rather than assumed: the existing catch sites begin after the provider is resolved, so they could not have caught an error raised during resolution. Both are now caught in get_provider(), the one place every call site passes through. Response bodies carry enum codes only, never the exception text, and the messages themselves record a status code and nothing else. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
The transport tests cover classification and the handler tests cover response shape, but the defect being guarded against lived between them: the transport raised a typed error that nothing on the request path caught, so it escaped as an untyped 500. Either side alone stayed green while production returned 500. These drive a real ARM status through the real classification, the real provider resolution, and the real handler, asserting a retryable status becomes 503 with Retry-After and that a missing group does not. Removing the catch in get_provider() fails five of the six, which is the seam the original defect occupied. The response body is also asserted not to carry the upstream status code, since the transport message includes it and the body must stay enum-only. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
Live qualification showed three of five concurrent admissions throttled by ARM. The previous change classified that correctly and returned 503 with Retry-After, but nothing acted on it, so the run still failed. Flex scales out and each cold instance resolves the group independently, so the throttling is inherent rather than a cache stampede; the singleton already serializes resolution within an instance and does not cache failures. The transport now absorbs a short blip with a bounded retry that covers only the transient error. Authorization and permanent binding failures are never retried, because repeating a rejected credential cannot succeed and risks lockout. Sustained throttling still exhausts the budget and surfaces as 503, so backpressure is preserved rather than hidden. The retry spends wall-clock the caller budgeted for setup, and every other retry test patches the sleep out, so nothing would notice the constants growing. A guard bounds the worst case against the setup budget itself; it fails on either a larger attempt count or a longer delay. The load qualification now honors Retry-After on 503 and 429 during admission, which is what a correct client does. Attempts stay bounded, retries past the setup deadline are refused, and the final attempt returns a terminal outcome, so admissions that never recover still fail the run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
…dget guard Review found two gaps, both in the new coverage rather than the retry itself. Mutating the admission branch to ignore Retry-After left every test green, so the client behavior honoring backpressure was unprotected. The decision is now a named function in the load support module with direct tests: throttled statuses retry, the final attempt never does, a missing or out-of-range header declines, and non-throttled statuses are untouched. Reverting either half of that decision now fails. The retry budget guard was looser than its own docstring claimed. It caught an absurd attempt count but not the plausible one -- raising attempts from three to four still passed under a five percent ceiling. It now asserts the intended one second worst case directly and keeps the setup-budget comparison alongside it, so the ratio stays visible. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
Live qualification reached the event stream and failed on a 503 from /events. That is the retryable status the controller now returns for transient sandbox group resolution, but both SSE readers treated any non-200 as terminal, so a signal the server explicitly marked as retryable ended the run. Not a regression: the same condition previously returned 500 and failed the same way, but the suite never got this far because it died during admission. Both readers now retry 503 and 429 within a bounded attempt count, honoring the server's delay. Every other status stays fatal, a missing or out-of-range header declines to retry rather than inventing a delay, and the final attempt always raises, so an endpoint that never recovers still fails the run. The retry sits inside the existing overall timeout, so it cannot outlive the caller's deadline. The bounded parse is shared rather than copied. The existing helper returns a fallback when the header is absent, which is right for a caller that has already decided to wait and wrong for one deciding whether to retry at all; the new variant returns None instead, and the load support module now uses it too rather than keeping a second copy of the same bounds. Auditing the remaining live client paths found no other place where a retryable status is treated as fatal. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
Review found the new tests exercised the pure decision function while the loops that call it went untested. That is the same layer seam that let the earlier defects reach a live run: the reader could stop sleeping, stop raising after exhausting its attempts, or ignore the decision entirely and every test stayed green. The loop is now driven directly with scripted responses: a throttled attempt followed by success honors the server delay exactly once, persistent throttling still raises after the bounded attempt count, a non-throttled status and a throttle without a usable header are not retried at all, and a retry preserves the caller's resume header so reconnecting cannot silently restart the stream. Reverting the sleep, the decision call, or the attempt bound now fails. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
The deployed suites had started failing on their own strictness rather than on defects. The clearest case was a hold-budget precondition that rejected a run the actual proof would have passed: it demanded 135 seconds of remaining hold because it borrowed the proof's polling timeout, while the proof itself needs two observations a second apart. The failing run had 54 seconds of real overlap. It failed because the service had asked the client to wait 120 seconds twice and the client obeyed. That precondition is removed. The concurrency property is still proven, and still fails when unprovable, by the observation loop that owns it. Transient throttling is now handled once in json_request rather than at each call site. Three of the last four failures were a retryable 503 reaching a caller that treated it as fatal, each found by a separate half-hour live run, and fifteen or so call sites remained un-covered. Only 429 and 503 with a usable Retry-After are retried; everything else is returned unchanged. A 504 carrying setup_deadline_exceeded is a real outcome several suites assert on, so the status-based rule leaves it alone and a test pins that. The retry ceiling is ten seconds, not the 120 the setup-timeout path uses. Every 503 this service emits asks for two. Admitting the larger value would let one request sleep for minutes and recreate the budget exhaustion this change exists to remove, so a guard bounds the worst case. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
Deployed load qualification hit a 409 from the data plane: resume was called on a sandbox another concurrent caller had already resumed. The resume path only handled authorization rejections, so the conflict escaped uncaught and reached the client as an untyped 500 with a body it could not parse. Resume is idempotent in intent, and a sandbox already in the requested state is a successful resume, not a failure. A 409 that reports some other state is a real conflict and now raises a typed error that the readiness and HTTP layers translate into a structured 409, so no unparseable response remains reachable from this path. Capacity was rejected as the carrier because it advertises a retry the caller should not attempt here. The already-running case is recognized from the provider's state text, which is brittle to rewording. The degradation is deliberate: an unrecognized 409 becomes the structured conflict rather than a 500, so a wording change costs a spurious conflict, never an unparseable response. The provider payload carries trace and request identifiers, so only an enum code crosses the boundary. Covered at the transport layer and across the seam to HTTP, because the last defect of this shape was fixed in transport while the seam stayed untested and reached production anyway. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 237a8bbf-6f1b-44b7-b905-34341fcbdb3b
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds deployed ACA qualification infrastructure, regional data-plane configuration, attestation, reliability handling, and expanded live coverage.
Changes:
- Adds ACA deployment, sweep, cold-start, and N=5 qualification pipeline assets.
- Introduces required Sandbox Group region configuration and structured transport errors.
- Improves result retention, retry behavior, and latency diagnostics.
Reviewed changes
Copilot reviewed 85 out of 85 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
README.md |
Documents required ACA region. |
docs/aca-sandbox-session-runtime.md |
Updates runtime configuration and failure semantics. |
docs/architecture.md |
Updates ACA architecture boundaries. |
docs/frds/0008-aca-sandbox-session-runtime.md |
Records qualification design decisions. |
docs/front-matter-reference.md |
Adds generated region reference. |
docs/front-matter-spec.md |
Adds region examples and validation. |
eng/ci/docs/aca-qualification.md |
Adds qualification runbook. |
eng/ci/e2e-tests.yml |
Wires ACA qualification stages. |
eng/constraints/README.md |
Documents fixture dependency locks. |
eng/constraints/aca-fixture-requirements.in |
Defines fixture dependency inputs. |
eng/constraints/aca-fixture-py313.txt |
Locks Python 3.13 dependencies. |
eng/constraints/aca-fixture-py314.txt |
Locks Python 3.14 dependencies. |
eng/scripts/README.md |
Documents qualification scripts. |
eng/scripts/aca_deployed_qualification.py |
Orchestrates deployed suites. |
eng/scripts/aca_qualification_pipeline.py |
Implements packaging, attestation, and sweep helpers. |
eng/scripts/aca_pr_smoke.py |
Requires the ACA region. |
eng/scripts/probe_aca_data_plane.py |
Uses the regional endpoint. |
eng/scripts/reap_aca_smoke_sandboxes.py |
Passes region during cleanup. |
eng/templates/official/jobs/aca-qualify.yml |
Defines deployment and qualification job. |
eng/templates/official/jobs/aca-sweep.yml |
Defines pre-run leak sweep. |
eng/templates/official/jobs/e2e-tests.yml |
Forwards ACA region to smoke jobs. |
src/azure_functions_agents/app.py |
Passes configured region to the adapter. |
src/azure_functions_agents/config/schema.py |
Adds required ACA region schema. |
src/azure_functions_agents/controller/http.py |
Maps structured ACA failures to HTTP. |
src/azure_functions_agents/controller/readiness.py |
Classifies activation failures and preserves results. |
src/azure_functions_agents/controller/reconciler.py |
Extends result retention and exact backing checks. |
src/azure_functions_agents/controller/sandbox_config.py |
Forwards region into sandboxes. |
src/azure_functions_agents/execution/__init__.py |
Exports indeterminate admission error. |
src/azure_functions_agents/execution/aca_sandbox.py |
Improves transport failure projection and retention. |
src/azure_functions_agents/execution/backend.py |
Adds durable indeterminate admission error. |
src/azure_functions_agents/sandbox_runtime_limits.py |
Defines result-hold duration. |
src/azure_functions_agents/session_state/store.py |
Supports minimum session expiry on adoption. |
src/azure_functions_agents/transport/aca_sdk.py |
Uses regional clients and typed failures. |
src/azure_functions_agents/transport/transport_models.py |
Revises transport errors and region validation. |
tests/doubles/fake_session_runtime.py |
Mirrors terminal-adoption retention behavior. |
tests/fixtures/config_scenarios/19_aca_sandbox_row2_workflows_enabled/agents.config.yaml |
Adds required region fixture value. |
tests/fixtures/config_scenarios/20_aca_sandbox_row3_dynamic_sessions_conflict/agents.config.yaml |
Adds required region fixture value. |
tests/fixtures/config_scenarios/21_aca_sandbox_row4_non_http_trigger/agents.config.yaml |
Adds required region fixture value. |
tests/fixtures/config_scenarios/25_aca_sandbox_row8_anonymous_auth/agents.config.yaml |
Adds required region fixture value. |
tests/fixtures/config_scenarios/26_aca_sandbox_row9_bad_auto_suspend_idle/agents.config.yaml |
Adds required region fixture value. |
tests/fixtures/config_scenarios/27_aca_sandbox_row10_bad_reclaim_idle/agents.config.yaml |
Adds required region fixture value. |
tests/fixtures/config_scenarios/29_aca_sandbox_valid/agents.config.yaml |
Adds valid region configuration. |
tests/fixtures/live_aca_deployed_agent_turn/agents.config.yaml |
Adds deployed fixture region. |
tests/fixtures/live_aca_setup_timeout_recovery/agents.config.yaml |
Adds timeout fixture region. |
tests/fixtures/live_aca_setup_timeout_recovery/controlled_setup_timeout.py |
Passes region to delayed provider. |
tests/live/aca_deployed_agent_support.py |
Adds bounded JSON and SSE retries. |
tests/live/aca_deployed_cold_start_support.py |
Adds expected-build identity reporting. |
tests/live/aca_deployed_lifecycle_support.py |
Adds lifecycle resource region. |
tests/live/aca_deployed_load_support.py |
Adds visibility and throttling metrics. |
tests/live/aca_deployed_loss_support.py |
Improves loss assertion diagnostics. |
tests/live/aca_smoke_support.py |
Adds region-aware smoke configuration. |
tests/live/apps/aca-qualification/.funcignore |
Excludes local fixture artifacts. |
tests/live/apps/aca-qualification/agents.config.yaml |
Configures deployed ACA fixture. |
tests/live/apps/aca-qualification/deployed_load.agent.md |
Adds load qualification agent. |
tests/live/apps/aca-qualification/deployed_turn.agent.md |
Adds turn qualification agent. |
tests/live/apps/aca-qualification/function_app.py |
Adds build-information endpoint. |
tests/live/apps/aca-qualification/host.json |
Configures the fixture host. |
tests/live/apps/aca-qualification/README.md |
Documents the fixture app. |
tests/live/apps/aca-qualification/requirements.txt |
Defines generated deployment requirements. |
tests/live/apps/aca-qualification/tools/qualification_hold.py |
Adds active-run hold tool. |
tests/live/README.md |
Documents pipeline-wired suites. |
tests/live/test_aca_deployed_cold_start.py |
Adds post-timing provenance verification. |
tests/live/test_aca_deployed_load.py |
Adds retries and visibility collection. |
tests/live/test_aca_deployed_loss.py |
Re-reads terminal public status. |
tests/live/test_aca_run_journal_acceptance.py |
Broadens accepted harness failures. |
tests/live/test_aca_sdk_smoke.py |
Supplies region to live adapter. |
tests/test_aca_deployed_agent_support.py |
Tests JSON and SSE retry behavior. |
tests/test_aca_deployed_cold_start.py |
Tests provenance reporting. |
tests/test_aca_deployed_load.py |
Tests result and admission recovery. |
tests/test_aca_deployed_load_support.py |
Tests latency and throttling helpers. |
tests/test_aca_deployed_agent_turn.py |
Adds region test setup. |
tests/test_aca_pr_smoke.py |
Tests required region preflight. |
tests/test_aca_smoke_support.py |
Tests region forwarding and reconstruction. |
tests/test_app.py |
Tests regional provider composition. |
tests/test_config_schema.py |
Tests region schema validation. |
tests/test_controller_http.py |
Tests structured public responses. |
tests/test_controller_readiness.py |
Tests deadlines, capacity, and result retention. |
tests/test_controller_reconciler.py |
Tests retention and paged reclaim recovery. |
tests/test_controller_sandbox_config.py |
Tests region forwarding. |
tests/test_execution_aca_sandbox.py |
Tests structured execution failures. |
tests/test_probe_aca_data_plane.py |
Tests region-aware probing. |
tests/test_session_runtime_validation.py |
Updates missing-field documentation. |
tests/test_session_state_store_errors.py |
Tests minimum expiry persistence. |
tests/test_aca_qualification_pipeline.py |
Tests qualification pipeline contracts. |
tests/test_transport_aca_sdk.py |
Tests regional transport classification. |
Suppressed comments (1)
eng/ci/e2e-tests.yml:87
- These jobs each deploy and immediately run the full expensive suite independently. This does not implement the PR's two-phase guarantee that both runtimes complete deploy, attestation, and cold start before either begins N=5 qualification; one leg can consume its full budget before the other exposes a cold-start regression. Split deploy/cold and qualification into separate dependent stages/jobs as described, or update the design claim.
# Parallel jobs keep one runtime's failure from suppressing the other.
- template: /eng/templates/official/jobs/aca-qualify.yml@self
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 89f4ac73-faf3-428f-8f36-1fa37e7ac412
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 89f4ac73-faf3-428f-8f36-1fa37e7ac412
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 89f4ac73-faf3-428f-8f36-1fa37e7ac412
hallvictoria
reviewed
Sep 1, 2026
hallvictoria
reviewed
Sep 1, 2026
hallvictoria
reviewed
Sep 1, 2026
hallvictoria
reviewed
Sep 1, 2026
hallvictoria
reviewed
Sep 1, 2026
This was referenced Sep 2, 2026
Contributor
Author
|
Superseded by native stack #199 (review bottom-to-top):
All prior review threads have been mapped to and resolved in their owning replacement layer. PR #169 is retained as historical context only. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #166.
Adds deployed Azure Container Apps Sandbox qualification to
eng/ci/e2e-tests.yml, complementing the current-checkout ACA smoke with validation of an actually deployed runtime build.What runs
Each runtime job installs shared tooling, assembles the committed fixture with the built wheel and pinned requirements, configures the authored Sandbox Group region, deploys through the Flex remote-build path, and runs one ordered suite:
Automatic deployed qualification runs only for
IndividualCI/BatchedCIbuilds ofmain. Manual runs remain available from any branch; PR and scheduled runs exclude these stages. The jobs remaincontinueOnErroruntil a separate human promotion decision.Key decisions
eng/ci/e2e-tests.yml:AcaSweepplus one matrix-backedAcaQualificationstageBUILD_INFO.jsonprovenance for build ID, commit SHA, and live Python minor after cold timingThe required ACA settings are ordinary pipeline variables configured directly on the E2E pipeline. Existing
- template:entries under YAMLvariables:import build-infrastructure variable templates; they are not ADO variable groups and do not provide ACA settings.The required authored Sandbox Group region selects the regional ACA data-plane endpoint directly. Runtime ARM discovery/fallback is removed, Function App and Sandbox Group may be in the same or different regions, and provider failures are translated into typed, redacted runtime responses.
Design and scope are recorded in finalized FRD 0008, including Decisions 189–196. Operational setup and triage live in
eng/ci/docs/aca-qualification.md.Verification
A green N=5 run is diagnostic evidence, not Decision #29's N=100 GA sign-off.