test(agent): disposable Azure end-to-end automation validation [Automation PR 5/5]

[ritiksah141]

Objective

Provision a disposable Azure validation environment and prove the complete OpenShield observe-reason-plan-approve-act-verify workflow. This is PR 5 of 5 and depends on #311-#314. It is the production-execution readiness gate, not a feature-expansion PR.

Safety boundary

• Use a dedicated non-production tenant/subscription or an explicitly isolated disposable subscription.
• Never target shared or production resources.
• Apply restrictive budget, region, naming, tagging, TTL, and resource-group boundaries.
• OpenShield execution remains disabled by default before and after tests unless a documented environment-specific flag is deliberately enabled.
• Human approval remains mandatory; the test harness must not forge or bypass it.

Infrastructure scope

Create version-controlled Terraform or Bicep for the minimum reviewed scenario:

• Dedicated resource group with ownership and automatic-expiry tags.
• A deliberately misconfigured but non-sensitive entry-point workload/public exposure suitable for the reviewed template.
• System- or user-assigned managed identity.
• Narrow test RBAC relationship connecting the identity to a disposable Key Vault target.
• Empty Key Vault containing no real secrets.
• Storage account(s) needed for the initially allowlisted reversible remediation controls.
• Log/diagnostic destination required for audit evidence.
• Separate scanner and remediation identities.
• Custom least-privilege remediation role containing only the exact actions required by allowlisted playbooks.

Do not create real customer data, broadly privileged identities, Owner/User Access Administrator assignments, public secrets, or reusable credentials.

Test orchestration

Provide documented, repeatable stages:

1. Preflight validates subscription allowlist, tenant, identity, budget/TTL tags, feature flags, and clean test state.
2. Provision isolated resources.
3. Seed the reviewed misconfiguration and relationship evidence.
4. Run OpenShield scan and assert complete rule/collector outcomes.
5. Assert finding lifecycle and evidence-backed attack path.
6. Create typed proposal and validate canonical plan/hash/approval packet.
7. Pause for a real authenticated human approval of the displayed hash.
8. Revalidate and execute through the remediation worker.
9. Independently read Azure state, rescan, refresh relationships, and recalculate path.
10. Assert VERIFIED and immutable before/after audit evidence.
11. Destroy disposable resources and verify cleanup.

The workflow may automate every stage except the actual human approval decision.

Required negative and recovery scenarios

• Rejection: no execution occurs.
• Expired approval: execution is blocked.
• Azure state changed after approval: approval is invalidated and fresh approval is required.
• Duplicate and concurrent approval: one execution maximum.
• Insufficient scanner permission: no false resolution/path.
• Insufficient remediation permission: accurate EXECUTION_FAILED, no privilege expansion.
• Command timeout/cancellation.
• Worker termination after claim and after Azure mutation.
• Exit code zero or successful SDK response while postcondition remains false.
• Finding removed but attack path remains, and path removed but finding remains: both yield VERIFICATION_FAILED.
• Kill switch activated before claim and during queued work.
• Redaction fixtures containing synthetic secrets.
• Cleanup retry after partial infrastructure deletion.

Measurable acceptance criteria

• Infrastructure deploy and destroy succeed from a clean environment twice consecutively.
• All created resources stay inside the allowlisted subscription/resource group and carry expiry/owner tags.
• Scanner and remediation identities are different; automated permission diff proves the remediation role contains only reviewed actions.
• Happy path reaches VERIFIED with matching plan hash and immutable before/after evidence.
• Human approval is demonstrably the only manual workflow decision.
• Every required negative scenario reaches the documented safe state with zero unauthorised or duplicate mutations.
• No real secrets or tenant identifiers enter fixtures, logs, screenshots, or committed artifacts.
• Resource cleanup leaves no billable test resources, role assignments, locks, or credentials.
• Test report records timings, retries, attack-path false results, execution success, independent verification result, and cleanup result.
• CI-equivalent tests and IaC security/lint/plan checks pass.

Production enablement decision

Attach a signed readiness report containing:

• Tested commit and registry versions.
• Azure environment and evidence timestamps with sensitive identifiers redacted.
• Results for every happy, negative, crash, replay, and cleanup scenario.
• Least-privilege review.
• Observed false-positive/false-negative notes.
• Alert, kill-switch, incident-response, retention/redaction, rollback/recovery, and operator runbook evidence.
• Explicit list of playbooks approved for staged enablement.

Passing this issue permits only staged enablement of the specifically tested playbooks:

proposal-only -> disposable Azure -> non-critical production subscription -> wider rollout

Failure keeps AUTOMATION_EXECUTION_ENABLED=false and creates follow-up issues; it must not be waived silently.

Dependencies

Requires #311, #312, #313, and #314 merged in sequence with execution disabled.

---

Mandatory safety and automation checklist

Environment guardrails

• Subscription and tenant are explicitly allowlisted as disposable/non-production.
• Preflight aborts on production/shared subscription, unexpected tenant, wrong identity, or missing safety configuration.
• All resources remain inside the dedicated resource group and carry owner, purpose, cost, and expiry tags.
• Budget/alerting, allowed regions/SKUs, TTL, naming prefix, and maximum resource count are enforced.
• No real customer data, secrets, broadly privileged roles, or reusable credentials are created.
• Terraform/Bicep state and outputs contain no committed sensitive values.

Identity and permissions

• Scanner and remediation identities are demonstrably different.
• Scanner identity has read-only permissions required by the reviewed collectors.
• Remediation identity has only the exact actions required by tested playbooks.
• Automated permission diff rejects Owner, Contributor, User Access Administrator, wildcard actions, and unexpected scopes.
• Credentials are short-lived/federated where supported and revoked during cleanup.

End-to-end happy path

• Infrastructure provisions from a clean environment.
• Seeded misconfiguration creates the expected finding with complete rule/collector outcomes.
• Current authoritative evidence creates the expected reviewed attack path.
• Typed proposal contains the correct immutable target, parameters, versions, expiry, and hash.
• A real authenticated human reviews and approves the displayed exact hash.
• Worker revalidates Azure state and executes only the allowlisted operation.
• Independent Azure read confirms the postcondition.
• Targeted rescan confirms finding removal.
• Relationship refresh confirms attack-path removal.
• Run reaches VERIFIED with immutable before/after evidence.

Mandatory negative scenarios

• Rejection produces zero execution.
• Expired approval is blocked.
• Post-approval state drift invalidates approval and requires fresh approval.
• Duplicate and concurrent approval executes at most once.
• Insufficient scanner permission cannot cause false resolution or false path removal.
• Insufficient remediation permission fails without privilege expansion.
• Timeout and cancellation reach documented safe states.
• Worker termination before and after mutation recovers without unsafe duplicate execution.
• Successful command/SDK response with unmet postcondition becomes VERIFICATION_FAILED.
• Finding-only removal and path-only removal both become VERIFICATION_FAILED.
• Kill switch prevents queued/new execution as documented.
• Synthetic secrets are redacted everywhere.

Cleanup and repeatability

• Destroy completes and verifies zero remaining billable resources, role assignments, locks, or credentials.
• Partial cleanup failure is retryable and produces an operator alert.
• Full provision-test-destroy cycle passes twice consecutively.
• Costs and total resource lifetime stay within the declared test budget.
• Test artifacts redact tenant, subscription, object, and resource identifiers as required.

Production-readiness evidence

• Signed report identifies tested commit, registry versions, environment, timestamps, and all scenario results.
• Report includes execution/verification success, retries, timing, false-result review, permission diff, and cleanup result.
• Alerting, kill switch, incident response, audit retention/redaction, recovery, and rollback/manual recovery are rehearsed.
• IaC format, validate, plan, lint, policy/security scanning, and application CI pass.
• Independent security reviewer approves the evidence.
• Only specifically tested playbooks are recommended for staged enablement.

Final exit gate

• Any failed or waived safety scenario keeps production execution disabled.
• No known critical/high security defect remains open.
• Maintainer and security approver sign the staged-enablement decision.
• Rollout begins proposal-only/non-critical and includes an immediate rollback/kill-switch owner.