feat(agent): human approval, controlled execution, and verification [Automation PR 4/5]

[TFT444]

Objective

Close the controlled automation loop while keeping an authenticated, authorised human as the final manual authority over one exact immutable change. This is PR 4 of 5 and depends on #311-#313.

Scope

State machine

Add APPROVED, EXECUTING, EXECUTION_FAILED, VERIFYING, VERIFIED, VERIFICATION_FAILED, and CANCELLED. Optional rollback states are allowed only for a rollback proven by #315. Preserve EXPIRED from #313.

Define allowed transitions, actor, reason, timestamp, attempt ID, and invariants. Transition history is append-only or tamper-evident.

Authorization and approval

• POST /api/v1/remediation-runs/{run_id}/approve
• Require authenticated server-authorised approver role for the exact tenant/subscription.
• Record approver identity, authorization decision, timestamp, exact target IDs, registry version, evidence version, expiry, and plan hash.
• Support separation-of-duties policy so production can prohibit the proposer from approving.
• Approval enqueues execution transactionally using an outbox or equivalent. There is no general execute endpoint.

Approval invalidation

Immediately before execution, reload Azure state, recheck permissions/preconditions/scope, confirm plan hash/evidence/registry versions, and acquire a normalized resource-scope lock.

If state changed, retain the invalidated approval in audit history, regenerate/revalidate the proposal, update evidence and hash as required, and require fresh approval. Never silently reuse approval.

Controlled worker

Use a separate remediation worker, queue, and short-lived least-privilege Azure identity. The scanner/API identity cannot mutate Azure.

• Atomic job claim and unique execution-attempt/idempotency key.
• Fixed registry executable/SDK operation only; argument arrays with shell=False where applicable.
• Timeout, concurrency, output-size, cancellation, redaction, and resource lock controls.
• Kill switch stops new execution claims without disabling scanning, rejection, inspection, or audit.
• Crash recovery inspects Azure state before retry: satisfied postcondition proceeds to verification; intact precondition may retry within policy; ambiguous/partial state stops for operator review.

Independent verification

After execution:

1. Read affected Azure configuration independently of command output.
2. Run affected scanner rules with successful completeness outcomes.
3. Refresh relevant relationships.
4. Recalculate the attack path.
5. Compare registered postconditions.

VERIFIED requires the postcondition, targeted finding removal, and affected attack-path removal. Exit code zero is never sufficient.

Feature controls

• AUTOMATION_PROPOSALS_ENABLED
• AUTOMATION_APPROVALS_ENABLED
• AUTOMATION_EXECUTION_ENABLED (default false)
• AUTOMATION_PLAYBOOK_ALLOWLIST
• AUTOMATION_KILL_SWITCH

Production execution remains disabled until #315 passes. Initially allowlist no more than two reviewed, reversible, single-resource Storage controls.

Measurable acceptance criteria

• Zero executions occur without valid authenticated and authorised approval.
• 100 duplicate or concurrent approval requests produce at most one execution attempt.
• Any hash, evidence, registry, target, scope, precondition, permission, or expiry mismatch blocks execution and requires fresh approval.
• Arbitrary commands, extra arguments, path traversal, environment injection, and scope expansion are impossible in adversarial tests.
• Timeout, cancellation, worker crash, retry, partial change, queue failure, and redaction outcomes are recorded accurately.
• Crash recovery never blindly repeats an ambiguous mutation.
• Exit zero plus remaining postcondition/finding/path produces VERIFICATION_FAILED.
• Tenant/subscription authorization is enforced on every new endpoint and worker claim.
• Logs/API/audit fixtures contain no configured test secrets after redaction tests.
• Operators can inspect rejected, expired, cancelled, failed, invalidated, and verification-failed runs.
• Unit, integration, authorization, replay, concurrency, injection, timeout, cancellation, crash-recovery, outbox, redaction, and verification tests pass.

Delivery evidence attached to PR

• State-transition and authorization matrices.
• Least-privilege role definitions for candidate playbooks.
• Kill-switch, incident-response, alerting, retention, redaction, and recovery runbooks.
• CI-equivalent/security-scan output and security-focused human review.
• Execution-disabled deployment showing proposal/approval telemetry.

Dependencies and handoff

Requires #311-#313. The code merges with execution disabled. #315 provisions disposable Azure resources and is the only gate that may recommend enabling selected playbooks beyond non-production.

Deferred

Broad production enablement, unproven automatic rollback, arbitrary/multi-step planning, and general-purpose execution.

---

Mandatory safety and automation checklist

Approval authority

• Approval requires authenticated, server-authorized approver role for the exact tenant/subscription.
• Approval is bound to one plan hash, evidence version, registry version, target set, and expiry.
• Approval cannot add or modify execution fields.
• Separation-of-duties policy can prevent a proposer approving their own production run.
• Rejected, expired, cancelled, or invalidated plans cannot execute.
• Any state/precondition/hash/evidence/permission drift preserves audit history and requires fresh approval.

Queue and worker safety

• Approval and enqueue use a transactional outbox or equivalent atomic guarantee.
• Job claim is atomic and has a unique execution-attempt/idempotency key.
• A normalized resource-scope lock prevents conflicting mutations.
• Duplicate/replayed/concurrent approvals produce at most one execution.
• Scanner, API, and remediation identities are separate.
• Remediation uses short-lived credentials and an exact least-privilege role.
• Worker accepts only allowlisted registry versions and fixed operations.
• Subprocess execution uses argument arrays, fixed paths, shell=False, minimal environment, safe working directory, and no inherited secrets where avoidable.
• Timeout, concurrency, output-size, cancellation, and redaction limits are enforced.
• Crash recovery inspects Azure state and never blindly repeats an ambiguous mutation.

Fail-safe operations

• Execution feature flag defaults to disabled.
• Empty allowlist means nothing can execute.
• Kill switch prevents new claims without disabling scanning, rejection, inspection, or audit.
• Public-demo mode and remediation execution cannot be enabled together.
• Partial change, permission loss, timeout, cancellation, and worker termination produce accurate safe states.
• Logs, metrics, API responses, and stored output redact secrets and sensitive values.
• Operators receive alerts for stuck, failed, invalidated, and verification-failed runs.

Independent verification

• Verification reads Azure independently of command/SDK success output.
• Relevant scanner rules complete successfully before a finding is considered removed.
• Relevant relationships and attack paths are refreshed from current evidence.
• VERIFIED requires registered postconditions, finding removal, and attack-path removal.
• Exit zero with any unmet condition becomes VERIFICATION_FAILED.
• Immutable before/after evidence and all transitions are retained.

Tests and proof

• 100 duplicate/concurrent approvals cause at most one execution attempt.
• Authorization, replay, race, injection, environment, traversal, timeout, cancellation, outbox, crash, retry, lock-expiry, redaction, and verification-failure tests pass.
• Synthetic secrets do not appear in logs, stored output, API responses, or test artifacts.
• Least-privilege role definitions and permission-diff evidence are reviewed.
• Kill-switch, incident-response, alerting, retention/redaction, rollback/recovery, and operator runbooks are complete.
• CI-equivalent validation and security scanning pass with evidence attached.
• Security-focused human review is approved.
• Execution-disabled deployment is observed before test(agent): disposable Azure end-to-end automation validation [Automation PR 5/5] #315 begins.

Exit gate

• AUTOMATION_EXECUTION_ENABLED remains false after merge.
• No more than two reviewed reversible single-resource controls are eligible for test(agent): disposable Azure end-to-end automation validation [Automation PR 5/5] #315.
• No known critical/high security defect remains open.
• Maintainer confirms feat(agent): human approval, controlled execution, and verification [Automation PR 4/5] #314 acceptance criteria and this checklist are complete before merge.