[IAmJSD]

Root cause

Deleting an editor-managed plan is refused (409 managed_plan_has_live_executions) while any execution may hold or buy capacity. requireManagedDeleteTx counts a failed row without a no_submission proof as live, because its outcome is unknown and it can still be retried (a retry of a row detached from a deleted plan would buy with no plan). Nothing could ever change that row: failed rows are never pruned, and the retry link isn't consulted. So even after a retry of the row completed, the plan could never be deleted, only disabled.

Fix

As the issue asks, this adds an explicit, audited operator action and leaves delete without a flag. Delete still fails closed on an unknown outcome until a named person has accounted for it.

• Migration 000143: purchase_executions.outcome_settled_at, outcome_settled_by (TEXT, so the audit survives user deletion and records the admin API key sentinel) and outcome_settlement (the complete fixed-format record), with a CHECK that the three are set together. 134 to 139 are taken by open PRs, and 140 to 142 by open PRs fix(recommendations): key stored recs, rec ID and purchase identity on RDS az_config (#408) #1239 and perf(api): page recommendations from an ID-ordered keyset scan instead of the full list #1240. Gaps are expected (previousMigrationVersion).
• SettleFailedExecution (internal/config/store_postgres_execution_settle.go):
  • Accepts only a failed plan execution with no no_submission proof. Any other status, a no_submission row, an ad-hoc row or a row whose plan is gone gets 409.
  • Requires Acknowledge and a reason, and stores operator:settled_unknown_outcome: ack=true chain_tail=<id> tail_status=<status> reason="...".
  • Locks the plan FOR SHARE, then the row FOR UPDATE, then its retry-chain tail FOR UPDATE. That's the same plan-first order as DeletePurchasePlan and the editor command footprint. It re-checks plan_id under the locks.
  • Refuses while the chain may still buy: a tail outside the terminal allowlist (pending, running and so on), a failed tail that can still be retried (unknown outcome and not itself settled), or a chain past LadderChainMaxDepth.
  • An identical repeat by the same actor replays (200, replayed: true). A different one gets 409.
• What settling does not do: it never changes status and never makes the row count as unbought. holdsCapacity, the ramp ledger (ListPlanCommittedTx), ramp_ledger_holds and the retention trigger are untouched, so a possibly-submitted purchase is never hidden or re-bought. The row stays failed in History after the plan is deleted (plan_id is set to NULL, as for completed rows today). RecoverReapedExecution can still record a late completion over it.
• requireManagedDeleteTx skips a settled failed row (blocksManagedDelete). Every other classification is unchanged.
• LinkRetryExecutionAtomic also requires outcome_settled_at IS NULL, so a settled row can never be retried. Retry's lost-race 409 now reads "already retried or settled; reload it".
• POST /api/purchases/{id}/settle, body {"reason": "...", "acknowledge_unknown_outcome": true}:
  • Requires execute:purchases (admin:* alone is not enough, matching the separation-of-duties carve-out) and account access to the execution (out of scope gets 404).
  • Checks the grant's constraints with StrictScope against what the execution would have bought: its recommendations' providers, services, regions and total commitment, plus its accounts. Any dimension the execution leaves unknown requires an unconstrained grant.
  • The reason must be 10 to 300 characters with no control characters. Unknown fields are rejected.
  • Logs purchase execution settled by operator with the actor, the credential kind and the stored record.
• known_issues/35_legacy_guard_followups.md: the feat(plans): settle a failed-with-submission row so a managed plan can be deleted #999 entry is marked resolved. The doc stays in place because its other entry belongs to the open PR fix(cleanup): only delete settled executions when their TTL lapses #1173. Whichever of the two merges second should move the doc to known_issues/resolved/. The audit-status line will conflict between the two PRs, and it's a one-line merge.

Closes #999

Verification

All of this ran locally on macOS against a real Postgres in Docker (testcontainers). It is unit and integration evidence, not real-cloud evidence. Tests ran with AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null because this machine's default AWS profile is invalid.

go build ./...
go test -tags integration -count=1 ./internal/config/                                   # ok (full package, 24s)
go test -tags integration -count=1 -run 'Retry|Managed|Settle|Delete|Reschedule|Resume' ./internal/api/   # ok
go test -tags integration -count=1 -run 'TestMigration_PurchaseExecutionsOutcomeSettlement' ./internal/database/postgres/migrations/  # ok
go test -count=1 ./internal/api/ ./internal/config/ ./internal/mocks/ ./internal/purchase/   # ok
go vet ./internal/api/ ./internal/config/ ./internal/database/...   (also with -tags integration)
golangci-lint v2.10.1 run --build-tags integration ./internal/api/... ./internal/config/... ./internal/database/...   # 0 issues (and without the tag)
gocyclo -over 10 -ignore "_test\.go" internal/api internal/config   # silent
bash scripts/gosec-hook.sh <changed .go files>   # clean
npx markdownlint-cli@0.47.0 known_issues/35_legacy_guard_followups.md   # clean
git-secrets (fresh repo, isolated HOME, changed files git-added, planted AKIA key flagged, clean once removed); scripts/test-git-secrets-allowlist.sh   # 137 PASS, 0 FAIL

New tests:

• internal/config/store_postgres_execution_settle_integration_test.go
  • TestSettleFailedExecutionLetsManagedDeleteProceed reproduces the issue scenario: a failed row whose retry completed blocks the delete. After a settle, the delete proceeds, the row is still failed and not no_submission, the audit survives, and ListPlanCommittedTx returns exactly the same ledger before and after.
  • TestSettleFailedExecutionRefusals covers completed, pending, canceled, failed+no_submission, retry tails pending/running/failed, no plan, not found, and no ack/actor/reason. None of them writes anything.
  • TestSettleFailedExecutionChainOfUnknownOutcomes checks that the head is refused while its failed tail can still be retried, and that the delete stays blocked until both are settled.
  • TestSettleFailedExecutionReplayAndRetry covers the replay, the 409 on a different reason, and the retry CAS refusing a settled row.
  • Races, each queued on a real lock and asserted via pg_stat_activity:
    • TestSettleWaitsForRetryAndRefusesItsLiveSuccessor: a retry linked first; the settle waits on the row lock, then refuses the pending successor.
    • TestSettleWaitsForDeleteAndRefusesDetachedRow: a delete holding the plan lock commits first; the settle waits on FOR SHARE, then refuses the detached row.
    • TestDeleteWaitsForSettleAndSeesIt: a delete queued behind a settle sees the committed settlement.
• internal/api/handler_execution_settle_integration_test.go
  • TestSettleUnblocksManagedPlanDeleteOverHTTP: delete 409, then POST /api/purchases/{id}/settle through HandleRequest (routing and CSRF) returns 200, then delete succeeds.
  • TestSettleAuthorizationAndValidation: admin without execute gets 403, out of account scope gets 404, execute constrained to another account or provider gets 403, a missing ack, short reason or unknown field gets 400, and a completed row gets 409. A refused settle writes nothing. Execute constrained to the plan's account succeeds.
• Unit tests: TestBlocksManagedDelete, TestSettlementRecordReplayMatchesOnlyTheSameReason, TestDecodeExecutionSettle, TestMapExecutionSettleError, plus the migration up/down/CHECK test.

Fail-before / pass-after (mutation probes on this branch, each reverted afterwards):

Mutation	Failing test(s)
guard ignores settlement (the pre-fix behavior)	...LetsManagedDeleteProceed, ...ChainOfUnknownOutcomes, TestDeleteWaitsForSettleAndSeesIt
retry CAS ignores settlement	...ReplayAndRetry
no plan_id re-check under the locks	TestSettleWaitsForDeleteAndRefusesDetachedRow
no plan FOR SHARE lock	TestSettleWaitsForDeleteAndRefusesDetachedRow
retryable failed tail allowed	...Refusals, ...ChainOfUnknownOutcomes
status check removed	...Refusals
handler skips constraints / account access / uses view instead of execute	TestSettleAuthorizationAndValidation

On main, TestDeleteManagedPlanByExecutionStatus/failed already shows the permanent 409, and no operation existed that could clear it.

Coverage gaps: nothing here touches a real cloud account. The operator's cloud-side check stays a human step, recorded only as the free-text reason and the acknowledgement. The full internal/api integration package was not run, only the subset above.

Out of scope

• The delete 409 still doesn't name the blocking execution ids. An operator finds them in History (failed rows of the plan). Adding blocking_execution_ids to the refusal details would be a small follow-up.
• History/execution GET responses don't expose the settlement columns yet. The settle response and the server log carry them.
• No structured cloud check (account/region/service/checked_at), unlike the ladder resolve, because a plan execution can span several services and regions. The reason field carries it.

Frontend follow-up for the UI owner

• On a 409 managed_plan_has_live_executions from plan delete, explain that failed purchases with an unknown outcome must be settled first, and link to the plan's failed executions in History.
• In History, for a failed plan execution that isn't no_submission, offer Settle next to Retry. It should open a confirm dialog that asks for a reason (10 to 300 characters) and a required "I checked the cloud account; this may have been bought, and it will never be retried" checkbox. It then sends POST /api/purchases/{id}/settle with {"reason", "acknowledge_unknown_outcome": true}. On 409, show the message (for example, a retry is still pending).
• Once settled, hide Retry for that row. That needs the settlement fields on the History payload (see the out-of-scope note above).

Maintainer: please mirror the issue's triage labels onto this PR: priority/p2, severity/medium, triaged, urgency/this-quarter, impact/few, effort/m, type/feat.

Known unrelated CI failure: govulncheck (GO-2026-6809) until #1236 merges.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}