Skip to content

fix(api/purchases): guard the email-failure write in finalizePurchaseStatus on pending - #1237

Merged
cristim merged 2 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-1232
Oct 25, 2026
Merged

cristim merged 2 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-1232

Conversation

@IAmJSD

@IAmJSD IAmJSD commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes item 1 of #1232: the unguarded failed write in finalizePurchaseStatus.

Refs #1232

This PR fixes only item 1 of #1232, so #1232 stays open. Item 2 (legacy EC2 executions without Details leaving a variant unsuppressed after a retry) is in the code #1110 added (SuppressionVariant / suppressionBucketOf in internal/config/suppressions_build.go). #1110 merged on 2026-10-23, so that gap is now on main. It needs its own small fix against main (treat an EC2 rec with empty Details as unknownVariant, which makes its row a 6-tuple wildcard), as the #1110 review suggested. It's tracked as a follow-up and isn't part of this PR.

Root cause

finalizePurchaseStatus (internal/api/handler_purchases.go) runs after the approval email send on both the web submit path (executePurchase) and the retry path (retryPurchase). When the send failed it set execution.Status = "failed" on the in-memory copy and wrote it with SavePurchaseExecution, a full-row upsert with no status guard. A session approve claims the row with a pending/notified to running CAS before calling the provider. If that claim landed while the email was in flight, the unguarded write overwrote running with failed. The commitment was being bought, but the row became retryable, so a retry could buy it again. The same applies to a cancel or pause landing during the send.

Fix

  • The failed write now goes through the existing SaveExecutionIfStatus(ctx, &failed, []string{"pending"}). It takes a FOR UPDATE row lock, checks the status and writes in one transaction, so it serializes with the approve CAS.
  • A lost guard (ErrExecutionNotInExpectedStatus) is a no-op with a warning log. The row keeps the other writer's status. The caller's in-memory copy is not mutated, because the write uses a copy that is only applied back on success. The response status is the row's re-read status (for example running) rather than a stale failed. If the re-read fails, it falls back to pending.
  • Any other save error keeps the existing best-effort contract: log it and return pending.

execution_status values are unchanged and there is no migration.

Verification

All commands were run from the repo root on macOS. Tests ran with AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null, because this machine's default AWS profile is invalid.

New tests:

  • internal/api/handler_purchases_finalize_test.go (unit, mock store). These pin the guarded call with []string{"pending"} and check that a lost guard returns the re-read status (approved/running/completed/canceled/paused) without mutating the caller's copy or calling SavePurchaseExecution. They also cover the re-read failure fallback, other save errors, and the email-sent path writing nothing.
  • internal/api/handler_purchases_finalize_race_integration_test.go (integration, real Postgres via testcontainers):
    • TestFinalizePurchaseStatus_ApproveDuringEmailSend drives the real persistExecutionAndSuppressions, then sendPurchaseApprovalEmail with a notifier that blocks. While the send is blocked, the test runs the approve claim CAS (pending to running), then fails the send and runs finalizePurchaseStatus. It asserts the row stays running with no error text and that the response reports running.
    • TestFinalizePurchaseStatus_ConcurrentWithApproveClaim races the failed write against the approve claim for 40 rounds. Exactly one may win each round: either the claim won and the row is running, or the claim got ErrExecutionNotInExpectedStatus and the row is failed.

Fail-before / pass-after (fix reverted with git stash push internal/api/handler_purchases.go):

go test -tags integration ./internal/api/ -run TestFinalizePurchaseStatus -count=1 -v
# before: FAIL ApproveDuringEmailSend (row "failed", want "running")
#         FAIL ConcurrentWithApproveClaim (claim won 35/40 rounds, each overwritten to "failed")
#         FAIL GuardsFailedWriteOnPending; LostGuardIsNoOp panics on the unexpected unguarded SavePurchaseExecution
# after:  all PASS (claim won 32/40 rounds, invariant held in every round)

Other checks:

go build ./...
go vet ./internal/api/ && go vet -tags integration ./internal/api/
go test ./internal/api/ -count=1                                   # ok
go test ./internal/mocks/ -count=1                                 # ok (repo-wide test guards)
go test -tags integration ./internal/api/ -run 'TestPersistExecution|TestFinalize|Retry|Notified|Purchase' -count=1   # ok
go test -short -race ./internal/api/ -run 'TestFinalizePurchaseStatus|retryPurchase|ExecutePurchase' -count=1          # ok
golangci-lint v2.10.1 run ./internal/api/...                        # 0 issues
golangci-lint v2.10.1 run --build-tags integration ./internal/api/... # 0 issues
gocyclo -over 10 -ignore "_test\.go" internal/api/handler_purchases.go  # silent
bash scripts/gosec-hook.sh internal/api/handler_purchases.go        # exit 0
bash scripts/gofmt-hook.sh <changed files>                          # exit 0
git secrets --scan (throwaway repo with the changed files and .gitallowed, plus a planted fake key that was flagged) # clean
bash scripts/test-git-secrets-allowlist.sh                          # 137 PASS, 0 FAIL

Evidence is from unit tests (mock store) and local integration tests against real Postgres 16 in testcontainers. Real SES sends and a real cloud purchase were not exercised. The email failure is simulated by a blocking stub notifier, and the approve is the same TransitionExecutionStatus claim CAS that purchase.Manager.claimRunning uses.

Out of scope

Frontend follow-up for the UI owner

None required. On the rare lost-guard path the submit and retry responses can now carry status: "running" (or approved, canceled, paused, ...) together with email_sent: false, where before they carried failed. The UI should already treat unknown or non-failed statuses as "see History".

Maintainers: please mirror the issue's triage labels onto this PR: priority/p1, severity/high, urgency/this-sprint, impact/few, effort/s, type/bug, triaged.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Fixed a race where purchase finalization could incorrectly mark a purchase as failed while approval was in progress. The purchase now retains its current status when approval and finalization happen at the same time, helping prevent conflicting updates and inaccurate failure reports.

…Status on pending

finalizePurchaseStatus stamped "failed" over a freshly inserted execution
with an unguarded full-row save whenever the approval email could not be
sent. A session approve (pending -> running claim), cancel or pause can
land while the email is in flight; the unguarded write then overwrote the
row, so a commitment that was being bought ended up "failed" and eligible
for retry, risking a second purchase.

The write now goes through SaveExecutionIfStatus with "pending" as the
only expected status (row lock + status check + write in one tx). A lost
guard is a no-op with a warning: the row keeps the other writer's status,
the caller's in-memory copy is left untouched, and the response reports
the row's re-read status instead of a stale "failed". Other save errors
keep the existing best-effort "pending" contract.

Tests: mock unit tests pin the guarded call and the lost-guard behavior;
integration tests on real Postgres drive the real send path with an
approve claim landing during a blocked email send, and race the failed
write against the approve claim for 40 rounds.

Refs LeanerCloud#1232

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Caution

Review failed

The pull request is closed.

Walkthrough

Purchase failure finalization now uses a pending-only status guard. If the guard loses, it returns the stored status when available. Unit and integration tests cover save outcomes and races with approval claims.

Changes: Purchase Finalization

Layer / File(s) Summary
Guarded failure write and status resolution
internal/api/handler_purchases.go, internal/api/handler_purchases_finalize_test.go
Failure finalization writes failed only when the stored status is pending. If the guard loses, it returns the current stored status; if that read fails, it returns pending. Unit tests cover these outcomes, other save errors, and successful email sends.
Approval claim race coverage
internal/api/handler_purchases_finalize_race_integration_test.go
Integration tests check that a concurrent approval claim can preserve running while email failure finalization completes. A 40-round test checks the outcomes of concurrent claims and failed finalization.

Priority: Not assessed

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Merge Risk: Minimal · up to 3086b

The guarded failure write preserves a status changed by an approval claim. No merge-blocking issue was found in this change.

Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Brings in main's CI fixes (LeanerCloud#1236, LeanerCloud#1252) and CI changes since, so this PR gets a fresh CI run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cristim

cristim commented Oct 25, 2026

Copy link
Copy Markdown
Member

Independent review 2 (claude-fleet, cold): APPROVE 3086bc8

Findings (none blocking):

The four questions:

  1. No overwrite of a moved row. The guard refuses to overwrite a row that moved to approved, running, completed, canceled or paused during the send.
    • The PR's ApproveDuringEmailSend shows this on real Postgres.
    • So does a scratch test of mine, which blocks the email send while pending -> approved -> running -> completed, pending -> canceled and pending -> paused happen. It passes on head, and the canceled case fails on base.
  2. It reuses the existing guarded save, SaveExecutionIfStatus (store_postgres_execution_guarded.go:44-63): FOR UPDATE, a status check and the write in one transaction, with no duplication.
    • Guarding on "pending" alone is correct, because both callers insert "pending" (handler_purchases.go:2579, 3533).
    • The only pending -> notified writer, RotatePendingApprovalToken, runs on plan-root rows only (notifications.go:193, 285), and submit and retry rows are never plan roots.
  3. A lost race is a no-op.
    • The write goes through a copy, so the caller's in-memory row is unchanged.
    • The caller logs a warning and returns the re-read status with email_sent=false.
    • Neither executePurchase (:3622) nor retryPurchase (:2299) returns an error to the client or reports failed.
  4. No other unguarded write. On the submit and retry paths, finalizePurchaseStatus is the only write after the insert. The only other unguarded post-CAS write nearby is the one in the second finding.

Mutation probes on a scratch copy, each restored and confirmed identical with cmp:

  • P1, the guard also accepts running, canceled, paused and completed: ApproveDuringEmailSend FAILS.
  • P2, a lost guard returns "failed": ApproveDuringEmailSend FAILS.
  • P3, the caller's copy changed before the save: LostGuardIsNoOp and WriteErrorKeepsPending FAIL.
  • P4, back to the unguarded SavePurchaseExecution: the integration tests FAIL.
  • P5, FOR UPDATE dropped in lockExecutionRow: ConcurrentWithApproveClaim FAILS, with rounds where running is overwritten to failed. So the race test catches a broken lock.
  • P6, every save error treated as a lost guard: WriteErrorKeepsPending FAILS.

Fail-before, on base: GuardsFailedWriteOnPending and LostGuardIsNoOp panic, ApproveDuringEmailSend fails with the row "failed", and ConcurrentWithApproveClaim fails.

Verified locally, with GOTOOLCHAIN=go1.26.10 and no cloud credentials, under the heavy lock:

  • go build ./..., and go vet ./internal/api/ plain and with -tags integration: OK. gofmt -l: clean.
  • go test -count=1 ./internal/api/ ./internal/mocks/: ok.
  • -tags integration -run TestFinalizePurchaseStatus (testcontainers): PASS. The claim won 27 of 40 race rounds.
  • -race -run TestFinalizePurchaseStatus: ok.
  • -tags integration -run 'TestPersistExecution|Retry|ExecutePurchase|Submit': ok.
  • Merged tree:
    • git merge-tree --write-tree origin/main 3086bc87 is clean.
    • Build, vet, the api unit tests and the integration Finalize tests pass on it.
  • No migration.

Reviewed by an independent agent with no author context (second reviewer). Review only: nothing was pushed, and the GitHub review state is unchanged.

@cristim

cristim commented Oct 25, 2026

Copy link
Copy Markdown
Member

VERDICT: REQUEST_CHANGES 3086bc8
(Code approved at this SHA; the only blocker is the PR body, an edit that leaves the SHA unchanged.)

Merge check: merge-tree vs origin/main 2ded648 is clean (tree 2993141d); go build ./... and go vet -tags integration ./internal/api/ pass on the merged tree.

Findings:

Attacked:

  • The guard: SaveExecutionIfStatus takes SELECT ... FOR UPDATE, then checks the status, then does a full-row write in one tx (store_postgres_execution_guarded.go:44-63). It serializes with the TransitionExecutionStatus claim. Expected is ["pending"] only, which is correct for both the submit and retry callers (2299, 3622), since both write fresh pending rows.
  • Lost guard: the caller's copy is not mutated, nothing is written, and the response carries the re-read status.
  • Unit: go test -run 'TestFinalizePurchaseStatus|Retry|ExecutePurchase' ./internal/api/ passed 64 tests.
  • Integration on the merged tree (testcontainers PG, under heavy.lock): TestFinalizePurchaseStatus_ApproveDuringEmailSend and _ConcurrentWithApproveClaim pass (claim won 16/40), and TestPersistExecution* and Notified* pass.
  • Mutation M1, revert to unguarded SavePurchaseExecution: both integration race tests FAIL. Killed.
  • Mutation M2, widen expected to {pending,running}: the 2 race tests and GuardsFailedWriteOnPending FAIL. Killed.
  • Mutation M3, lost guard returns "failed": LostGuardIsNoOp subtests FAIL. Killed.
  • CI at 3086bc8: all checks green, Snyk skipped.

@IAmJSD

IAmJSD commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: REQUEST_CHANGES 3086bc87cacae79c003011a6dc3882d6ad675b7d

Blocking:

Non-blocking:

  • none. I would approve the code as written.

Attacked:

  • The race itself: SaveExecutionIfStatus takes the FOR UPDATE row lock (lockExecutionRow), checks the status and writes in one transaction. The approve claim is an UPDATE ... WHERE status IN (pending, notified) CAS, so Postgres serializes the two and exactly one wins. Both callers create the row as pending (newPendingExecution and persistRetryExecution), and nothing between insert and finalize moves it off pending. So the guard cannot spuriously lose on the normal email-failure path, and the failed write still happens there. On a lost guard the caller's in-memory copy is left alone and the response reports the row's real status. A failed re-read falls back to pending. Other errors keep the old best-effort contract.
  • Fail-before: I reverted only internal/api/handler_purchases.go to the merge-base and kept the new tests. TestFinalizePurchaseStatus_ApproveDuringEmailSend then FAILs (row failed, want running, with the error text stamped on it), and TestFinalizePurchaseStatus_ConcurrentWithApproveClaim FAILs on several rounds where a won claim was overwritten with failed. At the head both pass (claim won 21 of 40 rounds, invariant held in each).
  • Ran: go build ./..., go vet ./internal/api/ with and without -tags integration, go test -race -short -count=1 ./internal/api/ (ok), and go test -race -tags integration -count=1 -run 'Purchase|Retry|Finalize|Submit|Persist' ./internal/api/ against testcontainers Postgres (ok, 390s). golangci-lint --allow-parallel-runners with and without --build-tags integration found 0 issues, gocyclo -over 10 internal/api/handler_purchases.go is silent, and the secrets-scan printed "clean; planted key flagged".

Body-stale:

Reviewer: rA3. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

@IAmJSD

IAmJSD commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Body updated for the review at 3086bc87: the sentence that GitHub would have read as closing #1232 is reworded, so #1232 stays open on merge. The stale claim that #1110 is still open is corrected: #1110 merged on 2026-10-23, so item 2 now needs its own fix against main. The code is unchanged (head still 3086bc87).

@IAmJSD

IAmJSD commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE 3086bc87cacae79c003011a6dc3882d6ad675b7d

Supersedes my earlier REQUEST_CHANGES on this same head (3086bc87). That review had one blocker, the PR body, and it is now resolved. The code is unchanged, so my earlier code review and checks still apply.

Blocking:

Non-blocking:

Attacked (from my earlier review of this sha, still valid since the code is unchanged):

  • SaveExecutionIfStatus takes the FOR UPDATE lock, checks the status and writes in one transaction, so Postgres serializes it with the approve claim CAS. Both callers create the row as pending, so the guard does not lose on the normal email-failure path.
  • Fail-before: I reverted only internal/api/handler_purchases.go to the merge-base, and TestFinalizePurchaseStatus_ApproveDuringEmailSend and _ConcurrentWithApproveClaim both FAIL. At the head they pass.
  • Ran: go build, go vet with and without the integration tag, go test -race -short ./internal/api/, and the purchase-related integration tests (-run 'Purchase|Retry|Finalize|Submit|Persist', ok). golangci-lint with and without the integration tag found 0 issues, gocyclo is silent, and the secrets-scan was clean.
  • For this re-check: I re-read the PR body, closingIssuesReferences and the branch's commit messages at 3086bc87.

Body-stale:

Reviewer: rA3. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

@cristim

cristim commented Oct 25, 2026

Copy link
Copy Markdown
Member

Local verification at 3086bc87cacae79c003011a6dc3882d6ad675b7d merged onto current main (51957bcf; merge-tree f1bbf632; macOS):

  • go build ./..., go vet -tags integration ./... and go test -race -short ./... pass (full suite).
  • go test -tags integration passes on real Postgres (testcontainers) for the touched packages: ./internal/api
  • Nothing was applied to any cloud.

@cristim

cristim commented Oct 25, 2026

Copy link
Copy Markdown
Member

Marked ready and merged by the owner's agent after 2 independent reviews and local verification (see the review and verification comments above, all at 3086bc87cacae79c003011a6dc3882d6ad675b7d).

@cristim
cristim marked this pull request as ready for review October 25, 2026 21:01
@cristim
cristim merged commit a59740d into LeanerCloud:main Oct 25, 2026
33 of 34 checks passed
IAmJSD added a commit to IAmJSD/cloud-commitments-platform that referenced this pull request Oct 26, 2026
This branch carried LeanerCloud#1237's commit (17654cc), which main now has as
the squash a59740d. Conflicts in handler_purchases.go and the two
finalize test files were resolved by taking main's side for LeanerCloud#1237's
changes and re-applying only this PR's own commit (91827b9) on top, so
the PR diff against main is exactly that commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/s Hours impact/few Limited audience priority/p1 Next up; this sprint severity/high Significant harm triaged Item has been triaged type/bug Defect urgency/this-sprint Within the current sprint

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants