Repository navigation
fix(server): SQS batch failure redelivers already-processed purchase messages (no BatchItemFailures) #108
Description
Reviewed commit: be11bdcb5. Note: origin/main moved to 3e9660d06 during the review; re-verify against current main before changing code, since a finding may have been fixed or moved.
Where
internal/server/lambda.go:226-248(handleLambdaSQSEvent), specifically the single batch-level error return atinternal/server/lambda.go:243-245internal/purchase/messages.go:113(theexecution not foundpath that produces the poison message)- No
BatchItemFailuresorReportBatchItemFailuresanywhere in the repo:git grepoverorigin/mainreturns zero hits, in Go source and in the Terraform event-source mappings alike
What
The handler loops over sqsEvent.Records, collects failed message IDs, and then returns a single error for the whole batch. Lambda's SQS integration treats a non-nil error as "the entire batch failed", so every record in the batch, including the ones HandleSQSMessage already processed successfully, becomes visible again on the queue.
Partial batch responses are the mechanism that exists to prevent this, and the repo has neither half of it: no events.SQSEventResponse return type in the handler, and no function_response_types = ["ReportBatchItemFailures"] on the event-source mapping.
Failure scenario
A batch of 5 messages arrives. Messages LeanerCloud/cloud-commitments-cli#1 to LeanerCloud/cloud-commitments-cli#4 are approve messages that succeed and drive real purchase-execution state transitions. Message LeanerCloud/cloud-commitments-cli#5 references a deleted execution and fails with execution not found (internal/purchase/messages.go:113).
handleLambdaSQSEvent returns an error, so SQS redelivers all 5.
On redelivery, LeanerCloud/cloud-commitments-cli#1 to LeanerCloud/cloud-commitments-cli#4 hit checkAutoExecuteGate and the status CAS, and now fail with "not eligible" or a status mismatch, because they already ran. That re-fails the batch, which redelivers again, and so on until maxReceiveCount moves all 5 to the DLQ.
Two consequences. First, the operator investigating the DLQ sees four phantom failures for purchases that actually completed, which is the worst possible signal during a money-path incident: the obvious remediation (replay the DLQ) is the wrong one. Second, the single poison message keeps the whole batch churning through the redrive budget, so genuinely retryable failures in the same batch burn their retries on someone else's poison.
Idempotency downstream is the only thing keeping this from being a straight double-spend: the CAS on the execution status is what turns the redelivery into a no-op rather than a second purchase. Any handler reachable on this path that is not CAS-guarded turns the same redelivery into a duplicate money action.
Fix direction
- Return
events.SQSEventResponse{BatchItemFailures: [...]}containing only the message IDs that actually failed, and return a nil error. - Set
function_response_types = ["ReportBatchItemFailures"]on the SQS event-source mapping in the Terraform module, or the response is silently ignored and the behaviour does not change. - Regression test: a batch where record N fails must leave records 1..N-1 acknowledged, asserted on the returned
BatchItemFailuresset rather than on the error value.
Point 2 is the easy one to miss: shipping only the Go change looks correct in review and in unit tests while behaving exactly as it does today in production.
Activity
cristim commentedon Sep 28, 2026
Fixed the Go-side defect in #382:
handleLambdaSQSEventnow returnsevents.SQSEventResponse{BatchItemFailures: [...]}naming only the records that actually failed, with a nil error, instead of one aggregate error for the whole batch.Terraform side could not be set: this repo's IaC provisions no SQS queue and no Lambda event source mapping at all. Confirmed with a full-repo search (
rg) foraws_sqs_queue,AWS::SQS::Queue,EventSourceMapping, andbatch_sizeacrossterraform/,cloudformation/,iac/, andarm/-- zero hits, matching the originating audit's own caveat atdocs/audits/codebase-audit-2026-09-02.md:6025which called the scenario "PLAUSIBLE" rather than confirmed for the same reason.function_response_types = ["ReportBatchItemFailures"]cannot be added to a Terraform resource that doesn't exist in this repository. Provisioning that queue (batch size, DLQ, redrive policy, visibility timeout, IAM grant) is a separate infrastructure decision. If the queue is provisioned outside this repo (private ops process, manual setup), whoever owns that needs to add the setting there for this fix to take effect in production.fix(server): fail closed on SQS partial batch response without the flag