Repository navigation
fix(marketplace): persist the listing ClientToken before the AWS call - #633
Merged
Merged
Conversation
The ClientToken was derived from the request, so a retry across a month boundary (the default price schedule depends on the remaining months) produced a new token and AWS silently created a second listing. A request that read the row before another one claimed it could also reuse the token of a canceled listing. Persist a random token and the price schedule in the UPDATE that claims the slot, before the AWS call (migration 000105). A retry resends the stored token and schedule. A pending row that holds a token can be claimed again; the retry first asks AWS (DescribeReservedInstancesListings) whether the earlier attempt created the listing and records it instead of creating another. Recording a listing, or a definitive AWS rejection, drops the token; an ambiguous error keeps it. Refs #525
Contributor
|
Warning Review limit reached
This review includes 18 billable files and costs up to $4.50.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 23 minutes for your next included review. View limit detailsLimit details: You’ve used the included review currently available. Your 68 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Review configuration: ⚙️ Run configuration
📒 Files selected for processing (18)
Comment |
A claim that raced another one derived Resumed and the prior state from a stale snapshot, so it skipped the reconcile and sent its own schedule with the first request's token. Lock the row first and derive Resumed from the returned token. Fence the release on the persisted token so a request only drops the attempt it owns. Also fix the lint findings (errcheck in the mock, rangeValCopy, gosec G117 on the token field). Refs #525
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #525 (PR 2 of 3; PR 3 keeps cancel/recover for pending rows, cancel-then-DB-failure retry, empty AWS status and the detached ctx for the release).
PR1 (#604) derived the ClientToken from the request. That left two gaps, both reproduced on current main through HandleRequest with a fake AWS that dedups on ClientToken (3 failing tests on main, run before this change):
TestMarketplaceListHTTP_RetryAcrossMonthBoundaryDoesNotDuplicate)...._CrashBetweenCreateAndPersistIsReconciled). The stale-row case:row.ListingIDis read before the claim, so a request holding a stale row replays a canceled listing and gets a 502 (..._StaleRowAfterCancelGetsNewListing).Change:
purchase_history.listing_client_token(TEXT) andlisting_price_schedule(JSONB), both nullable (up and down; integration test added). Next free number: 000104 is the highest on main and no open PR touches migrations.ClaimMarketplaceListingSlotwrites a random token and the schedule in the same conditional UPDATE that claims the slot, before the AWS call, and returns the claimed row's listing id, prior state, token and schedule (RETURNING). The handler uses only that, so a stale pre-claim row is never consulted. The derived-token function is removed.UpdatePurchaseHistoryListing(any recorded listing) clears the token and schedule; a newReleaseMarketplaceListingClaimrestores the prior state and keeps the attempt after an ambiguous error, drops it after a definitive client fault (400, AWS created nothing).Decisions the issue left open:
FindMarketplaceListingByToken), because the ec2 service wrapper in cloud-commitments-go has no lookup by token; no contract change there.mapAWSMarketplaceError.Tests: handler tests drive HandleRequest against an in-memory row with the store's claim/release/record semantics and a fake AWS (dedup on token, lost responses, parameter mismatch). Mutations that fail them: token not persisted before the AWS call (and persisted after), token re-derived on retry, schedule re-derived on retry, reconcile skipped, release always dropping the attempt. The claim/release SQL has a Postgres integration test (CI; also run locally against a throwaway Postgres 17 together with the 000105 up/down test, both passing). Gates: go test -race -short ./..., go vet ./..., go vet -tags integration ./internal/..., errcheck -blank, shadow, unparam, gocyclo, misspell -locale US on the touched code.
Review follow-up (second commit):
had_tokenand the prior state from a stale snapshot, so it skipped the reconcile and sent its own schedule with the first request's token. The claim now locks the row first (FOR UPDATEin a CTE) and derivesResumedfrom the returned row (its token differs from the one this claim tried to write). Integration testcheckConcurrentMarketplaceClaimholds a first claim in an open transaction, blocks a second claim on the row lock, commits, and asserts a resumed claim with the first token and schedule. Run against a throwaway Postgres 17: passes; with the old snapshot behavior it fails.ReleaseMarketplaceListingClaimtakes the token and only touches a pending row that holds it. A stale drop or keep release is a no-op (checkMarketplaceReleaseIsFenced; the mutant without the token condition fails it).MarketplaceListingClaim.price_schedulein the list response (marketplace endpoints are not in openapi.yaml, so nothing to update there).AmbiguousAWSErrorRetryDoesNotDuplicateListing->TestMarketplaceListHTTP_RetryAcrossMonthBoundaryDoesNotDuplicateand..._RetryReusesPersistedTokenAndSchedule;RetryAfterRolledBackPersistFailureGetsNewListingandReturnedCanceledStateIsAnError-> same-namedTestMarketplaceListHTTP_*;TestMarketplaceClientTokenand its replaying fake have no equivalent because the derivation function is gone (the fake AWS now checks the token and schedule are on the row before the call).