Skip to content

feat(aws): reconcile EC2/Redshift purchases against AWS after an unknown outcome #298

New issue
New issue
Closed
#708
Closed
feat(aws): reconcile EC2/Redshift purchases against AWS after an unknown outcome#298
#708
Labels
effort/mDaysimpact/fewLimited audiencepriority/p2Backlog-worthyseverity/mediumModerate harmtriagedItem has been triagedtype/bugDefecturgency/this-sprintWithin the current sprint

Description

@cristim
cristim
opened on Oct 9, 2026

Summary

After go#297 (bcc0dfc) an EC2 or Redshift purchase whose response is lost returns an error wrapping purchasecfg.ErrOutcomeUnknown, and the platform (PR cloud-commitments-platform#778) refuses to re-drive such rows. The operator then has to check the AWS account by hand. Reconcile automatically instead.

Current behaviour

purchasecfg.ErrOutcomeUnknown lives in providers/aws/internal/purchasecfg, so the platform cannot detect it with errors.Is; the persisted error is a flattened string. Reaped and stranded rows carry no error at all. Retry after a clean 4xx failure (for example "no offerings found") is refused too, because the platform cannot tell it from an ambiguous one.

Steps to verify the gap

Fail an EC2 purchase with a dropped connection; the row is failed, Retry returns 409 redrive_unsafe, and the only path forward is a manual check plus a new purchase.

Expected behaviour

Before any re-buy, look up what AWS holds (DescribeReservedInstances / DescribeReservedNodes, filtered by offering id, count, state and start time at or after the execution start) and mark the execution completed, or allow a retry only when nothing matches.

Proposed fix

  1. Export the outcome-unknown sentinel from pkg/common (or a public package) and keep it through the PurchaseCommitment result so the platform can persist a typed marker instead of a string.
  2. Add a reconcile call in the AWS provider (or platform) using the read calls above; confirm ec2:DescribeReservedInstances and redshift:DescribeReservedNodes are in the target-account IAM templates.
  3. Platform: relax recRedriveRefusalReason for a row whose reconcile found no matching commitment.

References

Parent: go#296, go#297 (bcc0dfc), platform PR #778. Related: #702 (ec2:CreateTags in member-account roles, the untagged-commit gap).

Severity: medium (the refusal is safe; this removes manual work and the lost-retry cost).

Activity

  1. cristim
    added
    priority/p2Backlog-worthy
    severity/mediumModerate harm
    triagedItem has been triaged
    urgency/this-sprintWithin the current sprint
    impact/fewLimited audience
    effort/mDays
    type/bugDefect
    on Oct 9, 2026

  2. IAmJSD commented on Oct 16, 2026

    @IAmJSD
    IAmJSD
    on Oct 16, 2026
    Collaborator

    Released: needs owner design decisions first (proposal below)

  3. IAmJSD commented on Oct 16, 2026

    @IAmJSD
    IAmJSD
    on Oct 16, 2026
    Collaborator

    Released: this needs owner decisions before code. I'm releasing the claim so it doesn't block anyone, and posting a design proposal below for you to decide on. Happy to pick it up again once you've decided.

    Already done

    Decisions needed

    1. How to match on EC2. types.ReservedInstances has no offering ID, and DescribeReservedInstances can't filter by one. So "filter by offering id" works only for Redshift (ReservedNodeOfferingId). For EC2 it would have to be an attribute match: instance type, product description, duration, offering class, offering type, tenancy, scope/AZ, fixed price, InstanceCount == rec.Count, and start within a window. Is that set acceptable?

    2. Which offering to match against. Running findOffering again at reconcile time can return a different offering than the one sent. Exact matching needs PurchaseResult to carry the attempted offering ID and attributes on an unknown outcome (a new public field), and the platform to store them.

    3. Can reconcile ever say "not purchased"? Describe results lag fresh purchases (the code already retries CreateTags on InvalidReservationID.NotFound), clocks can drift, and an RI can be modified or split inside the window. Options:

      • (a) Never: return only Found or Unknown, and keep retry manual.
      • (b) NotFound only after a settle window (for example 15 minutes or more), with every page read and no candidate in any state, including retired and payment-failed.
      • (c) NotFound only with operator confirmation.

      I recommend starting with (a).

    4. Ambiguity. I'd return Unknown in each of these cases:

      • more than one candidate in the window (for example a manual purchase of the same offering);
      • the count doesn't match;
      • the only candidate is payment-failed.

      I'd also exclude candidates tagged with a different idempotency token. Should exactly one untagged match count as Found? Getting that wrong under-buys rather than double-buys.

    5. Tag writing on Found. Should reconcile add the idempotency and CUDly tags on a best-effort basis (ec2:CreateTags, see platform#702) so later re-drive guards work? The verdict wouldn't depend on the tag write succeeding.

    6. API shape. I'd add an optional interface in pkg/provider rather than a new ServiceClient method, which would break every implementer and mock:

      • PurchaseReconciler.ReconcilePurchase(ctx, ReconcileRequest) (ReconcileResult, error)
      • ReconcileRequest: rec, IdempotencyToken, AttemptedOfferingID, AttemptStartedAt, AttemptEndedAt
      • ReconcileResult: Verdict, CommitmentID, Reason

      An empty Verdict, and any error, both mean Unknown.

    7. Where it lives. In the library, which owns the SDK calls and fakes. The platform only calls it.

    Fail-closed matrix (applies to every option)

    Situation Verdict
    Describe error, STS/account error, pagination cap, cancelled context Unknown
    Region mismatch, missing attempt time or offering, empty verdict Unknown
    More than one candidate, count mismatch, only payment-failed Unknown
    Exactly one match in active, payment-pending or queued state Found
    No candidates Unknown under (a); NotFound only under (b) or (c)

    Proposed split once you've decided

    This follows the GOWORK=off pin constraint, the same way #352 and #353 were split.

    • PR A (pkg): reconcile types, the optional interface, and PurchaseResult.AttemptedOfferingID.
    • PR B (providers/aws, after the pkg pin bump): EC2 and Redshift implementations, with fake-API tests and a mutation probe on every Unknown branch.
    • Platform follow-up:
      • store the attempted offering and attempt time;
      • call reconcile before any re-drive;
      • relax recRedriveRefusalReason only on NotFound;
      • allow retry after a definite 4xx via errors.Is;
      • check the IAM templates for ec2:DescribeReservedInstances and redshift:DescribeReservedNodes.

    Related: #443 (typed unknown result for exchanges, the same pattern) and #107 (Redshift tagging contract). The payment-failed and RI-modification behavior above comes from reading the SDK docs, not from AWS testing.

    🤖 Generated with Claude Code

  4. cristim commented on Oct 17, 2026

    @cristim
    cristim
    on Oct 17, 2026
    MemberAuthor

    Decision needed (one line): can reconcile ever return NotFound? The seven open design questions are in the earlier comment; step 1 is done (#329, #335). Safe default: Found/Unknown only (one untagged match counts as Found, any ambiguity is Unknown), then PR A (pkg interface plus PurchaseResult.AttemptedOfferingID) and PR B (aws, after the pin bump). Marking status/needs-info until that is confirmed.

  5. cristim
    added
    status/needs-infoAwaiting clarification from the asker
    on Oct 17, 2026

  6. cristim commented on Nov 5, 2026

    @cristim
    cristim
    on Nov 5, 2026
    MemberAuthor

    Owner decision (2026-11-05): Option A. Reconcile returns Found or Unknown only. One untagged match is Found; any ambiguity is Unknown, and the operator confirms manually. NotFound is never returned, so no automated re-buy. Next: implement reconcile with the Found/Unknown contract.

  7. cristim
    removed
    status/needs-infoAwaiting clarification from the asker
    on Nov 5, 2026

  8. cristim commented on Nov 5, 2026

    @cristim
    cristim
    on Nov 5, 2026
    MemberAuthor

    Claimed by claude-cc-libs: implementing the owner's option A. Reconcile returns Found or Unknown only: a single untagged match is Found, any ambiguity is Unknown and needs manual confirmation, and NotFound is never returned, so nothing re-buys automatically. Planning now, with two review passes before code. Builds wait for the current resource freeze to lift.

  9. cristim
    added 3 commits that reference this issue on Nov 6, 2026

    feat(pkg): add PurchaseReconciler and Found/Unknown reconcile result (#…

    5a9b4c5

    fix(aws): reject empty node type and bad payment option in Redshift r…

    9b7ef42

    feat(aws): reconcile EC2 and Redshift purchases after an unknown outc…

    5dcf711
  10. cristim
    closed this as completedin #708on Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort/mDaysimpact/fewLimited audiencepriority/p2Backlog-worthyseverity/mediumModerate harmtriagedItem has been triagedtype/bugDefecturgency/this-sprintWithin the current sprint

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions