fix(purchases): EC2 RI purchase times out under transient AWS SDK retries; Lambda 60s budget blown (P0) #667

Symptom

QA reproduced on PR #373's synchronous approve path:

Failed to approve: execution 3419bb8b-9785-4e8c-ae05-2193202d58af could not be approved:
some purchases failed: [t4g.nano: purchase failed: failed to find offering:
failed to describe offerings: operation error EC2: DescribeReservedInstancesOfferings,
https response error StatusCode: 0, RequestID: , canceled, context deadline exceeded]

StatusCode: 0, RequestID: "" = the HTTP request was cancelled at the network layer before AWS returned. Lambda function timeout = 60s.

Compounding symptom: execution status is left stuck at approved (not failed) — already tracked as #632 ("approve strands execution in 'approved' on interrupted sync execution"). Fix landing on this issue would unblock the time budget but #632 covers the state-machine repair separately.

Root cause

The AWS SDK v2 EC2 client has default behavior: 3 retries with exponential backoff, each up to a 30s request timeout. A single transient slow/failed DescribeReservedInstancesOfferings call can consume 30s × 3 + backoff = ~90-120s of wall clock. The synchronous approve handler is bounded by the Lambda invocation budget (60s on dev, see terraform/environments/aws/github-dev.tfvars:22 lambda_timeout = 60). Result: the SDK is still on its second retry when Lambda's deadline fires, the parent context cancels, the SDK reports "canceled, context deadline exceeded" on the in-flight HTTP call, and the purchase manager aggregates that as some purchases failed: [<rec>: ...].

This is NOT a pagination issue — buildOfferingFilters correctly narrows by instance-type + product-description + tenancy + scope + duration + offering-class (6 filters), so the result set is typically a single page.

Fix options (any one or combination)

1. Increase Lambda timeout for purchase endpoints — e.g. 300s. Cheapest fix; trades cost (longer invocations) for resilience. Update terraform/environments/aws/github-*.tfvars lambda_timeout. Note: API-Gateway-fronted paths have a 30s ceiling that we'd hit instead; Function URL has no such cap. Verify the deployment target.
2. Tighten SDK retry config on purchase-path clients — pass aws.Retryer with aws.WithMaxAttempts(2) and per-request http.Client{Timeout: 10*time.Second} so a transient slow API call fails fast (10s × 2 = 20s) and surfaces a retriable error to the user instead of stranding the execution. Concrete location: providers/aws/services/ec2/client.go NewClient (and the parity files for other services).
3. Make approve async again (revert fix(purchases): approve now executes synchronously and parallel across recs (closes #372) #373's synchronous semantics) and pick the purchase up via a worker / scheduled-task Lambda with a much longer timeout. Larger refactor but architecturally cleanest — the Lambda invocation budget should not be the purchase budget.

Recommendation

Ship (1) + (2) together as the immediate fix: bump dev/staging/prod lambda_timeout to 300s for the api function AND set EC2/RDS/ElastiCache/MemoryDB/OpenSearch/Redshift/SavingsPlans client retry config to MaxAttempts=2 + per-request 10s HTTP timeout. (3) is a separate architectural cleanup.

Acceptance criteria

• lambda_timeout raised for the api Lambda in all envs (or per-route override if possible)
• EC2/RDS/ElastiCache/MemoryDB/OpenSearch/Redshift/SavingsPlans AWS SDK clients configured with aws.WithRetryMaxAttempts(2) + per-request HTTP timeout ≤ 15s
• Regression test in providers/aws/services/ec2/client_test.go asserting that a slow-API mock fails within ≤30s wall clock instead of running to the SDK's default 90s+
• No regression in the rec-collection flow (which can be slow legitimately and should keep its current retry budget — confirm by reading providers/aws/recommendations/)

Cross-references

• fix(purchases): approve now executes synchronously and parallel across recs (closes #372) #373 (introduced synchronous approve)
• fix(purchases): every AWS purchase fails with 'invalid service details' after #373 (P0) #453 / PR fix(purchases): persist & round-trip full ServiceDetails (closes #453) #454 (codec round-trip)
• fix(frontend/api): Recommendation type missing details field — every dashboard purchase POSTs without per-rec details, silently defaults to Linux/Region/default #597 / PR fix(frontend/api): preserve details + engine on purchase POST (closes #597) #600 (frontend details preserve)
• fix(providers/aws/parser): EC2 parser writes "shared"/"region"/"availability-zone" — AWS RI filter API expects "default"/"Region"/"Availability Zone" #598 / PR fix(providers/aws): use canonical EC2 RI enum casing for Tenancy + Scope (closes #598) #599 (EC2 enum casing)
• fix(purchases): single-account path passes nil provCfg — Azure/GCP purchases fail with DefaultAzureCredential chain errors (P0) #602 / PR fix(purchases): resolve per-account credentials in single-account path (closes #602) #603 (single-account creds)
• bug(purchases): approve strands execution in 'approved' (purchase never runs, no error) on interrupted sync execution #632 (status stranding when the sync approve hits this kind of failure)