Skip to content

fix(server): gzip large JSON API responses on Lambda - #775

Merged
cristim merged 1 commit into
mainfrom
fix/723-compress-lambda-json-responses
Oct 9, 2026
Merged

cristim merged 1 commit into
mainfrom
fix/723-compress-lambda-json-responses

Conversation

@cristim

@cristim cristim commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Summary

  • The Lambda adapter sent API JSON uncompressed, so GET /api/recommendations for a large tenant exceeds the 6 MB Function URL response limit (502 for all users). CloudFront compression only covers the edge-to-viewer hop.
  • New compressLambdaResponse gzips JSON bodies of 64 KiB or more when the request accepts gzip and returns them base64-encoded with Content-Encoding: gzip and Vary: Accept-Encoding. Skipped for small, non-JSON, already-encoded or base64 responses and for clients without gzip. Wired into handleLambdaHTTPEvent for API responses only.
  • This is the smallest change that removes the failure for realistic sizes without touching the frontend or the API contract. It raises the ceiling but does not bound it; pagination and slimmer rows are tracked in feat(recommendations): bound GET /api/recommendations with pagination and slimmer rows #777 (scoping found the dashboard and Opportunities page compute totals client-side over the full set, and details is only ~a quarter of a row, so those are larger changes).

Verification

  • Unit tests: round-trip gunzip equals the original body, headers set, and each skip condition leaves the response untouched.
  • Wiring test through the real handleLambdaHTTPEvent fails when the call in lambda.go is removed (status 401 body not encoded) and passes with it (threshold lowered for the test).
  • go vet and go test ./internal/server pass on go1.26.9.
  • Not verified: behaviour on a real Lambda Function URL behind CloudFront (no deploy); the compressed size of a real production payload. Reviewer should check that Function URLs accept a base64 gzip body with Content-Encoding set, as the repo already does for binary static files.

Refs #723

🤖 Generated with Claude Code

GET /api/recommendations returns the whole cached set, and the Lambda
Function URL rejects responses over 6 MB, so a large tenant gets a 502
on the main page (#723). The Lambda adapter did not compress; CloudFront
compression only covers the edge-to-viewer hop, not the Lambda response.

- compressLambdaResponse gzips JSON bodies of 64 KiB or more when the
  request accepts gzip, returning them base64-encoded with
  Content-Encoding: gzip and Vary: Accept-Encoding.
- Skipped for small bodies, non-JSON, already-encoded or base64
  responses, and clients that do not accept gzip.
- Wired in handleLambdaHTTPEvent for API responses only; static files
  and OIDC are unchanged.

This raises the ceiling several-fold but does not bound the response;
pagination and slimmer rows are tracked separately.

Refs #723
@cristim cristim added urgency/this-sprint Within the current sprint triaged Item has been triaged priority/p1 Next up; this sprint severity/high Significant harm impact/all-users Affects every user effort/m Days type/bug Defect labels Oct 9, 2026
@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

This review includes 3 billable files and costs up to $0.75.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 8 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 85 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: LeanerCloud/cloud-commitments-platform/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: 506f161d-928d-44c0-a89c-2c83f64898ec

📥 Commits

Reviewing files that changed from the base of the PR and between 225ad35 and c875dc9.


📒 Files selected for processing (3)
  • internal/server/lambda.go
  • internal/server/lambda_compress.go
  • internal/server/lambda_compress_test.go


  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@cristim

cristim commented Oct 9, 2026

Copy link
Copy Markdown
Member Author

Independent review (cc-cli-w2; I did not write this PR) at c875dc940de07e7f97f8b7fe0331c4aeb25b0a26, base 225ad350.

Verdict: CLEAN, no blocking findings. There are 3 non-blocking nits, and CI was still pending at the head when I checked.

What I checked:

  • Wiring test fails before the fix. I removed the compressLambdaResponse(&request, resp) call from handleLambdaHTTPEvent (a -overlay mutation, the pre-fix wiring). TestHandleLambdaHTTPEvent_CompressesAPIJSONResponse then FAILS, and the two unit tests still pass. At the head, all 3 PR tests pass and go test ./internal/server passes in full (Go 1.26.9, -count=1, no -race).
  • The test drives the real handler. The wiring test goes through the actual api.NewHandler path: an unauthenticated GET /api/recommendations returns a 401 JSON body, which is compressed only because the threshold is lowered to 1 byte.
  • Other paths are unaffected.
    • OIDC (HandleOIDC) and static files (serveLambdaStatic) return before the new call.
    • API errors (err != nil) are returned untouched.
    • SQS and scheduled events never reach handleLambdaHTTPEvent.
  • CloudFront and the Function URL. Production reaches the Function URL only through CloudFront: the origin access control signs every request, and default_cache_behavior sets compress = true with legacy forwarded_values (terraform/modules/frontend/aws/main.tf:60-92). AWS's documentation (Serve compressed files) says:
    • With compression on, CloudFront "includes the Accept-Encoding header in the cache key and in origin requests automatically", so the missing Accept-Encoding in the forwarded-headers list doesn't matter.
    • It strips values other than gzip/br. Brotli needs cache policies, so with legacy settings the origin sees gzip.
    • "When a response from an origin includes the Content-Encoding header, CloudFront doesn't compress the object, regardless of the header's value", so nothing is encoded twice.
    • The Function URL response format honours isBase64Encoded, the same mechanism serveLambdaStatic already uses for binary files.
  • Header casing. isJSONResponse and acceptsGzip match keys case-insensitively. A probe with a lowercase content-type: application/json; charset=utf-8 compresses correctly.
  • Size headroom. A probe on a 5.5 MB JSON array compressed to 166 KB after base64. My data is more repetitive than real rows, so treat that ratio as an upper bound on the benefit. Any ratio better than about 0.75 still keeps a body under 6 MB, because base64 adds 33%.

Non-blocking nits:

  1. acceptsGzip treats gzip;q=0 as acceptance (a probe confirmed it compresses). RFC 9110 §12.5.3 says q=0 means "not acceptable". Browsers don't send that, and CloudFront normalizes the header, so the practical impact is nil. A one-line q-value check would make it exact.
  2. Vary is written by overwriting the key as "Vary". A response that already had a lowercase vary: Origin would end up with both keys (a probe showed Vary:Accept-Encoding alongside vary:Origin). No API path sets Vary today (grep finds none), so this is latent. Appending case-insensitively would future-proof it.
  3. application/problem+json and other +json types aren't compressed. The API emits none today, so this is informational.

Not verified: a real Lambda deploy, as the author also notes. A deployed check after merge would confirm a gzipped body on a large GET /api/recommendations through CloudFront.

@cristim

cristim commented Oct 9, 2026

Copy link
Copy Markdown
Member Author

Independent review: APPROVE at c875dc940de07e7f97f8b7fe0331c4aeb25b0a26

Reviewer: archera-quotes-codex. I did not write this PR. Base: 225ad350. Refs #723; pagination and slimmer rows are #777.

Behaviour, checked against source at this SHA:

  • compressLambdaResponse runs only on the API branch of handleLambdaHTTPEvent. It does not run on statics (serveLambdaStatic returns earlier) or on an error return.
  • It skips the response when any of these holds:
    • the response is nil or already base64;
    • the body is under 64 KiB;
    • the client does not accept gzip (case-insensitive header key);
    • the response is not application/json;
    • a Content-Encoding is already present, so nothing is encoded twice.
  • A nil headers map has no content-type, so it is skipped before any write and cannot panic.
  • The output is a standard-base64 gzip body with IsBase64Encoded=true and Content-Encoding: gzip. That is the same mechanism serveLambdaStatic already uses for binary bodies through the Function URL.
  • Vary is set to Accept-Encoding. The API sets no Vary header today, so nothing is overwritten.

CloudFront (terraform/modules/frontend/aws/main.tf, default behavior):

  • compress = true with legacy forwarded_values, and the header whitelist does not list Accept-Encoding.
  • The AWS docs (Serving compressed files) say that when CloudFront is configured to compress, it includes the normalized Accept-Encoding in origin requests automatically. So the Lambda does see gzip behind CloudFront.
  • An origin response that already has Content-Encoding is not compressed again.
  • This is doc-based. There was no real deploy, which the body states honestly.

Reproduced (go1.26.9, GOWORK=off, -mod=readonly, no -race, source archive of this SHA):

  • go vet ./internal/server is OK and go test ./internal/server passes. All 7 skip-condition subtests, the round-trip test and the wiring test pass.
  • Mutants (go test -overlay). All are KILLED:
    • the wiring call removed from lambda.go: the wiring test fails, confirming the author's claim;
    • the accept-gzip check dropped;
    • the JSON check dropped;
    • the already-base64 check dropped;
    • the size threshold dropped;
    • the already-encoded (content-encoding) check dropped;
    • the Content-Encoding header not set;
    • IsBase64Encoded not set.

Nits (optional, P3):

  • acceptsGzip treats Accept-Encoding: gzip;q=0 as accepting gzip. That is rare and harmless behind CloudFront, which normalizes the header.
  • Vary overwrites rather than appends. That is fine today; if a future handler sets Vary: Origin, append instead.

Scope: this raises the 6 MB ceiling by roughly the gzip ratio (base64 adds about 33% back) but does not bound it. Refs rather than Closes, with #777, is the right call.

CI at this SHA: 23 SUCCESS, with Integration Tests in progress when I checked. The merge needs it green.

@cristim
cristim merged commit 6f5a618 into main Oct 9, 2026
25 checks passed
@cristim

cristim commented Oct 9, 2026

Copy link
Copy Markdown
Member Author

Merged (squash) at reviewed head c875dc9 -> 6f5a618. Reviewers: cc-cli-w2 (verdict on PR), archera-quotes-codex; both clean at this SHA. CI: 25/25 checks pass. Local: go build ./... exit 0; go test ./internal/server/... exit 0 (server, scheduledauth ok). Still owed: post-merge check on the deployed stack that a large /api/recommendations response comes back gzipped through CloudFront (not run by the merge operator).

IAmJSD added a commit to IAmJSD/cloud-commitments-platform that referenced this pull request Oct 16, 2026
…ty paging params

A page was bounded only by row count, but the opaque Details blob makes a
row's size data-dependent, so 1000 large rows could still pass the 6 MiB
Lambda response cap. Pages now also stop once the rows, measured as they
appear inside the Lambda response envelope (the body re-encoded as a JSON
string, so quotes and backslashes count twice), plus the summary, regions,
keys and next_cursor would pass a 4 MiB budget. The budget is checked
against uncompressed bytes because the LeanerCloud#775 gzip only runs when the client
accepts it. A page always holds at least one row, so a row that alone is
over the budget is returned on its own page with a logged warning instead
of stalling the cursor walk.

Legacy unpaginated mode is now selected only when neither limit nor
cursor is present. A present but empty ?limit= or ?cursor= returns 400
instead of falling back to the unbounded response.

Refs LeanerCloud#723

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IAmJSD added a commit that referenced this pull request Oct 25, 2026
…dations (#974)

* fix(api): add opt-in keyset pagination to GET /api/recommendations

The list endpoint returned the whole filtered recommendations cache in a
single body, so a tenant with a few thousand cached rows pushed the
response past the 6 MB Lambda payload cap and the page failed with 502.
Gzip (#775) raised that ceiling but did not bound it.

Callers can now send limit (1-1000) and the returned next_cursor to read
the list in pages ordered by recommendation ID. The cursor is an opaque
base64url encoding of the last ID seen, so a row deleted between requests
does not shift later pages. summary and regions still describe the full
filtered set on every page, and compute capacity is only decoded for the
returned rows. Requests without limit or cursor keep the existing
unpaginated response, so the current frontend is unaffected until it
adopts the parameters. openapi.yaml documents the parameters and the
actual response shape.

Refs #777

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(api): cap recommendation pages by serialized bytes and reject empty paging params

A page was bounded only by row count, but the opaque Details blob makes a
row's size data-dependent, so 1000 large rows could still pass the 6 MiB
Lambda response cap. Pages now also stop once the rows, measured as they
appear inside the Lambda response envelope (the body re-encoded as a JSON
string, so quotes and backslashes count twice), plus the summary, regions,
keys and next_cursor would pass a 4 MiB budget. The budget is checked
against uncompressed bytes because the #775 gzip only runs when the client
accepts it. A page always holds at least one row, so a row that alone is
over the budget is returned on its own page with a logged warning instead
of stalling the cursor walk.

Legacy unpaginated mode is now selected only when neither limit nor
cursor is present. A present but empty ?limit= or ?cursor= returns 400
instead of falling back to the unbounded response.

Refs #723

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(api): reject duplicate recommendation IDs when paging and document 400s

- Correct the ordering comment: the paging key is the application
  recommendation ID built from the natural-key columns, not the cache
  table's UUID primary key. A paginated request now fails if two rows
  share an ID, since the strict "after" comparison would skip one of
  them at a page boundary.
- Describe the feature as opt-in in-memory cursor slicing, not a SQL
  keyset (store-side paging is #981).
- TestGetRecommendations_UnpaginatedShapeUnchanged copies the expected
  IDs before the call so an in-place reorder in legacy mode fails it.
- Add TestGetRecommendations_PaginationRejectsDuplicateIDs.
- openapi: document the 400 response for bad limit/cursor/filter
  values, and say next_cursor's absence is the only end-of-list signal.

Refs #723

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cristian Magherusan-Stanciu <cristim@users.noreply.github.com>
IAmJSD added a commit that referenced this pull request Oct 26, 2026
…d of the full list (#1240)

* fix(api): add opt-in keyset pagination to GET /api/recommendations

The list endpoint returned the whole filtered recommendations cache in a
single body, so a tenant with a few thousand cached rows pushed the
response past the 6 MB Lambda payload cap and the page failed with 502.
Gzip (#775) raised that ceiling but did not bound it.

Callers can now send limit (1-1000) and the returned next_cursor to read
the list in pages ordered by recommendation ID. The cursor is an opaque
base64url encoding of the last ID seen, so a row deleted between requests
does not shift later pages. summary and regions still describe the full
filtered set on every page, and compute capacity is only decoded for the
returned rows. Requests without limit or cursor keep the existing
unpaginated response, so the current frontend is unaffected until it
adopts the parameters. openapi.yaml documents the parameters and the
actual response shape.

Refs #777

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(api): cap recommendation pages by serialized bytes and reject empty paging params

A page was bounded only by row count, but the opaque Details blob makes a
row's size data-dependent, so 1000 large rows could still pass the 6 MiB
Lambda response cap. Pages now also stop once the rows, measured as they
appear inside the Lambda response envelope (the body re-encoded as a JSON
string, so quotes and backslashes count twice), plus the summary, regions,
keys and next_cursor would pass a 4 MiB budget. The budget is checked
against uncompressed bytes because the #775 gzip only runs when the client
accepts it. A page always holds at least one row, so a row that alone is
over the budget is returned on its own page with a logged warning instead
of stalling the cursor walk.

Legacy unpaginated mode is now selected only when neither limit nor
cursor is present. A present but empty ?limit= or ?cursor= returns 400
instead of falling back to the unbounded response.

Refs #723

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(api): reject duplicate recommendation IDs when paging and document 400s

- Correct the ordering comment: the paging key is the application
  recommendation ID built from the natural-key columns, not the cache
  table's UUID primary key. A paginated request now fails if two rows
  share an ID, since the strict "after" comparison would skip one of
  them at a page boundary.
- Describe the feature as opt-in in-memory cursor slicing, not a SQL
  keyset (store-side paging is #981).
- TestGetRecommendations_UnpaginatedShapeUnchanged copies the expected
  IDs before the call so an in-place reorder in legacy mode fails it.
- Add TestGetRecommendations_PaginationRejectsDuplicateIDs.
- openapi: document the 400 response for bad limit/cursor/filter
  values, and say next_cursor's absence is the only end-of-list signal.

Refs #723

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(db): index recommendations by byte-wise payload id for keyset scans

Migration 000142 adds recommendations_scan_key_idx on
(COALESCE(payload->>'id', '') COLLATE "C", id). The paginated
GET /api/recommendations scan (issue #981) walks the cache in chunks in
that order and resumes each chunk after the last (id, primary key) it
read; without the index every chunk is a full scan plus a sort.

COLLATE "C" is byte order, the same order Go's string comparison uses,
so SQL paging keeps the order the API already returns.

Refs #981

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(config): read stored recommendations in keyset chunks

ListStoredRecommendationsChunk returns at most limit rows matching a
RecommendationFilter, ordered by the byte-wise recommendation ID and then
the table's UUID primary key, resuming strictly after a given position.
The primary key keeps rows that share an ID in a total order, so a chunk
boundary inside such a run neither skips nor repeats a row. The
MinSavingsPct floor stays in-process (shared with
ListStoredRecommendations via decodeStoredRecommendation); rows it drops
still advance the position.

Integration tests pin the order to Go's sort.Strings on IDs that a
linguistic collation orders differently, cover duplicate IDs across every
chunk size, the pct floor emptying a whole chunk, the empty table, an
EXPLAIN check that recommendations_scan_key_idx serves the query without
a Sort node, and walks racing concurrent upserts.

Refs #981

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scheduler): scan cached recommendations chunk by chunk

ScanRecommendations walks the recommendations cache through
ListStoredRecommendationsChunk in byte-wise ID order and applies the same
suppressions and account overrides as ListRecommendations to each chunk,
handing the survivors to a callback. It holds one chunk at a time instead
of the whole filtered set. Suppressions are read once per scan and each
(account, provider, service) override is resolved once per scan, not once
per chunk. Read failures over-show exactly like ListRecommendations.

The freshness read, cold-start kick and stale-while-revalidate kick are
pulled out of ListRecommendations into startCacheRead/finishCacheRead so
both reads share them unchanged.

Tests pin the scan to ListRecommendations' output (sorted by ID) for chunk
sizes 1 to 31 with suppressions and overrides active.

Refs #981

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(api): page GET /api/recommendations from the chunked scan

A paginated request (limit or cursor sent) no longer loads the whole
filtered set through ListRecommendations and sorts it in memory. It runs
Scheduler.ScanRecommendations, applies the enabled-providers and account
scope filters to each chunk (their inputs read once per request), and
feeds the survivors to recommendationsPageCollector. The collector totals
every row into the summary and keeps only the rows after the cursor up to
limit+1 (the page plus one row proving a next page exists), then cuts the
page with the existing byte budget. Peak memory is one scan chunk plus
one page instead of the whole tenant.

Output is unchanged: rows arrive in byte-wise ID order (Go string order),
duplicate IDs still fail the request, and the summary still covers the
full filtered set. The unpaginated path is untouched.

The in-memory sortRecommendationsAfter/buildBudgetedPage are removed;
the existing #723 pagination tests now run against the scan path, with
buildBudgetedPage kept as a test helper over the collector.

Refs #981

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(api): pin paginated recommendations order, ties and last page

- PaginatedRequestDoesNotLoadFullList: the #981 regression; a paged
  request must use ScanRecommendations, never ListRecommendations.
- PagedWalkMatchesLegacyOrder: for an unrestricted and a scoped reader,
  every chunk size and page size (including the exact-fit last page), the
  walk returns the legacy rows sorted by Go string order in consecutive
  pages, next_cursor on all but the last page, and the legacy summary on
  every page. IDs mix case, "|", "-", "/", spaces and accents.
- PagedDuplicateIDs: ties fail every paged request at any chunk/page
  boundary; a twin removed by the scope filter is not a tie.
- Background-collection scope (#627), the one-page memory bound, the
  out-of-order guard and the empty cache.
- An integration test walks a real Postgres cache spanning several scan
  chunks with the min_savings_pct floor and matches the legacy output.

Refs #981

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(scheduler): scan with main's variant-aware suppression index

Merging main brought #1110 (issue #816), which changed
aggregateSuppressions to return the raw rows grouped by 6-tuple
(map[suppressionKey][]config.PurchaseSuppression) and moved the per-rec
aggregate into applySuppressionIndex, because which rows apply now depends
on the rec's variant (AZ, platform, tenancy). ScanRecommendations still
held the old map[suppressionKey]*suppressionAgg, so the merged tree did not
build.

loadSuppressionIndex now returns main's grouped index and the scan passes
it to the same applySuppressionIndex as ListRecommendations. Matching is
per rec, so applying it chunk by chunk suppresses exactly what the non-scan
path does.

A new test pins the scan to ListRecommendations at every chunk size on an
EC2 fixture with a variant purchase, a fully covered variant, a legacy
wildcard row and a rec with unreadable Details, and checks the expected
per-variant counts. Stripping variants from the scan's index fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(db): renumber recommendations_scan_key_idx migration 000142 -> 000140

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140, 000141, 000143 and
000144 belong to PRs held on owner decisions (#1239, #1241, #1253); this
PR takes 000140 so it can merge ahead of them, and they will take the
next free numbers when decided. Rename only: the SQL is unchanged, and
the test's version constant and the comments naming the migration
follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cristian Magherusan-Stanciu <cristim@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/m Days impact/all-users Affects every user priority/p1 Next up; this sprint severity/high Significant harm triaged Item has been triaged type/bug Defect urgency/this-sprint Within the current sprint

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant