fix(http): bound the size of buffered JSON responses

[Yazan-O]

Closes #83

What

HttpClient.request() reads every successful response with await response.json(), which buffers the entire body into memory before parsing, with no upper bound. A large test result --history page, or a run with a long steps[] array (getRun with includeSteps), grows the heap in proportion to the payload.

This adds a bounded read at that single choke point:

• A declared Content-Length over the cap is rejected before the body is read.
• Chunked bodies (no Content-Length) are bounded by counting bytes as they stream, stopping once the cap is crossed.
• On exceed, it throws a typed PAYLOAD_TOO_LARGE error (exit 5 — the same code the backend already returns for oversized request bodies), with a nextAction pointing at --page-size / --since.
• Decoding happens once, at the end, so multi-byte UTF-8 sequences that straddle a chunk boundary still decode correctly.

The cap is a new HttpClientOptions.maxResponseBytes (default 64 MiB — far above any legitimate metadata/history/steps response), injectable for tests and future env/flag wiring.

Scope

Focused on the unbounded typed-JSON read, which is the root cause described in #83. Pagination page-count bounding is handled separately (#248), so this does not touch pagination.ts. Error-response bodies (small) keep using the existing safeReadJson path. Happy to widen the scope if you'd prefer.

The default cap value and whether to make it env-configurable are easy to adjust to your preference.

Tests (src/lib/http.test.ts)

• Rejects an over-cap response via Content-Length.
• Rejects an over-cap chunked response that has no Content-Length.
• Reads a within-cap response unchanged.
• Updates the existing "timeout during body read" test to stall at the new read site (a body stream that errors) — same intent, still classified as RequestTimeoutError.

npm run typecheck, npm run lint, npm run format:check, and src/lib/http.test.ts all pass locally.

Summary by CodeRabbit

• New Features

  • Added protection against oversized JSON responses, with a default 64 MiB limit.
  • Requests exceeding the configured response size are rejected with a clear payload-too-large error.

• Bug Fixes

  • Improved handling of streamed responses so oversized payloads are detected while downloading.
  • Correctly classifies timeouts that occur while reading response content.
  • Preserved existing handling for malformed responses and interrupted requests.

[github-actions]

✅ This PR is linked to an issue assigned to @Yazan-O — thanks! The needs-issue label has been removed.

[coderabbitai]

Walkthrough

HttpClient now limits successful JSON response buffering to a configurable size, defaults to 64 MiB, and raises typed errors for oversized bodies. Tests cover declared and streamed limits, valid responses, and timeout errors during body reads.

Changes

Bounded HTTP response handling

Layer / File(s)	Summary
Response limit configuration and error contract src/lib/http.ts	HttpClientOptions accepts maxResponseBytes; HttpClient stores the configured value or a 64 MiB default.
Bounded JSON response reading src/lib/http.ts	Successful response bodies use bounded streaming reads before JSON.parse; oversized responses raise PAYLOAD_TOO_LARGE while existing parsing and abort errors remain classified.
Response limit and body-timeout tests src/lib/http.test.ts	Tests cover Content-Length, streamed responses, successful bounded parsing, and timeout errors raised during body consumption.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant HttpClient
  participant Response
  participant readBoundedText
  participant JSON.parse
  HttpClient->>Response: receive successful response
  HttpClient->>readBoundedText: read with maxResponseBytes
  readBoundedText->>Response: stream and count body bytes
  readBoundedText-->>HttpClient: bounded text or PAYLOAD_TOO_LARGE
  HttpClient->>JSON.parse: parse bounded text
  JSON.parse-->>HttpClient: parsed JSON or malformed response error

Loading

Possibly related PRs

• TestSprite/testsprite-cli#137: Overlaps with timeout classification during response body consumption.
• TestSprite/testsprite-cli#166: Overlaps with successful JSON parsing and typed error handling.

Suggested reviewers: ruili-testsprite, zeshi-du

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: bounding buffered JSON response sizes in the HTTP client.
Linked Issues check	✅ Passed	The PR implements bounded reads in the shared HTTP client path, addressing the unbounded buffering problem in #83.
Out of Scope Changes check	✅ Passed	The code changes stay focused on response-size bounding and related tests, with no clear unrelated additions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

src/lib/http.ts (2)

971-979: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Null-body fallback fully buffers before enforcing the cap.

When response.body is null, readBoundedText calls response.text() (fully materializing the body) and only checks its encoded length afterward — this defeats the memory-bounding goal this PR sets out to achieve for exactly the pathological/hostile-payload case it targets. It's only reachable when Content-Length is absent AND no stream is exposed, but real fetch implementations rarely hit this combination for non-empty bodies, so it's a residual gap rather than the common path.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/http.ts` around lines 971 - 979, Update readBoundedText’s stream ===
null fallback to avoid calling response.text() before enforcing maxBytes; use an
incremental or otherwise bounded read mechanism that rejects once the encoded
payload exceeds maxBytes while preserving the existing
responseTooLargeError(requestId, maxBytes) behavior and returning the text for
bodies within the cap.

---

989-992: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value

Unguarded reader.cancel() can mask the PAYLOAD_TOO_LARGE error.

If reader.cancel() rejects, the throw responseTooLargeError(...) on the next line never executes; the cancel rejection propagates instead, falls through the err instanceof ApiError check in requestWithMeta, and gets misclassified via malformedResponseError — silently breaking the exit-code-5 contract for this (admittedly rare) case.

🛡️ Defensive fix

       if (total > maxBytes) {
-        await reader.cancel();
+        await reader.cancel().catch(() => {});
         throw responseTooLargeError(requestId, maxBytes, total);
       }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/http.ts` around lines 989 - 992, Guard the await reader.cancel() call
in the response-size handling branch so any cancellation rejection is contained,
then always throw responseTooLargeError(requestId, maxBytes, total). Preserve
the existing oversized-payload behavior and ensure cancellation failures cannot
replace the intended ApiError.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/http.ts`:
- Around line 971-979: Update readBoundedText’s stream === null fallback to
avoid calling response.text() before enforcing maxBytes; use an incremental or
otherwise bounded read mechanism that rejects once the encoded payload exceeds
maxBytes while preserving the existing responseTooLargeError(requestId,
maxBytes) behavior and returning the text for bodies within the cap.
- Around line 989-992: Guard the await reader.cancel() call in the response-size
handling branch so any cancellation rejection is contained, then always throw
responseTooLargeError(requestId, maxBytes, total). Preserve the existing
oversized-payload behavior and ensure cancellation failures cannot replace the
intended ApiError.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 1f68da18-a6af-471f-9751-9298a900ac54

📥 Commits

Reviewing files that changed from the base of the PR and between fe07bc9 and 2d3fe69.

📒 Files selected for processing (2)

• src/lib/http.test.ts
• src/lib/http.ts