fix(download): cap fetch_json decoded bytes

[SebTardif]

What Problem This Solves

Fixes an issue where users running ocm runtime releases, ocm runtime install --manifest-url, or an official npm packument fetch would hang or run the process out of memory when the URL returned a huge, streaming, or gzip-expanded JSON body.

download_to_file already rejects bodies over 512 MiB (#92). fetch_json and fetch_json_with_accept still request gzip and parse the decoded stream with serde_json::from_reader and no decoded-byte cap. A 65 KB gzip payload in this session expanded to 64 MiB plus 11 bytes of JSON.

Why This Change Was Made

Copy the decoded JSON through the existing copy_capped helper into a bounded buffer, then parse that buffer. JSON is kept in memory, so the cap is 64 MiB, which is above today's official openclaw npm packument (about 16 MiB) and far below the 512 MiB artifact cap.

User Impact

A runtime manifest or npm packument that expands past 64 MiB now fails with download exceeded 67108864 bytes instead of growing without bound. Official ocm runtime releases still lists published OpenClaw versions.

Evidence

terminal output from the patched ocm binary against a local gzip-expanded JSON URL, then against registry.npmjs.org:

A 65,265-byte gzip body decoded to 67,108,875 bytes (11 bytes past the 64 MiB cap). The CLI rejected it:

$ ./target/debug/ocm runtime releases --manifest-url http://127.0.0.1:18765/manifest.json
ocm: failed to download runtime URL "http://127.0.0.1:18765/manifest.json": download exceeded 67108864 bytes

The same binary still loaded the official catalog (248 releases) and the stable channel (2026.7.1-2).

$ ./target/debug/ocm runtime releases --json
count 248
sample ['2026.9.1-beta.1', '2026.8.1-beta.3', '2026.8.1-beta.2']

$ ./target/debug/ocm runtime releases --channel stable --json
count 1
first 2026.7.1-2

Real behavior proof

• Behavior or issue addressed: fetch_json accepted unbounded decoded JSON (including gzip-expanded bodies) on ocm runtime releases and --manifest-url installs. Those commands now reject a body past 64 MiB decoded bytes.

• Real environment tested: macOS Darwin 25.6.0 arm64, rustc 1.98.0, ocm 0.2.33 built from this branch at /tmp/oc-pr-ocm-F003. Isolated OCM_HOME under /tmp/ocm-f003-proof-ocm.

• Exact steps or command run after this patch:

  Built ./target/debug/ocm. Served a gzip JSON body of 67,108,875 decoded bytes (65,265 bytes on the wire) from 127.0.0.1:18765. Then ran ocm runtime releases --manifest-url http://127.0.0.1:18765/manifest.json, then ocm runtime releases --json and ocm runtime releases --channel stable --json against registry.npmjs.org.

• Evidence after fix: terminal output from the patched CLI:

  $ ./target/debug/ocm runtime releases --manifest-url http://127.0.0.1:18765/manifest.json
  ocm: failed to download runtime URL "http://127.0.0.1:18765/manifest.json": download exceeded 67108864 bytes
  
  $ ./target/debug/ocm runtime releases --json
  count 248
  sample ['2026.9.1-beta.1', '2026.8.1-beta.3', '2026.8.1-beta.2']

• Observed result after fix: The gzip-expanded manifest URL exits 1 with the decoded-byte cap. The official npm catalog still returns 248 releases.

• What was not tested: A live attacker-controlled HTTPS host on the public internet. ocm self update GitHub release JSON (that path does not use fetch_json).

Related: #92 added ureq timeouts and the download_to_file size cap. Gzip request wrapping landed in f0b7f2d. fetch_json itself dates to 768baf1. Same class of bound as rust-lang/cargo#11151 (unpacked crate size).

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: blocked before merge. Reviewed September 11, 2026, 12:30 PM ET / 16:30 UTC (Revision 6).

ClawSweeper review

What this changes

Limits downloaded runtime manifests and npm metadata to 64 MiB after decompression and adds an oversized-gzip regression test.

Merge readiness

⛔ Blocked before merge - 4 items remain

This remains useful work: main and v0.2.44 still parse downloaded JSON without a decoded-byte limit. The implementation and supplied runtime proof are convincing; the previously identified custom-source compatibility decision remains unresolved.

Priority: P2
Reviewed head: f8f83cd3c178f0984c8d14a08d23d50775ebda3c
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused implementation with direct runtime evidence; compatibility acceptance and targeted upgrade validation remain before landing.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): The captured macOS terminal transcript exercises the patched CLI through real HTTP and gzip decoding, showing oversized-manifest rejection and successful official catalog loading; upgrade-state coverage is tracked separately.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): The captured macOS terminal transcript exercises the patched CLI through real HTTP and gzip decoding, showing oversized-manifest rejection and successful official catalog loading; upgrade-state coverage is tracked separately.
Evidence reviewed	9 items	Verified introduced change: The local delta matches the supplied introduction evidence: only the shared JSON parser and download tests change. The existing capped copier runs after gzip decoding and before JSON parsing. Still absent from main: Main calls serde_json::from_reader directly; its 512 MiB cap applies to artifact downloads, not JSON metadata. Latest release comparison: The v0.2.44 source also parses JSON directly without the proposed decoded-byte cap.
Findings	None	None.
Security	None	None.

How this fits together

OCM downloads release metadata to list available OpenClaw versions and select runtime installations or updates. Its shared JSON reader decompresses HTTP responses before passing metadata to release validation.

flowchart TD
  A[Release listing or runtime installation] --> B[Official registry or custom manifest]
  B --> C[HTTP reader and gzip decoding]
  C --> D{Decoded body within 64 MiB?}
  D -->|Yes| E[Parse and validate release metadata]
  D -->|No| F[Return download error]
  E --> G[List releases or select runtime]

Loading

Decision needed

Question	Recommendation
Should the 64 MiB decoded limit also reject previously usable custom runtime manifests?	Approve the universal bound: Accept the fixed 64 MiB limit with documented upgrade impact and verification that rejection preserves existing runtime files and metadata.

Why: The limit is intentional and well above the demonstrated official catalog size, but accepting a new failure mode for persisted custom sources requires repository-owner judgment.

Before merge

• Resolve merge risk (P1) - Existing custom manifests or alternate metadata sources exceeding 64 MiB decoded will stop loading after upgrade; no maintainer acceptance of that compatibility change is recorded.
• Resolve merge risk (P1) - The supplied real run covers release listing, while preservation of an installed runtime after oversized-source rejection remains unverified by an upgrade scenario.
• Complete next step (P2) - Record the maintainer's custom-source size-policy decision, document the chosen upgrade behavior, and verify oversized-source rejection preserves an existing runtime.
• Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production and test delta	Production +8/-2 (net +6); tests +23/-2 (net +21)	The small production increase reuses the existing capped copier and adds one focused gzip regression.

Merge-risk options

Maintainer options:

1. Approve and document the limit (recommended)
   Accept the intentional custom-source cutoff after documenting its upgrade impact and verifying safe rejection with an existing runtime.
2. Define a compatibility mode first
   Pause landing while maintainers choose how existing large custom sources remain supported alongside an explicit bounded mode.

Technical review

Best possible solution:

Use the shared decoded-byte bound with a maintainer-approved, documented custom-source policy and verified preservation of installed runtimes on rejection.

Do we have a high-confidence way to reproduce the issue?

Yes, source establishes the unbounded decoded JSON path on main, and an oversized gzip response provides a concrete trigger. No current-main execution was performed in this read-only review.

Is this the best way to solve the issue?

Yes for the mechanism: reusing the capped copier after decompression is narrow and maintainable. Applying the same fixed threshold to existing custom sources still needs explicit compatibility acceptance.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against c5ac5392e142.

Labels

Label justifications:

• P2: This is a focused download resource-limit improvement without evidence of an active widespread outage.
• merge-risk: 🚨 compatibility: The unconditional cap rejects previously accepted custom metadata and applies when reloading persisted runtime sources.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The captured macOS terminal transcript exercises the patched CLI through real HTTP and gzip decoding, showing oversized-manifest rejection and successful official catalog loading; upgrade-state coverage is tracked separately.
• proof: sufficient: Contributor real behavior proof is sufficient. The captured macOS terminal transcript exercises the patched CLI through real HTTP and gzip decoding, showing oversized-manifest rejection and successful official catalog loading; upgrade-state coverage is tracked separately.

Evidence

What I checked:

• Verified introduced change: The local delta matches the supplied introduction evidence: only the shared JSON parser and download tests change. The existing capped copier runs after gzip decoding and before JSON parsing. (src/infra/download.rs:114, f8f83cd3c178)
• Still absent from main: Main calls serde_json::from_reader directly; its 512 MiB cap applies to artifact downloads, not JSON metadata. (src/infra/download.rs:110, c5ac5392e142)
• Latest release comparison: The v0.2.44 source also parses JSON directly without the proposed decoded-byte cap. (src/infra/download.rs:110, 9ed6256922b3)
• Production-path proof: The supplied body, captured under sourceRevision 80a4b8f94424fb8597f1bebf56c161f7ca65382215ebe020663d0aacfdd84356, records a patched macOS CLI rejecting 67,108,875 decoded bytes delivered as 65,265 gzip bytes through runtime releases --manifest-url. The same binary returned 248 official releases and a stable selection. This exercises the actual HTTP, decompression, and JSON owner rather than a mocked reader.
• Existing-source upgrade boundary: Runtime updates reload the persisted source_manifest_url before applying an update. Thus existing sources over the new cap become unusable for updates. Resolution precedes runtime replacement, supporting safe rejection, but the submitted evidence does not exercise an installed runtime through that rejection. (src/runtime/install.rs:459, f8f83cd3c178)
• Fresh-install failure ordering: Custom-manifest loading and selection occur before calling the runtime installation owner, so an oversized manifest is rejected before preparing runtime files. (src/store/runtimes.rs:2626, f8f83cd3c178)

Likely related people:

• shakkernerd: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• SebTardif: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• Vincent Koc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Resolve and document the custom-source size policy, then verify oversized-source rejection preserves an existing runtime's files and metadata.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (5 earlier review cycles)

• reviewed 2026-08-29T22:52:41.238Z sha dfb553d :: needs maintainer review before merge. :: none
• reviewed 2026-09-05T00:02:01.965Z sha dfb553d :: blocked before merge. :: none
• reviewed 2026-09-07T14:22:40.714Z sha a87cfb4 :: blocked before merge. :: none
• reviewed 2026-09-07T14:25:59.721Z sha a87cfb4 :: blocked before merge. :: none
• reviewed 2026-09-09T01:40:28.027Z sha 9675e51 :: blocked before merge. :: none

[SebTardif]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.