perf(cursor)!: move request protobuf framing off the async runtime

[amondnet]

Summary

Move Cursor request protobuf framing off Tokio worker threads. build_run_frames previously ran synchronously inside open_turn: it took 424 µs for Claude Code's normal request shape (a 4 KiB prompt with 16 tools), roughly four times Tokio's 100 µs blocking budget, and 1.4–7.5 ms when a request included an image.

This change:

• Adds src/adapters/cursor/offload.rs with shared cpu_slots() and spawn_bounded() helpers, extracting the bounded-offload pattern from perf(cursor): decompress gzip response frames off the async runtime #256 instead of duplicating it a third time. The blocking closure acquires the permit so cancellation cannot release a slot while an unabortable task still runs. Cursor gzip decoding now uses the shared helper and deletes its private semaphore.
• Makes AgentRunParams owned so it can move into spawn_blocking, and changes open_turn to take it by value. forward adds no clones for the prompt, images, tools, or model_id; only cwd gains a .to_string().
• Offloads framing unconditionally once per request, before the body sender starts and before .send().await, covering every streaming path. No cheap input metric safely bounds framing work because tool-schema encoding grows nonlinearly: 25.9 µs for one tool versus 395.4 µs for 16 tools.
• Offloads base64 image decoding above 64 KiB of estimated decoded data, using base64's exact 4:3 expansion as a sound pre-decode upper bound.

Why 64 KiB instead of 128 KiB?

The image benchmark measures base64 decoding alone, so choosing the largest power of two below 100 µs would not provide a sound inline-work bound. Although the 128 KiB median was 93.17 µs, a rerun measured a 100.5 µs mean, already within noise of the budget. Before reaching the offload decision, the inline path also runs request::cursor_selected_images, which walks the message JSON and clones each base64 string—about 171 KiB of additional copying for a 128 KiB decoded image.

A 64 KiB threshold preserves headroom and remains conservatively below the approximately 105 KiB crossover measured in #259 for the pre-split decode_cursor_images, which included extraction.

Benchmarks

New CodSpeed benchmarks cover pure synchronous functions because CodSpeed runs under Valgrind. Apple Silicon medians are below; #259 used an i7-9700K, so the ordering is comparable but absolute values differ.

Benchmark	Median
Framing, 4 KiB / 0 tools	9.9 µs
Framing, 4 KiB / 16 tools	336.5 µs
Framing, 64 KiB / 16 tools	356.8 µs
Image decode, 32 / 64 / 128 / 256 KiB	23.4 / 46.1 / 92.3 / 185.7 µs

Known remaining inline cost

JSON extraction and its base64 string clone still run on the Tokio worker. cursor_selected_images borrows the request Value, so moving extraction into a 'static closure requires a larger refactor. This change does not regress that cost, and the retained cost is why the threshold keeps headroom.

Documentation

No documentation update is needed because this change does not alter observable behavior, config keys, endpoints, CLI, providers, models, or defaults. README.md, docs/, and site/ contain no inline-framing claim; wiki/ is generated. This matches the #256 precedent (c36c619), which changed only src/ and benches/.

Closes #259

Milestone / spec

Issue #259 — Cursor request framing performance; no frozen milestone/spec change.

Checklist

• cargo build passes
• cargo test passes (new behavior is covered; tests run without network/loopback where possible)
• cargo clippy --all-targets -- -D warnings clean
• cargo fmt --all --check clean
• Source files stay under 500 lines (new offload.rs is 129 lines; existing larger Cursor modules are not expanded structurally by this focused change)
• English only; matches surrounding style
• Frozen spec in docs/ updated if this change deviates from it (not applicable; no deviation)
• User-facing docs updated for behavior/config/endpoint/CLI/provider/model changes — README.md / site/ as applicable (wiki/ is generated; don't hand-edit) (not applicable; no user-facing change)
• Any new GitHub Action is pinned to a full commit SHA (not applicable; no Actions changes)

Notes for reviewers

Please review the 64 KiB image threshold and permit lifetime closely. The permit intentionally moves into the blocking closure so dropping the awaiting future cannot over-admit unabortable blocking work.

Sharing one semaphore across gzip decoding, framing, and image decoding introduced a test-isolation requirement. Slot-limit tests sample cpu_slots().available_permits() as their bound, so any test that holds a permit without the shared OFFLOAD_OBSERVER lock can understate capacity; an unguarded #[cfg(test)] global write can also be clobbered by an interleaving test. All 16 shared-state sites now take the lock, and the requirement is documented on the static.

Verification completed:

• cargo fmt --all --check
• cargo clippy --all-targets --all-features -- -D warnings
• cargo test --all-features --workspace (1009 library, 25 binary, and 244 integration tests)
• Five consecutive full-suite runs
• Twenty targeted stress runs of the shared-state tests

---

Summary by cubic

Moves Cursor request protobuf framing and large base64 image decoding off Tokio workers into a bounded blocking pool to reduce worker blocking and keep streams smooth. Addresses #259 and fixes failover behavior by treating request-prep errors as local, while isolating response-path gzip from request bursts.

• Bug Fixes

  • Framing/image-prep failures now return local Anthropic-shaped 500s and do not advance failover; framing moved to forward and errors mapped via own_error.
  • Split offload classes so response-path gzip isn’t blocked by request prep; permits stay inside blocking tasks so cancellation can’t over-admit work.

• Refactors

  • Added cursor::offload with gzip_slots(), request_prep_slots(), and spawn_bounded_*(); reused by request framing, large image decodes, and gzip (removed connect.rs’s private semaphore).
  • AgentRunParams is owned and open_turn now takes pre-built Vec<Bytes> frames. Framing is offloaded once per request; forward builds frames with no new clones except cwd.to_string(). Image decode offloads above a 64 KiB estimated decoded bound (exact 4:3 base64). New CodSpeed benches cover framing and extract+decode.

^{Written for commit e46704e. Summary will update on new commits.}

---

BREAKING CHANGE: shunt::adapters::cursor::agent::AgentRunParams changes from
borrowed fields (AgentRunParams<'a> with &str/&[T]) to owned String/Vec<T>,
and CursorAgentClient::open_turn changes from (&self, &str, &AgentRunParams<'_>)
to taking pre-built frames by value. Source-breaking for any external consumer of the
published shunt-gateway library's Cursor adapter API.

---

Reviewer note: known adjacent finding, tracked separately

A review pass on this PR flagged that Cursor-owned errors can still advance
ordered failover, contradicting docs/upstreams-failover.md:264-268.

That is real but pre-existing and out of scope here — the doc line,
map_client_error, and the open_turn header block are byte-identical to
origin/main. This PR moves conformance in the right direction: framing and
image-preparation failures now return failure: None and no longer advance the
chain. The residual path (a reqwest builder error from a malformed
SHUNT_CURSOR_CLIENT_VERSION) is tracked in #275.

[gemini-code-assist]

Code Review

This pull request refactors and centralizes the offloading of blocking CPU-bound tasks (such as gzip decompression, base64 image decoding, and request protobuf framing) into a new shared offload module that uses a process-wide semaphore to limit concurrency. It also introduces size-based thresholds to determine whether to decode images inline or offload them, and updates AgentRunParams to use owned types. Feedback on these changes suggests improving panic propagation in the new spawn_bounded helper by checking if a JoinError is a panic and resuming it, rather than unconditionally mapping it to a generic std::io::Error.

[codecov]

Codecov Report

❌ Patch coverage is 98.09689% with 11 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/adapters/cursor/mod.rs	96.41%	8 Missing ⚠️
src/adapters/cursor/offload.rs	99.10%	2 Missing ⚠️
src/adapters/cursor/agent.rs	98.96%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 72 untouched benchmarks
🆕 11 new benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
🆕	cursor_build_run_frames[(4096, 0)]	N/A	37.7 µs	N/A
🆕	cursor_build_run_frames[(4096, 16)]	N/A	513.3 µs	N/A
🆕	cursor_build_run_frames[(65536, 16)]	N/A	1.1 ms	N/A
🆕	cursor_decode_images[131072]	N/A	678.8 µs	N/A
🆕	cursor_decode_images[262144]	N/A	1.4 ms	N/A
🆕	cursor_decode_images[32768]	N/A	173.7 µs	N/A
🆕	cursor_decode_images[65536]	N/A	342.1 µs	N/A
🆕	cursor_extract_and_decode_images[131072]	N/A	1.1 ms	N/A
🆕	cursor_extract_and_decode_images[262144]	N/A	2.2 ms	N/A
🆕	cursor_extract_and_decode_images[32768]	N/A	290.7 µs	N/A
🆕	cursor_extract_and_decode_images[65536]	N/A	555.2 µs	N/A

---

_{Comparing amondnet/perf-cursor-move-request-protobuf-framing-off-th (e46704e) with main (37079c4)}

[greptile-apps]

Greptile Summary

This PR moves Cursor request framing and larger image decoding into bounded blocking tasks.

• Makes agent-run parameters owned and passes pre-built frames into open_turn.
• Introduces separate bounded admission classes for request preparation and response gzip decoding.
• Adds framing, image-decoding, concurrency, cancellation, and benchmark coverage.

Confidence Score: 4/5

The PR appears safe to merge, although the previously reported aggregate CPU-admission concern remains.

The request-preparation and gzip paths retain separate CPU-sized semaphores with no shared aggregate gate, so concurrent traffic can admit both classes at full capacity.

Files Needing Attention: src/adapters/cursor/offload.rs

Important Files Changed

Filename	Overview
src/adapters/cursor/offload.rs	Adds bounded blocking-task helpers with independent admission classes and tests for capacity, isolation, panic propagation, and cancellation.
src/adapters/cursor/mod.rs	Offloads image decoding above the configured threshold and request framing before opening the Cursor turn while preserving adapter-local error shaping.
src/adapters/cursor/agent.rs	Converts agent-run inputs to owned values and changes open_turn to consume pre-built request frames.
src/adapters/cursor/connect.rs	Migrates gzip decompression to the shared bounded-offload helper and verifies admission-class isolation.
benches/gateway.rs	Adds representative Cursor framing and image extraction/decoding benchmarks.

_{Reviews (2): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile}

[amondnet]

/gemini review

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
98.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[gemini-code-assist]

Code Review

This pull request introduces bounded CPU offloading for Cursor's request framing and large-image decoding to prevent blocking the Tokio worker pool. It isolates response-path gzip decompression from request-path preparation using separate, CPU-sized semaphores to eliminate cross-workload head-of-line blocking. The changes also include comprehensive unit tests, integration tests, and benchmarks to validate the offloading thresholds and concurrency limits. The review feedback suggests a minor optimization to use '.clone()' instead of '.to_string()' on 'cwd' to avoid unnecessary formatting overhead.