fix(llm-client): do not add a fifth cache_control block - #489
Conversation
enable_anthropic_prompt_caching marks the final message block unconditionally. Anthropic and Bedrock cap a request at four blocks carrying cache_control, counting tools, system and message blocks together, so a caller that already placed four breakpoints gets a fifth and the request fails upstream with HTTP 400: A maximum of 4 blocks with cache_control may be provided. Found 5. Claude Code hits this: its main turns mark the final block themselves, so the insert is a no-op and they pass, but the request shapes that leave that block unmarked while spending all four breakpoints elsewhere are rejected. Count first and abstain when the budget is spent. Signed-off-by: Federico Cia <fedecia@gmail.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. WalkthroughThe client now counts all existing Anthropic ChangesAnthropic prompt caching
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to This localized fix prevents adding a fifth cache-control marker while preserving behavior below the limit; no actionable merge-blocking risk remains after normal checks and review. Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
1 participant
What
enable_anthropic_prompt_cachingcounts thecache_controlmarkers already in therequest body and abstains once four are present, instead of always inserting one.
One guard plus a small recursive counter. No behaviour change below the limit.
Why
The function marks the final message content block for every
Backend::Anthropic.Anthropic and Bedrock both cap a request at four blocks carrying
cache_control,counting tools, system blocks and message blocks together. A caller that has
already spent the budget therefore gets a fifth marker and the request fails upstream:
The insert is idempotent per block, so a client that marks its own final block costs
nothing. The failing shape is a client that spends four breakpoints elsewhere — on
tools and system blocks — and leaves the last message block unmarked.
Claude Code does exactly that, on some request shapes and not others, which is why it
presents as flakiness rather than a hard failure. Observed with
switchyard/*routesto Bedrock via a PortKey gateway: 0 of 4 non-streaming calls succeeded before the
change, 31 of 31 after, then 133 calls in one live session with no failure of any
kind. Streaming turns were unaffected throughout, so a session keeps working while
some calls die.
Worth naming, because it is the symptom a user actually reports: Claude Code's
auto mode fails first. Its permission classifier is one of the affected request
shapes, so the tool-approval path breaks while ordinary conversation keeps working,
and the surfaced message is that Claude Code cannot reach a model. The real error is
only in the router log.
Introduced by #233, which enables Anthropic prompt caching by default. Its review
checklist covers the case where the final block is already marked — that case is
handled — but not the total across all blocks.
No issue filed; this PR is the report. Happy to open one if you would rather track it
that way.
How tested
The Python gates do not apply — this is a Rust-only change to
crates/libsy-llm-client.cargo fmt --all --checkcleancargo clippy -p switchyard-llm-client --all-targets -- -D warningscleancargo test -p switchyard-llm-clientgreen — 53 tests, including three newpreviously returned 400; a 5-marker body still returns 400, which is correct
because the caller, not the client, is over the limit
New tests:
anthropic_prompt_caching_abstains_at_the_cache_control_limit— a body at fourmarkers is left untouched
anthropic_prompt_caching_still_marks_one_below_the_limit— three markers still getthe fourth, so caching is not silently disabled
anthropic_prompt_caching_marks_final_message(existing) — unchangedChecklist
Signed-off-by) per the DCO--helpupdated — n/a, no customer-facing surface changeNotes for reviewers
The limit is a constant here (
MAX_CACHE_CONTROL_BLOCKS = 4). It matches whatAnthropic and Bedrock enforce today. If another Anthropic-compatible backend allows a
different number, this wants to come from the backend rather than the crate.
Abstaining is the deliberate choice over evicting. When the budget is spent, a
caller that placed four breakpoints itself needs them more than we need a fifth, and
dropping one of theirs to make room would silently change a caching decision they
made. The cost is that the final message block is not cached on those requests.
The counter walks the whole body. It is O(nodes) on a serialized request and runs
once per call. If that is not acceptable on the hot path, it could count only
tools,systemandmessages— but the recursive form cannot miss a marker in a shape wehave not anticipated.
Summary by CodeRabbit