Preserve Anthropic system-prompt cache_control on OpenAI→Anthropic path#337
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9cb39e3063
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
| if let UserContentPart::Text(t) = part { | ||
| let mut block = Map::new(); | ||
| block.insert("type".into(), Value::String("text".into())); | ||
| block.insert("text".into(), Value::String(t.text)); |
There was a problem hiding this comment.
When a leading system/developer prompt has multiple text parts or multiple leading messages and any text part has cache_control, this path switches away from the old flattening that inserted "\n" between parts and "\n\n" between messages, and instead emits one Anthropic system text block per part with only the raw text. That changes the system prompt only for cached OpenAI→Anthropic requests (for example, two leading instructions that previously became A\n\nB are sent as adjacent blocks with no separator). Add separator text to the generated blocks, or fold it into adjacent blocks while preserving the intended cache boundary.
Useful? React with 👍 / 👎.
The anthropic adapter flattened leading system messages into a plain string, discarding cache_control breakpoints. Anthropic prompt caching on the system prompt therefore never engaged via the OpenAI-compat path (cached_tokens stayed 0), while the native and user-message paths worked. Emit `system` as an array of text blocks preserving cache_control when a breakpoint is present; keep the string form otherwise (backward compatible). Fixes INF-170. Repro: Pylon #18806. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
paultancre-bt
force-pushed
the
paultancre/inf-170-anthropic-system-cache-control
branch
from
9cb39e3 to
d8d659e
Compare
|
do you mind adding a test case in |
Adds chatCompletionsSystemCacheControlParam (system message with a cache_control text part) plus a ChatCompletionSystemMessageWithCacheControl type. The chat-completions → anthropic transform snapshot confirms the system prompt is forwarded as a cache_control-bearing text block array. Captured with `make capture CASES=chatCompletionsSystemCacheControlParam` (anthropic only; other providers left uncaptured). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deserialize into CreateMessageParams and match on System instead of poking at serde_json::Value, satisfying the typed-boundary CI check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Ken Jiang (@knjiang) only supplied an Anthropic key for the test seems this make test-typescript failing do I need to supply all the keys? (- OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY,AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY) ? |
|
yep all those keys would be nice. this is to make sure we understand our change in the context of all the different model apis |
Adds the OpenAI/Bedrock/Google/Responses transform captures and the chat-completions provider snapshot for chatCompletionsSystemCacheControlParam, plus the vitest transform snapshots, so the payloads sync coverage check passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…thropic-system-cache-control
Main's reasoning/content merge for OpenAI chat completions (#331) changed the chat-completions → responses transform; refresh the case snapshot to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3 participants
Problem
On the OpenAI-compat
/chat/completionspath to a Claude model,cache_controlon the system message was silently dropped, so Anthropic prompt caching on the system prompt never engaged (cached_tokens: 0). User-message and native/anthropic/v1/messagespaths were unaffected.Approach
request_from_universal(anthropic adapter) flattened leading system messages into a plainString, keeping onlyt.text. The top-levelsystemparam supports an array of text blocks withcache_control, so:cache_control, emitsystemas an array of{type,text,cache_control}blocks (reusinganthropic_cache_control_from_universal, nowpub(crate)).Tests
test_anthropic_system_cache_control_preserved— asserts outboundsystemis an array whose block carriescache_control.type = "ephemeral". Verified red without the fix, green with it.test_anthropic_system_without_cache_control_stays_string— backward-compat guard.cargo fmt --check+cargo clippy -p linguaclean.Repro (prod,
x-bt-use-cache: never)Warm cache via native (
cache_creation=5012), then read identical prefix via openai-compat system → MISS before fix. Scripts:scripts/repro_pylon_18806_*.py(monorepo).Linear: INF-170 (https://linear.app/braintrustdata/issue/INF-170)
Pylon: #18806