feat(serve): automatic shared-prefix write at the system/developer frontier

[Astrangemaninhere]

Closes #142

What

Agent clients (Hermes, OpenCode, Pi, etc.) rarely send prompt_cache_breakpoint; on master they always miss the shared system/tools head. This adds one automatic write candidate at the leading system/developer frontier so the existing shared-prefix machinery can publish a prefix those clients actually share.

• Keeps the OpenAI explicit-breakpoint contract and the existing last-content implicit candidate (both writes coexist).
• Opt-out flag --no-auto-system-shared-prefix (extra write on by default) for the Responses path; the Chat path currently uses the default-on policy value (consistent with the issue discussion; wiring the flag into the Chat parse signature was deliberately left out to keep the diff minimal).
• No second store, no new catalog, no engine changes - the planner's existing shared_stable_prefix path does the work.

Per-file rationale (8 files, +65/-7, serve layer only)

1. src/serve/openai_common.h (+5) - OpenAIPromptCachePolicy gains auto_system_shared_prefix (default true). The policy is the single switch that reaches both Chat and Responses without threading a new parameter through every parser.
2. src/serve/openai_common.cpp (+31/-1) - the actual feature:
   • Locates the end of the contiguous leading system/developer run (not the first turn): requests that share both a system and a developer turn reuse the whole instruction prefix.
   • Publishes the DefaultAutomatic candidate on that last leading turn when unmarked (the existing explicit-breakpoint contract is untouched).
   • Reserves one of the four frontend marker slots for the new candidate (frontend.cpp rejects >4 markers); the explicit-write selection shrinks by one when the candidate is distinct, so a request with 3 explicit breakpoints + both automatic writes stays at 4.
3. src/serve/serve_options.h (+3) - ServeOptions gains auto_system_shared_prefix = true.
4. src/serve/serve_options.cpp (+4/-1) - parses --no-auto-system-shared-prefix and documents it in the usage text.
5. src/serve/openai_responses.h (+8/-1) - OpenAIResponsesPromptRequest carries the parsed cache_policy so the policy can be applied after prompt resolution (at parse time the leading turns are still in input_turns/instructions, not in generation.messages).
6. src/serve/openai_responses_request.cpp (+11/-3) - stores the policy on the prompt instead of applying it to the empty message list; the parser signature gains the opt-out bool from the HTTP layer.
7. src/serve/openai_responses_state.cpp (+7) - resolve_openai_responses_prompt applies the policy once generation.messages is assembled (instructions + input turns), which is the earliest point the leading-instruction candidate can be placed correctly. (Fixes the review finding that Responses requests saw no system/developer turns.)
8. src/serve/openai_responses_http.cpp (+3/-1) - passes options_.auto_system_shared_prefix into the parser; without this one line the flag would exist but never reach Responses requests.

Verification

• Build: clean on master da49c0d (WSL2, CUDA 13.3, sm_120a).
• Serve smoke: sibling Chat requests sharing a long system head reuse the published prefix without any client breakpoint; Responses requests place the candidate after the resolved leading instructions.
• --no-auto-system-shared-prefix restores strict OpenAI-implicit behavior on the Responses path.

Follow-ups agreed with the issue author (#142 discussion, not blockers)

• Frontier: if the Qwen template renders tools before the system block, the candidate should sit after the combined tools+system prefix. Open to folding in.
• Floor: skip the extra write below ~256 tokens of shared head.
• @steve8697 offered to re-run the 16k sibling probe against this patch.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 58ded12138

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Astrangemaninhere]

这次的pr修改程度较小，希望能得到采纳；如果不能接受，希望能给出意见。

[steve8697]

Thanks for landing this as a small serve-only patch, and for calling out the 16k sibling probe.

We can re-run the same Chat Completions probe we used in #142 (16k shared system, no prompt_cache_breakpoint, two sibling users; authoritative reuse= / cache= / ttft= from the serve log). Pass criteria on our 5090 box:

• sibling B should be shared_stable_prefix with cache ≈ system tokens, TTFT in the ~100 ms class (we measured 91 ms with an explicit breakpoint on master 3d9fda2; without a marker it was root / 2.4 s)
• same-turn continuation should still be private_turn_closure
• --no-auto-system-shared-prefix should restore the root miss on the sibling

Two practical notes, not blockers:

• This PR currently shows CONFLICTING against current master (a16b644, ~110 commits past our last 9811 build, mostly DFlash2). We will apply the patch on a clean 9811 worktree / rebase if needed; if that fails we will say so instead of inventing numbers.
• This machine cannot dual-run :9810 (daily fork) and a test serve. The probe waits on one GPU window.

Will post the table on this PR when the run finishes. Chat Completions is the path we can speak to; we did not exercise Responses in the original issue.

[steve8697]

16k sibling probe on this patch, RTX 5090 32GB, exclusive :9811.

Tree: cherry-pick of 937abea onto our last 9811 baseline 3d9fda2 → 4cdca00. We did not rebase onto current master (a16b644, still CONFLICTING).

Compile: as submitted, openai_responses.h uses OpenAIPromptCachePolicy without including openai_common.h. openai_responses_request.cpp includes the header first, so that TU fails. We added #include "serve/openai_common.h" only; no other local edits.

Serve flags: --host-kv-mib 16384 --host-state-slots 8 --device-state-slots 2 --max-shared-prefixes 4 --max-private-continuations 4 --max-concurrency 2 --kv-dtype int8 --vision --max-context 237568, MTP3. Chat Completions is the path from #142; Responses included because that is where the CLI opt-out is wired. Authoritative reuse= / cache= / ttft= from the serve log.

1. Default ON (no client prompt_cache_breakpoint)

scenario	reuse	cache	prompt	hit%	ttft ms	wall s
chat S1 cold 2k	root	0	2193	0.0	320	0.381
chat S1 warm continue	private_turn_closure	2188	2209	99.0	86	0.138
chat S2a sib A, implicit 16k	root	0	16484	0.0	2402	2.456
chat S2a sib B, implicit 16k	root	0	16485	0.0	2428	2.483
chat S2b sib A, explicit breakpoint	root	0	16484	0.0	2493	2.530
chat S2b sib B, explicit breakpoint	shared_stable_prefix	16466	16485	99.9	108	0.146
chat S4 conc CONC1 (after S2b)	shared_stable_prefix	16468	16484	99.9	240	0.173
chat S4 conc CONC2 (after S2b)	shared_stable_prefix	16468	16484	99.9	103	0.295
responses S2a sib A, implicit 16k	root	0	16484	0.0	2404	2.459
responses S2a sib B, implicit 16k	root	0	16485	0.0	2495	2.550

Historical same box on 3d9fda2 without this patch: S2a both root / 2363 ms; S2b sib B shared_stable_prefix / 91 ms / cache=16466. Same-turn was private_turn_closure / 92 ms.

2. --no-auto-system-shared-prefix

scenario	reuse	cache	prompt	ttft ms
opt-out chat S2a sib A/B	root / root	0	16485 / 16486	2404 / 2428
opt-out chat S1 warm continue	private_turn_closure	2189	2210	110
opt-out responses S2a sib A/B	root / root	0	16485 / 16486	2426 / 2425

Verdict

• Same-turn is fine. private_turn_closure still hits (~86–110 ms). Last-content implicit write is intact.
• Explicit breakpoint still works. S2b / S4 match the Agent sibling sessions miss shared prefix without prompt_cache_breakpoint #142 control: sibling B shared_stable_prefix, cache ≈ system tokens, TTFT in the ~100 ms class vs ~2.4 s root.
• The Agent sibling sessions miss shared prefix without prompt_cache_breakpoint #142 case does not. Chat and Responses 16k siblings with no client marker both stay root / ~2.4 s, same as unpatched master. Hermes / Pi / OpenCode would still re-prefill the shared system head.

So the extra write is not becoming a published shared_stable_prefix that a second user can reuse.

Likely reason, from reading current planner + this diff (not from extra instrumentation):

• The new candidate is CacheBoundary{ .evidence = DefaultAutomatic } on ChatTurn::cache_boundary_after (MessageBoundary).
• In resource_manager.h, DefaultAutomatic is only a surplus candidate. pressure_capable / explicit_shared_credit require ExplicitBoundary or RequestedAutomatic. The value model then has to beat baseline + split cost before anything is written into the shared catalog. That is already why today's last-content implicit write does not help siblings.
• The control that does publish is S2b: prompt_cache_breakpoint on the system part (ExplicitBoundary + LeadingInstructionBoundary).

--no-auto-system-shared-prefix could not be shown to restore a Chat miss, because Chat never hits in the first place, and the Chat parser still does not take options_.auto_system_shared_prefix (as the PR text said). Responses S2a also missed with the flag off.

Happy to re-run the same table if you:

1. #include "serve/openai_common.h" in openai_responses.h
2. Give the leading-instruction candidate RequestedAutomatic or ExplicitBoundary evidence so the planner treats it as declared
3. Place it as a leading-instruction part marker (the S2b path), not only a turn MessageBoundary
4. Optionally thread the CLI flag into the Chat parser

Raw: serve log reuse= lines from this run, ctx 237568, INT8, C=2.