feat(prefix-cache): sticky tools pins + DiffPin for warm agent TTFT

[davidmroth]

Summary

Follow-up to #492 (native tool-prefix caching).
That work made system/tools KV reusable across growing agent turns. Two gaps
remained in multi-chat / long-lived tool-heavy agent traffic:

1. Sticky protected tools-boundary pins — cold tool-heavy turns pin the
   system+tools head and mark those entries protected so deepen snaps from
   other sessions cannot thrash large tool prefixes out of the slot pool.
2. DiffPin (PPP) — when a small volatile island (session clock, banner) sits
   inside an otherwise-stable head, exact prefix match fails and the engine
   re-prefills tens of thousands of tokens. DiffPin diffs the head against
   recent traffic, annotates a sticky pin_end, and (optionally) relocates the
   ephemeral middle after the shared prefix+suffix so the contiguous stable
   blob can be pinned.

No conversation-id header or client protocol change is required. Defaults keep
DiffPin annotation on and token rearrange off (safe for tool JSON):

Env	Default	Role
DFLASH_PPP	1	LCP pin-end + sticky protect
DFLASH_PPP_REARRANGE	0	Opt-in token-level float rewrite
DFLASH_PPP_LCP_WINDOW / MIN_PIN / MAX_EPHEMERAL	(see docs)	DiffPin knobs

Docs: server/docs/PIN_FRIENDLY_PROMPT.md (linked from TOOL_PREFIX_CACHE.md).
usage.timings continues to report cache_hit, cached_prefix_tokens,
prefilled_tokens, and effective_prompt_tokens.

Why rearrange defaults off

An early DiffPin wiring applied token-level rewrite even when rearrange=0.
That could peel a mid-head island out of a large tools/system prefix
(diff-rewrite … middle≈170 on ~17–18K-token tool schemas) and scramble tool
JSON enough to yield empty post-tool completions. Gating rewrite behind
DFLASH_PPP_REARRANGE (default off) keeps the default path to pin-end
annotate + sticky protect only (pin-only / no rewrite in logs). Opt in
to rearrange when you deliberately want clock/banner float.

Measured warm reuse

Self-hosted production dflash_server (native HTTP path), ~28-tool agent
prompts in the ~17–24K token range:

Turn	Prompt	Cached prefix	Prefill	Restore	Notes
Cold tools head	19,502	0	86.3 s	false	pin tools boundary
Warm deepen	22,822	17,848 (78%)	25.5 s	true	sticky pin hit
Warm deepen (deeper)	23,243	22,249 (96%)	6.0 s	true	long decode turn
Warm deepen	23,903	23,173 (97%)	4.9 s	true	short stop turn
Warm deepen	23,993	23,889 (99.6%)	1.8 s	true	~13× fewer prefill seconds vs cold head

On the last warm turn, ~24K prompt / 1.8 s ≈ 13K effective t/s, while only
~100 uncached tokens were actually prefilling — the KV pin is doing the work.

Example log shape with safe defaults:

[ppp] enabled=1 rearrange=0 …
[ppp] pin_end=… (pin-only; prompt=…)
[pc] lookup hit … prefix_len=…
[server] chat CACHE … restore=true … prefill=1.8s

This is a prefill / TTFT result, not a claim about decode throughput.

Reproduce

Against a dflash_server with prefix cache enabled:

python3 server/scripts/benchmark_tool_prefix_cache.py \
  --url http://127.0.0.1:8080 \
  --json-out /tmp/diffpin-prefix.json

Confirm warm turns report cached_prefix_tokens > 0 and restore=true in
server logs ([pc] lookup hit, [server] chat CACHE … restore=true).

For DiffPin specifically, send two tool-heavy turns that differ only by a small
mid-head volatile line (e.g. session clock) and confirm either a sticky pin hit
without rewrite (pin-only / no rewrite), or — with DFLASH_PPP_REARRANGE=1 —
a PPP rearrange + protected pin of the stable blob.

Validation

• Unit coverage for DiffPin / sticky-pin helpers in test_server_unit
• dflash_server build with rearrange gated off by default
• Startup logs [ppp] enabled=1 rearrange=0
• Warm tool-heavy turns restore with large prefix_len near the tools pin
• Post-tool completions stable with default pin-only path
• Reviewer: tool-prefix benchmark still shows warm restore (feat(server): native tool-prefix caching for multi-turn agents #492 regression check)
• Reviewer: multi-chat deepen does not immediately evict protected tools pins

@davide221, Hey, thanks so much for helping out with that last PR! Just give me a shout if there's anything else you need me to do on this one to get it all wrapped up and merged.

[Copilot]

Pull request overview

This PR extends the server-side prefix-cache system to better support long-lived, tool-heavy agent traffic by (1) making the tools/system snapshot “sticky” via protected eviction behavior and (2) introducing the Pin-Friendly Prompt Processor (PPP / DiffPin) to improve cache reuse when small volatile text appears inside an otherwise-stable prompt head.

Changes:

• Add protected (sticky) tool-boundary pins and update eviction/snapshot-boundary selection to prefer the tools boundary until restored.
• Implement PPP (DiffPin): LCP-based pin_end annotation and optional token-level diff/rewrite (gated), plus request-time tracking of recent tool prefixes.
• Add unit tests and documentation for PPP/DiffPin behavior and the protected pin policy.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
server/test/test_server_unit.cpp	Adds unit tests for tools-boundary preference, PPP helpers, rewrite behavior, and protected-eviction behavior.
server/src/server/prefix_cache.h	Extends prefix-cache APIs for protected eviction, tools-boundary preference, and forced pin cuts.
server/src/server/prefix_cache.cpp	Implements protected eviction behavior, tools-boundary selection, forced-cut snapshotting, and direct prefix matching for non-boundary pins.
server/src/server/pin_friendly_prompt.h	Introduces PPP public interfaces/types for LCP pinning, diff/rewrite, and prefix memory helpers.
server/src/server/pin_friendly_prompt.cpp	Implements PPP (DiffPin) algorithms: LCP/suffix diffing, safe cut selection, optional rewrite, and prefix-ring maintenance.
server/src/server/http_server.h	Adds PPP configuration knobs and request pin-end plumbing plus recent tool-prefix storage.
server/src/server/http_server.cpp	Wires PPP into request handling (optional system-tail peel) and generation-cache preparation (pin-only by default; rewrite gated).
server/docs/TOOL_PREFIX_CACHE.md	Documents protected tools-boundary snapshot behavior and references PPP/DiffPin.
server/docs/PIN_FRIENDLY_PROMPT.md	Adds new PPP/DiffPin documentation explaining motivation and behavior.
server/CMakeLists.txt	Adds pin_friendly_prompt.cpp to the build.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cubic-dev-ai]

All reported issues were addressed across 10 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[davide221]

@davidmroth thanks for the PR! One problem seems that > DFLASH_PPP=0 disables PPP annotation/rewrite, but not sticky/protected tool-boundary pinning, despite the docs saying it controlsboth. Could you gate prefer_tools_boundary on config_.ppp_enabled?