feat: keep several prompt cache entries (--prompt-cache-entries) and evict under memory pressure

README.md

@@ -638,6 +638,7 @@ curl http://localhost:5413/v1/chat/completions \
 | `--gpu-layers` | `model_default`| Restrict the amount of layers allocated to GPU hardware |
 | `--stream-experts` | `false` | Enable SSD expert streaming for MoE models (10x speedup) |
 | `--turbo-kv` | `false` | Enable TurboQuant 3-bit KV cache compression (activates after 2048 tokens, server-wide) |
+| `--prompt-cache-entries` | `1` | Prompt cache entries kept in memory (LRU; values below 1 are raised to 1 with a warning). More entries let interleaved sessions each keep their prefix, at the cost of one KV copy per entry. A newer prompt that extends a saved one replaces it instead of taking a second slot; for attention-only models the same goes for a saved prompt it differs from only in its last few tokens (a re-rendered generation prompt, an edited last message), except once a sliding-window layer has outgrown its window. Hybrid (recurrent) models replace only on an exact extension. Under macOS memory pressure the cache keeps only its most recent entry (warning) or is emptied (critical), so with the default of 1 only a critical event drops it |
 | `--draft-model` | (none) | Draft model path/ID for speculative decoding. When used with `--stream-experts`, `--num-draft-tokens` is auto-capped to 1 to minimise SSD I/O fan-out (see performance note above). |
 | `--num-draft-tokens` | `4` | Tokens per speculation round. Auto-capped to 1 when combined with `--stream-experts`. |
 | `--dflash` | `false` | Enable DFlash block-diffusion speculative decoding. Requires a compatible DFlash draft model |