Semantic Cache

LLMOps layer for the AI-engineer roadmap: a FastAPI proxy that sits in front of Claude API calls (and the earlier hybrid-rag service) and answers repeated or near-duplicate prompts from a vector cache instead of calling the model.

How it works

1. Each incoming prompt is embedded locally (all-MiniLM-L6-v2, the same model hybrid-rag uses — no API key, vectors L2-normalized).
2. A FAISS IndexFlatIP index of past prompts is searched; inner product on normalized vectors = cosine similarity.
3. If the closest live (non-expired) entry is above SIMILARITY_THRESHOLD (default 0.90), the cached response is returned — no model call.
4. Otherwise the model is called and the new prompt + response is cached.
5. Entries expire after CACHE_TTL_SECONDS (default 1h); expiry is lazy on read, with /cache/purge to compact and /cache/invalidate to clear.

Without ANTHROPIC_API_KEY in .env, a mock LLM with simulated latency is used (same keyless-fallback pattern as agent-orchestrator).

Endpoints (port 8300)

Endpoint	What it does
POST /v1/complete	Cached Claude completion — {"prompt": "...", "bypass_cache": false}
POST /v1/proxy/rag	Cache-fronted proxy to hybrid-rag /query — {"question": "..."}
GET /metrics	Hit rate, avg hit/miss latency, tokens & cost saved
POST /cache/invalidate	Drop all cached answers
POST /cache/purge	Compact expired entries
GET /health	Liveness, active LLM backend, cache size

The agent /run endpoint is deliberately not proxied: agent runs mutate state (sessions, run logs, file writes), so replaying a cached response would skip real side effects. Caching is for read-only paths.

Run

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/python main.py                 # serves on :8300
.venv/bin/python benchmark.py            # 50-prompt workload, prints hit rate
.venv/bin/python -m pytest               # 25 tests

Benchmark results (mock LLM, this machine)

cache hits / misses    : 32 / 18  (hit rate 64%)
avg latency (hit)      :     37.5 ms
avg latency (miss)     :    674.2 ms
speedup on hits        :     18.0x

All 5 intentional paraphrases ("What is cosine similarity?" vs "what is cosine similarity?") matched above the 0.90 threshold. Against the live hybrid-rag service (local Ollama generation), a cached answer returned in ~84 ms vs ~34 s for the uncached call.

Design notes

• FAISS over Redis vector search: in-process, no server dependency, and at cache scale (thousands of entries) a flat index is exact and fast. TTL is per-entry timestamps rather than Redis EXPIRE.
• Expired-entry masking: lookups search the top 10 neighbors, not top 1, so a stale closest match cannot hide a fresh near match behind it.
• Cost estimate: output tokens avoided on hits × Claude Opus 4.8 output pricing ($25/MTok). Proxy responses estimate tokens at ~4 chars/token.
• Threshold tradeoff: 0.90 catches paraphrases while keeping distinct questions apart; raise it toward 0.95+ if wrong-answer-for-similar-question matters more than hit rate.