feat: enable 24h prompt cache retention + routing keys for gpt-5.1 (#4672)

[beastoin]

Summary

• Adds prompt_cache_retention: "24h" to all three gpt-5.1 ChatOpenAI clients via extra_body, extending cache from ~5-10min in-memory to 24h SSD-backed retention
• Adds per-function prompt_cache_key routing hints to improve cache hit rates by routing similar requests to the same cache host
• Part B of issue Enable 24h prompt cache retention on gpt-5.1 calls (fix ordering + add params) #4672 (Part A — message ordering fix — was shipped in PR fix: swap prompt message order to restore gpt-5.1 caching (#4654) #4670)

What changed

backend/utils/llm/clients.py

• Added extra_body={"prompt_cache_retention": "24h"} to llm_medium_experiment, llm_agent, and llm_agent_stream
• Note: must use extra_body not model_kwargs — the OpenAI SDK doesn't accept prompt_cache_retention as a direct kwarg

backend/utils/llm/conversation_processing.py

• extract_action_items() → .bind(prompt_cache_key="omi-extract-actions")
• get_transcript_structure() → .bind(prompt_cache_key="omi-transcript-structure")
• get_reprocess_transcript_structure() → .bind(prompt_cache_key="omi-transcript-structure")
• get_app_result() → .invoke(prompt, prompt_cache_key="omi-app-result")
• generate_summary_with_prompt() → .invoke(full_prompt, prompt_cache_key="omi-daily-summary")

backend/tests/unit/test_process_conversation_usage_context.py

• Updated regex patterns to handle .bind() and extra invoke kwargs

backend/tests/integration/test_prompt_caching_integration.py

• Added 5 new live API tests for retention and cache key params

Integration test results (live gpt-5.1 API)

Test	Cache Hit Rate
prompt_cache_retention="24h" accepted	PASS
24h retention + cross-conversation	87.7%
prompt_cache_key accepted	PASS
Cache key + cross-conversation	87.7%
Combined (retention + key)	91.4%

How it works

• prompt_cache_retention: "24h": OpenAI keeps cached prefixes on SSD for 24h instead of default ~5-10min in-memory. 90% discount on cached input tokens, zero write surcharge.
• prompt_cache_key: Combined with the prefix hash to route requests to machines holding the relevant cache. Each function gets a unique key so different instruction prefixes don't collide.

Expected impact

• PR fix: swap prompt message order to restore gpt-5.1 caching (#4654) #4670 (Part A) restored cache hits from ~6-10% to ~18%+ by fixing message ordering
• This PR (Part B) should push cache hits to 50-80% via 24h retention + routing
• Combined: estimated 25-45% reduction in gpt-5.1 input costs (~55% of total OpenAI spend)

Test plan

• All 228 backend unit tests pass
• 9 integration tests pass (4 message ordering + 5 retention/key)
• Verified extra_body is the correct LangChain mechanism (not model_kwargs)
• Post-deploy: monitor gpt-5.1 cached_tokens / prompt_tokens ratio — expect 50%+

Closes #4672

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request enables 24-hour prompt cache retention for gpt-5.1 models and adds prompt_cache_key routing hints to several LLM calls to improve cache hit rates. The changes in clients.py correctly configure the cache retention, and the test adjustments in test_process_conversation_usage_context.py accommodate the new calling patterns. My review focuses on the implementation of the cache keys in conversation_processing.py and I have one suggestion to improve consistency.

[gemini-code-assist]

The prompt_cache_key "omi-transcript-structure" is also used in get_transcript_structure. However, the system prompts for get_transcript_structure and get_reprocess_transcript_structure are different. According to the pull request description, 'Each function gets a unique key so different instruction prefixes don't collide.' Using the same key for functions with different prompt prefixes could be confusing and might not be optimal for caching. To align with the stated goal of this PR and improve clarity, consider using a unique key for this function, for example omi-reprocess-transcript-structure.

Suggested change

	chain = prompt | llm_medium_experiment.bind(prompt_cache_key="omi-transcript-structure") | parser
	chain = prompt | llm_medium_experiment.bind(prompt_cache_key="omi-reprocess-transcript-structure") | parser

[beastoin]

Smoke Test Results

Environment: Local backend (dev environment, based-hardware-dev Firestore + Redis)
Branch: feat/prompt-cache-retention
Backend: uvicorn main:app on port 8788

Results

#	Test	Status
1	Health check: GET /v1/conversations → HTTP 200	PASS
2	llm_medium_experiment: model=gpt-5.1, extra_body={"prompt_cache_retention": "24h"}	PASS
3	llm_agent: model=gpt-5.1, extra_body={"prompt_cache_retention": "24h"}	PASS
4	llm_agent_stream: model=gpt-5.1, extra_body={"prompt_cache_retention": "24h"}	PASS
5	prompt_cache_retention NOT in model_kwargs (SDK rejection fix)	PASS
6	get_transcript_structure has prompt_cache_key routing	PASS
7	extract_action_items has prompt_cache_key routing	PASS
8	2 distinct prompt_cache_key values: omi-transcript-structure, omi-extract-actions	PASS
9	18 unit tests pass	PASS
10	9 integration tests pass (live OpenAI API, 42.65s)	PASS

Integration Test Detail (live OpenAI API)

Test	Status
test_same_function_same_transcript_full_cache	PASS
test_same_language_different_transcripts	PASS
test_different_languages_share_instruction_cache	PASS
test_cross_user_vs_language_in_instructions	PASS
test_24h_retention_accepted_by_api	PASS
test_24h_retention_cache_hits	PASS
test_cache_key_accepted_by_api	PASS
test_same_key_cross_conversation_cache	PASS
test_retention_and_key_combined	PASS

All 27 tests passed (18 unit + 9 integration). Ready to merge.