Optimize conversation processing LLM costs (62% of total spend)

[beastoin]

Context

BQ export of users/{uid}/llm_usage shows the conversation processing pipeline consuming 62% of total LLM spend (updated Feb 6 data from @mon).

Feature	% of Calls	% of Cost	Model
conv_action_items	4.9%	24.1%	gpt-5.1
conversation_processing (legacy umbrella)	7.0%	18.5%	gpt-5.1
conv_structure	4.9%	17.9%	gpt-5.1
conv_apps	9.8%	13.5%	gpt-5.1 + mini

gpt-5.1 = 23.4% of calls but 86.4% of cost. The conversation pipeline alone = 74% of total LLM cost.

Root Cause

Every non-discarded conversation triggers 5-6 LLM calls. The same full transcript is sent to gpt-5.1 three separate times:

Step	Feature	Model	% of spend
1	conv_discard	gpt-4.1-mini	cheap
2	conv_structure	gpt-5.1	15.5%
3	conv_action_items	gpt-5.1	19.8%
4	conv_folder	gpt-4.1-mini	cheap
5	conv_apps (suggest)	gpt-4.1-mini	cheap
6	conv_apps (execute)	gpt-5.1	12.2%

Additional waste paths

• sync.py:606 and postprocess_conversation.py:115 both call process_conversation(force_process=True) on already-processed conversations → 2-3x LLM cost
• _trigger_apps() runs suggestion LLM even when user has a preferred app set
• extract_action_items() fetches 50 recent items for dedup context every call

Key Findings (from deep analysis)

1. Prompt caching is the safest win — restructure structure + action_items calls to share the transcript as a common prefix, letting OpenAI's automatic caching handle it. No quality risk. See Enable 24h prompt cache retention on gpt-5.1 calls (fix ordering + add params) #4672.
2. Merging structure + action_items into one call is risky — structure prompt encourages compression ("condense into summary") while action_items demands expansion ("Read the ENTIRE conversation"). Instruction conflict. Needs A/B eval on ~100 conversations before shipping.
3. Merging with app execute is not viable — app.memory_prompt is untrusted user-generated text. Mixing with strict schema extraction risks prompt injection.
4. conv_apps needs the full transcript — it's the main summarization step, structured overview is not sufficient.
5. conversations_to_string double-summarization — includes both overview AND app result (redundant content fed to daily summary LLM calls).

Sub-issues

#	Issue	Est. Savings	State	PR
#4636	Merge conv_structure + conv_action_items into single call	~15-20%	OPEN — needs A/B eval	PR #4689 (open)
#4639	Skip conv_apps suggestion when preferred app set	~2-3%	OPEN	PR #4683 (open)
#4640	Cap action item dedup context 50 → 10	~1-2%	OPEN	PR #4684 (open)
#4641	Avoid full reprocess in sync/postprocess paths	~5-10%	OPEN	—
#4637	Downgrade action_items to gpt-4.1-mini	—	CLOSED — no model downgrades	PR #4686 (closed)
#4638	Use structured overview for conv_apps	—	CLOSED — needs full transcript	—

Related (not sub-issues but address same pipeline)

#	Title	State	PR
#4672	Enable 24h prompt cache retention on gpt-5.1 (safe win)	OPEN	PR #4674 (open)
#4655	Fix double-summarization in conversations_to_string	OPEN	PR #4682 (open, replaces closed #4663)
#4656	Evaluate merged prompt quality with A/B comparison	OPEN	depends on #4636

Expected Outcome

Target: 25-30% reduction in conversation pipeline LLM costs without degrading output quality.

Data Source

BQ table: based-hardware:llm_usage.raw

---

Last updated: 2026-02-09 by @Chen — reflects new PRs from beastoin (#4682, #4683, #4684), closed sub-issues, and current state

[beastoin]

Sub-issues (ordered by estimated impact)

#	Issue	Savings	Priority
1	#4636 — Merge conv_structure + conv_action_items into single LLM call	~15-20%	P1
2	#4637 — Downgrade conv_action_items to gpt-4.1-mini	~10-15%	P1
3	#4638 — conv_apps: use overview instead of full transcript	~5-8%	P2
4	#4639 — Skip conv_apps suggestion when user has preferred app	~2-3%	P2
5	#4640 — Cap action item dedup context from 50 to 10 items	~2-3%	P2
6	#4641 — Avoid full reprocess in sync and postprocess paths	Variable	P2

Combined target: 25-30% reduction in conversation pipeline LLM costs.

Note: #4636 and #4637 can't both apply fully (merging eliminates the separate action_items call). Recommended approach: implement #4636 first, then evaluate if the merged call still benefits from model downgrade (#4637).

[beastoin]

📝 The proposed solutions here are based on code analysis and have not been verified by the CTO. If you have a better approach to address this, feel free to propose it.

[beastoin]

Updated: Feb 5 BQ Data from @mon

Fresh llm_usage export confirms the cost concentration:

Feature	% of Calls	% of Cost	Model
conv_action_items	4.9%	24.1%	gpt-5.1
conversation_processing	7.0%	18.5%	gpt-5.1
conv_structure	4.9%	17.9%	gpt-5.1
conv_apps	9.8%	13.5%	gpt-5.1 + mini
other (untracked)	29.6%	13.2%	mostly mini
daily_summary	0.6%	4.5%	gpt-5.1

Model split: gpt-5.1 handles only 23.4% of calls but drives 86.4% of cost. gpt-4.1-mini handles 76.4% of calls at just 13.6% of cost.

Bottom line: 5 features on gpt-5.1 = 78.5% of cost with only 27.2% of calls. The conversation pipeline sub-features alone (action_items + structure + apps + processing) = 74% of total cost.

Note: Per-user breakdown still blocked by #4616 (missing uid field in llm_usage docs).

[beastoin]

Deep Analysis: Conversation Processing LLM Cost Optimization

Pipeline Overview

Every non-discarded conversation triggers 5-6 LLM calls, 3 of which send the full transcript to gpt-5.1:

Step	Feature	Model	Input	% of spend
1	conv_discard	gpt-4.1-mini	transcript	cheap
2	conv_structure	gpt-5.1	full transcript	15.5%
3	conv_action_items	gpt-5.1	full transcript (again)	19.8%
4	conv_folder	gpt-4.1-mini	title+overview	cheap
5	conv_apps suggest	gpt-4.1-mini	structured data	cheap
6	conv_apps execute	gpt-5.1	full transcript (3rd time)	12.2%

Prompt Analysis

• conv_structure: ~362 words instructions. Asks for title, overview, emoji, category, events.
• conv_action_items: ~2095 words instructions. Highly complex — 9 task detection patterns, 6-step workflow, coreference resolution, timezone-aware due date extraction, dedup rules.
• conv_apps execute: ~5 lines of instructions + untrusted app.memory_prompt from Firestore. Output overrides structured.overview in the UI.

Key Findings

1. structured.overview is still needed even though the UI overrides it with app results — it's used by folder assignment, app suggestion, search embeddings, and daily summaries.

2. Merging structure + action_items into one call is risky — the structure prompt encourages compression ("condense into summary") while the action items prompt demands expansion ("Read the ENTIRE conversation", "resolve ALL vague references"). This instruction conflict risks degrading action item quality. Advisor recommends not merging without A/B evals first.

3. Merging with app execute is not viable — app.memory_prompt is untrusted user-generated text. Mixing it with strict schema extraction risks prompt injection and parse failures.

4. Prompt Caching is the safe win — OpenAI's prompt caching works on identical prefixes. If both gpt-5.1 calls (structure + action_items) share the transcript as a common prefix, the second call reuses cached tokens. gpt-5.1 supports 24h extended caching. This reduces cost without any quality risk.

5. conversations_to_string includes BOTH overview AND app result — this is redundant content being fed to daily summary LLM calls.

Proposed Sub-Tasks (ordered by impact and safety)

Safe wins (no quality risk):

• Prompt caching — Restructure structure + action_items calls to share transcript prefix, enabling OpenAI prompt caching on the second call
• Skip app suggestion when preferred app exists (Skip conv_apps suggestion when user has preferred app #4639) — _trigger_apps() always calls suggestion LLM even when preferred app overrides it
• Cap dedup context to 10 pending items (Cap action item dedup context from 50 to 10 items #4640) — currently fetches 50 items including completed ones
• Fix double-summarization in conversations_to_string — when app results exist, use them instead of duplicating overview + app summary

Deferred (needs evals):

• Merge structure + action_items (Merge conv_structure + conv_action_items into single LLM call #4636) — viable but must prove no quality regression via A/B eval on ~100 conversations first

Discarded:

• Downgrade conv_action_items to gpt-4.1-mini #4637 (downgrade model) — manager directive: no model downgrade
• conv_apps: use structured overview instead of full transcript for app execution #4638 (structured data for app execute) — app prompts are untrusted/freeform, can't predict what they need from transcript

---

Research by AI for @beastoin