claude-code-chat-browser: FTS5 search index with background rebuild and 30-day default window #112

Calendar Day

Monday, July 6, 2026 (PR 1 of 2 — search bundle)

Planned Effort

8 story points (Task) — sprint item #1

Companion PR: Tuesday PR 2 (search error codes + UX polish #2) — merge this PR first; shared cache dir under ~/.claude-code-chat-browser/.

Problem

GET /api/search is a brute-force substring scan: for every project, every .jsonl session file, it calls get_cached_session() (full JSONL parse) and walks every message in memory. Complexity is O(projects × sessions × messages) per query. The in-memory LRU session cache (PR #90) avoids re-parses within a server run but does not reduce work on the first search after restart. On a power-user install (20+ projects, 300+ sessions), a single search can take tens of seconds. tests/benchmarks/test_search_bench.py exercises live-scan only; benchmarks/baselines.json excludes test_search_full_corpus from the gate, so the search path is unguarded.

cppa-cursor-browser solved this with a derived search_index.sqlite (SQLite FTS5), mtime-keyed fingerprint invalidation, background rebuild, and live-scan fallback (cppa-cursor-browser PR #113). Claude has flat .jsonl files under ~/.claude/projects/ — the index must use line-by-line JSONL extraction, not a verbatim port of cppa's composer-bubble model.

Week 1 PR #111 (session summary disk cache) is merged; FTS index files live beside session_summary_cache.sqlite in the same cache root.

Goal

One merged PR that adds a local FTS5 search index with live-scan fallback, default 30-day search window (all_history opt-in), and re-enables the search benchmark regression gate. Structured error codes and search-page UX copy land in Tuesday PR 2.

Scope

Part A — FTS search index (utils/search_index.py, new)

• Cache directory: ~/.claude-code-chat-browser/ (same root as PR Session list performance: disk summary cache, display-name cache, and count alignment #111 session_summary_cache.sqlite).
• Index files: search_index.<uuid>.sqlite + search_index.active pointer (atomic swap — cppa pattern; safe on Windows).
• Fingerprint: projects_dir + sorted manifest of every .jsonl as (relative_path, mtime_ns, size_bytes) + hash of serialised exclusion rules.
• Schema: index_meta; sessions(session_id, project_name, title, first_ms, last_ms, file_path); messages_fts FTS5 (session_id, project_name, role, timestamp_ms, text, tokenize='unicode61').
• Build (line-by-line, not get_cached_session): walk projects via list_sessions(); line-by-line JSON decode (no full parse_session()); extract searchable text via utils/jsonl_helpers.py / session_peek.py. Deliberate deviation from operator issue-description wording that mentions get_cached_session during build — lower memory, faster rebuild, cppa PR claude-code-chat-browser: FTS search index, search UX, and distinct /api/search error codes #113 parity. get_cached_session remains source-of-truth for live-scan fallback and session detail/export only.
• Indexed text scope: user/assistant text and content on each message line plus tool-result searchable blobs (parity with fields api/search.py scans today around line 67). Only denormalized strings go into messages_fts, not full SessionDict.
• Query: FTS prefix match → SearchHitDict with ±80-char snippets; apply since_ms window; include undated sessions in windowed search (cppa _INCLUDE_UNKNOWN_TIMESTAMPS_IN_WINDOW parity); apply exclusion rules on hit candidates (prefer PR Session list performance: disk summary cache, display-name cache, and count alignment #111 summary cache when warm).
• Lifecycle: ensure_search_index, start_search_index_background (daemon on app.py startup), index_is_usable; bypass via CLAUDE_CODE_CHAT_BROWSER_NO_SEARCH_INDEX=1.
• threading.Lock for build; readers use ?mode=ro.

Part B — Search API window params (api/search.py)

• DEFAULT_SEARCH_WINDOW_DAYS = 30, resolve_search_since_ms(all_history, since_days).
• Query params: all_history=1 / true; since_days=N (validation tightening for invalid values → Tuesday #2 / SEARCH_INVALID_SINCE_DAYS).
• Flow: try FTS index when usable → fallback to existing live-scan loop.
• Happy-path JSON shape unchanged: list[SearchHitDict]; pagination _DEFAULT_LIMIT=50, _MAX_LIMIT=500 preserved.

Part C — Tests and benchmarks

• tests/test_search_index.py (new): schema, fingerprint, FTS hit, window filter, all_history, tool-result text indexed, NO_SEARCH_INDEX fallback, pointer swap.
• tests/benchmarks/: re-enable test_search_full_corpus in benchmarks/baselines.json gate (currently excluded).

Optional stretch (not release-blocking)

Wire PR #111 summary cache into live-scan fallback: consult session_summary_cache for (path, mtime) title/timestamp pre-filter before get_cached_session during live-scan. Defer to week 3 if time runs short.

Out of scope (Tuesday item #2)

• Structured error codes (SEARCH_EMPTY_QUERY, SEARCH_INDEX_UNAVAILABLE, etc.) — empty q may still return 200 [] until Tuesday merges.
• Search UI helper text, all-history checkbox, truncation warning, snippet term highlighting.
• docs/api-reference.md error catalog updates.

Follow-up (post-merge)

• Incremental per-line index updates on appended JSONL lines.
• Summary-cache pre-filter in live-scan if not done as stretch.

Acceptance Criteria

FTS index

• utils/search_index.py builds FTS5 index under ~/.claude-code-chat-browser/.
• Index rebuilds when any .jsonl (path, mtime, size) or rules fingerprint changes.
• Index build uses line-by-line JSON extraction (not get_cached_session / full parse_session).
• Indexed text includes message text/content and tool-result searchable blobs.
• GET /api/search uses index when usable; live-scan fallback when disabled/missing.
• Default 30-day window; all_history=1 searches full corpus; undated sessions in window.
• Background index build on startup without blocking first HTTP response.
• No new runtime dependency beyond Flask==3.1.3; sqlite3 is stdlib.
• tests/test_search_index.py passes; test_search_full_corpus re-enabled in benchmark gate.
• SearchHitDict response envelope and pagination preserved.
• (Optional) Live-scan fallback consults summary cache before full parse when cache row is warm.

General

• mypy --strict, full pytest, and ruff pass.
• PR approved by at least 1 reviewer.

Verification

cd C:\Users\Jasen\CppAliance\claude-code-chat-browser
.\.venv\Scripts\Activate.ps1
pytest tests/test_search_index.py -q
pytest tests/test_search.py tests/test_api_routes.py -q
pytest tests/benchmarks/test_search_bench.py -q
pytest -q
mypy .
ruff check .

Manual:

1. Start app with 50+ sessions; wait for background index build; search — sub-second on warm index.
2. Restart server; repeat search — still fast (disk index, fingerprint match).
3. GET /api/search?q=test (no all_history) — old session outside 30 days absent.
4. GET /api/search?q=test&all_history=1 — old session present.
5. CLAUDE_CODE_CHAT_BROWSER_NO_SEARCH_INDEX=1 — live-scan still works.