Refactor ingestion to use semantic chunking

[fronx]

Problem

The current ingestion pipeline is too fine-grained and slow:

Current approach:

• Parse markdown into sections -> paragraphs hierarchy
• Per paragraph: 1 embedding + 1 keyword extraction (Claude) + 3-5 keyword embeddings
• For agency-2.md (87 paragraphs): ~500+ API calls, very slow

Issues identified:

1. Keywords extracted per-paragraph are often redundant or nonsensical (e.g., "Content:", "agency-2")
2. The section/paragraph hierarchy doesn't fit all document types (survey found 54% have no headings, some are dialogs)
3. Too many small nodes in the database

Proposed Solution: Semantic Chunking

Use Claude Haiku to identify semantic boundaries in text, creating ~500-1500 token chunks based on meaning rather than arbitrary paragraph splits.

Prototype results on agency-2.md:

• 31 chunks instead of 87 paragraphs
• Average chunk: ~756 chars (~189 tokens)
• Chunk types detected: exposition, argument, list, dialog-turn, notes, section, conclusion
• Processing time: 26s (for chunking only, before embeddings)

New simplified schema:

documents (source file metadata)
    -> chunks (semantic units, ~500-1500 tokens each)
            -> keywords (extracted per chunk, not per paragraph)

Implementation Plan

1. Survey document formats to understand variety (done - see survey-documents.ts)
2. Prototype SemanticChunker with Haiku (done - see src/lib/chunker.ts)
3. Refine chunker prompts based on testing
4. Create new ingestion pipeline using chunks
5. Migrate database schema (chunks table, remove section/paragraph distinction)
6. Update search to work with chunks
7. Re-import all documents with new pipeline

Files

• src/lib/chunker.ts - Semantic chunker prototype
• scripts/test-chunker.ts - Test script for chunker
• scripts/survey-documents.ts - Document format survey

Related

Closes #1 (the original hierarchy bug becomes moot with new chunking approach)

[fronx]

Progress Update: Semantic Chunking Prototype

The chunker has been significantly refined since the initial prototype:

Changes Made

1. Direct text extraction (instead of character offsets)

• Haiku now returns chunk text verbatim in JSON
• Eliminates offset calculation errors and mid-word splits
• Simpler, more reliable approach

2. Stream processing with state handover

• Windows pass context between Haiku sessions:
  • remainder: incomplete text at window boundary
  • summary: brief context for continuity
• Allows processing documents of any length

3. Improved keyword extraction prompts

• Keywords should be specific domain terms from the text (e.g., "gradient descent", "cache invalidation")
• Not generic categories (e.g., "technology", "learning")
• Terms that get introduced/defined are especially valuable for cross-document connections

4. JSON error handling

• Retry logic with up to 2 retries
• Explains the likely cause to Haiku (unescaped quotes/newlines)
• Auto-repair for common JSON issues (literal newlines in strings)

Current State

• src/lib/chunker.ts - Working semantic chunker
• scripts/test-chunker.ts - Test script
• Tested on agency-2.md documents

Remaining Work

1. Test on variety of document types (dialogs, notes, etc.)
2. Create new ingestion pipeline using chunks
3. Migrate database schema (documents → chunks)
4. Update search to work with chunks
5. Re-import all documents with new pipeline

[fronx]

Semantic Chunking Prototype Complete

Branch chunking now has a working prototype. Key commits:

• 05dab95 - Semantic chunker with Haiku (main implementation)
• c516088 - Force reimport functionality for testing
• 7d06859 - Parser tests (closes Import creates incorrect node hierarchy for articles with large frontmatter #1)

Chunker Features

• Streaming: Yields chunks as they are extracted
• State handover: Windows pass remainder text + context summary to next session
• Keyword extraction: Domain terms and named references (not generic meta-keywords)
• JSON recovery: Retry logic with diagnostic output for parse errors
• Verbatim text: Chunks contain exact text from source, never summaries

Test Results

Tested on agency-2.md and Active inference discussion.md:

• Keywords capture domain concepts: Friston, Markov blanket, free energy minimization, Levin, nested selves
• ~15-20 chunks per document vs 80+ paragraphs
• Processing time: ~60-90s for 24k char document

Remaining Work

1. Add chunk node type to database schema
2. Create ingestArticleWithChunks function
3. Update search to work with chunks
4. Re-import test documents with new pipeline
5. Compare search quality between old/new approaches