Skip to content

Refresh OCR model lineup: local-first routing, drop dead-weight engines #46

Description

@r-uben

Problem

The OCR engine lineup no longer reflects measured quality/cost on the owner's hardware and born-digital academic workload. Several engines are mis-ranked or dead weight:

  • DeepSeek-OCR scores ~0.085 (socOCRbench, near dead last) yet sits in AUTO_ENGINE_ORDER and _LOCAL_ENGINE_ORDER.
  • Mistral OCR (~0.45) is ranked as the premium rung above Gemini, but it is worse AND ~5x pricier than Gemini — strictly dominated.
  • Figure description is hardwired to the Gemini API with no local-first path.
  • judge / math-recovery defaults lean on slow/timeout-prone local models instead of the practical winner qwen3.5:cloud.

Policy

Best cheap combination, local-first → Ollama Cloud → Gemini/Mistral edge-case only. Native PyMuPDF stays the real default for born-digital text and table values.

Measured winners (owner's 64GB Mac, real workload)

  • qwen3.5:cloud (Ollama Cloud, no extra key): ~0.57, ~49s/page, only engine to clear all 3 hard page types → practical workhorse VLM.
  • Gemini 3.x: best quality (0.60-0.64) but paid + occasional empty pages → edge escalation only.
  • DeepSeek-OCR / Mistral: drop from auto ladders.

Scope (this issue)

  1. New AUTO_ENGINE_ORDER: QWEN → GEMINI → MARKER → GLM → NOUGAT (drop DeepSeek + Mistral from auto).
  2. Demote Mistral to manual-only fallback; drop DeepSeek from _LOCAL_ENGINE_ORDER.
  3. Flip judge_model / math_model defaults to qwen3.5:cloud.
  4. Add an LLM-friendly MODELS.md documenting the routing policy per sub-task (text/formula, figure description, table layout).

Follow-ups (separate)

  • Local-first figure description in gemini_api.py.
  • Split provider identity by engine+backend+model (QWEN local vs cloud) + fix QwenEngine.is_available() cloud gating.

Design second opinion: Codex (gpt-5.5) — agreed with the lineup above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions