feat: crop phone photos, re-read small text enlarged, preprocess hook

CHANGELOG.md

@@ -31,6 +31,19 @@ exported from `docket`, the `docket` / `docket-api` commands, the HTTP API in
 
 ### Added
 
+- Phone photos: with the new `[photo]` extra (OpenCV), a photographed
+  receipt or invoice is cropped to the paper and flattened before OCR
+  (`DOCKET_OCR_CROP_PHOTOS`, on by default). Crops that would cut text — a
+  table frame on a scan, a page that already fills the frame — are refused.
+  `PageLayout.crop_quad` records where the page was in the original photo.
+- Small text: when Tesseract's typical word is shorter than
+  `DOCKET_OCR_MIN_TEXT_HEIGHT` (20 px), the page is re-read enlarged (up to
+  3×) and the enlarged reading is kept only if Tesseract is at least as sure
+  of it. On an 11 px synthetic invoice this turns 16/03/2028 and 316.00 back
+  into the printed 15/03/2026 and 315.00.
+- `OcrOptions(preprocess=fn)`: your own `(image, page_number) -> image` step
+  on every page image, after cropping and before OCR (Docling reads the
+  file itself and skips it).
 - Real LLM usage per document: `ProcessingMetrics.llm_input_tokens` /
   `llm_output_tokens` from what the provider reports (Ollama
   `prompt_eval_count`/`eval_count`, OpenAI `usage`), `llm_unreported_calls`

README.md

@@ -112,7 +112,7 @@ Method, per-source results and how to rerun: [docs/BENCHMARKS.md](https://github
 
 ## How it works
 
-Text comes from the cheapest source that works, page by page: PDF text layer, then OCR (Tesseract, PaddleOCR, Docling or a plugin), then a vision model only when OCR is unusable. Classification tries keyword rules, TF-IDF, then an LLM. Extraction fills a Pydantic schema and cites the verbatim line for every value; schema errors go back to the model. Validation never calls a model: arithmetic to the cent, dates, IBAN, VAT and tax-ID check digits, and that every cited line contains the value. Anything uncertain goes to review instead of being silently fixed.
+Text comes from the cheapest source that works, page by page: PDF text layer, then OCR (Tesseract, PaddleOCR, Docling or a plugin), then a vision model only when OCR is unusable. Before OCR, a phone photo of a receipt or invoice is cropped to the paper and flattened (the `[photo]` extra), and Tesseract re-reads small text enlarged; `OcrOptions(preprocess=fn)` adds your own step on every page image. Classification tries keyword rules, TF-IDF, then an LLM. Extraction fills a Pydantic schema and cites the verbatim line for every value; schema errors go back to the model. Validation never calls a model: arithmetic to the cent, dates, IBAN, VAT and tax-ID check digits, and that every cited line contains the value. Anything uncertain goes to review instead of being silently fixed.
 
 ```
 document → text layer / OCR / VLM → classify → extract + cite → validate → JSON or review
@@ -131,6 +131,8 @@ Priority, lowest to highest: defaults, a TOML file (`--config` or `DOCKET_CONFIG
 | `DOCKET_OCR_BACKEND` | `auto` | `tesseract`, `paddle`, `docling`, `auto` or a plugin name |
 | `DOCKET_OCR_FALLBACKS` | `vlm` | Backends tried when a page's reading is rejected |
 | `DOCKET_OCR_LANGUAGES` | `en` | ISO 639-1 codes, e.g. `en,de,fr` |
+| `DOCKET_OCR_CROP_PHOTOS` | `true` | Crop and flatten the document in a photo (needs `[photo]`) |
+| `DOCKET_OCR_MIN_TEXT_HEIGHT` | `20` | Tesseract re-reads a page enlarged when words are shorter than this (px); `0` off |
 | `DOCKET_MIN_CONFIDENCE` | `0.55` | Classification confidence below which a document goes to review |
 | `DOCKET_MIN_SOURCE_CONFIDENCE` | `0.8` | OCR confidence a key field's cited words need, or the document goes to review |
 | `DOCKET_REVIEW_QUEUE_ENABLED` | `false` | Persist flagged documents in the review queue (`[review]` extra) |
@@ -163,6 +165,7 @@ pip install "docket-idp[einvoice]"  # official e-invoice validation
 pip install "docket-idp[review]"    # persistent review queue (SQLAlchemy; [postgres] for PostgreSQL)
 pip install "docket-idp[paddle]"    # PaddleOCR backend
 pip install "docket-idp[docling]"   # Docling/TableFormer backend
+pip install "docket-idp[photo]"     # crop and flatten documents in phone photos (OpenCV)
 pip install "docket-idp[all]"       # + Langfuse tracing and e-invoice validation
 ```
 

docket.example.toml

@@ -55,6 +55,13 @@
 # detect_rotation = true
 # Correct fine scan skew before raster OCR. Env: DOCKET_OCR_DESKEW.
 # deskew = true
+# Find the document in a photo (receipt or invoice on a table) and flatten it
+# before OCR; needs the [photo] extra, otherwise photos are read as they are.
+# Env: DOCKET_OCR_CROP_PHOTOS.
+# crop_photos = true
+# Tesseract re-reads a page enlarged when its typical word is shorter than
+# this many pixels (low-resolution scans); 0 disables. Env: DOCKET_OCR_MIN_TEXT_HEIGHT.
+# min_text_height = 20.0
 # Resolution PDF pages are rendered at for OCR. Env: DOCKET_OCR_DPI.
 # dpi = 200
 # Tesseract page-segmentation mode (3 automatic; 6 can suit dense tables). Env:

docs/openapi.json

@@ -4197,6 +4197,31 @@
             "description": "Fine clockwise correction in degrees, applied after orthogonal rotation.",
             "default": 0.0
           },
+          "crop_quad": {
+            "anyOf": [
+              {
+                "items": {
+                  "prefixItems": [
+                    {
+                      "type": "number"
+                    },
+                    {
+                      "type": "number"
+                    }
+                  ],
+                  "type": "array",
+                  "maxItems": 2,
+                  "minItems": 2
+                },
+                "type": "array"
+              },
+              {
+                "type": "null"
+              }
+            ],
+            "title": "Crop Quad",
+            "description": "When the page was cropped out of a photo: its corners (top-left, top-right, bottom-right, bottom-left) in the original image, normalized 0..1. Word boxes are on the flattened page."
+          },
           "backend": {
             "type": "string",
             "title": "Backend",

pyproject.toml

@@ -50,6 +50,10 @@ api = [
     "uvicorn>=0.29",
     "python-multipart>=0.0.9",
 ]
+# Crop and flatten documents in phone photos before OCR.
+photo = [
+    "opencv-python-headless>=4.8",
+]
 # Persistent review queue (ReviewOptions(enqueue=True)); SQLite by default.
 review = [
     "SQLAlchemy>=2.0,<3",
@@ -83,13 +87,13 @@ einvoice = [
     "saxonche>=12.5",
 ]
 all = [
-    "docket-idp[api,tracing,einvoice,postgres]",
+    "docket-idp[api,tracing,einvoice,postgres,photo]",
 ]
 eval = [
     "datasets>=2.19",
 ]
 dev = [
-    "docket-idp[api,eval,einvoice]",
+    "docket-idp[api,eval,einvoice,photo]",
     # The Streamlit demo (examples/streamlit_demo.py) is a repo-only dev tool.
     "streamlit>=1.49",
     "pytest>=8.0",

src/docket/config.py

@@ -119,6 +119,11 @@ def _cpu_half() -> int:
             "A page reading is accepted at this confidence (0..1).", minimum=0, maximum=1),
     Setting("OCR_DETECT_ROTATION", "ocr.detect_rotation", "DOCKET_OCR_DETECT_ROTATION", "bool", True,
             "Detect and undo page rotation on scans (Tesseract OSD)."),
+    Setting("OCR_CROP_PHOTOS", "ocr.crop_photos", "DOCKET_OCR_CROP_PHOTOS", "bool", True,
+            "Find the document in a photo and flatten it before OCR (needs the [photo] extra)."),
+    Setting("OCR_MIN_TEXT_HEIGHT", "ocr.min_text_height", "DOCKET_OCR_MIN_TEXT_HEIGHT", "float", 20.0,
+            "Tesseract re-reads a page enlarged when its typical word is shorter than this (px); 0 disables.",
+            minimum=0, maximum=200),
     Setting("OCR_DESKEW", "ocr.deskew", "DOCKET_OCR_DESKEW", "bool", True,
             "Correct fine scan skew before raster OCR."),
     Setting("OCR_DPI", "ocr.dpi", "DOCKET_OCR_DPI", "int", 200,
@@ -419,6 +424,8 @@ def describe() -> list[dict[str, Any]]:
 OCR_MIN_CONFIDENCE: float
 OCR_DETECT_ROTATION: bool
 OCR_DESKEW: bool
+OCR_CROP_PHOTOS: bool
+OCR_MIN_TEXT_HEIGHT: float
 OCR_DPI: int
 TESSERACT_PSM: int
 OCR_CONCURRENCY: int

src/docket/layout/models.py

@@ -176,6 +176,13 @@ class PageLayout(BaseModel):
         le=15.0,
         description="Fine clockwise correction in degrees, applied after orthogonal rotation.",
     )
+    crop_quad: list[tuple[float, float]] | None = Field(
+        default=None,
+        description=(
+            "When the page was cropped out of a photo: its corners (top-left, top-right, bottom-right, "
+            "bottom-left) in the original image, normalized 0..1. Word boxes are on the flattened page."
+        ),
+    )
     backend: str = Field(description="OCR backend that produced this page.")
     confidence: float | None = Field(
         default=None,

src/docket/ocr/acquire.py

@@ -288,9 +288,13 @@ def acquire(path: str | Path, options: AcquisitionOptions | None = None) -> Acqu
             dpi=options.settings.dpi,
             max_pages=options.max_pages,
             max_pixels=options.max_pixels,
+            crop_photos=options.settings.crop_photos,
+            preprocess=options.settings.preprocess,
         ) as source:
             for page in source.pages():
                 layout, record = _read_page(page, chain, options, pdf_text)
+                if page.crop_quad is not None:
+                    layout = layout.model_copy(update={"crop_quad": page.crop_quad})
                 pages.append(layout)
                 records.append(record)
     except UnsupportedDocument:

src/docket/ocr/base.py

@@ -19,9 +19,11 @@ def recognize_page(self, page: PageSource) -> PageLayout: ...
 from __future__ import annotations
 
 from abc import ABC, abstractmethod
+from collections.abc import Callable
 from typing import TYPE_CHECKING, ClassVar, Literal
 
-from pydantic import BaseModel, Field
+from PIL import Image
+from pydantic import BaseModel, ConfigDict, Field
 
 from ..errors import ConfigurationError
 from ..layout import DocumentLayout, PageLayout
@@ -76,6 +78,8 @@ class BackendStatus(BaseModel):
 class OcrSettings(BaseModel):
     """Engine-independent settings each backend maps to its own options."""
 
+    model_config = ConfigDict(arbitrary_types_allowed=True)
+
     languages: list[str] = Field(
         default_factory=lambda: ["en"], description="ISO 639-1 codes, e.g. ['en', 'de']."
     )
@@ -86,6 +90,18 @@ class OcrSettings(BaseModel):
         default=True,
         description="Correct small scan angles before OCR; orthogonal rotation is controlled separately.",
     )
+    crop_photos: bool = Field(
+        default=True,
+        description="Find the document in a photo and flatten it before OCR (needs the [photo] extra).",
+    )
+    min_text_height: float = Field(
+        default=20.0, ge=0.0,
+        description="Tesseract re-reads a page enlarged when its typical word is shorter than this, in px; 0 disables.",
+    )
+    preprocess: Callable[[Image.Image, int], Image.Image] | None = Field(
+        default=None,
+        description="Your own step on every page image (image, page number) -> image, after cropping, before OCR.",
+    )
     word_confidence_floor: float = Field(
         default=0.60, ge=0.0, le=1.0,
         description="Words below this confidence neither count toward page confidence nor act as witnesses.",

src/docket/ocr/preprocess.py

@@ -0,0 +1,123 @@
+"""Page image preparation before OCR: photo cropping and small-text upscaling.
+
+A phone photo of a receipt or invoice holds the document somewhere in a
+frame of table, hand and shadow, at an angle. OCR reads the table grain as
+characters and the tilted lines as a slope. `crop_document` finds the
+paper's four corners and warps it flat. It needs OpenCV (the `[photo]`
+extra); without it photos are read as they are.
+
+Tesseract reads text best when capital letters are 20–30 pixels tall; a
+low-resolution scan gives it 10. `upscale_factor` says how far to enlarge a
+page whose words came out that small (see TesseractBackend).
+
+Every step here is conservative: when in doubt it leaves the image alone,
+because a wrong crop loses text and nothing downstream can bring it back.
+"""
+from __future__ import annotations
+
+from collections.abc import Callable
+from statistics import median
+
+from PIL import Image
+
+# Page image, page number -> page image. Runs on every rendered PDF page and
+# image page before any OCR engine sees it (Docling reads the file itself).
+Preprocessor = Callable[[Image.Image, int], Image.Image]
+
+# Four corners (top-left, top-right, bottom-right, bottom-left) of the
+# document in the original image, normalized to 0..1.
+Quad = list[tuple[float, float]]
+
+# A crop must keep at least this share of the frame (smaller is more likely
+# a label, a table or a card than the page), and leave out at least this
+# much (otherwise the page already fills the frame: a scan).
+_MIN_AREA, _MAX_AREA = 0.20, 0.92
+# The frame outside the page must be background, not more paper: the page
+# must be clearly brighter than what surrounds it (median grey levels), and
+# the surroundings must not be busy with edges. A table frame on a scanned
+# page fails the first test — white paper on both sides of the line.
+_MIN_PAPER_CONTRAST = 25
+_MAX_OUTSIDE_EDGES = 0.02
+# Work on a copy this wide; corner finding needs shape, not detail.
+_DETECT_WIDTH = 800
+
+
+def crop_document(image: Image.Image) -> tuple[Image.Image, Quad | None]:
+    """The document flattened out of a photo, and where it was; or the image
+    unchanged and None when no page outline is clear enough to trust."""
+    try:
+        import cv2
+        import numpy as np
+    except ImportError:
+        return image, None
+
+    rgb = np.asarray(image.convert("RGB"))
+    height, width = rgb.shape[:2]
+    scale = min(1.0, _DETECT_WIDTH / width)
+    small = cv2.resize(rgb, (round(width * scale), round(height * scale))) if scale < 1 else rgb
+    gray = cv2.GaussianBlur(cv2.cvtColor(small, cv2.COLOR_RGB2GRAY), (5, 5), 0)
+    edges = cv2.Canny(gray, 50, 150)
+    closed = cv2.dilate(edges, np.ones((3, 3), np.uint8), iterations=2)
+    contours, _ = cv2.findContours(closed, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+    frame = float(small.shape[0] * small.shape[1])
+
+    quad = None
+    for contour in sorted(contours, key=cv2.contourArea, reverse=True)[:5]:
+        approx = cv2.approxPolyDP(contour, 0.02 * cv2.arcLength(contour, True), True)
+        if len(approx) == 4 and cv2.isContourConvex(approx):
+            share = cv2.contourArea(approx) / frame
+            if _MIN_AREA <= share <= _MAX_AREA:
+                quad = approx.reshape(4, 2).astype("float32")
+            break  # only the largest outline can be the page
+    if quad is None:
+        return image, None
+
+    mask = np.zeros(gray.shape, np.uint8)
+    cv2.fillConvexPoly(mask, quad.astype(np.int32), (255,))
+    # Leave the outline's own edge band out of the "outside".
+    inside = cv2.dilate(mask, np.ones((9, 9), np.uint8))
+    outside = inside == 0
+    if not outside.any():
+        return image, None
+    contrast = float(np.median(gray[mask > 0])) - float(np.median(gray[outside]))
+    if contrast < _MIN_PAPER_CONTRAST or (edges[outside] > 0).mean() > _MAX_OUTSIDE_EDGES:
+        return image, None
+
+    corners = _ordered(quad / scale)
+    top = np.linalg.norm(corners[1] - corners[0])
+    bottom = np.linalg.norm(corners[2] - corners[3])
+    left = np.linalg.norm(corners[3] - corners[0])
+    right = np.linalg.norm(corners[2] - corners[1])
+    out_w, out_h = round(max(top, bottom)), round(max(left, right))
+    target = np.array([[0, 0], [out_w - 1, 0], [out_w - 1, out_h - 1], [0, out_h - 1]], dtype="float32")
+    warp = cv2.getPerspectiveTransform(corners.astype("float32"), target)
+    flat = cv2.warpPerspective(rgb, warp, (out_w, out_h), flags=cv2.INTER_CUBIC, borderValue=(255, 255, 255))
+    normalized = [(float(x) / width, float(y) / height) for x, y in corners]
+    return Image.fromarray(flat), normalized
+
+
+def _ordered(points):
+    """Corners as top-left, top-right, bottom-right, bottom-left."""
+    import numpy as np
+
+    by_sum = points.sum(axis=1)
+    by_diff = np.diff(points, axis=1).ravel()
+    return np.array([
+        points[by_sum.argmin()], points[by_diff.argmin()], points[by_sum.argmax()], points[by_diff.argmax()],
+    ])
+
+
+def upscale_factor(word_heights: list[float], *, min_height: float, target: float = 30.0,
+                   max_factor: float = 3.0) -> float:
+    """How much to enlarge a page whose typical word is `word_heights` tall:
+    1.0 when it is at least `min_height` px (or nothing was read), else
+    enough to reach `target`, at most `max_factor`."""
+    if min_height <= 0 or not word_heights:
+        return 1.0
+    typical = median(word_heights)
+    if typical <= 0 or typical >= min_height:
+        return 1.0
+    return min(max_factor, target / typical)
+
+
+__all__ = ["Preprocessor", "Quad", "crop_document", "upscale_factor"]

src/docket/ocr/source.py

@@ -14,6 +14,7 @@
 from PIL import Image
 
 from .. import pdf as pdf_render
+from .preprocess import Preprocessor, Quad, crop_document
 
 PDF_SUFFIXES = {".pdf"}
 IMAGE_SUFFIXES = {".png", ".jpg", ".jpeg", ".tif", ".tiff", ".bmp", ".webp"}
@@ -35,8 +36,12 @@ def __init__(
         dpi: int = 200,
         max_pages: int | None = None,
         max_pixels: int | None = None,
+        crop_photos: bool = False,
+        preprocess: Preprocessor | None = None,
     ):
         self.path = Path(path)
+        self.crop_photos = crop_photos
+        self.preprocess = preprocess
         suffix = self.path.suffix.lower()
         if suffix in PDF_SUFFIXES:
             self.kind: SourceKind = "pdf"
@@ -122,6 +127,7 @@ def __init__(self, document: DocumentSource, number: int):
         self.document = document
         self.number = number
         self._image: Image.Image | None = None
+        self.crop_quad: Quad | None = None
 
     @property
     def kind(self) -> SourceKind:
@@ -132,17 +138,24 @@ def plumber_page(self):
         return self.document.plumber.pages[self.number - 1]
 
     def image(self) -> Image.Image:
+        """The page as OCR sees it: rendered or decoded, cropped out of a
+        photo when that is on, then through the caller's preprocessor."""
         if self._image is None:
             if self.kind == "pdf":
-                self._image = pdf_render.render_page(
-                    self.document.path, self.number - 1, self.document.dpi
-                )
+                image = pdf_render.render_page(self.document.path, self.number - 1, self.document.dpi)
             elif self.kind == "image":
-                with Image.open(self.document.path) as image:
-                    image.seek(self.number - 1)
-                    self._image = image.convert("RGB")
+                with Image.open(self.document.path) as source:
+                    source.seek(self.number - 1)
+                    image = source.convert("RGB")
+                # A rendered PDF page is already the page; only a picture
+                # can hold the document inside a larger frame.
+                if self.document.crop_photos:
+                    image, self.crop_quad = crop_document(image)
             else:
                 raise UnsupportedDocument("a plain-text source has no page image")
+            if self.document.preprocess is not None:
+                image = self.document.preprocess(image, self.number)
+            self._image = image
         return self._image
 
     def image_png(self) -> bytes: