Repository navigation
ev2r: read PDFs page by page with per-page OCR, retry and budgets - #89
Merged
Merged
Conversation
Unstaple each PDF in an isolated pypdfium2 worker; keep each page's text layer or OCR that page alone through the pinned Docling adapter. Every unit is retried once on timeout/crash, then the page fails alone with a placeholder and code. Budgets (pages, lines, OCR seconds) replace the 100-page refusal; confidence is the union of page kinds plus incomplete-rendition. pdf_image keeps its identity and reads images. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ut, placeholders within the line cap Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The falcon sample's slowest page took 160 s under 4-way load, too close to 180. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Kata ev2r (owner: "unstaple PDFs and extract text or process with OCR per page. We must keep retry."). Builds on #86 (ye2s). Cohort PDFs are now read page by page through two new modules,
tools/extraction/pdf_pages.py(coordinator, units, retry, budgets, assembly) andtools/extraction/pdf_page_worker.py(isolated pypdfium2 worker: inventory + single-page render). pypdfium2 is reused from the pinned Docling environment; no new dependency, and requirements-docling.* are unchanged. Images still go throughpdf_imageunchanged.Behaviour
Split and route per page. An isolated inventory unit reads every page's text layer and image count. A page keeps its text layer when it has ≥200 alphanumeric chars that make up ≥50% of its visible chars. Otherwise it is OCRed: it is rendered alone (216 dpi, downscaled per page below Docling's pixel cap, so oversized sheets no longer refuse the document) and converted by the pinned Docling worker. Docling identity files are not touched.
Retry kept. Every unit (the inventory and each OCR page) is retried once on timeout or crash. Deterministic refusals such as
pixel-limitormalformed-pageare not retried. A page that still fails becomes a failed page with its code and attempt count; the rest of the document is kept.One Markdown rendition (
pdf-pages) with pages in order. Failed pages get a visible placeholder like[Page N not extracted: extract-<code> after 2 attempts].payload.pagesrecords the outcome of every page (text-layer/ocr/failed+code+attempts, plus OCRrender_scaleand confidence).unread_pagescovers pages beyond the budget.Refused only when the document cannot be opened (encrypted or malformed), every page failed (their shared code, else
extract-all-pages-failed), or no page has text (extract-empty-conversion).Budgets replace the >100-page refusal.
extract-page-budgetextract-output-limitextract-record-limitbehind one range placeholderextract-time-budgetwithout runningNothing is truncated without a record.
Confidence is the union of the page kinds: pdf-text gives
converted-format, ocr givesmachine-ocr. Missing or failed pages add a new reason,incomplete-rendition(low; details list pages and codes, capped at 20).Concurrency. All units of all documents share one
--extract-concurrencypool.No orphaned work. Each unit runs as PID 1 of its own PID namespace (
unshare -Ur --pid --fork --kill-child). When it ends or is killed, every page or Docling worker it started dies with it, even workers in their own sessions.python -m tools.extraction.pdf_pages proberuns the measurement.Identity and rebuild impact
extract.py,policy.py,mhtml.py,docling_*,pdf_image.py,pdf_text_worker.pyandrequirements-docling.*are unchanged.pdf-pages.correction_ofrequest. Replay keeps the old intent.changed_extraction_identity. Let in-flight cohorts register before upgrading.renditions.pyandconfidence.pychange the index engine fingerprint, which causes a one-off index rebuild with no re-extraction.Live-instance rollout (falcon)
--extract-concurrency 4.correction_ofrequests for PDFs ye2s: read PDFs and images in cohort intake (text layer first, hardened Docling OCR) #86 refused.Measurements
The sample was 58 falcon PDFs (seed 2107), copied read-only to a scratch directory and deleted afterwards. It totals 4,454 pages and includes both 102-page scans (refused by #86), a 210-page mixed document, the 34- and 55-page scans, a 1,265-page manual and an oversized-page document. The run used concurrency 4 on a 14-core / 47 GiB host with the real runtime.
Tests
tests/test_pdf_pages.pywith 18 tests: routing; assembly, placeholders, budgets, the line cap and confidence; hardening with stand-in units (retry, timeouts, crashes, no surviving processes including other sessions, the time budget and the shared concurrency bound); and real-runtime tests (mixed text, scanned and oversized PDFs, a 120-page PDF, encrypted and malformed PDFs).tests/test_pdf_image.pycohort tests were updated.Roborev
Job 834: Fail with 3 findings, all fixed in a4c2a66 (one fix pass) and recorded with
roborev job comment 834.extract-output-limit.Not done
correction_of).