Adopt canonical output contract; standardize metadata + checksum resume - #22
Merged
Conversation
…metadata + checksum resume
…tence bugs Adopts ocr-output-contract v0.1.1 and fixes the PR #22 review blockers. Contract v0.1.1 adoption: - Bump pin to @v0.1.1. - Replace hand-rolled discovery (utils.get_supported_files excluded any dir literally named 'ocr' + redundant name match) with the contract's iter_input_files(scan_root, output_root, suffixes), which excludes only the RESOLVED output root. Fixes the "files under any dir named 'ocr' silently skipped" bug (acutely fatal under .../toolkits/ocr/...). Dry-run now resolves the same output root so it mirrors the real run. - Pass a run_fingerprint(model, backend, task=OCR-3 toggles) to RootIndex.is_completed on both skip paths, so a re-run under a different model/--table-format/--extract-* reprocesses instead of reusing stale output (also gains v0.1.1's on-disk output-existence check for free). Review blockers: - MEDIUM (regression): render structured page.tables into the page body per --table-format. With --table-format=markdown|html Mistral returns tables in the structured field, NOT inline, so the flag was a silent no-op and OCR-3 table extraction was lost. Tables now replace an in-body placeholder (by table id) when present, else are appended; never discarded. (test_save_results) - MEDIUM: checksum-resume skip now emits the existing .md path on both single- file and directory skip branches, so the quiet one-path-per-line scripting contract reports skipped-but-valid docs. - MEDIUM: persistence (save/metadata/index) failures are wrapped in _persist so a failure is recorded status=failed (best-effort, both metadata levels) and the batch continues instead of aborting. - LOW: under --quiet, per-file failures go to a dedicated stderr console and the failed-doc placeholder .md path is no longer echoed to stdout as a success; only successful outputs populate the quiet stdout list. Tests: +tables-preserved, +quiet-skip-path, +changed-fingerprint-reprocesses, +persistence-failure-doesn't-abort, +quiet-failure-to-stderr; existing tests updated for v0.1.1's <stem>_<ext> doc-dir disambiguation. 137 passed.
…t v0.1.2 Clears the round-2 review blockers for PR #22. HIGH - dangling inline image placeholders -> conformance FAIL on real docs: Real Mistral page.markdown carries inline  placeholders whose target is page.images[i].id. The body kept them verbatim while appending a separate ./figures/figure_N_pageP.png link, so the v0.1.2 conformance harness (which now resolves every inline  on disk) failed with "dangling inline image link". Fix: carry image.id through OCRResult.page_images as (id, bytes) pairs; _render_page_with_figures rewrites each placeholder in place to its canonical figure link (no duplicate appended link), appends figures the model did not place inline, and strips any unresolved local placeholder (e.g. a figure that failed to save) so the body never ships a dangling local link. - Bump pin ocr-output-contract @v0.1.1 -> @v0.1.2 (uv.lock not tracked). - run_fingerprint: pass resolved OCR-3 flags (table_format, extract_header, extract_footer, include_images) via extra={...} instead of mangling task. - Idempotency pre-check uses safe_checksum: an input unreadable at the pre-check (permission denied, deleted/replaced mid-run, broken symlink) is recorded status=failed and the batch CONTINUES (SYS-02), instead of an OSError aborting the whole run. Tests (141 pass): byte-faithful embedded-image-links-resolve conformance test (real placeholder, passes now / failed before); unwritten-placeholder-stripped; unreadable-input-failed-batch-continues (chmod-000); fingerprint-extra-keyed. ruff format + check clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adopts the canonical OCR output contract via the shared
ocr-output-contractpackage (v0.1.0), making this engine's output byte-structure-identical to the rest of the fleet:<input-parent>/ocr/<rel>/<stem>/<stem>.mdwith## Page N, no frontmatter, dual-levelmetadata.json(per-doc + root index, input-relative keyed), and a uniform nonzero-exit-on-failure policy.Adopt canonical output contract; standardize metadata + checksum resume.
Test plan
126tests pass (incl. conformance viaocr_output_contract.conformance.assert_conforms), backend/API mocked (no GPU/keys needed in CI).Part of the fleet-wide output-contract rollout (see
../ocr/docs/plans/00-output-contract). Do not merge before review.