Skip to content

Add Image OCR converter, import docs, README/license updates, and expand CI/tests - #14

Merged
etherlink-intern merged 2 commits into
mainfrom
codex/identify-acceptable-document-types-for-ingestion-wzuajz
Jun 2, 2026
Merged

etherlink-intern merged 2 commits into
mainfrom
codex/identify-acceptable-document-types-for-ingestion-wzuajz

Conversation

@etherlink-intern

Copy link
Copy Markdown
Owner

Motivation

  • Enable image-based OCR imports and recognize common image file types in the native converter pipeline so apps can import photos or images and get Markdown output when Apple Vision is available.
  • Provide guidance and examples for wiring app import surfaces (fileImporter, PhotosPicker, Share Extensions) into the conversion core.
  • Surface a clear plan for future native/FOSS backends (PDF/DOCX/PPTX/XLSX) and add an explicit license for the project.

Description

  • Add ImageOCRConverter that uses Apple Vision (behind #if canImport(Vision) && canImport(CoreGraphics) && canImport(ImageIO)) to extract text lines from images and return a MarkdownDocument, and register it in MarkItDown.defaultConverters.
  • Extend DocumentFormat with image formats (png, jpeg, heic, tiff, gif) and a imageFormats helper set, and update inference from filename and content-type.
  • Harden JSONConverter to throw a targeted malformedInput on empty or unparseable JSON inputs and improve error mapping.
  • Add documentation Docs/ImportButtonIntegration.md and Docs/NativeConverterBackends.md, update README.md with supported formats and planned backends, and add an LICENSE (MIT).
  • Expand CI and developer tooling: bump actions/checkout to v5 and runner to macos-15 in .github/workflows/ci.yml, enhance Scripts/smoke-test.sh to cover empty/unsupported cases, add many fixture files and an empty.md expected output, and extend unit tests to cover image formats, fixture-backed conversions, and unsupported/reserved-format routing.

Testing

  • Ran the unit test suite with swift test, which completed successfully.
  • Ran the CLI fixture smoke tests with Scripts/smoke-test.sh, which completed successfully and verifies supported fixtures and expected error messages for unsupported/reserved formats.

Codex Task

… updates

### Motivation

- Add on-device image OCR support and recognize image inputs in the native conversion pipeline so apps can import photos and perform OCR on Apple platforms.
- Make reserved document-format implementation plans explicit and provide app integration guidance for import buttons, PhotosPicker, and Share Extensions.
- Harden the CLI smoke tests and unit tests to cover empty inputs and routing for reserved/unsupported formats.

### Description

- Add `ImageOCRConverter` with conditional `Vision`/`CoreGraphics`/`ImageIO` compilation that extracts text via Vision and returns a `MarkdownDocument`, and register it in `MarkItDown.defaultConverters`.
- Extend `DocumentFormat` to include image formats (`png`, `jpeg`, `heic`, `tiff`, `gif`) and add `imageFormats` helper; update `JSONConverter` to surface targeted errors for empty or invalid JSON inputs.
- Add comprehensive unit tests in `SwiftMarkItDownTests` covering format inference, fixture-backed conversions, JSON error cases, image OCR routing (conditional), and unsupported/reserved format errors, plus supporting fixture files and an empty expected file.
- Improve `Scripts/smoke-test.sh` to include error-case checks, add many fixture files, add `Docs/ImportButtonIntegration.md` and `Docs/NativeConverterBackends.md` with integration guidance and backend plans, update `README.md`, add an `MIT` `LICENSE`, and bump GitHub Actions checkout to `actions/checkout@v5`.

### Testing

- Ran the unit test suite with `swift test` which succeeded against the updated test cases.
- Ran the CLI fixture smoke tests with `Scripts/smoke-test.sh` which completed successfully and validated both success and error-path smoke cases.
- CI is configured to run the same checks on push and PRs via the updated `.github/workflows/ci.yml`.
@etherlink-intern
etherlink-intern merged commit 4babe93 into main Jun 2, 2026
1 check failed
@chatgpt-codex-connector

Copy link
Copy Markdown

💡 Codex Review

https://github.com/etherlink-intern/codexplayground/blob/e60e5ab0d90c6a30f12562b4584beea67df22afa/Sources/SwiftMarkItDown/Converters/ImageOCRConverter.swift#L27
P2 Badge Preserve image orientation for OCR

For JPEG/HEIC/TIFF photos whose EXIF orientation is not .up (common for portrait iPhone shots), constructing the Vision handler with only the decoded CGImage drops the ImageIO orientation metadata, so Vision reads the raw pixel orientation and OCR can return empty or garbled text even though the image decodes. Please read the orientation from imageSource (for example kCGImagePropertyOrientation) and use the VNImageRequestHandler(cgImage:orientation:options:) initializer.


https://github.com/etherlink-intern/codexplayground/blob/e60e5ab0d90c6a30f12562b4584beea67df22afa/Docs/ImportButtonIntegration.md#L375-L376
P2 Badge Copy temporary shared files before returning

For share-extension attachments loaded with loadFileRepresentation, the URL is a temporary file that is only guaranteed while the completion handler is running. This continuation returns that URL after the handler has completed, so the caller's convertFile(url) path can try to read a file that has already been removed; copy the file or read its data inside the completion before resuming.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant