Repository navigation
Conversation
Keep resource paths UTF-8 at string boundaries and use native filesystem paths when locating and opening glyph, encoding, AFM and CMap resources. Use the same conversions in the serial/threaded parsers and native tools. Add subprocess regressions for ASCII, Chinese and mixed-script package locations through the serial, threaded and rendered public APIs. Fixes docling-project#115. Signed-off-by: ainuo-AI <2470447645@qq.com>
|
✅ DCO Check Passed Thanks @ainuo-AI, all your commits are properly signed off. 🎉 |
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
| #ifndef PDF_UTILS_FILES_H | ||
| #define PDF_UTILS_FILES_H | ||
|
|
||
| #include "utf8_path.h" |
There was a problem hiding this comment.
Can you just add it here: src/parse.h
There was a problem hiding this comment.
Thanks, Peter. I've moved the UTF-8 path helper include into src/parse.h, before resources.h, and removed the three includes you pointed out. I also moved the existing parse.h include ahead of docling_resources.h in both the serial and threaded bindings, so the resource initialization sees the helper definitions first.
I rebuilt the native extension and the parse, render, run_scaling, and structure_concurrency targets on Windows x64 (GCC 13 / Python 3.12). All 66 focused tests passed, including the ASCII, Chinese, and mixed-script package-location tests for serial, threaded, and rendered parsing.
I also reran dln-v1.pdf (page 1) and 1702.05596v1.pdf (all 13 pages) from fresh ASCII and Chinese package locations. DocumentConverter succeeded with both ASCII and Chinese input paths using NativePdfPipeline. The text-cell, Markdown, and page-image hashes match the previous version of this PR. The native CLI also passed with both resource paths. These checks don't cover the model/OCR pipeline.
Ruff and the lock-file check passed. The local MyPy run still reports the same 19 diagnostics as unmodified main, with no new diagnostics.
Updated in a94d62f. The follow-up only changes include placement and adds the required SPDX header to parse.h.
| #include <Python.h> | ||
|
|
||
| #include <resources.h> | ||
| #include <parse/utils/utf8_path.h> |
|
|
||
| #define LOGURU_WITH_STREAMS 1 | ||
| #include <loguru.hpp> | ||
| #include <parse/utils/utf8_path.h> |
There was a problem hiding this comment.
can yoy add it here: src/parse.h
Signed-off-by: ainuo-AI <2470447645@qq.com>
Resolves #115.
With docling-parse installed under a non-ASCII directory on Windows,
DoclingPdfParser()can fail while openingglyphs/standard/additional.dat, even though the file is present. The same package works from an ASCII installation path. This happens during resource initialization, before it gets to the PDF.The native resource loaders mix UTF-8 strings from Python with narrow filesystem and stream APIs. On Windows, the narrow file opens interpret those bytes using the ANSI codepage. This patch keeps resource-directory strings in UTF-8 and converts them explicitly to native
std::filesystem::pathobjects for filesystem operations.That covers resource discovery, directory enumeration, and the glyph, encoding, AFM and CMap readers. The serial and threaded parsers, native tools and native test callers all use the same convention when passing the resource directory around. The default build-time resource root is decoded the same way. There is no new setting or public API change.
The regression test copies the installed package into ASCII, Chinese and mixed Latin/Hangul/Cyrillic directories, then parses through the public serial, threaded and rendered APIs. Each case runs in a fresh process so resource caches from the original installation cannot hide a failure. Its small generated PDF checks both Latin text and a predefined
GBK-EUC-HCMap; the real-document checks are described below.Evidence:
The existing regression document
regression/dln-v1.pdf, page 1 reproduces this without changing the PDF. The dataset revision is12531342c926708dcb1387ea25213db4603a7255.To reproduce, install the package in a Windows virtual environment whose path contains non-ASCII characters, save the linked PDF as
dln-v1.pdf, and run:On unmodified main, the constructor raises this error (installation prefix shortened):
The before/after builds used the same source base, 88f1c38, with the same GCC 13.1, CMake 4.2.2, pybind11 3.1.0 and dependency binaries. Python files, resource files and runtime DLLs were held constant; only the native extension was replaced. Both builds were Release builds. The machine was Windows 11 x64 with Python 3.12.10 and ANSI codepage 936.
For
dln-v1.pdf, page 1:additional.datThe resource file exists and has the same SHA-256 in every installation. Running Python with
-X utf8does not fix the main build; the system ANSI codepage remains 936.The failing Chinese installation produces no cells before the fix. After the fix, its cells match the working main ASCII installation exactly, for both serial and threaded parsing:
The page's native RGB output is also unchanged: mean absolute difference 0, changed-pixel ratio 0, compared with the working main ASCII installation. This comparison is against main, not pypdfium2.
As a second real-document check, all 13 pages of the unmodified arXiv paper 1702.05596v1 were parsed. Across both documents, both parsing APIs and both patched installation paths, all 168 per-page character/word/line cell comparisons match main ASCII.
DocumentConverterwithNativePdfFormatOptionalso converts both documents from ASCII and Chinese input paths under both installations; the exported Markdown hashes match the working main build.The native
--configCLI was checked separately: main fails with the Chinese resource directory, and the patched build writes the expected page JSON. The config, input PDF and output JSON paths were kept ASCII for that check, so it isolates resource loading.Local tests and remaining validation
The nine new package-location cases give 6 failures and 3 passes on main: each non-ASCII installation fails in all three parser modes, while the ASCII controls pass. All nine pass with this patch.
The focused suite passed 66 tests, with no skips, covering:
Ruff formatting and lint,
uv lock --check --offline, andgit diff --checkpassed. Ruff 0.17.1, MyPy 1.20.2 and pytest 9.1.1 match the lock file.Full-project MyPy still reports 19 errors in this Windows environment. Running it against unmodified main produces the same diagnostics: optional benchmark dependencies are missing, and some benchmark code uses POSIX-only interfaces. There are no new diagnostics; this is not a claim that the full pre-commit run is green.
The native extension,
parse,render,run_scalingandstructure_concurrencycompile. The macOS/Linux memory probe usesgetrusage/sysconfand fails to build on Windows on both versions; this patch does not port that probe.Linux/macOS, MSVC/ARM64, the full test suite and the CI Python matrix have not been run locally. Builds used an ASCII source mirror; the non-ASCII compile-time
ROOT_PATHcase was reviewed in source, rather than built end to end. The Docling conversion checks used the supported native PDF pipeline, without the model/OCR pipeline.This changes resource location and I/O, with no rendering-algorithm or font-selection changes. The selected-page RGB check above checks for a side effect; corpus-wide pypdfium2 rendering comparisons have not been run.
Ground truth changes:
None.
HF_DATASET_REVISIONand the existing expected results are unchanged. The existing dataset document already reproduces the failure, so no dataset PR is needed.Checklist:
main, no merge commits