Repository navigation
Description
Environment
docling-parse 7.21.0 (win_amd64 wheel), Python 3.12, Windows 11, and GitHub's windows-latest runner. Linux and macOS are unaffected. Docling 2.128 and later reach this through the default ThreadedDoclingParseDocumentBackend, which renders pages by default (render_pages=True).
Symptom
faulthandler prints Windows fatal exception: access violation with no Current thread line, and the process keeps running. Later, unrelated Python objects are corrupt, or the process segfaults. For example, we saw re.error on a valid pattern, a pydantic error about a missing attribute on a RefItem, and a DoclingDocument failing validation with "Duplicate ref".
Minimal reproduction (docling-parse only)
import faulthandler, sys, tempfile, pathlib
faulthandler.enable()
import pypdfium2 as pdfium
from docling_parse.pdf_parser import DoclingThreadedPdfParser, ThreadedPdfParserConfig, DecodeConfig, RenderConfig
render = sys.argv[1] == "render"
path = pathlib.Path(tempfile.mkdtemp()) / "blank.pdf"
doc = pdfium.PdfDocument.new()
for _ in range(3):
doc.new_page(612, 792)
doc.save(path)
doc.close()
for i in range(1000):
parser = DoclingThreadedPdfParser(
ThreadedPdfParserConfig(threads=4, render_config=RenderConfig() if render else None), DecodeConfig()
)
key = parser.load(path)
for result in parser.iterate_results():
pass
parser.unload(key)
print("done")On an i7-7700 with Windows 11:
python repro.py render: 18 access violations, then a segfault (exit 139) before the loop finished.python repro.py parse: no access violations, and the loop completed.
Through Docling's backend, over 300 loads of a 3-page PDF, rendering faulted 7 times with 4 threads and 4 times with 1. With render_pages=False it didn't fault at either thread count.
Likely cause
- feat: optimization of the parse/render with up to 3.84× speedup #333 (7.17.0) added
thread_localobjects with non-trivial destructors to the Blend2D renderer. These are the canvas pool (thread_local BLImage,src/render/blend2d_renderer.haround line 231) and the CMYK memo vectors (around lines 1419-1420). - The win_amd64 wheel is built with MinGW GCC using emulated TLS:
pdf_parsers.*.pydimports__emutls_get_addressfrom the bundledlibgcc_s_seh-1.dll. The render workers arestd::threads, so they are winpthreads threads. - When one of those threads exits, winpthreads runs its key destructors before
_endthreadex. These include libgcc'semutls_destroy, which frees the thread's TLS storage. The C++thread_localdestructors registered through the CRT'stls_atexitrun later, atDLL_THREAD_DETACH, on memory that has already been freed. The fault happens inside the loader's thread-detach callback, which is why Windows swallows it instead of ending the process. This is the long-standing MinGW problem in mingw-w64 bugs 727 and 859. - Workers exit as soon as the task queue is empty, so this can happen on every document.
Suggested fix
Avoid thread_local objects with non-trivial destructors in the renderer on Windows. For example, make the canvas pool and the CMYK memo per-worker objects owned by the worker loop and passed into the renderer, as the FreeType font cache and glyph bbox cache already are (docling_threaded_renderer.h). A Windows CI job, as in #332, would catch a regression: the access violations print on every run.
Workaround
We turn the renderer off with ThreadedDoclingParseBackendOptions(render_pages=False) and render page images with pypdfium2 instead.
Activity
SergiiShcherbak commentedon Oct 20, 2026
Still reproduces with docling-parse 7.22.3 and Docling 2.140.0 (Python 3.14, Windows 11, default
ThreadedDoclingParseDocumentBackend): 14 of 27 processes exited with0xC0000374(heap corruption) while converting their first PDF, also withparser_threads=1. faulthandler shows the page producer in_timings_from_decoderwhile the preprocessing stage is inget_image.usercjr commentedon Oct 31, 2026
Still reproduces on 7.24.1, and I have a fix along the lines @mickey-mikey suggested, tested on Windows 11 with a local MinGW build.
Reproduction. Using the script above with
render, 300 loads:main, local MinGW buildWith
parseinstead ofrenderthere are none, as reported.Fix (branch ready, not opened as a PR yet). The renderer had three
thread_localobjects with non-trivial destructors inblend2d_renderer.h:thread_local BLImage) becomes astd::shared_ptr<BLImage>owned by the render worker indocling_threaded_renderer.h::worker_loop(), next to the FreeType and glyph-bbox caches it already owns. It's passed to each page renderer through a new optional constructor argument; a renderer built without one gets its own, so the other constructors behave as before.thread_local std::vector×2) becomes zero-initialisedthread_localarrays, which have no destructor. Slots storekey + 1, so 0 marks an empty one; the key is 33 bits, sokey + 1never wraps.The remaining
thread_locals in the tree (int, a pointer, a POD timer struct) are trivially destructible and unaffected.Test. A Windows-only unit test runs 120 rendered loads of a small 3-page PDF in a subprocess with
faulthandlerenabled, and fails on any "access violation" or a non-zero exit. Onmainthe subprocess dies with0xC0000005(2 of 2 runs); with the fix it passes in ~30 s.pytest tests/test_unit_*.py: 292 passed, 12 skipped. As a side effect,test_unit_bitmap_artifacts.pyno longer prints the intermittent access violations it showed on my local builds.@cau-git, how would you like this submitted under #428? It changes
blend2d_renderer.h, which is in the renderer topic, but none of the hunks of #386, #390, #397 or #413 touch the canvas pool or the CMYK memo. Can it go as a standalone crash fix, or should it wait for that topic? It changes no pixels, so I'd expect the pypdfium2 metrics to be identical; I can include the corpus table to confirm.(Prepared with Claude Code; I reviewed the change and ran the tests above myself.)
usercjr commentedon Nov 5, 2026
A document in the regression dataset reproduces this on current
main(f4b12cf).I rendered
17898535961965146676-4.pdfp1, one of the cited pages that #439 added, withtests.pdf_builder.render_page()and one worker thread. Each of the 20 runs per build used a fresh process, on a local MinGW build (MSYS2 g++ 15.2, the toolchain the Windows wheels use):0xC0000374(heap corruption)mainbefore #439main(f4b12cf)main+ the fix described aboveSo
soft-masked-drop-shadowintest_regression_pypdfium_cited_pages.pyis flaky on Windows. CI runs the tests on Ubuntu only, which is why it doesn't show there. The renderer changes in #439 only make the crash more likely; on the same page it already happened before them.The fix is rebased on
main. It includes a Windows-only regression test that fails onmainand passes with the fix.pytest tests/test_unit_*.py: 396 passed, 16 skipped, no access violations. Without the fix, the same run prints several, fromtest_unit_bitmap_artifacts.pyandtest_unit_ccitt_images.py.@cau-git, the renderer topic is merged and no open PR touches
blend2d_renderer.h. Unless you'd rather handle it differently, I'll open this as a standalone crash fix following the #428 / CONTRIBUTING rules.(Prepared with Claude Code; I ran the builds and tests above myself.)
ionutvoinea191978-sys commentedon Nov 10, 2026
We see what looks like this fault on docling-parse 7.22.3 with Docling 2.140.0, and can add the native crash position from minidumps, the rates we measured and two small synthetic PDFs. We have not tried 7.24.x or 7.25.0. Everything attached is synthetic (below).
Environment
tags/v3.13.16:cbc944f, MSC v.1944), started with-I -X faulthandler.pdf_parsers.cp313-win_amd64.pydwith itslibstdc++-6.dll,libgcc_s_seh-1.dllandlibwinpthread-1.dll), docling-core 2.103.0, docling-ibm-models 4.0.3, torch 2.14.1 (CPU), torchvision 0.29.1, opencv-python-headless 5.0.0.93, numpy 2.5.4, pillow 12.3.0, pypdfium2 5.14.0, pydantic 2.14.1, onnxruntime 1.31.0, rapidocr 3.10.0, transformers 5.20.2. Layout model "heron" (docling-project/docling-layout-heronat8f39ad3c0b4c58e9c2d2c84a38465abf757272d8) and TableFormer accurate (docling-project/docling-modelsatfc0f2d45e2218ea24bce5045f58a389aed16dc23, tag v2.3.0), the revisions Docling 2.140.0 names, from local files; no network.ThreadedDoclingParseDocumentBackend, with its default options (pages rendered);PdfPipelineOptions(do_ocr=False, do_table_structure=True, table_structure_options=TableStructureOptions(mode=TableFormerMode.ACCURATE, do_cell_matching=True), generate_parsed_pages=True). Our longer runs also used RapidOCR (onnxruntime) and Tesseract on scanned pages; the crash happened with and without OCR.Setup (a clean Windows 11 x64 machine;
C:\reprois an example folder)Python 3.13.16 x64 from python.org (the installer, or the
python3.13.16 package on nuget.org, the same build).The Microsoft Visual C++ 2015–2022 Redistributable x64, 14.44 or later (https://aka.ms/vs/17/release/vc_redist.x64.exe), unless
C:\Windows\System32\msvcp140.dllis already there: PyTorch and ONNX Runtime loadmsvcp140.dll,msvcp140_1.dll,msvcp140_atomic_wait.dllandvcruntime140_threads.dll, which neither Windows nor Python brings. (Or copy those four files into themsvcfolder of step 5.)A venv with the pins (PyPI's Windows torch wheel is the CPU build, so no extra index):
The other dependencies resolve to their latest releases: on 8 November 2026, 19 of them were newer than in our runs (for example pandas 3.1.1 for 3.0.6, python-dateutil 4.0.0 for 2.9.0.post0); we can send our full list.
The two models, at the revisions above (the
hfcommand comes withhuggingface_hub, installed in step 3):model.safetensorsis SHA-25600333a43451945aaf89db8ca9c0a17e75d1537c17db60fdb91aa95f4c7929e0candtableformer_accurate.safetensors2a7d6c924b3cd12fb99a09280ca9c33a89c5d60b93253617d2e088c1a40374d9.Create the empty folder
C:\repro\pack\msvc(or put the four files of step 2 in it).C:\repro\packis the<folder>thatrepro.pytakes: it passes<folder>\modelsto Docling asartifacts_path, adds<folder>\msvcto the DLL search and appends<folder>\site-packagestosys.path, which the venv's Python does not need (the folder need not exist). Give the folders as absolute paths:os.add_dll_directoryrefuses a relative one (WinError 87).Steps
DocumentConverter. Before each conversion it also reads the same PDF withDoclingPdfParser(loglevel="fatal")(DecodeConfig(enforce_same_font=True), line cells) and unloads it.repro.py(attached) does exactly this in a loop, here for 720 seconds:C:\repro\venv\Scripts\python.exe -I -X faulthandler repro.py C:\repro\pack C:\repro\work1 720 minutes-a.pdf minutes-b.pdf. Our run E (below) started four of these at once, each with its own work folder. Each conversion prints a line (1 minutes-a.pdf lines=34 items=11 11.4s). Python 3.13 also prints aSyntaxWarningfor\sin the script's docstring at start-up; it is harmless.Expected: every conversion returns a document.
Actual: now and then the process ends with an unhandled access violation (exit code -1073741819, 0xC0000005), or reports one and goes on. Our runs:
pdf_parsers.cp313-win_amd64.pyd+0x7e7bd7pdf_parsers…+0x7e7bd7; 2 atpython313.dll+0x2b80c0and+0x2b80ddrepro.pyalone, no debugger, 4 processes at once for 12 minutes,minutes-a.pdfandminutes-b.pdfWith a debugger attached the rate is much higher. The debugger saw docling-parse throw and catch between 24 and 1,660 GCC C++ exceptions (
0x20474343) in each watched process, each a round trip to the debugger, which slows the parser's threads and probably widens a race. The two smallest PDFs that crashed are attached:minutes-a.pdf(3,103 bytes, 2 pages, a native text page with a table) crashed two processes in a row in run B, andminutes-b.pdf(3,368 bytes, 2 pages) crashed one. Both convert normally almost every time.The crash, from the minidumps (14 dumps; exception record and module list; the stack is a raw scan of the faulting thread's stack for addresses inside loaded modules, from the stack pointer upward, so a line may be a stale value and not a real frame):
In 12 of 14:
ACCESS_VIOLATION reading 0xFFFFFFFFFFFFFFFF(one:0xFFFFFFFFFFFFFFF8) atpdf_parsers.cp313-win_amd64.pyd+0x7e7bd7, on a thread that is not the main thread (not the dump's first thread). The same scan in all 12:The ntdll addresses under the docling-parse frames may be the thread-exit path (TLS callbacks), which would fit Windows: threaded renderer corrupts the heap when its worker threads exit (thread_local destructors run after MinGW emutls frees their storage) #357's account; we have no symbols to confirm it. The
cv2.pydlines are probably stale values (OpenCV is loaded by the layout stage).In 2 of 14:
ACCESS_VIOLATIONinpython313.dll+0x2b80c0(reading0x1CCD9280128) and+0x2b80dd(reading0x100000007), also on a non-main thread, with python313.dll and ntdll.dll addresses on its stack. This may be the later effect of a corrupted heap that Windows: threaded renderer corrupts the heap when its worker threads exit (thread_local destructors run after MinGW emutls frees their storage) #357 describes.Python's faulthandler wrote "Windows fatal exception: access violation" and every thread's stack; we kept only its last 2 KiB, whose last line was our worker's main loop,
File "extract_worker.py", line 2082 in <module>.In run E two processes printed "Windows fatal exception: access violation" once each during the 12 minutes and went on: every later conversion returned a document and both exited 0. Each report lists 12 Python threads and no "Current thread" line, so the faulting thread ran no Python code (as #357 says). At that moment the main thread was in
DocumentConverter.convert→StandardPdfPipeline._build_document→get_batch(waiting), Docling's stage threads were inget_batchor_run_model, and one thread was in the layout model's forward pass (torch.nn.Conv2d._conv_forwardin report 1,torch.nn.functional._max_pool2dunderRTDetrV2in report 2). Both reports are attached as written, the package folder shown as<site-packages>.Attachments
minutes-a.pdf6dcc2f381036bad855e8ff1680ec59f51d48d5ae870120f65b7cfdb247126885minutes-b.pdf50236065268641f7f669c0460645882a5fb34c469f7273235300c90f3e404d61repro.pyc9d27eae8725a9cd4c73f8d23513aa081e96f9318b9aba426018e3f51b2f4a7ffaulthandler-run1.txtf8f4593b21238e4c6a9fc7da54abfaea529850862a09ac90d5ddf6818373c405faulthandler-run2.txt6026465bfda17a01cc28e9bec0e6f96605983089ebecaf29b916eeb5df7b08beEvery attachment is synthetic. The two PDFs were made by a test generator for an invented project ("North Quay Works, Phase 2 (synthetic)"); they describe no real person, organisation, place or document, and their metadata is empty. The script and the two faulthandler logs are ours; in the logs the package folder is shown as
<site-packages>and the script by its file name. We do not attach the minidumps: they hold the paths and account details of the computer they were written on. We can read further values from them on request (registers, other threads' stacks).The files. They are inline below rather than uploaded (posted through the API, which takes no attachments). The hashes in the table are of the original bytes: the text files with LF line endings as shown, the PDFs after decoding the base64 (
certutil -decode minutes-a.b64 minutes-a.pdf, orbase64 -d).repro.py
faulthandler-run1.txt
faulthandler-run2.txt
minutes-a.pdf (base64)
minutes-b.pdf (base64)
usercjr commentedon Nov 13, 2026
Thank you, this is the first independent reproduction on this issue with dumps, and it lines up with the report:
src/render/blend2d_renderer.h, the spare page canvas (thread_local BLImage) and the CMYK memo (twothread_local std::vectors). Both have destructors, and MinGW's emulated TLS frees a thread's storage before those run.I can't map
+0x7e7bd7in the PyPI wheel to a source line, so that part stays an inference.The fix is in #462: the canvas is now owned by the worker loop, and the memo is a set of arrays without destructors. On my Windows build, a regression-dataset page that crashes 7 of 20 fresh-process renders on
maincrashes 0 of 20 with it, and no pixel changes across the corpus. CI runs the tests on Ubuntu only, so if it's easy for you, it would help to know whether the access violations also stop on your workload with that branch.