Skip to content

Latest commit

 

History

History
341 lines (278 loc) · 17.4 KB

README.md

File metadata and controls

341 lines (278 loc) · 17.4 KB

Benchmarking tools

For the current publication workflow, use one launcher:

bash scripts/benchmarking/refresh_performance_benchmarks.sh
# Include cross-package parsing/rendering and repeated measurements:
REPEATS=3 RUN_COMPARISON=1 bash scripts/benchmarking/refresh_performance_benchmarks.sh

See the performance page for workload contracts, timing boundaries and publication placeholders. The launcher rebuilds the native executable and Python extension, verifies the pinned corpus, runs each native/replay profile in a fresh process, and writes a unique output bundle. It does not update the documentation. Full runs can take hours.

Entry point Role Inputs Outputs
refresh_performance_benchmarks.sh Build and run the publication workflow Pinned HF dataset; environment settings shown by --help Raw measurements, provenance, generated report and PNGs
run_performance_report.py Validate and summarize a launcher bundle Bundle containing metadata.json report/summary.md, CSV tables, PNGs
recover_performance_benchmarks.py Reuse completed core results and recollect comparisons Existing bundle; normally invoked through RECOVER_FROM New bundle with original core artifacts and separate comparison provenance
run_performance_benchmarking_with_docling_settings.py Standalone backend replay PDFs or pinned HF revision report.json and page metadata JSONL
run_performance_benchmarking.py Python API scaling and cross-package measurement PDFs or HF dataset Console tables and per-page CSV
run_performance_eval.py Explore Python comparison page timings Comparison per-page CSV Histograms, scaling/correlation PNGs, document statistics
run_performance_analysis.py Replay slow pages for detailed native timings Comparison per-page CSV and original PDFs Analysis CSV or detailed timing table
run_quality_benchmarking.py Compare text and raster quality PDFs or HF dataset Quality CSVs and comparison visualizations

Install and rebuild

For individual tools:

uv sync --group build --group dev --group perf
# Rebuild the extension after C++ changes:
uv run --no-sync python local_build.py
uv run --no-sync cmake --build build --target run_scaling.exe --parallel 8

The launcher defaults to BUILD_MODE=incremental: existing build/ and externals/ are retained. It configures/builds/installs the Python extension and checks/builds the standalone executable; CMake recompiles affected targets and reuses unchanged ones. Both checks run before any measurements.

The launcher checks cached repository, Python, SDK and compiler paths before dependency synchronization, and again afterwards in case Python changed. An incompatible cache stops the run with a BUILD_MODE=clean instruction. Use clean mode after SDK/toolchain changes or bundled dependency pin/patch upgrades: it archives existing directories as OUT/previous-build and OUT/previous-externals, then builds from scratch. It selects the current macOS SDK and preserves old outputs rather than deleting them. Failed commands print their exit status, log path and final 40 log lines before stopping. Each experiment displays its number, repetition, backend and thread count. Live runner output includes its existing page progress bar and is also saved in full. Completion messages show command elapsed time, separate from benchmark wall time. The launcher defaults to BENCHMARK_OUTPUT=compact: live progress, short run headers, result tables and error messages remain visible, while repeated config tables and native timing tables are suppressed. Use BENCHMARK_OUTPUT=full to restore the full benchmark output. Setup/build output is unaffected.

Standalone runners retain their full output by default. All three accept --suppress-config; the native executable also accepts --suppress-timings. With --enable-timing --suppress-timings, native timing summaries go to <timing-csv>.summary.log, including exact wall time, stage totals, the timing tree and slowest pages/documents. The report reads that file when present and also supports older console-only logs. Timing collection and CSV data are unchanged. Config suppression retains the replay's untimed parser initialization.

# Force a fresh dependency and native build when needed.
BUILD_MODE=clean DATASET=bo20 \
  bash scripts/benchmarking/refresh_performance_benchmarks.sh

BENCHMARK_OUTPUT=full DATASET=bo20 \
  bash scripts/benchmarking/refresh_performance_benchmarks.sh

The perf group includes the five reference packages, Matplotlib, PDFium and R-tree support. Selected comparison backends that cannot import are skipped by the old measurement runner; the new bundle report rejects the resulting incomplete suite.

Publication bundle and plots

bash scripts/benchmarking/refresh_performance_benchmarks.sh --help
# Use a new output path; existing paths are rejected to prevent CSV appends.
THREADS=1,4,8 REPEATS=1 OUT=scratch-performance-new-run \
  bash scripts/benchmarking/refresh_performance_benchmarks.sh

Choose the corpus with DATASET=bo20, DATASET=bo767 (default), or DATASET=bo10k. Each selects the matching docling-project/performance-dataset-<name> repository and a pinned revision:

Dataset Default revision PDFs Pages
bo20 65b9971b3fa939fe4ef89be678237e6928049b67 20 496
bo767 221c19e716d6ede5d18fd1560bc938ab19cc1a70 767 54,730
bo10k f85e203f95879e2f5d77059a6bfeffe667cd359e 10,000 213,538
# Smaller corpus and thread sweep; builds incrementally by default.
DATASET=bo20 THREADS=1,4 \
  bash scripts/benchmarking/refresh_performance_benchmarks.sh

DATASET=bo10k \
  bash scripts/benchmarking/refresh_performance_benchmarks.sh

DATASET_REVISION overrides the selected repository's default pin. The requested revision, resolved commit, repository, file hashes and counts are recorded in metadata.json. Default snapshots are checked against the counts above. The historical Bo767 snapshot c684df54de2081a6de1192cee12713b54759c6bb contained 753 PDFs / 54,584 pages; its existing reports retain those inputs. Compare runs with matching corpus revisions and file identities.

Reference preflight, checkpoints and recovery

The perf dependency group pins PyMuPDF 1.27.2.3. Version 1.28.2 aborts in MuPDF display-list construction on Bo767 1293644.pdf, page 3. The pinned version was checked on all 453 pages with text extraction, rasterization and PIL conversion. See the matching upstream report. When RUN_COMPARISON=1, the launcher probes this page in a disposable subprocess before any corpus measurements if the PDF is present in the selected dataset. A native abort or timeout fails the preflight with the document/page and package version, instead of wasting the preceding scaling runs.

Comparison runs checkpoint each completed backend/thread series with its wall time, page CSV, CSV hash, source/package provenance and input/settings identity. Saving happens after that series' timer stops. The launcher supplies --checkpoint-dir; standalone comparison runs can use the same option and add --resume to reuse matching complete checkpoints. A failed backend's partial pages are not treated as a completed measurement.

To recover a failed refresh without repeating completed native/replay measurements:

RECOVER_FROM=/absolute/path/to/scratch-performance-refresh-RUN \
  bash scripts/benchmarking/refresh_performance_benchmarks.sh

Recovery creates a new bundle using the dataset, repetitions and thread counts from the source bundle. It synchronizes the pinned dependencies while retaining the installed local project, checks the recorded machine, Python, core packages, native binary/extension hashes and QPDF version, then validates the complete core results and dataset hashes. It links the original native/replay artifacts, so keep the original bundle. It does not rebuild native targets. If scripts were committed during collection, replay provenance is accepted only when the commit patch exactly matches the recorded build diff and the replay script hash matches that commit. The report explains this equivalence.

Both comparison modes are recollected under the current reference versions. Compatible checkpoints from a previous recovery are copied and reused; a reference-version or source change prevents checkpoint reuse. The original metadata is retained as original-metadata.json. comparison_provenance records the new environment separately, and report tables show package versions. The completed original parse comparison remains in its original bundle.

If recovery is interrupted, use the newly printed recovery directory as RECOVER_FROM to retain compatible completed comparison checkpoints. OUT may select a new output directory; BUILD_MODE=clean is incompatible with recovery because the recorded core binaries must remain unchanged.

You can also inspect the completed core results independently:

uv run --no-sync python scripts/benchmarking/run_performance_report.py \
  /absolute/path/to/scratch-performance-refresh-RUN --core-only

This validates all requested native/replay profiles and writes report-core/, clearly labeling comparisons as excluded. The normal report still requires the complete comparison suite and rejects missing profiles or mixed provenance.

For a finished comparison whose report failed, regenerate directly without recovery or benchmark reruns:

uv run --no-sync python scripts/benchmarking/run_performance_report.py \
  /absolute/path/to/scratch-performance-recovery-RUN

A reference series with page errors is listed under Excluded reference comparisons, with attempted/successful/failed page counts. Its timings remain in excluded_comparisons.csv, and document/page/error details are written to comparison_errors.csv. All repetitions of that series are excluded from performance tables, speedups and figures. Reporting requires full attempted-page coverage, matching checkpoint provenance and checksums; missing pages and native errors still fail validation. Original CSVs, checkpoints and metadata are retained.

Native stage timers and CSV output are enabled; glyph diagnostics and PNG saving are disabled. Every repetition runs both render-only (word/line cell construction disabled) and render-with-text (both enabled), with separate fresh processes for each profile/thread count. Both decode characters and bitmap geometry needed for drawing; Python models and separate bitmap-bound outputs are not materialized. The native wall clock excludes document preload. Replay wall time sums document totals including construction/load, processing and unload. Comparison wall time comes from the exact completed-series checkpoint, including document loading. Older bundles without checkpoints use rounded console wall times. The report checks console rounding against checkpoint timings, including older tables rounded to six significant digits. These remain separate result series.

Every output bundle preserves metadata.json, exact commands, build logs, tests, raw native CSV/log pairs, replay JSON/JSONL and optional comparison CSVs. To regenerate tables and figures:

uv run --no-sync python scripts/benchmarking/run_performance_report.py \
  scratch-performance-new-run

Results go to INPUT/report, or the directory supplied by --output-dir. The report validates complete page sets, repetitions and native log/CSV sums. It writes summary.md, summary.csv, runs.csv, phase CSVs, optional canvas checks, and runtime/scaling/phase/distribution PNGs. Native CSVs are stored under native/<profile>/r*/t*.csv. Profiles are recorded in metadata, and native baselines, stage totals and histograms stay separate. New native figures use native_render_only_* and native_render_with_text_*; legacy bundles keep their original single profile and figure names.

The generated Figures section places C++ before Python, with explicit workload subtitles for images, text geometry and bitmap bounding boxes. Each plot has a caption, and each profile includes shell commands that reproduce its measurements with fresh output paths. A complete launcher command regenerates the whole bundle.

Native histograms use page worker elapsed. Replay histograms use document totals: its page JSONL stores outputs, not per-page timings. Phase bars show the run nearest each median total. Error bars show observed minimum–maximum wall time, not confidence intervals.

Individual Python API comparison

The currently supported modes are parse-without-bitmap, parse-with-bitmap, render, default (parse without bitmap bytes plus render), and all.

uv run --group perf python scripts/benchmarking/run_performance_benchmarking.py \
  ./pdfs --mode render --threads 1,4,8,12 --scale 1 \
  --max-concurrent-results 32 --3rd-party-backends all --only-threaded \
  --output-dir scratch-comparison-new-run

Inputs may be a local PDF/directory or an HF dataset ID. This runner has no revision flag: for a pinned comparison, pass the same snapshot directory used by the other runners, as the launcher does. Add --recursive for nested local PDFs.

Supported flags include:

  • --3rd-party-backends: all, none, or a semicolon-separated list of pymupdf;pypdfium2;pdfplumber;pdfminer.six;pypdf.
  • --only-threaded: skip the sequential Docling parse baseline. Reference packages remain controlled by --3rd-party-backends.
  • --threads, --scale, --max-concurrent-results, --max-pages, --pdf-selection, --bytesio, --output-dir.
  • --keep-char-cells, --create-word-cells, --create-line-cells, --keep-shapes, --keep-bitmaps.
  • --materialize-char-cells, --materialize-word-cells, --materialize-line-cells, --materialize-shapes, --materialize-bitmaps.

Boolean flags take true or false. Bitmap-byte export is selected by the experiment mode; it has no separate CLI switch. Third-party packages are excluded from the bitmap-byte-export mode.

The runner writes <cpu>_<input-name>_<mode>.csv under the supplied output directory and prints the result/configuration tables to stdout. Capture stdout as a log to retain the corpus wall times. It currently does not write a Markdown report. There are no --compare, --other, --pages-csv or --mode both options.

Backend-settings replay

uv run --group perf python \
  scripts/benchmarking/run_performance_benchmarking_with_docling_settings.py \
  docling-project/performance-dataset-bo767 \
  --dataset-revision 221c19e716d6ede5d18fd1560bc938ab19cc1a70 \
  --backend threaded --threads 8 --output-dir scratch-replay-new-run

Choose --backend pdfium, threaded or both. A comma-separated thread list is supported, but use separate invocations for independent process/RSS measurements. --threads default uses accelerator environment resolution; explicit counts are labeled scaling overrides.

Outputs contain source/package/configuration provenance, per-document phase timings, API/output counters and streamed page metadata. The replay includes Python model materialization, separators and RGB images and excludes model inference and Docling framework construction. See the script header for the pinned source inspection and limitations. Native CSV, replay JSONL and comparison CSV are different formats; they must not be passed interchangeably to plotting tools.

Existing page-level exploration tools

_common.py defines the comparison PageRow schema:

backend,task,threads,doc_key,page_number,success,elapsed_s,wall_gap_s,
image_width,image_height,make_page_decoder_s,decode_page_s,
create_word_cells_s,create_line_cells_s,render_page_s,error_message
uv run --group perf python scripts/benchmarking/run_performance_eval.py \
  scratch-comparison-new-run/COMPARISON.csv --viz-dir scratch-comparison-plots

uv run --group perf python scripts/benchmarking/run_performance_analysis.py \
  scratch-comparison-new-run/COMPARISON.csv --backend docling-parse \
  --task 'parse+render' --threads 8 --top 20 --render 1.0

Replace COMPARISON.csv with the filename printed by the measurement runner. Evaluation generates histograms, hexbins, per-document statistics and scaling plots. Its scaling plots sum arrival gaps, which omit document loading and may omit native processing already performed during submission. They do not share the corpus-wall boundary of the publication tables. Use run_performance_report.py for publication runtime and efficiency figures.

Docling page durations are native worker elapsed; reference page durations are wall measurements around page operations. Neither distribution is the amortized corpus wall / page count. Equal-size renders are not automatically pixel-equivalent.

Analysis replays pages and requires the original files. Pass the measured scale and content/decode flags explicitly; defaults may differ from the original run.

Quality benchmarking

uv run --group perf python scripts/benchmarking/run_quality_benchmarking.py ./pdfs
uv run --group perf python scripts/benchmarking/run_quality_benchmarking.py ./pdfs \
  --compare render --reference-renderer pypdfium2 --renderers docling-parse,pymupdf

The default input is the pinned regression dataset in tests.constants. Outputs include render_quality.csv, text_quality.csv, render comparison panels and quality-distribution plots. Quality evidence is separate from the performance refresh; the launcher's damaged-page preflight is not a complete render-quality regression run.