Skip to content

docs+bench: measured out-of-core evidence; correct native-handle recipe and polars-lazy claims (#52, #53) - #124

Merged
JohnnyWilson16 merged 1 commit into
mainfrom
fix/outofcore-evidence-jwd
Jul 11, 2026
Merged

JohnnyWilson16 merged 1 commit into
mainfrom
fix/outofcore-evidence-jwd

Conversation

@JohnnyWilson16

Copy link
Copy Markdown
Contributor

Summary

Evidence pass for the out-of-core claims behind issues #52 / #53. 1.1.0 already shipped the honest handles (output_format="duckdb" / "polars-lazy", report.materialized, streaming-safe dedupe), but there was no measured evidence, and two documented recipes turned out not to match actual behavior when measured.

What was measured

New benchmarks/bench_outofcore.py: each engine/output-format combination cleans the same generated parquet fixture in an isolated subprocess (so allocator reuse can't blur numbers); peak RSS is read from ru_maxrss at exit. The fixture itself is written batch-by-batch, never held in memory.

Results on this machine (5M rows, mixed dtypes, strategy="conservative", fix_dtypes=False):

engine output_format materialized peak RSS (MB) seconds
duckdb pandas True 1771 5.6
duckdb duckdb False 723 3.5
polars pandas True 2195 4.2
polars polars-lazy False 2222 5.0

Reproduce: python benchmarks/bench_outofcore.py --rows 5000000

Two doc corrections the measurements forced

  1. The documented native-handle recipe was incomplete. Docs said strategy="conservative" keeps the native handle. It does not: default fix_dtypes=True runs sampled pandas heuristics, triggers the (honestly recorded) fallback, and materializes — report.materialized=True, result is a pandas frame. The working recipe is strategy="conservative" and fix_dtypes=False. Both backends.md and limitations.md now say so.

  2. polars-lazy was called "larger-than-RAM safe end to end" — measurement says otherwise. The Polars pipeline collects intermediates eagerly between stages (the residue of Polars streaming is disabled whenever default duplicate removal runs #53), so the lazy handle defers only the final materialization: peak RSS during cleaning is the same as eager output (2222 vs 2195 MB). The DuckDB handle is the genuinely lower-peak path (723 vs 1771 MB, 2.4×). backends.md and limitations.md now state the boundary per backend; the README's "larger-than-memory data" bullet now points at the measured boundary instead of claiming it flat.

No code behavior changes; no public API changes. The CLAIM_REGISTRY-tracked claims are untouched (tests/test_cleanbench_public_release.py green).

Verification

  • python benchmarks/bench_outofcore.py run at 200k / 2M / 5M rows — pattern stable across sizes.
  • Full gate pytest -m "not online and not large": green, coverage ≥ 93%.
  • ruff check src tests + new benchmark file clean (PLC0415 per-file-ignore added, matching the existing bench_streaming/bench_report entries); mypy src/freshdata clean.

…pe and polars-lazy claims (#52, #53)

New benchmarks/bench_outofcore.py measures peak RSS (subprocess-isolated
ru_maxrss) for the four engine/output-format combinations on a generated
parquet fixture. Measured at 5M rows:

  duckdb/pandas       1771 MB   materialized
  duckdb/duckdb        723 MB   native handle (2.4x lower peak)
  polars/pandas       2195 MB   materialized
  polars/polars-lazy  2222 MB   handle, but NO peak reduction

Two doc corrections the measurements forced:
- The documented native-handle recipe was incomplete: default
  fix_dtypes=True runs sampled pandas heuristics and triggers the
  (recorded) fallback even under strategy='conservative', materializing
  the result. backends.md/limitations.md now state the working recipe:
  strategy='conservative' AND fix_dtypes=False.
- polars-lazy was described as larger-than-RAM safe end to end; in fact
  the Polars pipeline collects intermediates eagerly between stages, so
  the lazy handle defers only the final materialization and peak RSS
  during cleaning matches eager output. Docs now state the per-backend
  boundary; README points at the measured boundary instead of a flat
  larger-than-memory claim.

No code behavior changes. CLAIM_REGISTRY claims untouched.
@strix-security

strix-security Bot commented Jul 11, 2026 •

Copy link
Copy Markdown
Contributor

Strix Security Review

No security issues found.

Updated for 860c015.


Reviewed by Strix
Re-run review · Configure security review settings

@coderabbitai

coderabbitai Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@JohnnyWilson16, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 21 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c05e2543-1cb6-4b85-9b45-8f7dcbd43b5d

📥 Commits

Reviewing files that changed from the base of the PR and between a5c1bdc and 860c015.

📒 Files selected for processing (6)
  • CHANGELOG.md
  • README.md
  • benchmarks/bench_outofcore.py
  • docs/backends.md
  • docs/limitations.md
  • pyproject.toml
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/outofcore-evidence-jwd

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

FreshData benchmark report — 2026-07-11T04:44:12Z

  • freshdata: 1.1.1
  • python: 3.12.13
  • platform: Linux-6.17.0-1018-azure-x86_64-with-glibc2.39
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %
crm 10,200 40 1.314 1.315 14.0 100.0 0.0 100.0 93.84 ✅ 100.0
event_log 10,000 25 0.502 0.506 5.1 100.0 0.0 100.0 99.677 ✅ 100.0
finance 10,200 60 1.697 1.703 18.3 100.0 0.0 100.0 99.499 ✅ 100.0
gold 10,200 7 0.196 0.199 3.7 100.0 0.0 100.0 98.3 ✅ 100.0
provenance 10,000 18 0.411 0.416 7.2 100.0 0.0 100.0 99.765 ✅ 100.0
wide_schema 10,000 100 3.109 3.117 13.4 100.0 0.0 100.0 96.042 ✅ 100.0

Authored-code reduction (Metric 6)

  • FreshData: 3 lines
  • pandas baseline: 26 lines (88.46% reduction)

@JohnnyWilson16
JohnnyWilson16 merged commit 426d57a into main Jul 11, 2026
15 checks passed
@kevincostner17
kevincostner17 deleted the fix/outofcore-evidence-jwd branch September 14, 2026 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants