Repository navigation
docs+bench: measured out-of-core evidence; correct native-handle recipe and polars-lazy claims (#52, #53) - #124
Conversation
…pe and polars-lazy claims (#52, #53) New benchmarks/bench_outofcore.py measures peak RSS (subprocess-isolated ru_maxrss) for the four engine/output-format combinations on a generated parquet fixture. Measured at 5M rows: duckdb/pandas 1771 MB materialized duckdb/duckdb 723 MB native handle (2.4x lower peak) polars/pandas 2195 MB materialized polars/polars-lazy 2222 MB handle, but NO peak reduction Two doc corrections the measurements forced: - The documented native-handle recipe was incomplete: default fix_dtypes=True runs sampled pandas heuristics and triggers the (recorded) fallback even under strategy='conservative', materializing the result. backends.md/limitations.md now state the working recipe: strategy='conservative' AND fix_dtypes=False. - polars-lazy was described as larger-than-RAM safe end to end; in fact the Polars pipeline collects intermediates eagerly between stages, so the lazy handle defers only the final materialization and peak RSS during cleaning matches eager output. Docs now state the per-backend boundary; README points at the measured boundary instead of a flat larger-than-memory claim. No code behavior changes. CLAIM_REGISTRY claims untouched.
Strix Security ReviewNo security issues found. Updated for Reviewed by Strix |
|
Warning Review limit reached
Next review available in: 21 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|---|---|---|---|---|---|---|---|---|---|---|
| crm | 10,200 | 40 | 1.314 | 1.315 | 14.0 | 100.0 | 0.0 | 100.0 | 93.84 | ✅ | 100.0 |
| event_log | 10,000 | 25 | 0.502 | 0.506 | 5.1 | 100.0 | 0.0 | 100.0 | 99.677 | ✅ | 100.0 |
| finance | 10,200 | 60 | 1.697 | 1.703 | 18.3 | 100.0 | 0.0 | 100.0 | 99.499 | ✅ | 100.0 |
| gold | 10,200 | 7 | 0.196 | 0.199 | 3.7 | 100.0 | 0.0 | 100.0 | 98.3 | ✅ | 100.0 |
| provenance | 10,000 | 18 | 0.411 | 0.416 | 7.2 | 100.0 | 0.0 | 100.0 | 99.765 | ✅ | 100.0 |
| wide_schema | 10,000 | 100 | 3.109 | 3.117 | 13.4 | 100.0 | 0.0 | 100.0 | 96.042 | ✅ | 100.0 |
Authored-code reduction (Metric 6)
- FreshData: 3 lines
- pandas baseline: 26 lines (88.46% reduction)
Summary
Evidence pass for the out-of-core claims behind issues #52 / #53. 1.1.0 already shipped the honest handles (
output_format="duckdb"/"polars-lazy",report.materialized, streaming-safe dedupe), but there was no measured evidence, and two documented recipes turned out not to match actual behavior when measured.What was measured
New
benchmarks/bench_outofcore.py: each engine/output-format combination cleans the same generated parquet fixture in an isolated subprocess (so allocator reuse can't blur numbers); peak RSS is read fromru_maxrssat exit. The fixture itself is written batch-by-batch, never held in memory.Results on this machine (5M rows, mixed dtypes,
strategy="conservative",fix_dtypes=False):Reproduce:
python benchmarks/bench_outofcore.py --rows 5000000Two doc corrections the measurements forced
The documented native-handle recipe was incomplete. Docs said
strategy="conservative"keeps the native handle. It does not: defaultfix_dtypes=Trueruns sampled pandas heuristics, triggers the (honestly recorded) fallback, and materializes —report.materialized=True, result is a pandas frame. The working recipe isstrategy="conservative"andfix_dtypes=False. Both backends.md and limitations.md now say so.polars-lazywas called "larger-than-RAM safe end to end" — measurement says otherwise. The Polars pipeline collects intermediates eagerly between stages (the residue of Polars streaming is disabled whenever default duplicate removal runs #53), so the lazy handle defers only the final materialization: peak RSS during cleaning is the same as eager output (2222 vs 2195 MB). The DuckDB handle is the genuinely lower-peak path (723 vs 1771 MB, 2.4×). backends.md and limitations.md now state the boundary per backend; the README's "larger-than-memory data" bullet now points at the measured boundary instead of claiming it flat.No code behavior changes; no public API changes. The
CLAIM_REGISTRY-tracked claims are untouched (tests/test_cleanbench_public_release.pygreen).Verification
python benchmarks/bench_outofcore.pyrun at 200k / 2M / 5M rows — pattern stable across sizes.pytest -m "not online and not large": green, coverage ≥ 93%.ruff check src tests+ new benchmark file clean (PLC0415 per-file-ignore added, matching the existing bench_streaming/bench_report entries);mypy src/freshdataclean.