Skip to content

fix: keep DuckDB and Polars results out of pandas unless requested - #191

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/out-of-core-fetch
Sep 14, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/out-of-core-fetch

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Closes #52. Addresses #53.

Summary

#52 — DuckDB: with engine="duckdb", every non-native output format ran conn.execute(sql).fetchdf(). Arrow or Polars output then converted that pandas frame a second time (pa.Table.from_pandas / pl.from_pandas), so the whole result sat in memory twice for callers who never asked for pandas.

#53 — Polars: the dedup stage still counted rows with a plain lf.select(pl.len()).collect(), three times: before dedup, for duplicate detection, and after dedup. Each count re-evaluated the entire upstream plan in memory instead of streaming.

Changes

  • execution/backends/_duckdb.py: the pipeline returns an unfetched relation; the engine then fetches in the requested format:
    • arrow → to_arrow_table(), or fetch_arrow_table() on older duckdb. In duckdb 1.5 .arrow() returns a RecordBatchReader, so it isn't used.
    • polars → .pl()
    • pandas → fetchdf(), unchanged.
    • When semantic_enabled, fetchdf() is kept, because the native semantic stage accepts pandas, Polars and DuckDB relations but not Arrow tables.
  • execution/_report.py: _frame_stats handles pyarrow.Table (rows, columns, null count, nbytes).
  • execution/__init__.py: _convert_output passes an Arrow table straight through.
  • execution/backends/_polars.py: new _count() sends row counts through _collect(), the streaming path when EngineConfig.streaming. _collect now accepts engine_config=None, treated as the streaming default.

Not changed

Subset dedup still uses unique(maintain_order=True), and the report still says that stage evaluates eagerly. I tried a with_row_index + min().over(subset) filter, but explain(engine="streaming") didn't show it streaming any better than unique, so the added complexity wasn't justified. I left #53 open for that part.

Behaviour note

Arrow output from DuckDB now keeps DuckDB's native column types instead of going through pandas dtypes, e.g. DECIMAL stays decimal128 instead of becoming float64. The values match the pandas path. This is noted in the CHANGELOG.

Tests

New tests/test_execution/test_out_of_core_fetch.py (12 cases):

  • DuckDB arrow/polars output never calls fetchdf()/df() (both are patched to raise), and the report's row and column counts match the result.
  • The direct fetch equals the pandas-path output; missing_after and duplicates_removed match.
  • Polars dedup counts use engine="streaming" when streaming is on, and a plain collect when it is off, for both detection and removal.
  • Polars dedup after an empty-row drop matches pandas for full-row and subset dedup with keep="first"/"last": same frame, duplicates_removed and rows_after.

Results:

  • pytest tests/test_execution tests/test_backend_honesty.py tests/test_engine_duplicates.py tests/test_semantic_cleaning.py plus the new file: all pass.
  • ruff check and mypy src/freshdata: clean.
  • make truthbench-pr: backend_parity_comparison: pass, "no gate-level changes vs baseline", overall: PASS.

@coderabbitai

coderabbitai Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: c25e3ce7-5569-4e22-b03e-1c8240d15062


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

DuckDB (#52): output_format="arrow" / "polars" fetched every result with
fetchdf() and then converted the pandas frame again. Fetch directly with
to_arrow_table() (fetch_arrow_table() on older duckdb) or .pl() instead,
keeping fetchdf() for pandas output and when the native semantic stage,
which does not accept Arrow tables, runs afterwards. _frame_stats learns
pyarrow tables and _convert_output passes them through.

Polars (#53): duplicate detection and dedup counted rows with plain
.collect() calls that re-evaluate the whole upstream plan in memory. Route
them through the streaming collect path via a new _count helper; _collect
now accepts engine_config=None (streaming default).
@kevincostner17
kevincostner17 merged commit 20c9d3a into main Sep 14, 2026
16 of 17 checks passed
@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DuckDB engine materializes final results with fetchdf(), defeating out-of-core execution

1 participant