You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Polars streaming is disabled whenever default duplicate removal runs #53
The Polars backend advertises LazyFrame streaming, and EngineConfig.streaming defaults to True. However, the default clean configuration also enables drop_duplicates, which makes the native path choose an order-preserving unique operation that Polars documents as incompatible with its streaming engine. Large native Polars cleans therefore do not receive the advertised streaming behavior by default.
Steps to reproduce
Use a Parquet or LazyFrame source with the native-only configuration:
It calls lf.unique(keep=config.duplicate_keep, maintain_order=True) before _collect requests engine="streaming".
Expected behavior
A run configured for streaming should remain streamable, or clearly report that exact duplicate-order semantics require materialization.
Actual behavior
Polars documents that LazyFrame.unique(..., maintain_order=True) blocks streaming. Because drop_duplicates=True and duplicate_keep="first" are defaults, this applies to the normal native Polars execution path.
Likely root cause
Pandas parity requires the surviving first or last duplicate and original row order, while the Polars streaming-compatible keep="any" path intentionally makes no ordering guarantee.
Suggested first investigation path
Make this tradeoff explicit in the plan. Consider an opt-in unordered duplicate mode that uses a streamable unique operation, or surface a clear warning and benchmark note whenever order-preserving dedup forces eager execution. Add tests that assert the planned backend remains streaming only for streamable stage combinations.
The remaining part is resolved by disclosure, which the issue lists as an acceptable outcome ("clearly report that exact duplicate-order semantics require materialization"). #191 routed the dedup row counts through the streaming collect. On main (execution/backends/_polars.py ~L463-485), full-row dedup under streaming uses a streamable unordered unique, and EngineConfig(streaming_dedup=False) restores order at the cost of materializing. Subset dedup keeps maintain_order=True and the report says the stage "evaluates eagerly rather than streaming". The same tradeoff is documented in docs/backends.md, docs/fallback-matrix.md and docs/limitations.md, and tests/test_backend_honesty.py checks it. #191 also tried a row-index/window rewrite for subset dedup, but explain(engine="streaming") showed no streaming gain. Please reopen if a streamable ordered-subset strategy becomes available upstream in Polars.
Activity
Merge pull request #124 from FreshCode-Org/fix/outofcore-evidence-jwd
fix: keep DuckDB and Polars results out of pandas unless requested
fix: keep DuckDB and Polars results out of pandas unless requested (#191
kevincostner17 commentedon Sep 14, 2026
The remaining part is resolved by disclosure, which the issue lists as an acceptable outcome ("clearly report that exact duplicate-order semantics require materialization"). #191 routed the dedup row counts through the streaming collect. On main (
execution/backends/_polars.py~L463-485), full-row dedup under streaming uses a streamable unorderedunique, andEngineConfig(streaming_dedup=False)restores order at the cost of materializing. Subset dedup keepsmaintain_order=Trueand the report says the stage "evaluates eagerly rather than streaming". The same tradeoff is documented indocs/backends.md,docs/fallback-matrix.mdanddocs/limitations.md, andtests/test_backend_honesty.pychecks it. #191 also tried a row-index/window rewrite for subset dedup, butexplain(engine="streaming")showed no streaming gain. Please reopen if a streamable ordered-subset strategy becomes available upstream in Polars.