Skip to content

Polars streaming is disabled whenever default duplicate removal runs #53

Closed

Description

@JohnnyWilson16

Why it matters

The Polars backend advertises LazyFrame streaming, and EngineConfig.streaming defaults to True. However, the default clean configuration also enables drop_duplicates, which makes the native path choose an order-preserving unique operation that Polars documents as incompatible with its streaming engine. Large native Polars cleans therefore do not receive the advertised streaming behavior by default.

Steps to reproduce

  1. Use a Parquet or LazyFrame source with the native-only configuration:
from freshdata import clean
from freshdata.config import CleanConfig
from freshdata.execution import EngineConfig

clean(
    "large.parquet",
    config=CleanConfig(strategy="conservative", fix_dtypes=False),
    engine="polars",
    engine_config=EngineConfig(engine="polars", streaming=True),
)
  1. Follow PolarsEngine._stage_drop_duplicates.
  2. It calls lf.unique(keep=config.duplicate_keep, maintain_order=True) before _collect requests engine="streaming".

Expected behavior

A run configured for streaming should remain streamable, or clearly report that exact duplicate-order semantics require materialization.

Actual behavior

Polars documents that LazyFrame.unique(..., maintain_order=True) blocks streaming. Because drop_duplicates=True and duplicate_keep="first" are defaults, this applies to the normal native Polars execution path.

Likely root cause

Pandas parity requires the surviving first or last duplicate and original row order, while the Polars streaming-compatible keep="any" path intentionally makes no ordering guarantee.

Suggested first investigation path

Make this tradeoff explicit in the plan. Consider an opt-in unordered duplicate mode that uses a streamable unique operation, or surface a clear warning and benchmark note whenever order-preserving dedup forces eager execution. Add tests that assert the planned backend remains streaming only for streamable stage combinations.

Activity

  1. added a commit that references this issue on Jul 11, 2026
  2. kevincostner17 commented on Sep 14, 2026

    @kevincostner17
    Contributor

    The remaining part is resolved by disclosure, which the issue lists as an acceptable outcome ("clearly report that exact duplicate-order semantics require materialization"). #191 routed the dedup row counts through the streaming collect. On main (execution/backends/_polars.py ~L463-485), full-row dedup under streaming uses a streamable unordered unique, and EngineConfig(streaming_dedup=False) restores order at the cost of materializing. Subset dedup keeps maintain_order=True and the report says the stage "evaluates eagerly rather than streaming". The same tradeoff is documented in docs/backends.md, docs/fallback-matrix.md and docs/limitations.md, and tests/test_backend_honesty.py checks it. #191 also tried a row-index/window rewrite for subset dedup, but explain(engine="streaming") showed no streaming gain. Please reopen if a streamable ordered-subset strategy becomes available upstream in Polars.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    benchmarksPerformance benchmarksbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions