Skip to content

Static planner: null-aware anti join loses its probe-side coalesce under an aggregate #2849

Open

Description

@ewoodbury

Describe the bug

With the static planner, a NOT IN subquery planned as a null-aware anti hash join can return wrong results, with no error, when an aggregate sits above it and the join's stage is split into tasks.

A null-aware anti join has to see the whole right side in one task, so lower_null_aware_join puts a CoalescePartitionsExec under its probe side. That coalesce sits inside the join's stage. When an aggregate is above the join, the planner runs EnsureRequirements over that stage again, and it removes the coalesce because the join does not ask for one. The join then runs in several tasks, each seeing only part of the right side.

To Reproduce

A standalone cluster with:

  • ballista.planner.adaptive.enabled = false
  • datafusion.optimizer.prefer_hash_join = true
  • ballista.optimizer.not_in_subquery_rewrite = false
  • ballista.scheduler.max_partitions_per_task = 1

Run a GROUP BY over a NOT IN subquery on a table read from several files. The counts came back as 1190/1155/1155, against 170/165/165 from DataFusion.

Expected behavior

The same counts as DataFusion.

Additional context

#2848 hits the same problem for nested-loop joins and avoids it by giving the probe side a one-partition stage of its own instead of a coalesce inside the join's stage. The same approach should work here.

Activity

  1. milenkovicm commented on Nov 16, 2026

    @milenkovicm
    Contributor

    Can't this issue be closed @ewoodbury ?

  2. NoahKusaba commented on Nov 16, 2026

    @NoahKusaba
    Contributor

    Can't this issue be closed @ewoodbury ?

    Sorry @milenkovicm I deleted my last comment, I wasn't 100% sure this ticket can be deleted. I still hope that older issues can be cleaned-up, as it's hard to parse what's important in there.

  3. milenkovicm commented on Nov 16, 2026

    @milenkovicm
    Contributor

    Can't this issue be closed @ewoodbury ?

    Sorry @milenkovicm I deleted my last comment, I wasn't 100% sure this ticket can be deleted. I still hope that older issues can be cleaned-up, as it's hard to parse what's important in there.

    No worries, tag me for other issues you find

  4. ewoodbury commented on Nov 19, 2026

    @ewoodbury
    ContributorAuthor

    Still an issue, just opened #2898 to address!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions