Skip to content

Make bloom filters / ArrowSbbf it's own crate #11681

Open

Description

@adriangb

I'm interested in using our bloom filter implementation in DataFusion outside of the context of parquet. To do that we'd want to have it be it's own crate. It seems like a good general implementation of bloom filters that is compatible w/ the arrow ecosystem. Shall we make it it's own crate so that it can be used outside of the context of parquet?

Activity

  1. adriangb commented on Oct 18, 2026

    @adriangb
    ContributorAuthor
  2. Rich-T-kid commented on Oct 19, 2026

    @Rich-T-kid
    Contributor

    related issue #11518

  3. alamb commented on Oct 19, 2026

    @alamb
    Contributor

    Seems like a reasonable idea to me. Maybe a crate like parquet-bloom-filter ?

  4. Rich-T-kid commented on Oct 19, 2026

    @Rich-T-kid
    Contributor

    this can be distinct from parquet, right?

  5. adriangb commented on Oct 19, 2026

    @adriangb
    ContributorAuthor

    That's what I'm not sure of: in implementation, yes certainly. In spirit: maybe not? If we are going to choose to match the encodings to the Parquet spec / implementations (so that these bloom filters and encodings can be used in parquet or interact w/ parquet's bloom filters) then having parquet in the name sounds reasonable?

  6. alamb commented on Oct 26, 2026

    @alamb
    Contributor

    I think calling it parquet-bloom-filter would be appropriate as it implements the bloom filters specified in the Parquet spec; So even if you want to use it outside of parquet files, it would still be correct to label them "Parquet Bloom Filters" I think

  7. etseidl commented on Oct 28, 2026

    @etseidl
    Contributor

    Late to the party, but this seems like a good idea to me too. I've been playing around with trying to do for Bloom filters what I've been doing for the page index (#11475 and prior), but I think I'll hold off for a bit and see where this work goes. Thanks @adriangb

  8. alamb commented on Oct 28, 2026

    @alamb
    Contributor

    I think this would probably be a pretty simple job for the AI machine to pull out into its own crate. We should then make sure the docs,etc are reasonable but othrwise let's try and do it for 60.2.0

  9. etseidl commented on Oct 28, 2026

    @etseidl
    Contributor

    I've got #11975 (on top of #11953) submitted now that may simplify this some 🤷

    PTAL 🙏

  10. self-assigned this
    on Nov 5, 2026
  11. adriangb commented on Nov 12, 2026

    @adriangb
    ContributorAuthor

    related: #8277

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions