You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Make bloom filters / ArrowSbbf it's own crate #11681
I'm interested in using our bloom filter implementation in DataFusion outside of the context of parquet. To do that we'd want to have it be it's own crate. It seems like a good general implementation of bloom filters that is compatible w/ the arrow ecosystem. Shall we make it it's own crate so that it can be used outside of the context of parquet?
That's what I'm not sure of: in implementation, yes certainly. In spirit: maybe not? If we are going to choose to match the encodings to the Parquet spec / implementations (so that these bloom filters and encodings can be used in parquet or interact w/ parquet's bloom filters) then having parquet in the name sounds reasonable?
I think calling it parquet-bloom-filter would be appropriate as it implements the bloom filters specified in the Parquet spec; So even if you want to use it outside of parquet files, it would still be correct to label them "Parquet Bloom Filters" I think
Late to the party, but this seems like a good idea to me too. I've been playing around with trying to do for Bloom filters what I've been doing for the page index (#11475 and prior), but I think I'll hold off for a bit and see where this work goes. Thanks @adriangb
I think this would probably be a pretty simple job for the AI machine to pull out into its own crate. We should then make sure the docs,etc are reasonable but othrwise let's try and do it for 60.2.0
Activity
adriangb commentedon Oct 18, 2026
cc @Jefffrey @alamb
Rich-T-kid commentedon Oct 19, 2026
related issue #11518
alamb commentedon Oct 19, 2026
Seems like a reasonable idea to me. Maybe a crate like
parquet-bloom-filter?Rich-T-kid commentedon Oct 19, 2026
this can be distinct from parquet, right?
adriangb commentedon Oct 19, 2026
That's what I'm not sure of: in implementation, yes certainly. In spirit: maybe not? If we are going to choose to match the encodings to the Parquet spec / implementations (so that these bloom filters and encodings can be used in parquet or interact w/ parquet's bloom filters) then having
parquetin the name sounds reasonable?alamb commentedon Oct 26, 2026
I think calling it
parquet-bloom-filterwould be appropriate as it implements the bloom filters specified in the Parquet spec; So even if you want to use it outside of parquet files, it would still be correct to label them "Parquet Bloom Filters" I thinketseidl commentedon Oct 28, 2026
Late to the party, but this seems like a good idea to me too. I've been playing around with trying to do for Bloom filters what I've been doing for the page index (#11475 and prior), but I think I'll hold off for a bit and see where this work goes. Thanks @adriangb
alamb commentedon Oct 28, 2026
I think this would probably be a pretty simple job for the AI machine to pull out into its own crate. We should then make sure the docs,etc are reasonable but othrwise let's try and do it for 60.2.0
etseidl commentedon Oct 28, 2026
I've got #11975 (on top of #11953) submitted now that may simplify this some 🤷
PTAL 🙏
adriangb commentedon Nov 12, 2026
related: #8277