Repository navigation
Description
Describe the bug
read_parquet with a filter on an INT96 timestamp column throws if the column chunk has min/max statistics:
CUDF failure at: cpp/src/io/parquet/stats_filter_helpers.hpp:85: Parquet reader encountered a mismatch in size of statistics vector and the cudf storage type
The Type::INT96 case in stats_caster_base::convert (stats_filter_helpers.hpp L145-149) decodes the 12-byte value with decode_fixed_width_value<int64_t> / <int32_t>, so the size check throws. Past that, it builds (nanos << 32) | julian_day rather than a timestamp, and ts_scale is 0 because INT96 has no logical type.
Per the spec, INT96 min/max have no defined order unless the column declares INT96_TIMESTAMP_ORDER (apache/parquet-format#584), and readers should ignore them otherwise. libcudf only parses the TYPE_ORDER arm of ColumnOrder, never reads column_orders, and falls back to the deprecated min/max fields for timestamps.
Writers that emit INT96 min/max: fastparquet (deprecated min/max), parquet-java when min == max (Spark writes INT96 by default), and arrow-rs 60.x with INT96_TIMESTAMP_ORDER (also on parquet-java master).
Steps/Code to reproduce bug
Write the file with fastparquet. libcudf and pyarrow don't write INT96 stats:
import fastparquet
import pandas as pd
df = pd.DataFrame(
{"ts": pd.to_datetime(["2023-01-01", "2023-06-01", "2024-01-01"]).as_unit("ns")}
)
fastparquet.write("int96_stats.parquet", df, times="int96", stats=True)C++:
#include <cudf/ast/expressions.hpp>
#include <cudf/io/parquet.hpp>
#include <cudf/scalar/scalar.hpp>
#include <iostream>
int main(int argc, char** argv)
{
// ts >= 2023-06-01
auto value = cudf::timestamp_scalar<cudf::timestamp_ms>{
cudf::timestamp_ms{cudf::duration_ms{1'685'577'600'000}}, true};
auto literal = cudf::ast::literal{value};
auto col = cudf::ast::column_name_reference{"ts"};
auto filter = cudf::ast::operation{cudf::ast::ast_operator::GREATER_EQUAL, col, literal};
auto options = cudf::io::parquet_reader_options::builder(cudf::io::source_info{argv[1]})
.timestamp_type(cudf::data_type{cudf::type_id::TIMESTAMP_MILLISECONDS})
.filter(filter)
.build();
try {
std::cout << "rows: " << cudf::io::read_parquet(options).tbl->num_rows() << "\n";
} catch (std::exception const& e) {
std::cout << "threw: " << e.what() << "\n";
}
}$ ./repro int96_stats.parquet
threw: CUDF failure at: .../stats_filter_helpers.hpp:85: Parquet reader encountered a mismatch in size of statistics vector and the cudf storage type
$ ./repro int96_nostats.parquet # same data via pyarrow use_deprecated_int96_timestamps=True
rows: 2
Possible pytest, with int96_stats.parquet added to the test data:
def test_parquet_reader_int96_stats_filter(datadir):
fname = datadir / "int96_stats.parquet"
filters = [("ts", ">=", pd.Timestamp("2023-06-01"))]
expect = pd.read_parquet(fname, filters=filters)
got = cudf.read_parquet(fname, filters=filters)
assert_eq(expect, got, check_index_type=False)Expected behavior
Returns 2 rows. Specifically:
- INT96 min/max (including the deprecated
min/maxfields) are ignored unless the column order isINT96_TIMESTAMP_ORDER.null_countstays usable. - With
INT96_TIMESTAMP_ORDER, they're decoded with the same Julian-day conversion as the data path (page_data.cuh) and scaled totimestamp_type. - The same applies to the page-index path (
experimental/page_index_filter.cu).
Environment overview (please complete the following information)
- Environment location: Docker
- Method of cuDF install: from source (libcudf built as part of Velox)
Environment details
libcudf 369efe9 (2026-11-09); same code at c9086e0. CentOS Stream 9, CUDA 12.9, driver 580.82.07, NVIDIA B200.
Additional context
Found while pushing TIMESTAMP filters into the Velox cuDF scan (facebookincubator/velox#20196). Related: #10438 (INT96 stats writing disabled).
Activity
Ignore unordered INT96 min/max in Parquet stats filters (#25177)