Skip to content

[BUG] Parquet stats filter throws on INT96 min/max statistics #25165

Closed

Description

@mhaseeb123

Describe the bug

read_parquet with a filter on an INT96 timestamp column throws if the column chunk has min/max statistics:

CUDF failure at: cpp/src/io/parquet/stats_filter_helpers.hpp:85: Parquet reader encountered a mismatch in size of statistics vector and the cudf storage type

The Type::INT96 case in stats_caster_base::convert (stats_filter_helpers.hpp L145-149) decodes the 12-byte value with decode_fixed_width_value<int64_t> / <int32_t>, so the size check throws. Past that, it builds (nanos << 32) | julian_day rather than a timestamp, and ts_scale is 0 because INT96 has no logical type.

Per the spec, INT96 min/max have no defined order unless the column declares INT96_TIMESTAMP_ORDER (apache/parquet-format#584), and readers should ignore them otherwise. libcudf only parses the TYPE_ORDER arm of ColumnOrder, never reads column_orders, and falls back to the deprecated min/max fields for timestamps.

Writers that emit INT96 min/max: fastparquet (deprecated min/max), parquet-java when min == max (Spark writes INT96 by default), and arrow-rs 60.x with INT96_TIMESTAMP_ORDER (also on parquet-java master).

Steps/Code to reproduce bug
Write the file with fastparquet. libcudf and pyarrow don't write INT96 stats:

import fastparquet
import pandas as pd

df = pd.DataFrame(
    {"ts": pd.to_datetime(["2023-01-01", "2023-06-01", "2024-01-01"]).as_unit("ns")}
)
fastparquet.write("int96_stats.parquet", df, times="int96", stats=True)

C++:

#include <cudf/ast/expressions.hpp>
#include <cudf/io/parquet.hpp>
#include <cudf/scalar/scalar.hpp>

#include <iostream>

int main(int argc, char** argv)
{
  // ts >= 2023-06-01
  auto value   = cudf::timestamp_scalar<cudf::timestamp_ms>{
    cudf::timestamp_ms{cudf::duration_ms{1'685'577'600'000}}, true};
  auto literal = cudf::ast::literal{value};
  auto col     = cudf::ast::column_name_reference{"ts"};
  auto filter  = cudf::ast::operation{cudf::ast::ast_operator::GREATER_EQUAL, col, literal};

  auto options = cudf::io::parquet_reader_options::builder(cudf::io::source_info{argv[1]})
                   .timestamp_type(cudf::data_type{cudf::type_id::TIMESTAMP_MILLISECONDS})
                   .filter(filter)
                   .build();
  try {
    std::cout << "rows: " << cudf::io::read_parquet(options).tbl->num_rows() << "\n";
  } catch (std::exception const& e) {
    std::cout << "threw: " << e.what() << "\n";
  }
}
$ ./repro int96_stats.parquet
threw: CUDF failure at: .../stats_filter_helpers.hpp:85: Parquet reader encountered a mismatch in size of statistics vector and the cudf storage type
$ ./repro int96_nostats.parquet   # same data via pyarrow use_deprecated_int96_timestamps=True
rows: 2

Possible pytest, with int96_stats.parquet added to the test data:

def test_parquet_reader_int96_stats_filter(datadir):
    fname = datadir / "int96_stats.parquet"
    filters = [("ts", ">=", pd.Timestamp("2023-06-01"))]
    expect = pd.read_parquet(fname, filters=filters)
    got = cudf.read_parquet(fname, filters=filters)
    assert_eq(expect, got, check_index_type=False)

Expected behavior
Returns 2 rows. Specifically:

  • INT96 min/max (including the deprecated min/max fields) are ignored unless the column order is INT96_TIMESTAMP_ORDER. null_count stays usable.
  • With INT96_TIMESTAMP_ORDER, they're decoded with the same Julian-day conversion as the data path (page_data.cuh) and scaled to timestamp_type.
  • The same applies to the page-index path (experimental/page_index_filter.cu).

Environment overview (please complete the following information)

  • Environment location: Docker
  • Method of cuDF install: from source (libcudf built as part of Velox)

Environment details
libcudf 369efe9 (2026-11-09); same code at c9086e0. CentOS Stream 9, CUDA 12.9, driver 580.82.07, NVIDIA B200.

Additional context
Found while pushing TIMESTAMP filters into the Velox cuDF scan (facebookincubator/velox#20196). Related: #10438 (INT96 stats writing disabled).

Activity

  1. self-assigned this
    on Nov 10, 2026
  2. added a commit that references this issue on Nov 13, 2026
    322470e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions