Skip to content

feat(parquet): write Arrow timestamps to INT96 columns - #12263

Open
ranflarion wants to merge 1 commit into
apache:mainfrom
ranflarion:parquet-write-int96
Open

ranflarion wants to merge 1 commit into
apache:mainfrom
ranflarion:parquet-write-int96

Conversation

@ranflarion

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

ArrowWriter panics when the supplied Parquet schema declares an INT96 column, so Arrow timestamps cannot be written as INT96, Spark's default timestamp type.

What changes are included in this PR?

  • write_leaf writes Timestamp arrays of any unit to INT96 columns as the Julian day and the nanoseconds of the day, converting only non-null values. Other Arrow types get a "Cannot coerce" error, as for the other physical types, and a value whose Julian day does not fit in i32 gets an error.
  • A crate-private Int96::from_epoch_nanos for the conversion, compiled with the arrow feature.

Are these changes tested?

Yes: a round trip through INT96 of second, microsecond (with a time zone) and nanosecond columns and a list of millisecond timestamps, with nulls and values before the epoch; an out-of-range error; unit tests of the conversion.

Are there any significant user-facing changes?

No public API changes. Writers that do not declare INT96 in their Parquet schema are unaffected.

AI usage disclosure

AI tools were used to write the code, tests and this description. I own this change.

@boring-cyborg boring-cyborg Bot added the parquet Changes to the parquet crate label Nov 15, 2026
@Rich-T-kid Rich-T-kid added the enhancement New feature/addition label Nov 16, 2026
@alamb

alamb commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

@mbutrovich can you help review this?

Is the usecase to create files to read by Spark?

@ranflarion

Copy link
Copy Markdown
Contributor Author

@mbutrovich can you help review this?

Is the usecase to create files to read by Spark?

yeah that's the use case

@mbutrovich

mbutrovich commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

@mbutrovich can you help review this?

Is the usecase to create files to read by Spark?

Sure thing. My gut reaction is that I really struggle to justify putting more INT96 writers into the world (a long-deprecated type). But maybe that's a topic for the mailing list.

@mbutrovich mbutrovich left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ranflarion. Following up on my comment above, I'm not convinced arrow-rs should add an INT96 writer, and I'd like us to settle that question before the code goes further.

The spec says new writers should not write INT96 (parquet.thrift). For Arrow input I can't find a case where INT96 is the better choice:

  • It adds no range or resolution. An Arrow millisecond, microsecond, or nanosecond value is an i64 count of its unit, so it fits INT64 TIMESTAMP of that unit exactly. INT96's 32-bit Julian day is narrower than Arrow seconds and milliseconds, which is why this PR needs an out-of-range error at all. For a range wider than INT64 nanoseconds, the spec now allows TIMESTAMP on FIXED_LEN_BYTE_ARRAY(12) (LogicalTypes.md, apache/parquet-format#601).

  • It's larger. INT96 values take 12 bytes instead of 8, and the spec allows neither DELTA_BINARY_PACKED nor BYTE_STREAM_SPLIT for INT96 (Encodings.md, Encodings.md). Here are file sizes for 1M Timestamp(Microsecond, "UTC") values written with this PR:

    Input Compression INT64, defaults INT96, defaults INT64, DELTA_BINARY_PACKED
    Sorted, 1 s apart plus random microseconds none 8.27 MB 12.18 MB 2.66 MB
    Sorted, 1 s apart plus random microseconds zstd 4.08 MB 4.54 MB 2.66 MB
    Random over 50 years none 8.27 MB 12.18 MB 6.55 MB
    Random over 50 years zstd 7.56 MB 9.82 MB 6.55 MB
  • Its statistics are less useful. Readers that don't know the INT96_TIMESTAMP_ORDER column order have to ignore min and max for the column (parquet.thrift), so they can't prune on it.

For Spark, INT96 isn't needed. Spark reads INT64 TIMESTAMP(MICROS) with isAdjustedToUTC set to true as TimestampType (ParquetSchemaConverter.scala). Spark's own source calls that type recommended and keeps INT96 as the default only for backward compatibility (ParquetSchemaConverter.scala). arrow-rs already writes that type for Timestamp(Microsecond, Some(tz)) (schema/mod.rs). Which reader do you need to support that can't read it? If there is one, I think the case for an INT96 writer belongs on the dev mailing list.

Spark 3.x may also refuse to read what this writes. A file without org.apache.spark.version in its footer gets its INT96 calendar rebase mode from spark.sql.parquet.int96RebaseModeInRead (DataSourceUtils.scala). In Spark 3.5 that defaults to EXCEPTION, which fails the read "if it sees ancient timestamps that are ambiguous between the two calendars" (SQLConf.scala). Spark master defaults to CORRECTED (SQLConf.scala). So whether Spark can read old timestamps from this writer depends on its version and settings.

There's also the question of who maintains this. Since March 2025 arrow-rs has merged eight INT96 changes for reading and statistics alone: #7285, #7367, #7687, #7975, #8034, #8603, #10106, and #11348. A writer is new surface that someone has to keep correct as the spec and other readers change.

),
ColumnWriter::Int96ColumnWriter(_typed) => {
unreachable!("Currently unreachable because data type not supported")
ColumnWriter::Int96ColumnWriter(typed) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With bloom filters enabled, ArrowWriter now writes a bloom filter for an INT96 column, but ArrowSbbf can't check it. I wrote Timestamp(Microsecond, "UTC") values to an INT96 column with set_bloom_filter_enabled(true). ArrowSbbf::try_new with that Arrow type then returned a Cannot check a bloom filter of type INT96 with ... error. validate only accepts the types that arrow_to_parquet_type produces, and that function never produces INT96 (bloom_filter.rs). The INT96 arm of check also still says ArrowWriter does not write INT96 columns (bloom_filter.rs).

ArrowSbbf exists so that callers can check Arrow values converted the same way ArrowWriter converts them. Could the conversion move into a visit_int96 that both write_leaf and ArrowSbbf::check call, the way visit_i64 works, with validate accepting Timestamp for INT96 columns?

Comment on lines +1772 to +1788
let (values, nanos_per_unit) = match column.data_type() {
ArrowDataType::Timestamp(TimeUnit::Second, _) => (
column.as_primitive::<TimestampSecondType>().values(),
1_000_000_000,
),
ArrowDataType::Timestamp(TimeUnit::Millisecond, _) => (
column.as_primitive::<TimestampMillisecondType>().values(),
1_000_000,
),
ArrowDataType::Timestamp(TimeUnit::Microsecond, _) => (
column.as_primitive::<TimestampMicrosecondType>().values(),
1_000,
),
ArrowDataType::Timestamp(TimeUnit::Nanosecond, _) => {
(column.as_primitive::<TimestampNanosecondType>().values(), 1)
}
d => return Err(ParquetError::General(format!("Cannot coerce {d} to INT96"))),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This writes Timestamp(_, None) and Timestamp(_, Some(tz)) as the same INT96 values, and INT96 has no isAdjustedToUTC to tell them apart. Spark reads every INT96 column as TimestampType, an instant (ParquetSchemaConverter.scala). For INT64 it reads isAdjustedToUTC = false, which arrow-rs writes for Timestamp(_, None), as TimestampNTZType (ParquetSchemaConverter.scala). So a timestamp without a time zone becomes an instant in Spark only when it's written as INT96. Spark itself never writes TimestampNTZType as INT96 (ParquetSchemaConverter.scala).

The arrow-rs reader makes the opposite assumption. Without the embedded Arrow schema, it reads INT96 as Timestamp(Nanosecond, None) (primitive.rs).

Should INT96 accept only timestamps with a time zone, and return the "Cannot coerce" error for Timestamp(_, None)?

Comment on lines +1790 to +1801
let values = indices
.iter()
.map(|&i| {
Int96::from_epoch_nanos(values[i] as i128 * nanos_per_unit).ok_or_else(|| {
ParquetError::General(format!(
"{} value {} is out of range for INT96",
column.data_type(),
values[i]
))
})
})
.collect::<Result<Vec<_>>>()?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

values[i] as i128 * nanos_per_unit makes every value an i128 division, which compiles to a call to __divti3 per value on both aarch64 and x86_64. Could we divide the i64 value by the unit's count per day with arrow_buffer::DivisorI64 instead? Build it once per batch, match on its variant before the loop, and move the variant into the closure, the way divisor_op in arrow-arith does (numeric.rs). The divisor is always positive, so MIN / -1 can't happen, and flooring is one step: subtract one from the quotient and add the divisor to the remainder when the remainder is negative. When I tried this, each value became a high multiply and shifts (smulh on aarch64, a one-operand imul on x86_64), with no division and no zero or overflow checks. #11467 measured DivisorI64 at 2.4x to 4.9x faster than i64 / and % on x86_64 for divisors that aren't powers of two, its smallest gain on any target, and the code here starts from an i128 division.

collect::<Result<Vec<_>>>() also calls GenericShunt::next out of line for each value, so the divisor's fields are reloaded from memory every time. The DivisorI64 docs warn about that (divisor.rs). Could this be a plain loop that pushes into Vec::with_capacity(indices.len()) and returns on the first error?

}

#[test]
fn arrow_writer_timestamp_out_of_int96_range() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test checks one value far past the range. Could we test at the boundaries instead? For Timestamp(Second), 185331720383999 and -185753453990400 are the largest and smallest values whose Julian day fits in i32. They should round trip, and one second past each should return the error.

The conversion skips null slots on purpose, so an out-of-range value under a null doesn't fail the write. Could a test cover that too, with a null slot that holds i64::MAX?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature/addition parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet: write Arrow timestamps to INT96 columns

4 participants