Repository navigation
feat(parquet): write Arrow timestamps to INT96 columns - #12263
ranflarion wants to merge 1 commit into
Conversation
|
@mbutrovich can you help review this? Is the usecase to create files to read by Spark? |
yeah that's the use case |
Sure thing. My gut reaction is that I really struggle to justify putting more INT96 writers into the world (a long-deprecated type). But maybe that's a topic for the mailing list. |
There was a problem hiding this comment.
Thanks @ranflarion. Following up on my comment above, I'm not convinced arrow-rs should add an INT96 writer, and I'd like us to settle that question before the code goes further.
The spec says new writers should not write INT96 (parquet.thrift). For Arrow input I can't find a case where INT96 is the better choice:
-
It adds no range or resolution. An Arrow millisecond, microsecond, or nanosecond value is an
i64count of its unit, so it fits INT64TIMESTAMPof that unit exactly. INT96's 32-bit Julian day is narrower than Arrow seconds and milliseconds, which is why this PR needs an out-of-range error at all. For a range wider than INT64 nanoseconds, the spec now allowsTIMESTAMPonFIXED_LEN_BYTE_ARRAY(12)(LogicalTypes.md, apache/parquet-format#601). -
It's larger. INT96 values take 12 bytes instead of 8, and the spec allows neither
DELTA_BINARY_PACKEDnorBYTE_STREAM_SPLITfor INT96 (Encodings.md, Encodings.md). Here are file sizes for 1MTimestamp(Microsecond, "UTC")values written with this PR:Input Compression INT64, defaults INT96, defaults INT64, DELTA_BINARY_PACKEDSorted, 1 s apart plus random microseconds none 8.27 MB 12.18 MB 2.66 MB Sorted, 1 s apart plus random microseconds zstd 4.08 MB 4.54 MB 2.66 MB Random over 50 years none 8.27 MB 12.18 MB 6.55 MB Random over 50 years zstd 7.56 MB 9.82 MB 6.55 MB -
Its statistics are less useful. Readers that don't know the
INT96_TIMESTAMP_ORDERcolumn order have to ignore min and max for the column (parquet.thrift), so they can't prune on it.
For Spark, INT96 isn't needed. Spark reads INT64 TIMESTAMP(MICROS) with isAdjustedToUTC set to true as TimestampType (ParquetSchemaConverter.scala). Spark's own source calls that type recommended and keeps INT96 as the default only for backward compatibility (ParquetSchemaConverter.scala). arrow-rs already writes that type for Timestamp(Microsecond, Some(tz)) (schema/mod.rs). Which reader do you need to support that can't read it? If there is one, I think the case for an INT96 writer belongs on the dev mailing list.
Spark 3.x may also refuse to read what this writes. A file without org.apache.spark.version in its footer gets its INT96 calendar rebase mode from spark.sql.parquet.int96RebaseModeInRead (DataSourceUtils.scala). In Spark 3.5 that defaults to EXCEPTION, which fails the read "if it sees ancient timestamps that are ambiguous between the two calendars" (SQLConf.scala). Spark master defaults to CORRECTED (SQLConf.scala). So whether Spark can read old timestamps from this writer depends on its version and settings.
There's also the question of who maintains this. Since March 2025 arrow-rs has merged eight INT96 changes for reading and statistics alone: #7285, #7367, #7687, #7975, #8034, #8603, #10106, and #11348. A writer is new surface that someone has to keep correct as the spec and other readers change.
| ), | ||
| ColumnWriter::Int96ColumnWriter(_typed) => { | ||
| unreachable!("Currently unreachable because data type not supported") | ||
| ColumnWriter::Int96ColumnWriter(typed) => { |
There was a problem hiding this comment.
With bloom filters enabled, ArrowWriter now writes a bloom filter for an INT96 column, but ArrowSbbf can't check it. I wrote Timestamp(Microsecond, "UTC") values to an INT96 column with set_bloom_filter_enabled(true). ArrowSbbf::try_new with that Arrow type then returned a Cannot check a bloom filter of type INT96 with ... error. validate only accepts the types that arrow_to_parquet_type produces, and that function never produces INT96 (bloom_filter.rs). The INT96 arm of check also still says ArrowWriter does not write INT96 columns (bloom_filter.rs).
ArrowSbbf exists so that callers can check Arrow values converted the same way ArrowWriter converts them. Could the conversion move into a visit_int96 that both write_leaf and ArrowSbbf::check call, the way visit_i64 works, with validate accepting Timestamp for INT96 columns?
| let (values, nanos_per_unit) = match column.data_type() { | ||
| ArrowDataType::Timestamp(TimeUnit::Second, _) => ( | ||
| column.as_primitive::<TimestampSecondType>().values(), | ||
| 1_000_000_000, | ||
| ), | ||
| ArrowDataType::Timestamp(TimeUnit::Millisecond, _) => ( | ||
| column.as_primitive::<TimestampMillisecondType>().values(), | ||
| 1_000_000, | ||
| ), | ||
| ArrowDataType::Timestamp(TimeUnit::Microsecond, _) => ( | ||
| column.as_primitive::<TimestampMicrosecondType>().values(), | ||
| 1_000, | ||
| ), | ||
| ArrowDataType::Timestamp(TimeUnit::Nanosecond, _) => { | ||
| (column.as_primitive::<TimestampNanosecondType>().values(), 1) | ||
| } | ||
| d => return Err(ParquetError::General(format!("Cannot coerce {d} to INT96"))), |
There was a problem hiding this comment.
This writes Timestamp(_, None) and Timestamp(_, Some(tz)) as the same INT96 values, and INT96 has no isAdjustedToUTC to tell them apart. Spark reads every INT96 column as TimestampType, an instant (ParquetSchemaConverter.scala). For INT64 it reads isAdjustedToUTC = false, which arrow-rs writes for Timestamp(_, None), as TimestampNTZType (ParquetSchemaConverter.scala). So a timestamp without a time zone becomes an instant in Spark only when it's written as INT96. Spark itself never writes TimestampNTZType as INT96 (ParquetSchemaConverter.scala).
The arrow-rs reader makes the opposite assumption. Without the embedded Arrow schema, it reads INT96 as Timestamp(Nanosecond, None) (primitive.rs).
Should INT96 accept only timestamps with a time zone, and return the "Cannot coerce" error for Timestamp(_, None)?
| let values = indices | ||
| .iter() | ||
| .map(|&i| { | ||
| Int96::from_epoch_nanos(values[i] as i128 * nanos_per_unit).ok_or_else(|| { | ||
| ParquetError::General(format!( | ||
| "{} value {} is out of range for INT96", | ||
| column.data_type(), | ||
| values[i] | ||
| )) | ||
| }) | ||
| }) | ||
| .collect::<Result<Vec<_>>>()?; |
There was a problem hiding this comment.
values[i] as i128 * nanos_per_unit makes every value an i128 division, which compiles to a call to __divti3 per value on both aarch64 and x86_64. Could we divide the i64 value by the unit's count per day with arrow_buffer::DivisorI64 instead? Build it once per batch, match on its variant before the loop, and move the variant into the closure, the way divisor_op in arrow-arith does (numeric.rs). The divisor is always positive, so MIN / -1 can't happen, and flooring is one step: subtract one from the quotient and add the divisor to the remainder when the remainder is negative. When I tried this, each value became a high multiply and shifts (smulh on aarch64, a one-operand imul on x86_64), with no division and no zero or overflow checks. #11467 measured DivisorI64 at 2.4x to 4.9x faster than i64 / and % on x86_64 for divisors that aren't powers of two, its smallest gain on any target, and the code here starts from an i128 division.
collect::<Result<Vec<_>>>() also calls GenericShunt::next out of line for each value, so the divisor's fields are reloaded from memory every time. The DivisorI64 docs warn about that (divisor.rs). Could this be a plain loop that pushes into Vec::with_capacity(indices.len()) and returns on the first error?
| } | ||
|
|
||
| #[test] | ||
| fn arrow_writer_timestamp_out_of_int96_range() { |
There was a problem hiding this comment.
This test checks one value far past the range. Could we test at the boundaries instead? For Timestamp(Second), 185331720383999 and -185753453990400 are the largest and smallest values whose Julian day fits in i32. They should round trip, and one second past each should return the error.
The conversion skips null slots on purpose, so an out-of-range value under a null doesn't fail the write. Could a test cover that too, with a null slot that holds i64::MAX?
Which issue does this PR close?
Rationale for this change
ArrowWriterpanics when the supplied Parquet schema declares an INT96 column, so Arrow timestamps cannot be written as INT96, Spark's default timestamp type.What changes are included in this PR?
write_leafwritesTimestamparrays of any unit to INT96 columns as the Julian day and the nanoseconds of the day, converting only non-null values. Other Arrow types get a "Cannot coerce" error, as for the other physical types, and a value whose Julian day does not fit ini32gets an error.Int96::from_epoch_nanosfor the conversion, compiled with thearrowfeature.Are these changes tested?
Yes: a round trip through INT96 of second, microsecond (with a time zone) and nanosecond columns and a list of millisecond timestamps, with nulls and values before the epoch; an out-of-range error; unit tests of the conversion.
Are there any significant user-facing changes?
No public API changes. Writers that do not declare INT96 in their Parquet schema are unaffected.
AI usage disclosure
AI tools were used to write the code, tests and this description. I own this change.