Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 134 additions & 3 deletions parquet/src/arrow/arrow_writer/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ use crate::column::writer::encoder::ColumnValueEncoder;
use crate::column::writer::{
ColumnCloseResult, ColumnWriter, ColumnWriterImpl, GenericColumnWriter, get_column_writer,
};
use crate::data_type::{ByteArray, FixedLenByteArray, FixedLenByteArrayType};
use crate::data_type::{ByteArray, FixedLenByteArray, FixedLenByteArrayType, Int96};
use std::collections::HashSet;
type DistinctValuesSet = HashSet<u64>;
#[cfg(feature = "encryption")]
Expand Down Expand Up @@ -1768,8 +1768,46 @@ fn write_leaf(
levels,
},
),
ColumnWriter::Int96ColumnWriter(_typed) => {
unreachable!("Currently unreachable because data type not supported")
ColumnWriter::Int96ColumnWriter(typed) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With bloom filters enabled, ArrowWriter now writes a bloom filter for an INT96 column, but ArrowSbbf can't check it. I wrote Timestamp(Microsecond, "UTC") values to an INT96 column with set_bloom_filter_enabled(true). ArrowSbbf::try_new with that Arrow type then returned a Cannot check a bloom filter of type INT96 with ... error. validate only accepts the types that arrow_to_parquet_type produces, and that function never produces INT96 (bloom_filter.rs). The INT96 arm of check also still says ArrowWriter does not write INT96 columns (bloom_filter.rs).

ArrowSbbf exists so that callers can check Arrow values converted the same way ArrowWriter converts them. Could the conversion move into a visit_int96 that both write_leaf and ArrowSbbf::check call, the way visit_i64 works, with validate accepting Timestamp for INT96 columns?

let (values, nanos_per_unit) = match column.data_type() {
ArrowDataType::Timestamp(TimeUnit::Second, _) => (
column.as_primitive::<TimestampSecondType>().values(),
1_000_000_000,
),
ArrowDataType::Timestamp(TimeUnit::Millisecond, _) => (
column.as_primitive::<TimestampMillisecondType>().values(),
1_000_000,
),
ArrowDataType::Timestamp(TimeUnit::Microsecond, _) => (
column.as_primitive::<TimestampMicrosecondType>().values(),
1_000,
),
ArrowDataType::Timestamp(TimeUnit::Nanosecond, _) => {
(column.as_primitive::<TimestampNanosecondType>().values(), 1)
}
d => return Err(ParquetError::General(format!("Cannot coerce {d} to INT96"))),
Comment on lines +1772 to +1788

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This writes Timestamp(_, None) and Timestamp(_, Some(tz)) as the same INT96 values, and INT96 has no isAdjustedToUTC to tell them apart. Spark reads every INT96 column as TimestampType, an instant (ParquetSchemaConverter.scala). For INT64 it reads isAdjustedToUTC = false, which arrow-rs writes for Timestamp(_, None), as TimestampNTZType (ParquetSchemaConverter.scala). So a timestamp without a time zone becomes an instant in Spark only when it's written as INT96. Spark itself never writes TimestampNTZType as INT96 (ParquetSchemaConverter.scala).

The arrow-rs reader makes the opposite assumption. Without the embedded Arrow schema, it reads INT96 as Timestamp(Nanosecond, None) (primitive.rs).

Should INT96 accept only timestamps with a time zone, and return the "Cannot coerce" error for Timestamp(_, None)?

};
let values = indices
.iter()
.map(|&i| {
Int96::from_epoch_nanos(values[i] as i128 * nanos_per_unit).ok_or_else(|| {
ParquetError::General(format!(
"{} value {} is out of range for INT96",
column.data_type(),
values[i]
))
})
})
.collect::<Result<Vec<_>>>()?;
Comment on lines +1790 to +1801

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

values[i] as i128 * nanos_per_unit makes every value an i128 division, which compiles to a call to __divti3 per value on both aarch64 and x86_64. Could we divide the i64 value by the unit's count per day with arrow_buffer::DivisorI64 instead? Build it once per batch, match on its variant before the loop, and move the variant into the closure, the way divisor_op in arrow-arith does (numeric.rs). The divisor is always positive, so MIN / -1 can't happen, and flooring is one step: subtract one from the quotient and add the divisor to the remainder when the remainder is negative. When I tried this, each value became a high multiply and shifts (smulh on aarch64, a one-operand imul on x86_64), with no division and no zero or overflow checks. #11467 measured DivisorI64 at 2.4x to 4.9x faster than i64 / and % on x86_64 for divisors that aren't powers of two, its smallest gain on any target, and the code here starts from an i128 division.

collect::<Result<Vec<_>>>() also calls GenericShunt::next out of line for each value, so the divisor's fields are reloaded from memory every time. The DivisorI64 docs warn about that (divisor.rs). Could this be a plain loop that pushes into Vec::with_capacity(indices.len()) and returns on the first error?

typed.write_batch_internal(
&values,
None,
levels.def_level_data().as_ref(),
levels.rep_level_data().as_ref(),
None,
None,
None,
)
}
ColumnWriter::FloatColumnWriter(typed) => {
let array = column.as_primitive::<Float32Type>();
Expand Down Expand Up @@ -6342,4 +6380,97 @@ mod tests {
assert_eq!(fixed, (-1, 1), "incorrect Arrow FixedSizeBinaryArray stats");
assert_eq!(binary, (-1, 1), "incorrect Arrow BinaryArray stats");
}

#[test]
fn arrow_writer_timestamps_to_int96() {
let seconds =
TimestampSecondArray::from(vec![Some(-1), None, Some(1_700_000_000), None, None]);
let micros = TimestampMicrosecondArray::from(vec![
Some(0),
Some(-1),
None,
Some(-86_400_000_001),
Some(1_700_000_000_123_456),
])
.with_timezone("UTC");
let nanos = TimestampNanosecondArray::from(vec![
Some(1),
Some(-1),
Some(1_700_000_000_123_456_789),
None,
Some(-86_400_000_000_001),
]);
let millis = TimestampMillisecondArray::from(vec![Some(-1), None, Some(1_700_000_000_123)]);
let list = ListArray::new(
Arc::new(Field::new("element", millis.data_type().clone(), true)),
OffsetBuffer::from_lengths([2, 0, 1, 0, 0]),
Arc::new(millis),
None,
);
let schema = Arc::new(Schema::new(vec![
Field::new("s", seconds.data_type().clone(), true),
Field::new("us", micros.data_type().clone(), true),
Field::new("ns", nanos.data_type().clone(), true),
Field::new("list", list.data_type().clone(), false),
]));
let batch = RecordBatch::try_new(
schema.clone(),
vec![
Arc::new(seconds),
Arc::new(micros),
Arc::new(nanos),
Arc::new(list),
],
)
.unwrap();
let parquet_schema = crate::schema::parser::parse_message_type(
"message schema {
OPTIONAL INT96 s;
OPTIONAL INT96 us;
OPTIONAL INT96 ns;
REQUIRED group list (LIST) {
REPEATED group list {
OPTIONAL INT96 element;
}
}
}",
)
.unwrap();
let options = ArrowWriterOptions::new()
.with_parquet_schema(SchemaDescriptor::new(Arc::new(parquet_schema)));
let mut buf = Vec::new();
let mut writer = ArrowWriter::try_new_with_options(&mut buf, schema, options).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();

let builder = ParquetRecordBatchReaderBuilder::try_new(Bytes::from(buf)).unwrap();
let descr = builder.metadata().file_metadata().schema_descr_ptr();
assert!(
descr
.columns()
.iter()
.all(|c| c.physical_type() == crate::basic::Type::INT96)
);
let read = builder.build().unwrap().next().unwrap().unwrap();
assert_eq!(read, batch);
}

#[test]
fn arrow_writer_timestamp_out_of_int96_range() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test checks one value far past the range. Could we test at the boundaries instead? For Timestamp(Second), 185331720383999 and -185753453990400 are the largest and smallest values whose Julian day fits in i32. They should round trip, and one second past each should return the error.

The conversion skips null slots on purpose, so an out-of-range value under a null doesn't fail the write. Could a test cover that too, with a null slot that holds i64::MAX?

let seconds = TimestampSecondArray::from(vec![i64::MAX]);
let schema = Arc::new(Schema::new(vec![Field::new(
"s",
seconds.data_type().clone(),
false,
)]));
let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(seconds)]).unwrap();
let parquet_schema =
crate::schema::parser::parse_message_type("message schema { REQUIRED INT96 s; }")
.unwrap();
let options = ArrowWriterOptions::new()
.with_parquet_schema(SchemaDescriptor::new(Arc::new(parquet_schema)));
let mut writer = ArrowWriter::try_new_with_options(Vec::new(), schema, options).unwrap();
let err = writer.write(&batch).unwrap_err();
assert!(err.to_string().contains("out of range for INT96"), "{err}");
}
}
36 changes: 36 additions & 0 deletions parquet/src/data_type.rs
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,18 @@ impl Int96 {
.wrapping_add(nanos)
}

/// Creates an INT96 from nanoseconds since EPOCH, or `None` if the Julian day exceeds `i32`
#[cfg(feature = "arrow")]
pub(crate) fn from_epoch_nanos(nanos: i128) -> Option<Self> {
let nanos_in_day = NANOSECONDS_IN_DAY as i128;
let day =
i32::try_from(nanos.div_euclid(nanos_in_day) + JULIAN_DAY_OF_EPOCH as i128).ok()?;
let nanos_of_day = nanos.rem_euclid(nanos_in_day) as u64;
let mut result = Self::new();
result.set_data(nanos_of_day as u32, (nanos_of_day >> 32) as u32, day as u32);
Some(result)
}

#[inline]
fn get_days(&self) -> i32 {
self.data()[2] as i32
Expand Down Expand Up @@ -1400,6 +1412,30 @@ mod tests {
);
}

#[test]
#[cfg(feature = "arrow")]
fn test_int96_from_epoch_nanos() {
for nanos in [
0,
1,
-1,
86_400_000_000_000,
-86_400_000_000_001,
1_700_000_000_123_456_789,
] {
assert_eq!(
Int96::from_epoch_nanos(nanos).unwrap().to_nanos() as i128,
nanos
);
}
let nanos_of_day: u64 = 86_399_999_999_000;
assert_eq!(
Int96::from_epoch_nanos(-1_000).unwrap().data(),
&[nanos_of_day as u32, (nanos_of_day >> 32) as u32, 2_440_587]
);
assert!(Int96::from_epoch_nanos(i64::MAX as i128 * 1_000_000_000).is_none());
}

#[test]
fn test_byte_array_from() {
assert_eq!(ByteArray::from(b"ABC".to_vec()).data(), b"ABC");
Expand Down
Loading