Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/source/contributor-guide/timezones.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,12 +185,13 @@ value by a timezone:
| `TIMESTAMP`, `isAdjustedToUTC=false` | `TimestampType` | Relabelled without shifting, as Spark does |
| `TIMESTAMP`, `isAdjustedToUTC=false` | `TimestampNTZType` | Read as is |
| `TIMESTAMP`, `isAdjustedToUTC=true` | `TimestampNTZType` | Rejected on Spark 3.x (SPARK-36182). Relabelled without shifting on Spark 4.0+ (SPARK-47447). |
| `DATE` | `TimestampNTZType` | Rejected on Spark 3.5. Scaled from days to microseconds with an overflow check on Spark 4.0+. |
| `INT96` | `TimestampType` | Coerced to microseconds and labelled `"UTC"` |

The `INT96` coercion is configured by `coerce_int96` and `coerce_int96_tz` in `parquet_exec.rs`. An
`INT96` column read as `TimestampNTZType` follows the `isAdjustedToUTC=true` row.

The timestamp-to-timestamp adaptations go through `CometCastColumnExpr` and
The date-to-NTZ and timestamp-to-timestamp adaptations go through `CometCastColumnExpr` and
`parquet_convert_array` in `native/core/src/parquet/`, not through Spark's `Cast`. A Spark cast
between `TimestampType` and `TimestampNTZType` would apply the session timezone, and the Parquet
reader does not.
Expand Down Expand Up @@ -222,6 +223,13 @@ path cannot handle through the JVM codegen dispatcher, as they do for a session
`timeZoneId` stamped on the expression, so the results match Spark, and its Arrow output is
labelled `"UTC"` like everything else.

`CAST(date AS TIMESTAMP)` uses codegen dispatch for region zones and offsets with seconds.
Spark's `LocalDate.atStartOfDay` applies the JVM's transition rules even for years beyond
chrono's calendar. This also covers implicit casts when comparing a date with a timestamp or
passing a date to `hour` or `date_trunc`. For a fixed offset in whole minutes, the Rust kernel
subtracts the offset from the start of the epoch day in seconds before converting to
microseconds. This order preserves instants that fit only after applying the offset.

## Guidelines

- Never read `TimeZone.getDefault`, `ZoneId.systemDefault` or the host's local time in Comet code.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -123,18 +123,27 @@ Supported input formats match Spark exactly:

## Date to Timestamp

Comet's native `CAST(date AS TIMESTAMP)` is compatible with Spark. The cast interprets each
date as midnight in the session timezone and converts to a UTC epoch value. DST transitions
are handled correctly, including spring-forward gaps (where midnight may not exist) and
fall-back ambiguity (where Comet picks the earlier/DST occurrence, matching Spark's
`LocalDate.atStartOfDay(zoneId)` behavior).
`CAST(date AS TIMESTAMP)` converts the start of the date in the session timezone to microseconds
since the Unix epoch. For fixed offsets in whole minutes, including `Etc/UTC` and `GMT+05:30`,
Comet computes this in Rust. For region zones such as `America/Los_Angeles` and offsets with
seconds, it uses codegen dispatch so Spark applies the JVM's timezone rules across the full date
range. The cast falls back to Spark when dispatch is unavailable.

## Date to TimestampNTZ

Comet's native `CAST(date AS TIMESTAMP_NTZ)` is compatible with Spark. The cast is
timezone-independent: each date is converted to midnight as pure arithmetic
(`days * 86,400,000,000` microseconds) with no session timezone offset applied. The result
is the same regardless of the session timezone setting.
`CAST(date AS TIMESTAMP_NTZ)` multiplies the epoch day by 86,400,000,000 to get microseconds at
midnight, independently of the session timezone.

For either timestamp type, `CAST` raises `ArithmeticException("long overflow")` if the result
exceeds the range of a signed 64-bit integer, even with ANSI mode disabled. `TRY_CAST` returns
`NULL` for the overflowing timestamp. When Spark declares that timestamp non-nullable, Comet
falls back to Spark because the declared Arrow field cannot represent the null.

As with [other casts evaluated in batches](../../index.md#ansi-errors-from-rows-skipped-by-limit),
Comet can reach an overflowing row that Spark skips under a `LIMIT`. With the optional
`spark.comet.parquet.rowFilterPushdown.enabled` setting, overflow while evaluating the
pushed predicate can surface as a generic query error because the reader converts predicate
errors to strings ([DataFusion #24638](https://github.com/apache/datafusion/pull/24638)).

## Date to Numeric Types

Expand Down
4 changes: 4 additions & 0 deletions docs/source/user-guide/latest/parquet.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,10 @@ reads that fail where Spark succeeds are listed in
TIMESTAMP_MILLIS). On Spark 4.0+, this read is permitted
([SPARK-47447](https://issues.apache.org/jira/browse/SPARK-47447)) and Comet matches Spark's
behavior.
- Reading a Parquet `DATE` as `TIMESTAMP_NTZ` on Spark 3.5. Spark 3.5 rejects this type
promotion. On Spark 4.0 and later, the reader converts epoch days to microseconds at midnight.
If a decoded date overflows during projection, the reader raises a file-read error caused
by `ArithmeticException("long overflow")`.

### Missing Nested Fields

Expand Down
26 changes: 25 additions & 1 deletion native/common/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,10 @@ pub enum SparkError {
function_name: String,
},

/// Plain ArithmeticException from Math.multiplyExact, independent of ANSI mode.
#[error("long overflow")]
LongOverflow,
Comment thread
sunchao marked this conversation as resolved.
Resolved

#[error("[ARITHMETIC_OVERFLOW] Overflow in integral divide. Use 'try_divide' to tolerate overflow and return NULL instead. If necessary set \"spark.sql.ansi.enabled\" to \"false\" to bypass this error.")]
IntegralDivideOverflow,

Expand Down Expand Up @@ -269,7 +273,7 @@ pub enum SparkError {
spark_type: String,
},

/// Overflow in Parquet's millis-to-micros conversion. The per-file reader fills in the
/// Overflow converting Parquet days or milliseconds to microseconds. The reader fills in the
/// original Spark path before the JVM wraps this in cannotReadFilesError.
#[error("long overflow")]
ParquetTimestampOverflow { file_path: String },
Expand Down Expand Up @@ -360,6 +364,7 @@ impl SparkError {
SparkError::CastOverFlow { .. } => "CastOverFlow",
SparkError::CannotParseDecimal => "CannotParseDecimal",
SparkError::ArithmeticOverflow { .. } => "ArithmeticOverflow",
SparkError::LongOverflow => "LongOverflow",
SparkError::IntegralDivideOverflow => "IntegralDivideOverflow",
SparkError::DecimalSumOverflow { .. } => "DecimalSumOverflow",
SparkError::DivideByZero => "DivideByZero",
Expand Down Expand Up @@ -720,6 +725,8 @@ impl SparkError {
/// Returns the appropriate Spark exception class for this error
pub fn exception_class(&self) -> &'static str {
match self {
SparkError::LongOverflow => "java/lang/ArithmeticException",

// ArithmeticException
SparkError::DivideByZero
| SparkError::RemainderByZero
Expand Down Expand Up @@ -840,6 +847,7 @@ impl SparkError {
SparkError::RemainderByZero => Some("REMAINDER_BY_ZERO"),
SparkError::IntervalDividedByZero => Some("INTERVAL_DIVIDED_BY_ZERO"),
SparkError::ArithmeticOverflow { .. } => Some("ARITHMETIC_OVERFLOW"),
SparkError::LongOverflow => None,
SparkError::IntegralDivideOverflow => Some("ARITHMETIC_OVERFLOW"),
SparkError::DecimalSumOverflow { .. } => Some("ARITHMETIC_OVERFLOW"),
SparkError::BinaryArithmeticOverflow { .. } => Some("BINARY_ARITHMETIC_OVERFLOW"),
Expand Down Expand Up @@ -1141,6 +1149,22 @@ mod tests {
assert!(json.contains("\"errorClass\":\"REMAINDER_BY_ZERO\""));
}

#[test]
fn test_long_overflow_json() {
let error = SparkError::LongOverflow;
let parsed: serde_json::Value = serde_json::from_str(&error.to_json()).unwrap();
assert_eq!(
parsed,
serde_json::json!({
"errorType": "LongOverflow",
"errorClass": "",
"params": {},
})
);
assert_eq!(error.exception_class(), "java/lang/ArithmeticException");
assert_eq!(error.to_string(), "long overflow");
}

#[test]
fn test_arithmetic_overflow_suggestion_json_and_display() {
for function in ["", "try_add", "try_subtract", "try_multiply"] {
Expand Down
91 changes: 89 additions & 2 deletions native/common/src/struct_nulls.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.

//! Pushing a struct's null mask down into its children.
//! Propagating parent visibility to nested children.
//!
//! Arrow stores a `StructArray`'s children with their own validity, independent of the parent's
//! null buffer, so at a row where the struct itself is null a child buffer can still hold a value
Expand All @@ -28,10 +28,14 @@
//! whatever sits in the child slot
//! (<https://github.com/apache/datafusion-comet/issues/5753>).

use arrow::array::{make_array, Array, ArrayRef, StructArray};
use arrow::array::{
make_array, Array, ArrayRef, AsArray, GenericListViewArray, NullBufferBuilder, OffsetSizeTrait,
StructArray,
};
use arrow::buffer::NullBuffer;
use arrow::datatypes::DataType;
use arrow::error::ArrowError;
use std::cell::LazyCell;
use std::sync::Arc;

/// The parent's nulls unioned into one child, or the child unchanged when there is nothing to push.
Expand Down Expand Up @@ -76,6 +80,89 @@ pub fn children_with_parent_nulls(struct_array: &StructArray) -> Result<Vec<Arra
}
}

/// Slots visible through the ancestors of a nested array. A checked leaf requests its
/// mask after field resolution, so each ancestor computes the mask at most once per
/// batch and only when one of its selected children needs it.
pub type Visibility<'a> = LazyCell<Option<NullBuffer>, &'a dyn Fn() -> Option<NullBuffer>>;

/// Map the visible slots of a container to the indices of its children.
pub fn child_visibility(array: &dyn Array, parent: &Visibility<'_>) -> Option<NullBuffer> {
let visible = NullBuffer::union(array.nulls(), parent.as_ref());
let nulls = visible.as_ref();
match array.data_type() {
DataType::List(_) => {
let list = array.as_list::<i32>();
repeated_visibility(list.value_offsets(), list.values().len(), nulls)
}
DataType::LargeList(_) => {
let list = array.as_list::<i64>();
repeated_visibility(list.value_offsets(), list.values().len(), nulls)
}
DataType::FixedSizeList(_, size) => {
// Arrow slices the child values along with a fixed-size list. Expand in those
// child coordinates; applying a parent slice offset again would shift the mask.
nulls.map(|nulls| nulls.expand(*size as usize))
}
DataType::ListView(_) => list_view_visibility(array.as_list_view::<i32>(), nulls),
DataType::LargeListView(_) => list_view_visibility(array.as_list_view::<i64>(), nulls),
DataType::Map(_, _) => {
let map = array.as_map();
repeated_visibility(map.value_offsets(), map.entries().len(), nulls)
}
_ => visible,
}
}

fn list_view_visibility<O: OffsetSizeTrait>(
list: &GenericListViewArray<O>,
nulls: Option<&NullBuffer>,
) -> Option<NullBuffer> {
let mut valid = vec![false; list.values().len()];
for (row, (offset, size)) in list
.value_offsets()
.iter()
.zip(list.value_sizes())
.enumerate()
{
if nulls.is_none_or(|nulls| nulls.is_valid(row)) {
let start = offset.as_usize();
// Views can overlap or be out of order. Accumulate the union of visible ranges;
// a null parent must never clear values another parent can see.
valid[start..start + size.as_usize()].fill(true);
}
}
Some(NullBuffer::from(valid))
}

fn repeated_visibility<O: OffsetSizeTrait>(
offsets: &[O],
len: usize,
nulls: Option<&NullBuffer>,
) -> Option<NullBuffer> {
let start = offsets[0].as_usize();
let end = offsets.last().unwrap().as_usize();
let mut visible = NullBufferBuilder::new(len);
if start > 0 {
visible.append_n_nulls(start);
}
if let Some(nulls) = nulls.filter(|n| n.null_count() > 0) {
for (valid, range) in nulls.iter().zip(offsets.windows(2)) {
let count = range[1].as_usize() - range[0].as_usize();
if valid {
visible.append_n_non_nulls(count);
} else if count > 0 {
visible.append_n_nulls(count);
}
}
} else {
visible.append_n_non_nulls(end - start);
}
if end < len {
visible.append_n_nulls(len - end);
}
visible.finish()
}

#[cfg(test)]
mod tests {
use super::*;
Expand Down
Loading
Loading