diff --git a/docs/source/contributor-guide/timezones.md b/docs/source/contributor-guide/timezones.md index 951b8b93ee5..55790913f1b 100644 --- a/docs/source/contributor-guide/timezones.md +++ b/docs/source/contributor-guide/timezones.md @@ -185,12 +185,13 @@ value by a timezone: | `TIMESTAMP`, `isAdjustedToUTC=false` | `TimestampType` | Relabelled without shifting, as Spark does | | `TIMESTAMP`, `isAdjustedToUTC=false` | `TimestampNTZType` | Read as is | | `TIMESTAMP`, `isAdjustedToUTC=true` | `TimestampNTZType` | Rejected on Spark 3.x (SPARK-36182). Relabelled without shifting on Spark 4.0+ (SPARK-47447). | +| `DATE` | `TimestampNTZType` | Rejected on Spark 3.5. Scaled from days to microseconds with an overflow check on Spark 4.0+. | | `INT96` | `TimestampType` | Coerced to microseconds and labelled `"UTC"` | The `INT96` coercion is configured by `coerce_int96` and `coerce_int96_tz` in `parquet_exec.rs`. An `INT96` column read as `TimestampNTZType` follows the `isAdjustedToUTC=true` row. -The timestamp-to-timestamp adaptations go through `CometCastColumnExpr` and +The date-to-NTZ and timestamp-to-timestamp adaptations go through `CometCastColumnExpr` and `parquet_convert_array` in `native/core/src/parquet/`, not through Spark's `Cast`. A Spark cast between `TimestampType` and `TimestampNTZType` would apply the session timezone, and the Parquet reader does not. @@ -222,6 +223,13 @@ path cannot handle through the JVM codegen dispatcher, as they do for a session `timeZoneId` stamped on the expression, so the results match Spark, and its Arrow output is labelled `"UTC"` like everything else. +`CAST(date AS TIMESTAMP)` uses codegen dispatch for region zones and offsets with seconds. +Spark's `LocalDate.atStartOfDay` applies the JVM's transition rules even for years beyond +chrono's calendar. This also covers implicit casts when comparing a date with a timestamp or +passing a date to `hour` or `date_trunc`. For a fixed offset in whole minutes, the Rust kernel +subtracts the offset from the start of the epoch day in seconds before converting to +microseconds. This order preserves instants that fit only after applying the offset. + ## Guidelines - Never read `TimeZone.getDefault`, `ZoneId.systemDefault` or the host's local time in Comet code. diff --git a/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md b/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md index 4dbf360aeba..3d3aebe4de0 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md +++ b/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md @@ -123,18 +123,27 @@ Supported input formats match Spark exactly: ## Date to Timestamp -Comet's native `CAST(date AS TIMESTAMP)` is compatible with Spark. The cast interprets each -date as midnight in the session timezone and converts to a UTC epoch value. DST transitions -are handled correctly, including spring-forward gaps (where midnight may not exist) and -fall-back ambiguity (where Comet picks the earlier/DST occurrence, matching Spark's -`LocalDate.atStartOfDay(zoneId)` behavior). +`CAST(date AS TIMESTAMP)` converts the start of the date in the session timezone to microseconds +since the Unix epoch. For fixed offsets in whole minutes, including `Etc/UTC` and `GMT+05:30`, +Comet computes this in Rust. For region zones such as `America/Los_Angeles` and offsets with +seconds, it uses codegen dispatch so Spark applies the JVM's timezone rules across the full date +range. The cast falls back to Spark when dispatch is unavailable. ## Date to TimestampNTZ -Comet's native `CAST(date AS TIMESTAMP_NTZ)` is compatible with Spark. The cast is -timezone-independent: each date is converted to midnight as pure arithmetic -(`days * 86,400,000,000` microseconds) with no session timezone offset applied. The result -is the same regardless of the session timezone setting. +`CAST(date AS TIMESTAMP_NTZ)` multiplies the epoch day by 86,400,000,000 to get microseconds at +midnight, independently of the session timezone. + +For either timestamp type, `CAST` raises `ArithmeticException("long overflow")` if the result +exceeds the range of a signed 64-bit integer, even with ANSI mode disabled. `TRY_CAST` returns +`NULL` for the overflowing timestamp. When Spark declares that timestamp non-nullable, Comet +falls back to Spark because the declared Arrow field cannot represent the null. + +As with [other casts evaluated in batches](../../index.md#ansi-errors-from-rows-skipped-by-limit), +Comet can reach an overflowing row that Spark skips under a `LIMIT`. With the optional +`spark.comet.parquet.rowFilterPushdown.enabled` setting, overflow while evaluating the +pushed predicate can surface as a generic query error because the reader converts predicate +errors to strings ([DataFusion #24638](https://github.com/apache/datafusion/pull/24638)). ## Date to Numeric Types diff --git a/docs/source/user-guide/latest/parquet.md b/docs/source/user-guide/latest/parquet.md index 42f450afdc1..ced04271267 100644 --- a/docs/source/user-guide/latest/parquet.md +++ b/docs/source/user-guide/latest/parquet.md @@ -111,6 +111,10 @@ reads that fail where Spark succeeds are listed in TIMESTAMP_MILLIS). On Spark 4.0+, this read is permitted ([SPARK-47447](https://issues.apache.org/jira/browse/SPARK-47447)) and Comet matches Spark's behavior. +- Reading a Parquet `DATE` as `TIMESTAMP_NTZ` on Spark 3.5. Spark 3.5 rejects this type + promotion. On Spark 4.0 and later, the reader converts epoch days to microseconds at midnight. + If a decoded date overflows during projection, the reader raises a file-read error caused + by `ArithmeticException("long overflow")`. ### Missing Nested Fields diff --git a/native/common/src/error.rs b/native/common/src/error.rs index c227c3f1b8e..9d7109eff08 100644 --- a/native/common/src/error.rs +++ b/native/common/src/error.rs @@ -81,6 +81,10 @@ pub enum SparkError { function_name: String, }, + /// Plain ArithmeticException from Math.multiplyExact, independent of ANSI mode. + #[error("long overflow")] + LongOverflow, + #[error("[ARITHMETIC_OVERFLOW] Overflow in integral divide. Use 'try_divide' to tolerate overflow and return NULL instead. If necessary set \"spark.sql.ansi.enabled\" to \"false\" to bypass this error.")] IntegralDivideOverflow, @@ -269,7 +273,7 @@ pub enum SparkError { spark_type: String, }, - /// Overflow in Parquet's millis-to-micros conversion. The per-file reader fills in the + /// Overflow converting Parquet days or milliseconds to microseconds. The reader fills in the /// original Spark path before the JVM wraps this in cannotReadFilesError. #[error("long overflow")] ParquetTimestampOverflow { file_path: String }, @@ -360,6 +364,7 @@ impl SparkError { SparkError::CastOverFlow { .. } => "CastOverFlow", SparkError::CannotParseDecimal => "CannotParseDecimal", SparkError::ArithmeticOverflow { .. } => "ArithmeticOverflow", + SparkError::LongOverflow => "LongOverflow", SparkError::IntegralDivideOverflow => "IntegralDivideOverflow", SparkError::DecimalSumOverflow { .. } => "DecimalSumOverflow", SparkError::DivideByZero => "DivideByZero", @@ -720,6 +725,8 @@ impl SparkError { /// Returns the appropriate Spark exception class for this error pub fn exception_class(&self) -> &'static str { match self { + SparkError::LongOverflow => "java/lang/ArithmeticException", + // ArithmeticException SparkError::DivideByZero | SparkError::RemainderByZero @@ -840,6 +847,7 @@ impl SparkError { SparkError::RemainderByZero => Some("REMAINDER_BY_ZERO"), SparkError::IntervalDividedByZero => Some("INTERVAL_DIVIDED_BY_ZERO"), SparkError::ArithmeticOverflow { .. } => Some("ARITHMETIC_OVERFLOW"), + SparkError::LongOverflow => None, SparkError::IntegralDivideOverflow => Some("ARITHMETIC_OVERFLOW"), SparkError::DecimalSumOverflow { .. } => Some("ARITHMETIC_OVERFLOW"), SparkError::BinaryArithmeticOverflow { .. } => Some("BINARY_ARITHMETIC_OVERFLOW"), @@ -1141,6 +1149,22 @@ mod tests { assert!(json.contains("\"errorClass\":\"REMAINDER_BY_ZERO\"")); } + #[test] + fn test_long_overflow_json() { + let error = SparkError::LongOverflow; + let parsed: serde_json::Value = serde_json::from_str(&error.to_json()).unwrap(); + assert_eq!( + parsed, + serde_json::json!({ + "errorType": "LongOverflow", + "errorClass": "", + "params": {}, + }) + ); + assert_eq!(error.exception_class(), "java/lang/ArithmeticException"); + assert_eq!(error.to_string(), "long overflow"); + } + #[test] fn test_arithmetic_overflow_suggestion_json_and_display() { for function in ["", "try_add", "try_subtract", "try_multiply"] { diff --git a/native/common/src/struct_nulls.rs b/native/common/src/struct_nulls.rs index 64d8c4ece88..16be6a99769 100644 --- a/native/common/src/struct_nulls.rs +++ b/native/common/src/struct_nulls.rs @@ -15,7 +15,7 @@ // specific language governing permissions and limitations // under the License. -//! Pushing a struct's null mask down into its children. +//! Propagating parent visibility to nested children. //! //! Arrow stores a `StructArray`'s children with their own validity, independent of the parent's //! null buffer, so at a row where the struct itself is null a child buffer can still hold a value @@ -28,10 +28,14 @@ //! whatever sits in the child slot //! (). -use arrow::array::{make_array, Array, ArrayRef, StructArray}; +use arrow::array::{ + make_array, Array, ArrayRef, AsArray, GenericListViewArray, NullBufferBuilder, OffsetSizeTrait, + StructArray, +}; use arrow::buffer::NullBuffer; use arrow::datatypes::DataType; use arrow::error::ArrowError; +use std::cell::LazyCell; use std::sync::Arc; /// The parent's nulls unioned into one child, or the child unchanged when there is nothing to push. @@ -76,6 +80,89 @@ pub fn children_with_parent_nulls(struct_array: &StructArray) -> Result = LazyCell, &'a dyn Fn() -> Option>; + +/// Map the visible slots of a container to the indices of its children. +pub fn child_visibility(array: &dyn Array, parent: &Visibility<'_>) -> Option { + let visible = NullBuffer::union(array.nulls(), parent.as_ref()); + let nulls = visible.as_ref(); + match array.data_type() { + DataType::List(_) => { + let list = array.as_list::(); + repeated_visibility(list.value_offsets(), list.values().len(), nulls) + } + DataType::LargeList(_) => { + let list = array.as_list::(); + repeated_visibility(list.value_offsets(), list.values().len(), nulls) + } + DataType::FixedSizeList(_, size) => { + // Arrow slices the child values along with a fixed-size list. Expand in those + // child coordinates; applying a parent slice offset again would shift the mask. + nulls.map(|nulls| nulls.expand(*size as usize)) + } + DataType::ListView(_) => list_view_visibility(array.as_list_view::(), nulls), + DataType::LargeListView(_) => list_view_visibility(array.as_list_view::(), nulls), + DataType::Map(_, _) => { + let map = array.as_map(); + repeated_visibility(map.value_offsets(), map.entries().len(), nulls) + } + _ => visible, + } +} + +fn list_view_visibility( + list: &GenericListViewArray, + nulls: Option<&NullBuffer>, +) -> Option { + let mut valid = vec![false; list.values().len()]; + for (row, (offset, size)) in list + .value_offsets() + .iter() + .zip(list.value_sizes()) + .enumerate() + { + if nulls.is_none_or(|nulls| nulls.is_valid(row)) { + let start = offset.as_usize(); + // Views can overlap or be out of order. Accumulate the union of visible ranges; + // a null parent must never clear values another parent can see. + valid[start..start + size.as_usize()].fill(true); + } + } + Some(NullBuffer::from(valid)) +} + +fn repeated_visibility( + offsets: &[O], + len: usize, + nulls: Option<&NullBuffer>, +) -> Option { + let start = offsets[0].as_usize(); + let end = offsets.last().unwrap().as_usize(); + let mut visible = NullBufferBuilder::new(len); + if start > 0 { + visible.append_n_nulls(start); + } + if let Some(nulls) = nulls.filter(|n| n.null_count() > 0) { + for (valid, range) in nulls.iter().zip(offsets.windows(2)) { + let count = range[1].as_usize() - range[0].as_usize(); + if valid { + visible.append_n_non_nulls(count); + } else if count > 0 { + visible.append_n_nulls(count); + } + } + } else { + visible.append_n_non_nulls(end - start); + } + if end < len { + visible.append_n_nulls(len - end); + } + visible.finish() +} + #[cfg(test)] mod tests { use super::*; diff --git a/native/core/src/parquet/eager_page_index_reader_factory.rs b/native/core/src/parquet/eager_page_index_reader_factory.rs index 00220e26666..72f62a6dd65 100644 --- a/native/core/src/parquet/eager_page_index_reader_factory.rs +++ b/native/core/src/parquet/eager_page_index_reader_factory.rs @@ -94,7 +94,7 @@ //! bring it back. use crate::parquet::datetime_rebase::stamp_int96_leaves; -use crate::parquet::parquet_support::SparkParquetOptions; +use crate::parquet::parquet_support::{contains_ntz, contains_type, SparkParquetOptions}; use crate::parquet::schema_adapter::{ check_file_conversions, RejectOnNonEmpty, NANOS_TIMESTAMP_KEY, REPEATED_PRIMITIVE_KEY, }; @@ -135,7 +135,7 @@ use parquet::file::metadata::{ FooterTail, PageIndexPolicy, ParquetMetaData, ParquetMetaDataReader, }; use parquet::schema::types::{ColumnDescPtr, SchemaDescriptor, Type as ParquetType}; -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use std::fmt::{Debug, Display, Formatter}; use std::ops::Range; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; @@ -233,7 +233,7 @@ pub struct EagerPageIndexReaderFactory { // does when the requested schema carries one and `ignoreMissing` is not set. require_field_ids: bool, stamp_int96_leaves: bool, - conversion_check: Option<(SchemaRef, SparkParquetOptions, bool)>, + conversion_check: Option>, deferred_rejections: Arc>>, next_file_prefetch: Option>, } @@ -272,16 +272,28 @@ impl EagerPageIndexReaderFactory { } } - /// Retain raw LIST encoding and `TIMESTAMP(NANOS)` leaves for open-time validation. When row - /// filters are enabled, or the file has a `TIMESTAMP(NANOS)` leaf, enforce deferred failures - /// at the first data-page read, after format pruning. + /// Check physical encodings that Arrow normalizes. Enforce deferred schema errors on + /// the first data-page read, after pruning and before row filtering. pub(crate) fn with_conversion_check( mut self, required: SchemaRef, options: SparkParquetOptions, row_filters: bool, ) -> Self { - self.conversion_check = Some((required, options, row_filters)); + let ntz_roots = required + .fields() + .iter() + .filter(|f| contains_ntz(f.data_type())) + .map(|f| f.name().to_ascii_lowercase()) + .collect::>(); + let match_any_root = options.use_field_id || ntz_roots.iter().any(|name| !name.is_ascii()); + self.conversion_check = Some(Arc::new(FileConversionCheck { + required, + options, + row_filters, + ntz_roots, + match_any_root, + })); self } @@ -393,7 +405,7 @@ struct EagerPageIndexReader { spark_variant_schema: bool, require_field_ids: bool, stamp_int96_leaves: bool, - conversion_check: Option<(SchemaRef, SparkParquetOptions, bool)>, + conversion_check: Option>, deferred_rejections: Arc>>, deferred_rejection: Option, next_file_prefetch: Option>, @@ -482,19 +494,21 @@ fn spark_enum_schema(schema: &SchemaDescriptor) -> ParquetResult> ))) } +fn has_arrow_schema(file: &FileMetaData) -> bool { + file.key_value_metadata() + .into_iter() + .flatten() + .any(|kv| kv.key == ARROW_SCHEMA_META_KEY) +} + /// Arrow restores advisory `ARROW:schema` types that can differ from Spark's physical Parquet /// interpretation. Replace that hint with physical inference and the ENUM string mapping. /// Rebuild only the returned metadata; the shared cache retains the original footer. /// https://github.com/apache/datafusion-comet/issues/5477 fn with_spark_arrow_schema(metadata: Arc) -> ParquetResult> { let file = metadata.file_metadata(); - let has_arrow_schema = file.key_value_metadata().is_some_and(|key_values| { - key_values - .iter() - .any(|key_value| key_value.key == ARROW_SCHEMA_META_KEY) - }); let enum_schema = spark_enum_schema(file.schema_descr())?; - if !has_arrow_schema && enum_schema.is_none() { + if !has_arrow_schema(file) && enum_schema.is_none() { return Ok(metadata); } @@ -540,6 +554,43 @@ fn is_nanos_timestamp(column: &ColumnDescPtr) -> bool { ) } +fn is_plain_int64(column: &ColumnDescPtr) -> bool { + column.physical_type() == parquet::basic::Type::INT64 + && matches!( + column.converted_type(), + ConvertedType::NONE | ConvertedType::INT_64 + ) + && !matches!(column.logical_type_ref(), Some(LogicalType::Timestamp(_))) +} + +#[derive(Debug)] +struct FileConversionCheck { + required: SchemaRef, + options: SparkParquetOptions, + row_filters: bool, + ntz_roots: HashSet, + match_any_root: bool, +} + +impl FileConversionCheck { + // The adapter resolves exact nested paths, field IDs and Unicode names after this check. + fn may_read_hinted_int64(&self, file: &FileMetaData) -> bool { + !self.ntz_roots.is_empty() + && has_arrow_schema(file) + && file + .schema_descr() + .columns() + .iter() + .filter(|c| is_plain_int64(c)) + .any(|c| { + let root = &c.path().parts()[0]; + self.match_any_root + || !root.is_ascii() + || self.ntz_roots.contains(&root.to_ascii_lowercase()) + }) + } +} + /// Arrow normalizes both standard and legacy LISTs to the same type, and gives INT96 and /// `TIMESTAMP(NANOS)` leaves the same nanosecond timestamp type. Carry the repeated primitive /// and `TIMESTAMP(NANOS)` distinctions in an ephemeral schema used by the conversion check, @@ -571,6 +622,12 @@ fn mark_parquet_leaves(schema: &Schema, parquet: &SchemaDescriptor) -> Schema { }); let nanos = column.is_some_and(is_nanos_timestamp); let mut field = field.as_ref().clone(); + // A Date64 hint can describe INT32 DATE or an unannotated INT64. + if column.is_some_and(is_plain_int64) + && contains_type(field.data_type(), &DataType::Date64) + { + field = field.with_data_type(DataType::Int64); + } if repeated || nanos { let mut metadata = field.metadata().clone(); if repeated { @@ -769,30 +826,37 @@ impl AsyncFileReader for EagerPageIndexReader { } else { metadata }; - if let Some((required, conversion_options, row_filters)) = - self.conversion_check.as_ref() - { + if let Some(check) = &self.conversion_check { let file = metadata.file_metadata(); let columns = file.schema_descr().columns(); // Only this check sees the Parquet type of a nanosecond timestamp leaf, so a // `TIMESTAMP(NANOS)` rejection is deferred from here even without row filters. let has_nanos = columns.iter().any(is_nanos_timestamp); - let needs_check = *row_filters + let needs_check = check.row_filters || has_nanos || columns.iter().any(|column| { column.self_type().get_basic_info().repetition() == parquet::basic::Repetition::REPEATED }); - if !needs_check { + let check_hint = check.may_read_hinted_int64(file); + if !needs_check && !check_hint { return Ok(metadata); } let schema = parquet_to_arrow_schema(file.schema_descr(), file.key_value_metadata())?; + let has_date_hint = check_hint + && schema + .fields() + .iter() + .any(|f| contains_type(f.data_type(), &DataType::Date64)); + if !needs_check && !has_date_hint { + return Ok(metadata); + } let schema = mark_parquet_leaves(&schema, file.schema_descr()); let rejection = check_file_conversions( - Arc::clone(required), + Arc::clone(&check.required), Arc::new(schema), - conversion_options.clone(), + check.options.clone(), ) .map_err(|e| match e { // Keep structured Spark errors directly inside the Parquet wrapper so @@ -800,7 +864,7 @@ impl AsyncFileReader for EagerPageIndexReader { DataFusionError::External(source) => ParquetError::External(source), other => ParquetError::External(Box::new(other)), })?; - if *row_filters || has_nanos { + if check.row_filters || has_nanos || has_date_hint { if let Some(rejection) = &rejection { self.deferred_rejections.lock().unwrap().insert( object_meta.location.clone(), diff --git a/native/core/src/parquet/parquet_exec.rs b/native/core/src/parquet/parquet_exec.rs index f2e3e1e4b6c..d76f4851f6c 100644 --- a/native/core/src/parquet/parquet_exec.rs +++ b/native/core/src/parquet/parquet_exec.rs @@ -21,7 +21,7 @@ use crate::parquet::encryption_support::{CometEncryptionConfig, ENCRYPTION_FACTO use crate::parquet::file_error_context::ParquetErrorContext; use crate::parquet::name_fold::fold_schema_names; use crate::parquet::parquet_support::{ - object_store_authority, ObjectStoreBackend, SparkParquetOptions, + contains_ntz, object_store_authority, ObjectStoreBackend, SparkParquetOptions, }; use crate::parquet::schema_adapter::SparkPhysicalExprAdapterFactory; use arrow::datatypes::{Field, FieldRef, Schema, SchemaRef}; @@ -123,7 +123,7 @@ pub(crate) fn init_datasource_exec( &session_config.options().execution.parquet, ); spark_parquet_options.use_field_id = use_field_id; - // Spark can discard filtered-out values before timestamp conversion using statistics, + // Spark can discard filtered-out values before TIMESTAMP_MILLIS conversion using statistics, // dictionary, and row-level filters. Comet cannot mirror every pruning path, so applying // checked conversion in a filtered scan can fail on values Spark never reads. Preserve the // existing safe cast for filtered scans and use checked conversion only when every value is @@ -311,11 +311,22 @@ pub(crate) fn init_datasource_exec( _ => Arc::new(parquet_source), }; - let file_source = if spark_parquet_options.checked_timestamp_overflow { + // A projected NTZ may need a checked DATE conversion when the physical file opens. + let file_source = if spark_parquet_options.checked_timestamp_overflow + || (allow_type_promotion + && required_schema + .fields() + .iter() + .any(|f| contains_ntz(f.data_type()))) + { ParquetErrorContext::wrap(file_source) } else { file_source }; + // Row filters need the checked cast when promotion is supported. On Spark 3.5 + // the footer check above rejects promotion at the first data-page read. + spark_parquet_options.prune_date_nulls = + !allow_type_promotion || !session_config.options().execution.parquet.pushdown_filters; let expr_adapter_factory: Arc = Arc::new( SparkPhysicalExprAdapterFactory::new(spark_parquet_options, default_values), ); diff --git a/native/core/src/parquet/parquet_support.rs b/native/core/src/parquet/parquet_support.rs index 2b42b01b767..59436fd1b61 100644 --- a/native/core/src/parquet/parquet_support.rs +++ b/native/core/src/parquet/parquet_support.rs @@ -19,12 +19,12 @@ use crate::execution::operators::ExecutionError; use crate::parquet::field_resolver::FieldResolver; use crate::parquet::name_fold::fold_names; use arrow::array::{ - make_array, BooleanBufferBuilder, FixedSizeBinaryArray, GenericListViewArray, MapArray, - OffsetSizeTrait, StringArray, + make_array, BooleanBufferBuilder, FixedSizeBinaryArray, MapArray, PrimitiveArray, StringArray, + TimestampMicrosecondArray, }; use arrow::buffer::NullBuffer; use arrow::compute::{can_cast_types, take}; -use arrow::datatypes::{FieldRef, Fields}; +use arrow::datatypes::{ArrowPrimitiveType, Date32Type, Date64Type, FieldRef, Fields}; use arrow::{ array::{ cast::AsArray, new_null_array, types::TimestampMicrosecondType, @@ -41,6 +41,7 @@ use datafusion::execution::object_store::{ }; use datafusion::execution::runtime_env::RuntimeEnv; use datafusion::physical_plan::ColumnarValue; +use datafusion_comet_common::struct_nulls::{child_visibility, Visibility}; use datafusion_comet_common::SparkError; use datafusion_comet_spark_expr::{spark_cast, EvalMode, SparkCastOptions}; use log::debug; @@ -121,6 +122,8 @@ pub struct SparkParquetOptions { /// (overflow -> NULL), because Spark may discard values through pruning paths that /// DataFusion cannot fully mirror before conversion. pub checked_timestamp_overflow: bool, + /// Whether null predicates can use the physical date during NTZ promotion. + pub prune_date_nulls: bool, /// When true, resolve each file's datetime calendar-rebase policy from its parquet footer /// metadata (`org.apache.spark.legacyDateTime` and friends) and rebase -- or refuse -- /// affected values, mirroring Spark's per-file `DataSourceUtils.datetimeRebaseSpec` @@ -152,6 +155,7 @@ impl SparkParquetOptions { allow_type_promotion: false, allow_timestamp_ltz_to_ntz: false, checked_timestamp_overflow: true, + prune_date_nulls: false, rebase_from_file_metadata: false, datetime_rebase_mode_in_read: String::new(), int96_rebase_mode_in_read: String::new(), @@ -170,6 +174,7 @@ impl SparkParquetOptions { allow_type_promotion: false, allow_timestamp_ltz_to_ntz: false, checked_timestamp_overflow: true, + prune_date_nulls: false, rebase_from_file_metadata: false, datetime_rebase_mode_in_read: String::new(), int96_rebase_mode_in_read: String::new(), @@ -210,7 +215,14 @@ fn parquet_convert_array( to_type: &DataType, parquet_options: &SparkParquetOptions, ) -> DataFusionResult { - parquet_convert_array_impl(array, to_type, parquet_options, None, false).map(|(array, _)| array) + parquet_convert_array_impl( + array, + to_type, + parquet_options, + &Visibility::new(&|| None), + false, + ) + .map(|(array, _)| array) } // The boolean records whether any requested leaf exists in the file schema, independent @@ -221,48 +233,40 @@ fn parquet_convert_array_impl( array: ArrayRef, to_type: &DataType, parquet_options: &SparkParquetOptions, - parent_nulls: Option<&NullBuffer>, + parent_nulls: &Visibility<'_>, in_map_key: bool, ) -> DataFusionResult<(ArrayRef, bool)> { use DataType::*; let from_type = array.data_type(); - // Only checked millis-to-micros casts consume ancestor visibility. In particular, - // unchanged array/map siblings must not expand a mask over their backing values. - let checked_timestamp_overflow = parquet_options.checked_timestamp_overflow - && has_timestamp_unit(from_type, TimeUnit::Millisecond) - && has_timestamp_unit(to_type, TimeUnit::Microsecond); - let visible = if checked_timestamp_overflow { - NullBuffer::union(array.nulls(), parent_nulls) - } else { - None - }; + let child_mask = || child_visibility(array.as_ref(), parent_nulls); + let children = Visibility::new(&child_mask); // Try Comet specific handlers first, then arrow-rs cast if supported, and fail otherwise. let converted: DataFusionResult = match (from_type, to_type) { - (Struct(_), Struct(_)) => return parquet_convert_struct_to_struct( - array.as_struct(), - from_type, - to_type, - parquet_options, - visible.as_ref(), - in_map_key, - ), + (Struct(_), Struct(_)) => { + return parquet_convert_struct_to_struct( + array.as_struct(), + from_type, + to_type, + parquet_options, + &children, + in_map_key, + ) + } ( List(_) | LargeList(_) | FixedSizeList(_, _) | ListView(_) | LargeListView(_), - List(to_inner_type) | LargeList(to_inner_type) | FixedSizeList(to_inner_type, _) - | ListView(to_inner_type) | LargeListView(to_inner_type), + List(to_inner_type) + | LargeList(to_inner_type) + | FixedSizeList(to_inner_type, _) + | ListView(to_inner_type) + | LargeListView(to_inner_type), ) => { let data = array.to_data(); - let child_visibility = if checked_timestamp_overflow { - list_child_visibility(array.as_ref(), visible.as_ref()) - } else { - None - }; let (cast_field, field_overlap) = parquet_convert_array_impl( make_array(data.child_data()[0].clone()), to_inner_type.data_type(), parquet_options, - child_visibility.as_ref(), + &children, in_map_key, )?; if !field_overlap && parquet_options.return_null_struct_if_all_fields_missing { @@ -279,10 +283,12 @@ fn parquet_convert_array_impl( _ => unreachable!(), }; // Retain the source offsets, sizes and null buffer while replacing its values. - let resolved = make_array(data.into_builder() - .data_type(resolved_type) - .child_data(vec![cast_field.to_data()]) - .build()?); + let resolved = make_array( + data.into_builder() + .data_type(resolved_type) + .child_data(vec![cast_field.to_data()]) + .build()?, + ); let output = if resolved.data_type() == to_type { resolved } else { @@ -290,10 +296,9 @@ fn parquet_convert_array_impl( }; return Ok((output, field_overlap)); } - ( - Timestamp(TimeUnit::Millisecond, _), - Timestamp(TimeUnit::Microsecond, target_tz), - ) if checked_timestamp_overflow => { + (Timestamp(TimeUnit::Millisecond, _), Timestamp(TimeUnit::Microsecond, target_tz)) + if parquet_options.checked_timestamp_overflow => + { // Spark's Parquet reader calls the checked `millisToMicros` conversion for both // direct and dictionary values, independent of CAST evaluation mode: // https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L817-L833 @@ -302,20 +307,42 @@ fn parquet_convert_array_impl( // // Filtered scans retain safe conversion until DataFusion can mirror Spark's // pruning paths, including nested predicates. - let millis = array.as_primitive::(); - // Ignore values hidden by null ancestors or by sliced list/map offsets. - // Restore the original child validity: required fields must remain non-null. - let micros = arrow::array::TimestampMillisecondArray::new( - millis.values().clone(), visible) - .try_unary::<_, TimestampMicrosecondType, _>(|value| { - value.checked_mul(1_000).ok_or_else(|| SparkError::ParquetTimestampOverflow { - file_path: String::new(), - }) - })?; - let micros = arrow::array::TimestampMicrosecondArray::new( - micros.values().clone(), millis.nulls().cloned()) - .with_timezone_opt(target_tz.clone()); - Ok(Arc::new(micros)) + checked_micros( + array.as_primitive::(), + parent_nulls.as_ref(), + 1_000, + target_tz.clone(), + ) + } + (Date32, Timestamp(TimeUnit::Microsecond, None)) => { + // Spark 4.x's DateToTimestampNTZUpdater calls daysToMicros at UTC. + checked_micros( + array.as_primitive::(), + parent_nulls.as_ref(), + 86_400_000_000, + None, + ) + } + (Date64, Timestamp(TimeUnit::Microsecond, None)) => { + // ARROW:schema can restore a physical INT32 DATE as milliseconds. + checked_micros( + array.as_primitive::(), + parent_nulls.as_ref(), + 1_000, + None, + ) + } + (Dictionary(_, values), _) if may_read_date_as_ntz(values, to_type) => { + // An ARROW:schema dictionary can include days no row references. + let dictionary = array.as_any_dictionary(); + let decoded = take(dictionary.values().as_ref(), dictionary.keys(), None)?; + return parquet_convert_array_impl( + decoded, + to_type, + parquet_options, + parent_nulls, + in_map_key, + ); } (Timestamp(TimeUnit::Microsecond, None), Timestamp(TimeUnit::Microsecond, Some(tz))) => { Ok(Arc::new( @@ -325,9 +352,16 @@ fn parquet_convert_array_impl( .with_timezone(Arc::clone(tz)), )) } - (Map(_, ordered_from), Map(_, ordered_to)) if ordered_from == ordered_to => - return parquet_convert_map_to_map(array.as_map(), to_type, parquet_options, *ordered_to, visible.as_ref(), checked_timestamp_overflow, in_map_key) - , + (Map(_, ordered_from), Map(_, ordered_to)) if ordered_from == ordered_to => { + return parquet_convert_map_to_map( + array.as_map(), + to_type, + parquet_options, + *ordered_to, + &children, + in_map_key, + ); + } // Iceberg stores UUIDs as 16-byte fixed binary but Spark expects string representation. // Arrow doesn't support casting FixedSizeBinary to Utf8, so we handle it manually. (FixedSizeBinary(16), Utf8) => { @@ -340,9 +374,8 @@ fn parquet_convert_array_impl( .iter() .map(|opt_bytes| { opt_bytes.map(|bytes| { - let uuid = uuid::Uuid::from_bytes( - bytes.try_into().expect("Expected 16 bytes") - ); + let uuid = + uuid::Uuid::from_bytes(bytes.try_into().expect("Expected 16 bytes")); uuid.to_string() }) }) @@ -381,7 +414,9 @@ fn parquet_convert_array_impl( // the batch into one string array with 32-bit offsets, so a batch whose rows reference // wide entries many times can decode past `i32::MAX` bytes and fail where Spark, which // keeps the dictionary, succeeds (#7254). - (Dictionary(_, value_type), Utf8) if matches!(value_type.as_ref(), Binary | LargeBinary) => { + (Dictionary(_, value_type), Utf8) + if matches!(value_type.as_ref(), Binary | LargeBinary) => + { let dictionary = array.as_any_dictionary(); let keys = dictionary.normalized_keys(); let key_nulls = dictionary.keys().nulls(); @@ -389,7 +424,7 @@ fn parquet_convert_array_impl( referenced_dictionary_values(dictionary.values(), &keys, key_nulls)?, to_type, parquet_options, - None, + &Visibility::new(&|| None), in_map_key, )?; let decoded_bytes = @@ -419,82 +454,57 @@ fn parquet_convert_array_impl( converted.map(|array| (array, true)) } -// Struct fields are matched by name/field ID later. This type-only check is conservative -// until that matching occurs; each selected child is checked again before conversion. -fn has_timestamp_unit(data_type: &DataType, unit: TimeUnit) -> bool { - match data_type { - DataType::Timestamp(timestamp_unit, _) => *timestamp_unit == unit, - DataType::Struct(fields) => fields - .iter() - .any(|field| has_timestamp_unit(field.data_type(), unit)), - DataType::List(field) - | DataType::LargeList(field) - | DataType::FixedSizeList(field, _) - | DataType::ListView(field) - | DataType::LargeListView(field) - | DataType::Map(field, _) => has_timestamp_unit(field.data_type(), unit), - _ => false, - } +fn checked_micros( + input: &PrimitiveArray, + parent_nulls: Option<&NullBuffer>, + micros_per_unit: i64, + timezone: Option>, +) -> DataFusionResult +where + i64: From, +{ + let visible = NullBuffer::union(input.nulls(), parent_nulls); + let micros = PrimitiveArray::::new(input.values().clone(), visible) + .try_unary::<_, TimestampMicrosecondType, _>(|value| { + i64::from(value).checked_mul(micros_per_unit).ok_or(()) + }) + .map_err(|_| SparkError::ParquetTimestampOverflow { + file_path: String::new(), + })?; + // Restore the original child validity: required fields must remain non-null. + let micros = TimestampMicrosecondArray::new(micros.values().clone(), input.nulls().cloned()) + .with_timezone_opt(timezone); + Ok(Arc::new(micros)) } -// List/map values can include entries outside a slice or beneath a null parent. -fn repeated_visibility( - offsets: &[O], - len: usize, - nulls: Option<&NullBuffer>, -) -> Option { - if nulls.is_none() && offsets[0].as_usize() == 0 && offsets.last().unwrap().as_usize() == len { - return None; - } - let mut valid = vec![false; len]; - for (row, range) in offsets.windows(2).enumerate() { - if nulls.is_none_or(|nulls| nulls.is_valid(row)) { - valid[range[0].as_usize()..range[1].as_usize()].fill(true); +pub(super) fn may_read_date_as_ntz(physical_type: &DataType, target_type: &DataType) -> bool { + match (physical_type, target_type) { + (DataType::Date32 | DataType::Date64, DataType::Timestamp(TimeUnit::Microsecond, None)) => { + true } + (DataType::Dictionary(_, value), _) => may_read_date_as_ntz(value, target_type), + _ => false, } - Some(NullBuffer::from(valid)) } -fn list_child_visibility(array: &dyn Array, nulls: Option<&NullBuffer>) -> Option { - match array.data_type() { - DataType::List(_) => { - let list = array.as_list::(); - repeated_visibility(list.value_offsets(), list.values().len(), nulls) - } - DataType::LargeList(_) => { - let list = array.as_list::(); - repeated_visibility(list.value_offsets(), list.values().len(), nulls) - } - DataType::FixedSizeList(_, size) => { - // Arrow slices the child values along with a fixed-size list. Expand in those - // child coordinates; applying a parent slice offset again would shift the mask. - nulls.map(|nulls| nulls.expand(*size as usize)) - } - DataType::ListView(_) => list_view_visibility(array.as_list_view::(), nulls), - DataType::LargeListView(_) => list_view_visibility(array.as_list_view::(), nulls), - _ => unreachable!("only called for list arrays"), - } +pub(super) fn contains_ntz(data_type: &DataType) -> bool { + contains_type(data_type, &DataType::Timestamp(TimeUnit::Microsecond, None)) } -fn list_view_visibility( - list: &GenericListViewArray, - nulls: Option<&NullBuffer>, -) -> Option { - let mut valid = vec![false; list.values().len()]; - for (row, (offset, size)) in list - .value_offsets() - .iter() - .zip(list.value_sizes()) - .enumerate() - { - if nulls.is_none_or(|nulls| nulls.is_valid(row)) { - let start = offset.as_usize(); - // Views can overlap or be out of order. Accumulate the union of visible ranges; - // a null parent must never clear values another parent can see. - valid[start..start + size.as_usize()].fill(true); - } +pub(super) fn contains_type(data_type: &DataType, leaf: &DataType) -> bool { + match data_type { + DataType::Struct(fields) => fields + .iter() + .any(|field| contains_type(field.data_type(), leaf)), + DataType::List(field) + | DataType::LargeList(field) + | DataType::FixedSizeList(field, _) + | DataType::ListView(field) + | DataType::LargeListView(field) + | DataType::Map(field, _) => contains_type(field.data_type(), leaf), + DataType::Dictionary(_, values) => contains_type(values, leaf), + _ => data_type == leaf, } - Some(NullBuffer::from(valid)) } /// Read the Parquet field id stored under arrow-rs's `PARQUET_FIELD_ID_META_KEY`. @@ -548,7 +558,7 @@ fn parquet_convert_struct_to_struct( from_type: &DataType, to_type: &DataType, parquet_options: &SparkParquetOptions, - parent_nulls: Option<&NullBuffer>, + parent_nulls: &Visibility<'_>, in_map_key: bool, ) -> DataFusionResult<(ArrayRef, bool)> { match (from_type, to_type) { @@ -603,8 +613,7 @@ fn parquet_convert_map_to_map( to_data_type: &DataType, parquet_options: &SparkParquetOptions, to_ordered: bool, - parent_nulls: Option<&NullBuffer>, - checked_timestamp_overflow: bool, + parent_nulls: &Visibility<'_>, in_map_key: bool, ) -> Result<(ArrayRef, bool), DataFusionError> { match to_data_type { @@ -616,23 +625,18 @@ fn parquet_convert_map_to_map( "map is missing value field".to_string(), ))?; - let child_visibility = if checked_timestamp_overflow { - repeated_visibility(from.value_offsets(), from.keys().len(), parent_nulls) - } else { - None - }; let (key_array, key_overlap) = parquet_convert_array_impl( Arc::clone(from.keys()), key_field.data_type(), parquet_options, - child_visibility.as_ref(), + parent_nulls, true, )?; let (value_array, value_overlap) = parquet_convert_array_impl( Arc::clone(from.values()), value_field.data_type(), parquet_options, - child_visibility.as_ref(), + parent_nulls, in_map_key, )?; let field_overlap = key_overlap || value_overlap; @@ -2536,6 +2540,57 @@ mod tests { } } + #[test] + fn widening_sibling_of_date_and_ntz_skips_parent_visibility() { + use super::{parquet_convert_array_impl, SparkParquetOptions, Visibility}; + use arrow::array::new_null_array; + use arrow::datatypes::{DataType, Field, TimeUnit}; + use datafusion_comet_spark_expr::EvalMode; + + let list = |int| { + let fields = vec![ + Field::new("d", DataType::Date32, true), + Field::new("t", DataType::Timestamp(TimeUnit::Microsecond, None), true), + Field::new("i", int, true), + ]; + DataType::new_list(DataType::Struct(fields.into()), true) + }; + let array = new_null_array(&list(DataType::Int32), 1); + let target = list(DataType::Int64); + let opts = SparkParquetOptions::new(EvalMode::Legacy, "UTC"); + parquet_convert_array_impl( + array, + &target, + &opts, + &Visibility::new(&|| panic!("no leaf needs the parent mask")), + false, + ) + .unwrap(); + } + + #[test] + fn date_to_ntz_ignores_unreferenced_dictionary_values() { + use super::{parquet_convert_array, SparkParquetOptions}; + use arrow::array::{ArrayRef, AsArray, Date32Array, DictionaryArray, Int32Array}; + use arrow::datatypes::{DataType, Int32Type, TimeUnit, TimestampMicrosecondType}; + use datafusion_comet_spark_expr::EvalMode; + use std::sync::Arc; + + let target = DataType::Timestamp(TimeUnit::Microsecond, None); + let mut options = SparkParquetOptions::new(EvalMode::Legacy, "UTC"); + // Filtering only relaxes the overflow check for TIMESTAMP_MILLIS. + options.checked_timestamp_overflow = false; + let values: ArrayRef = Arc::new(Date32Array::from(vec![i32::MAX, 1, i32::MIN])); + let dictionary = + DictionaryArray::::new(Int32Array::from(vec![Some(1), None]), values); + let decoded = parquet_convert_array(Arc::new(dictionary), &target, &options).unwrap(); + assert_eq!( + decoded.as_primitive::().value(0), + 86_400_000_000 + ); + assert!(decoded.is_null(1)); + } + #[test] fn test_millis_to_micros_overflow_checked_in_nested_fields() { use crate::parquet::parquet_support::{parquet_convert_array, SparkParquetOptions}; @@ -2703,11 +2758,11 @@ mod tests { } #[test] - fn test_millis_to_micros_nested_visibility() { + fn test_checked_micros_nested_visibility() { use super::{parquet_convert_array, SparkParquetOptions}; use arrow::array::{ - Array, ArrayRef, ListArray, MapArray, StructArray, TimestampMicrosecondArray, - TimestampMillisecondArray, + Array, ArrayRef, Date32Array, ListArray, MapArray, StructArray, + TimestampMicrosecondArray, TimestampMillisecondArray, }; use arrow::buffer::{NullBuffer, OffsetBuffer}; use arrow::datatypes::{DataType, Field, TimeUnit}; @@ -2715,13 +2770,19 @@ mod tests { use std::sync::Arc; for timezone in [None::>, Some(Arc::from("UTC"))] { + let mut cases: Vec<(ArrayRef, i64)> = Vec::new(); for overflow in [i64::MAX, i64::MIN] { + let millis = TimestampMillisecondArray::from(vec![overflow, 7, overflow]) + .with_timezone_opt(timezone.clone()); + cases.push((Arc::new(millis), 7000)); + } + if timezone.is_none() { + let dates = Date32Array::from(vec![i32::MIN, 1, i32::MAX]); + cases.push((Arc::new(dates), 86_400_000_000)); + } + for (input, expected) in cases { let options = SparkParquetOptions::new(EvalMode::Legacy, "UTC"); - let millis: ArrayRef = Arc::new( - TimestampMillisecondArray::from(vec![overflow, 7, overflow]) - .with_timezone_opt(timezone.clone()), - ); - let field = Arc::new(Field::new("ts", millis.data_type().clone(), false)); + let field = Arc::new(Field::new("ts", input.data_type().clone(), false)); let target_field = Arc::new(Field::new( "ts", DataType::Timestamp(TimeUnit::Microsecond, timezone.clone()), @@ -2730,22 +2791,22 @@ mod tests { let validity = Some(NullBuffer::from(vec![false, true, false])); let strukt: ArrayRef = Arc::new(StructArray::new( vec![Arc::clone(&field)].into(), - vec![Arc::clone(&millis)], + vec![Arc::clone(&input)], validity.clone(), )); let list: ArrayRef = Arc::new(ListArray::new( Arc::clone(&field), OffsetBuffer::new(vec![0, 1, 2, 3].into()), - Arc::clone(&millis), + Arc::clone(&input), validity.clone(), )); let entries = StructArray::new( vec![ Arc::clone(&field), - Arc::new(Field::new("value", millis.data_type().clone(), false)), + Arc::new(Field::new("value", input.data_type().clone(), false)), ] .into(), - vec![Arc::clone(&millis), Arc::clone(&millis)], + vec![Arc::clone(&input), Arc::clone(&input)], None, ); let map: ArrayRef = Arc::new(MapArray::new( @@ -2812,7 +2873,10 @@ mod tests { .as_any() .downcast_ref::() .unwrap(); - assert_eq!(values.value(if values.len() == 3 { 1 } else { 0 }), 7000); + assert_eq!( + values.value(if values.len() == 3 { 1 } else { 0 }), + expected + ); assert_eq!(values.null_count(), 0); } // Removing parent nulls makes the overflow visible and must fail. diff --git a/native/core/src/parquet/schema_adapter.rs b/native/core/src/parquet/schema_adapter.rs index b3f93e18762..c3357babee6 100644 --- a/native/core/src/parquet/schema_adapter.rs +++ b/native/core/src/parquet/schema_adapter.rs @@ -23,8 +23,8 @@ use crate::parquet::datetime_rebase::{ use crate::parquet::field_resolver::{duplicate_name_error, FieldResolver, Selection}; use crate::parquet::name_fold::{fold_name, fold_names, fold_schema_names}; use crate::parquet::parquet_support::{ - duplicate_parquet_field_error, field_id, match_struct_fields, spark_parquet_convert, - SparkParquetOptions, + duplicate_parquet_field_error, field_id, match_struct_fields, may_read_date_as_ntz, + spark_parquet_convert, SparkParquetOptions, }; use arrow::array::new_empty_array; use arrow::compute::can_cast_types; @@ -32,7 +32,7 @@ use arrow::datatypes::{DataType, Field, FieldRef, Schema, SchemaRef, TimeUnit}; use arrow::record_batch::RecordBatch; use datafusion::common::tree_node::{Transformed, TransformedResult, TreeNode, TreeNodeRecursion}; use datafusion::common::{DataFusionError, Result as DataFusionResult}; -use datafusion::physical_expr::expressions::Column; +use datafusion::physical_expr::expressions::{Column, IsNotNullExpr, IsNullExpr}; use datafusion::physical_expr::PhysicalExpr; use datafusion::physical_plan::ColumnarValue; use datafusion::scalar::ScalarValue; @@ -388,7 +388,7 @@ fn parquet_primitive_name(dt: &DataType) -> &'static str { DataType::Float64 => "DOUBLE", DataType::Utf8 | DataType::LargeUtf8 | DataType::Binary | DataType::LargeBinary => "BINARY", // Spark stores DATE as INT32 with a DATE logical-type annotation. - DataType::Date32 => "INT32", + DataType::Date32 | DataType::Date64 => "INT32", // Spark stores TIMESTAMP as INT64 with a timestamp annotation, or as // INT96 (legacy nanos). arrow-rs surfaces both as `Timestamp`; without // the original physical name we report INT64, which matches the @@ -611,7 +611,7 @@ fn spark_has_updater( } // Type promotion (widening). When `allow_type_promotion` is false, - // reject the three widenings (INT32→INT64, FLOAT→DOUBLE, INT32→DOUBLE) + // reject the widenings (INT32→INT64, FLOAT→DOUBLE, INT32→DOUBLE, DATE→NTZ) // that Spark 3.x's vectorized reader rejects. The flag tracks Comet's // per-Spark-version constant in ShimCometConf. if !options.allow_type_promotion { @@ -620,6 +620,10 @@ fn spark_has_updater( (DataType::Int32, DataType::Int64) | (DataType::Float32, DataType::Float64) | (DataType::Int32, DataType::Float64) + | ( + DataType::Date32 | DataType::Date64, + DataType::Timestamp(_, None) + ) ); if is_disallowed_promotion { return false; @@ -1404,11 +1408,35 @@ impl SparkPhysicalExprAdapter { } } } - Ok(Transformed::no(e)) + self.adapt_date_null_check(e) }) .data() } + /// DataFusion prunes by null count only on a bare column. Spark prunes before it + /// selects a DATE -> NTZ updater. The projected column still checks decoded values. + fn adapt_date_null_check( + &self, + expr: Arc, + ) -> DataFusionResult>> { + if self.parquet_options.prune_date_nulls + && (expr.is::() || expr.is::()) + { + let cast = expr.children()[0]; + if cast.is::() || cast.is::() { + let source = cast.children()[0]; + if may_read_date_as_ntz( + &source.data_type(&self.physical_file_schema)?, + &cast.data_type(&self.physical_file_schema)?, + ) { + let source = Arc::clone(source); + return Ok(Transformed::yes(expr.with_new_children(vec![source])?)); + } + } + } + Ok(Transformed::no(expr)) + } + /// Replace CastExpr (DataFusion's cast) with Spark's Cast expression. /// /// Only the casts that reconcile a file column with its table-schema type are Parquet read @@ -1540,9 +1568,9 @@ impl SparkPhysicalExprAdapter { } // Complex casts (including changes in list representation), timestamp tz relabel - // (e.g. Timestamp(us, None) -> Timestamp(us, Some("UTC")) for INT96 reads), and - // Timestamp -> Int64 - // (Spark's `nanosAsLong`) need spark_parquet_convert: it handles nested + // (e.g. Timestamp(us, None) -> Timestamp(us, Some("UTC")) for INT96 reads), + // Timestamp -> Int64 (Spark's `nanosAsLong`), and DATE -> NTZ need + // spark_parquet_convert: it handles checked reader conversions, nested // field selection, metadata-only tz changes, and raw-value reinterpretation // that Spark's Cast would otherwise convert incorrectly. if matches!( @@ -1563,7 +1591,8 @@ impl SparkPhysicalExprAdapter { | (DataType::Map(_, _), DataType::Map(_, _)) | (DataType::Timestamp(_, _), DataType::Timestamp(_, _)) | (DataType::Timestamp(_, _), DataType::Int64) - ) { + ) || may_read_date_as_ntz(physical_type, target_type) + { let comet_cast: Arc = Arc::new( CometCastColumnExpr::try_new( child, @@ -1599,7 +1628,7 @@ impl SparkPhysicalExprAdapter { )?)); } - Ok(Transformed::no(expr)) + self.adapt_date_null_check(expr) } /// Replace missing columns with defaults cast to the logical schema, propagating conversion @@ -2704,6 +2733,48 @@ pub(crate) mod test { Ok(()) } + #[tokio::test] + async fn parquet_date64_hint_read_as_ntz_checks_physical_type() -> Result<(), DataFusionError> { + use arrow::array::Date64Array; + use parquet::file::properties::WriterProperties; + + let days = i64::MAX / 86_400_000_000 + 1; + let values: ArrayRef = Arc::new(Date64Array::from(vec![days * 86_400_000])); + let batch = RecordBatch::try_from_iter([("d", values)])?; + let ntz = DataType::Timestamp(TimeUnit::Microsecond, None); + let required = Arc::new(Schema::new(vec![Field::new("d", ntz, false)])); + // Coercion writes INT32 DATE, otherwise the writer stores INT64. + // The Arrow footer hint says Date64 in both files. + for coerce in [true, false] { + let props = WriterProperties::builder().set_coerce_types(coerce).build(); + let filename = get_temp_filename().to_str().unwrap().to_string(); + let mut writer = + ArrowWriter::try_new(File::create(&filename)?, batch.schema(), Some(props))?; + writer.write(&batch)?; + writer.close()?; + + for promote in [true, false] { + let mut options = default_options(); + options.allow_type_promotion = promote; + let mut stream = scan_parquet_file( + filename.clone(), + Arc::clone(&required), + options, + None, + false, + )?; + let expected = match (coerce, promote) { + (true, true) => "long overflow", + (true, false) => "Found: INT32", + (false, _) => "Found: INT64", + }; + let error = stream.next().await.unwrap().unwrap_err().to_string(); + assert!(error.contains(expected), "unexpected error: {error}"); + } + } + Ok(()) + } + /// `Date32 -> Timestamp(_, Some(_))` (LTZ). Spark's vectorized reader /// allows `Date -> TimestampNTZ` but not `Date -> Timestamp(LTZ)`. #[tokio::test] diff --git a/native/spark-expr/src/conversion_funcs/cast.rs b/native/spark-expr/src/conversion_funcs/cast.rs index eb115979363..2e7161e2960 100644 --- a/native/spark-expr/src/conversion_funcs/cast.rs +++ b/native/spark-expr/src/conversion_funcs/cast.rs @@ -44,8 +44,8 @@ use crate::{cast_whole_num_to_binary, BinaryOutputStyle}; use crate::{EvalMode, SparkError}; use arrow::array::builder::{GenericStringBuilder, StringBuilder}; use arrow::array::{ - new_null_array, BinaryBuilder, GenericByteArray, GenericStringArray, ListArray, MapArray, - StringArray, StructArray, + new_null_array, BinaryBuilder, BooleanArray, Date32Array, GenericByteArray, GenericStringArray, + ListArray, MapArray, StringArray, StructArray, }; use arrow::buffer::{BooleanBuffer, Buffer, NullBuffer, OffsetBuffer}; use arrow::datatypes::{ @@ -58,7 +58,7 @@ use arrow::{ cast::AsArray, types::Decimal128Type, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array, OffsetSizeTrait, }, - compute::{cast_with_options, rescale_decimal, CastOptions}, + compute::{cast_with_options, kernels::zip::zip, rescale_decimal, CastOptions}, record_batch::RecordBatch, util::display::FormatOptions, }; @@ -68,6 +68,7 @@ use datafusion::common::{internal_err, DataFusionError, Result as DataFusionResu use datafusion::physical_expr::PhysicalExpr; use datafusion::physical_plan::ColumnarValue; use datafusion_comet_common::decode_utf8_spark_lossy; +use datafusion_comet_common::struct_nulls::{child_visibility, Visibility}; use std::{ fmt::{Debug, Display, Formatter, Write}, hash::Hash, @@ -265,6 +266,15 @@ pub(crate) fn cast_array( array: ArrayRef, to_type: &DataType, cast_options: &SparkCastOptions, +) -> DataFusionResult { + cast_array_impl(array, to_type, cast_options, &Visibility::new(&|| None)) +} + +fn cast_array_impl( + array: ArrayRef, + to_type: &DataType, + cast_options: &SparkCastOptions, + parent_visibility: &Visibility<'_>, ) -> DataFusionResult { use DataType::*; let from_type = array.data_type().clone(); @@ -302,6 +312,8 @@ pub(crate) fn cast_array( } } + let child_mask = || child_visibility(array.as_ref(), parent_visibility); + let children = Visibility::new(&child_mask); let cast_result = match (&from_type, to_type) { // Null arrays carry no concrete values, so Arrow's native cast can change only the // logical type while preserving length and nullness. @@ -440,6 +452,7 @@ pub(crate) fn cast_array( &from_type, to_type, cast_options, + &children, )?), (List(_), Utf8) => Ok(cast_array_to_string(array.as_list(), cast_options)?), (List(_), List(to)) => { @@ -449,10 +462,11 @@ pub(crate) fn cast_array( let casted_values = match (list_array.values().data_type(), to.data_type()) { // Spark legacy array casts produce null elements for array -> array. (Date32, Int32) => new_null_array(to.data_type(), list_array.values().len()), - _ => cast_array( + _ => cast_array_impl( Arc::clone(list_array.values()), to.data_type(), cast_options, + &children, )?, }; Ok(Arc::new(ListArray::new( @@ -462,14 +476,31 @@ pub(crate) fn cast_array( list_array.nulls().cloned(), )) as ArrayRef) } - (Map(_, _), Map(_, _)) => Ok(cast_map_to_map(&array, &from_type, to_type, cast_options)?), + (Map(_, _), Map(_, _)) => Ok(cast_map_to_map( + &array, + &from_type, + to_type, + cast_options, + &children, + )?), (UInt8 | UInt16 | UInt32 | UInt64, Int8 | Int16 | Int32 | Int64) if cast_options.allow_cast_unsigned_ints => { Ok(cast_with_options(&array, to_type, &CAST_OPTIONS)?) } (Binary, Utf8) => Ok(cast_binary_to_string::(&array, cast_options)?), - (Date32, Timestamp(_, tz)) => Ok(cast_date_to_timestamp(&array, cast_options, tz)?), + (Date32, Timestamp(_, tz)) => { + // An invisible date can still be required, e.g. a map key. Use the epoch as its + // placeholder so both checked and TRY casts preserve required child validity. + let dates = + if let Some(visible) = parent_visibility.as_ref().filter(|n| n.null_count() != 0) { + let visible = BooleanArray::new(visible.inner().clone(), None); + zip(&visible, &array, &Date32Array::new_scalar(0))? + } else { + array + }; + Ok(cast_date_to_timestamp(&dates, cast_options, tz)?) + } (Int8, Binary) if (eval_mode == Legacy) => cast_whole_num_to_binary!(&array, Int8Array, 1), (Int16, Binary) if (eval_mode == Legacy) => { cast_whole_num_to_binary!(&array, Int16Array, 2) @@ -562,22 +593,21 @@ fn cast_struct_to_struct( from_type: &DataType, to_type: &DataType, cast_options: &SparkCastOptions, + child_visibility: &Visibility<'_>, ) -> DataFusionResult { match (from_type, to_type) { - (DataType::Struct(from_fields), DataType::Struct(to_fields)) => { - let cast_fields: Vec = from_fields + (DataType::Struct(_), DataType::Struct(to_fields)) => { + let cast_fields = array + .columns() .iter() - .enumerate() - .zip(to_fields.iter()) - .map(|((idx, _from), to)| { - let from_field = Arc::clone(array.column(idx)); - let array_length = from_field.len(); - let cast_result = spark_cast( - ColumnarValue::from(from_field), - to.data_type(), + .zip(to_fields) + .map(|(child, field)| { + cast_array_impl( + Arc::clone(child), + field.data_type(), cast_options, - )?; - cast_result.to_array(array_length) + child_visibility, + ) }) .collect::>>()?; @@ -611,6 +641,7 @@ fn cast_map_to_map( from_type: &DataType, to_type: &DataType, cast_options: &SparkCastOptions, + child_visibility: &Visibility<'_>, ) -> DataFusionResult { let map_array = array .as_any() @@ -673,13 +704,23 @@ fn cast_map_to_map( let cast_keys = if key_type_unchanged { Arc::clone(keys) } else { - cast_array(Arc::clone(keys), to_fields[0].data_type(), cast_options)? + cast_array_impl( + Arc::clone(keys), + to_fields[0].data_type(), + cast_options, + child_visibility, + )? }; let values = map_array.values(); let cast_values = if value_type_unchanged { Arc::clone(values) } else { - cast_array(Arc::clone(values), to_fields[1].data_type(), cast_options)? + cast_array_impl( + Arc::clone(values), + to_fields[1].data_type(), + cast_options, + child_visibility, + )? }; // `None` rather than the source entries null buffer, which is always absent. @@ -2034,6 +2075,79 @@ mod tests { SparkCastOptions::new(EvalMode::Legacy, "UTC") } + #[test] + fn nested_timestamp_cast_checks_only_visible_dates() { + use arrow::array::{Date32Array, TimestampMicrosecondArray}; + + // One child per row, even when the struct/list/map is null. + fn containers(values: ArrayRef) -> [ArrayRef; 4] { + let field = Arc::new(Field::new("value", values.data_type().clone(), false)); + let offsets = OffsetBuffer::new(vec![0, 1, 2, 3, 4, 5].into()); + let nulls = Some(NullBuffer::from(vec![true, true, false, true, true])); + let map = |keys: ArrayRef| -> ArrayRef { + let key_field = Arc::new(Field::new("key", keys.data_type().clone(), false)); + let entries = StructArray::new( + vec![key_field, Arc::clone(&field)].into(), + vec![keys, Arc::clone(&values)], + None, + ); + Arc::new(MapArray::new( + Arc::new(Field::new("entries", entries.data_type().clone(), false)), + offsets.clone(), + entries, + nulls.clone(), + false, + )) + }; + [ + map(Arc::clone(&values)), + map(Arc::new(StringArray::from(vec!["a", "b", "c", "d", "e"]))), + Arc::new(StructArray::new( + vec![Arc::clone(&field)].into(), + vec![Arc::clone(&values)], + nulls.clone(), + )), + Arc::new(ListArray::new(field, offsets, values, nulls)), + ] + } + + let dates = containers(Arc::new(Date32Array::from(vec![ + i32::MIN, + 0, + i32::MAX, + 1, + i32::MAX, + ]))); + let timestamps = containers(Arc::new(TimestampMicrosecondArray::from(vec![ + 0, + 0, + 0, + 86_400_000_000, + 0, + ]))); + for (dates, expected) in dates.into_iter().zip(timestamps) { + for mode in [EvalMode::Legacy, EvalMode::Ansi, EvalMode::Try] { + let options = SparkCastOptions::new(mode, "UTC"); + let cast = |input| cast_array(input, expected.data_type(), &options); + for (start, len) in [(1, 3), (1, 1), (2, 1), (2, 0)] { + let actual = cast(dates.slice(start, len)).unwrap(); + actual.to_data().validate_full().unwrap(); + assert_eq!(actual.to_data(), expected.slice(start, len).to_data()); + if let Some(map) = dates + .as_map_opt() + .filter(|m| m.keys().data_type() == &DataType::Utf8) + { + assert!(Arc::ptr_eq(actual.as_map().keys(), map.keys())); + } + } + if mode != EvalMode::Try { + let error = cast(dates.slice(1, 4)).unwrap_err(); + assert!(error.to_string().contains("long overflow")); + } + } + } + } + /// Build a `Map` MapArray (Parquet-style "key_value" field names). fn build_str_i32_map( keys: Vec<&str>, diff --git a/native/spark-expr/src/conversion_funcs/string.rs b/native/spark-expr/src/conversion_funcs/string.rs index b4194e0a8ac..779ae7c427e 100644 --- a/native/spark-expr/src/conversion_funcs/string.rs +++ b/native/spark-expr/src/conversion_funcs/string.rs @@ -1702,7 +1702,7 @@ fn timestamp_parser( /// /// Hours must be 0–18 and minutes 0–59, with a maximum absolute offset of 18:00. /// A trailing colon ("+8:") is rejected. -fn parse_sign_offset(s: &str) -> Option { +pub(super) fn parse_sign_offset(s: &str) -> Option { if s.is_empty() { return Some(0); } diff --git a/native/spark-expr/src/conversion_funcs/temporal.rs b/native/spark-expr/src/conversion_funcs/temporal.rs index 15926047f73..ed04c471bce 100644 --- a/native/spark-expr/src/conversion_funcs/temporal.rs +++ b/native/spark-expr/src/conversion_funcs/temporal.rs @@ -15,15 +15,12 @@ // specific language governing permissions and limitations // under the License. -use crate::utils::resolve_local_datetime; -use crate::{EvalMode, SparkCastOptions, SparkResult}; -use arrow::array::timezone::Tz; -use arrow::array::{ArrayRef, AsArray, TimestampMicrosecondBuilder}; -use arrow::compute::{cast_with_options, CastOptions}; -use arrow::datatypes::{DataType, Date32Type, TimeUnit}; -use chrono::NaiveDate; -use datafusion::common::format::DEFAULT_CAST_OPTIONS; -use std::str::FromStr; +use crate::conversion_funcs::string::parse_sign_offset; +use crate::conversion_funcs::utils::{timestamp_micros_with_tz, MICROS_PER_SECOND}; +use crate::{EvalMode, SparkCastOptions, SparkError, SparkResult}; +use arrow::array::{ArrayRef, AsArray}; +use arrow::datatypes::{DataType, Date32Type}; +use arrow::error::ArrowError; use std::sync::Arc; pub(crate) fn is_df_cast_from_date_spark_compatible(to_type: &DataType) -> bool { @@ -42,121 +39,88 @@ pub(crate) fn cast_date_to_timestamp( cast_options: &SparkCastOptions, target_tz: &Option>, ) -> SparkResult { - if target_tz.is_none() { - // TIMESTAMP_NTZ ignores session TZ. Only TRY_CAST turns overflow into null. - return Ok(cast_with_options( - array_ref.as_ref(), - &DataType::Timestamp(TimeUnit::Microsecond, None), - &CastOptions { - safe: cast_options.eval_mode == EvalMode::Try, - ..DEFAULT_CAST_OPTIONS - }, - )?); - } - - let date_array = array_ref.as_primitive::(); - let mut builder = TimestampMicrosecondBuilder::with_capacity(date_array.len()); - // TIMESTAMP: midnight in session TZ → UTC epoch μs - let tz_str = if cast_options.timezone.is_empty() { - "UTC" + // CometTimeZone canonicalizes fixed offsets. Spark handles region zones, whose + // start-of-day rules extend beyond chrono-tz's transition tables. + let offset = if target_tz.is_none() || cast_options.timezone == "UTC" { + 0 } else { - cast_options.timezone.as_str() + parse_sign_offset(&cast_options.timezone).ok_or_else(|| { + ArrowError::ParseError(format!( + "Expected a fixed offset for DATE to TIMESTAMP, got {}", + cast_options.timezone + )) + })? }; - // safe to unwrap since we are falling back to UTC above - let tz = Tz::from_str(tz_str)?; - let epoch = NaiveDate::from_ymd_opt(1970, 1, 1).unwrap(); - for date in date_array.iter() { - match date { - Some(d) => { - // safe to unwrap since chrono's range ( 262,143 yrs) is higher than - // number of years possible with days as i32 (~ 6 mil yrs) - // convert date in session timezone to timestamp in UTC - let naive_date = epoch + chrono::Duration::days(d as i64); - let local_midnight = naive_date.and_hms_opt(0, 0, 0).unwrap(); - // Use resolve_local_datetime to correctly handle DST transitions: - // - Single: normal case, uses the given offset - // - Ambiguous (fall back): uses the earlier/DST occurrence, matching Spark - // - None (spring forward gap at midnight, e.g. America/Sao_Paulo): uses the - // pre-transition offset to compute the correct UTC time, matching Spark's - // LocalDate.atStartOfDay(zoneId) behaviour. - let local_midnight_in_microsec = - resolve_local_datetime(&tz, local_midnight).timestamp_micros(); - builder.append_value(local_midnight_in_microsec); - } - None => { - builder.append_null(); - } - } - } - Ok(Arc::new( - builder.finish().with_timezone_opt(target_tz.clone()), - )) + // Every Date32 fits in seconds. Spark applies the offset before multiplying + // by micros per second, so an offset can move midnight across the i64 boundary. + let to_micros = |day| { + let seconds = i64::from(day) * 86_400 - i64::from(offset); + seconds.checked_mul(MICROS_PER_SECOND) + }; + let dates = array_ref.as_primitive::(); + let timestamps = if cast_options.eval_mode == EvalMode::Try { + dates.unary_opt(to_micros) + } else { + dates + .try_unary(|day| to_micros(day).ok_or(())) + .map_err(|()| SparkError::LongOverflow)? + }; + Ok(timestamp_micros_with_tz(timestamps, target_tz)) } #[cfg(test)] mod tests { use super::*; - use std::sync::Arc; + use arrow::array::{Date32Array, TimestampMicrosecondArray}; + use arrow::datatypes::TimeUnit; + #[test] fn test_cast_date_to_timestamp() { - use crate::EvalMode; - use arrow::array::Date32Array; - use arrow::array::{Array, ArrayRef}; - use arrow::datatypes::TimestampMicrosecondType; - - // verifying epoch , DST change dates (US) and a null value (comprehensive tests on spark side) - let dates: ArrayRef = Arc::new(Date32Array::from(vec![ - Some(0), - Some(19723), - Some(19793), - None, - ])); - - let non_dst_date = 1704067200000000i64; - let dst_date = 1710115200000000i64; - let seven_hours_ts = 25200000000i64; - let eight_hours_ts = 28800000000i64; - - // validate UTC - let target_tz: Option> = Some("UTC".into()); - let result = cast_date_to_timestamp( - &dates, - &SparkCastOptions::new(EvalMode::Legacy, "UTC"), - &target_tz, - ) - .unwrap(); - let ts = result.as_primitive::(); - assert_eq!(ts.value(0), 0); - assert_eq!(ts.value(1), non_dst_date); - assert_eq!(ts.value(2), dst_date); - assert!(ts.is_null(3)); - - // validate LA timezone (follows Daylight savings) - let result = cast_date_to_timestamp( - &dates, - &SparkCastOptions::new(EvalMode::Legacy, "America/Los_Angeles"), - &target_tz, - ) - .unwrap(); - let ts = result.as_primitive::(); - assert_eq!(ts.value(0), eight_hours_ts); - assert_eq!(ts.value(1), non_dst_date + eight_hours_ts); - // should adjust for DST - assert_eq!(ts.value(2), dst_date + seven_hours_ts); - assert!(ts.is_null(3)); + // 95,026,237 is +262143-01-01, the day after chrono's maximum date. + let days = [Some(0), Some(95_026_237), Some(-95_026_237), None]; + let dates: ArrayRef = Arc::new(Date32Array::from_iter(days)); + for (zone, offset) in [("UTC", 0), ("-08:00", -28_800), ("+05:30", 19_800)] { + let actual = cast_date_to_timestamp( + &dates, + &SparkCastOptions::new(EvalMode::Legacy, zone), + &Some("UTC".into()), + ) + .unwrap(); + let expected = TimestampMicrosecondArray::from_iter( + days.map(|day| day.map(|d| i64::from(d) * 86_400_000_000 - offset * 1_000_000)), + ) + .with_timezone("UTC"); + assert_eq!(actual.as_ref(), &expected, "{zone}"); + } + } - // Phoenix timezone (does not follow Daylight savings) - let result = cast_date_to_timestamp( - &dates, - &SparkCastOptions::new(EvalMode::Legacy, "America/Phoenix"), - &target_tz, - ) - .unwrap(); - let ts = result.as_primitive::(); - assert_eq!(ts.value(0), seven_hours_ts); - assert_eq!(ts.value(1), non_dst_date + seven_hours_ts); - assert_eq!(ts.value(2), dst_date + seven_hours_ts); - assert!(ts.is_null(3)); + #[test] + fn test_cast_date_to_timestamp_offset_at_boundary() { + // The same day can fit or overflow depending on the offset. + for (days, zone, expected) in [ + (106_751_991, "-18:00", None), + (-106_751_991, "+18:00", None), + (106_751_991, "+18:00", Some(9_223_371_957_600_000_000)), + (-106_751_991, "-18:00", Some(-9_223_371_957_600_000_000)), + (i32::MAX, "UTC", None), + (i32::MIN, "UTC", None), + ] { + let dates: ArrayRef = Arc::new(Date32Array::from(vec![days])); + for mode in [EvalMode::Legacy, EvalMode::Ansi, EvalMode::Try] { + let result = cast_date_to_timestamp( + &dates, + &SparkCastOptions::new(mode, zone), + &Some("UTC".into()), + ); + if expected.is_none() && mode != EvalMode::Try { + assert!(matches!(result, Err(SparkError::LongOverflow))); + } else { + let expected = + TimestampMicrosecondArray::from(vec![expected]).with_timezone("UTC"); + assert_eq!(result.unwrap().as_ref(), &expected); + } + } + } } #[test] @@ -212,15 +176,12 @@ mod tests { for day in [106_751_992, -106_751_992, i32::MIN, i32::MAX] { let dates: ArrayRef = Arc::new(Date32Array::from(vec![None, Some(day)])); - assert!( - cast_date_to_timestamp( - &dates, - &SparkCastOptions::new(EvalMode::Legacy, "UTC"), - &ntz_target, - ) - .is_err(), - "day={day}" + let result = cast_date_to_timestamp( + &dates, + &SparkCastOptions::new(EvalMode::Legacy, "UTC"), + &ntz_target, ); + assert!(matches!(result, Err(SparkError::LongOverflow)), "day={day}"); } } diff --git a/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala b/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala index fb72412d722..efdf1cddf04 100644 --- a/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala +++ b/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala @@ -189,6 +189,9 @@ object CometBatchKernelCodegen extends Logging with CometExprTraitShim with Come "codegen dispatch: the output can carry a null map key, which Arrow cannot hold " + "(https://github.com/apache/datafusion-comet/issues/6172)") } + if (boundExpr.exists(CometCast.mayNullRequiredTimestamp)) { + return Some(CometCast.requiredTimestampReason) + } // Mirror WSCG's `spark.sql.codegen.maxFields` gate. Wide schemas blow the generated class's // typed input field count, the typed-getter switch, and the constant pool. Refuse here so the // operator falls back to Spark cleanly rather than tripping a Janino compile failure diff --git a/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala b/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala index 72c9c8eb546..8ac2256b315 100644 --- a/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala +++ b/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala @@ -121,7 +121,9 @@ object CometCast // would only duplicate the matrix and risk drifting from it. override def getSupportLevel(cast: Cast): SupportLevel = { - if (cast.child.isInstanceOf[Literal]) { + if (mayNullRequiredTimestamp(cast)) { + Unsupported(Some(requiredTimestampReason)) + } else if (cast.child.isInstanceOf[Literal]) { // A cast whose child is a literal is folded by Spark at planning time via `cast.eval()` // (see `convert`), so the cast never executes natively and the result matches Spark by // definition. `CometLiteral` then validates the resulting literal's data type, except @@ -338,6 +340,11 @@ object CometCast isSupported(from_map.keyType, to_map.keyType, timeZoneId, evalMode), isSupported(from_map.valueType, to_map.valueType, timeZoneId, evalMode))) } + case (DataTypes.DateType, DataTypes.TimestampType) + if !CometTimeZone.hasNativeFixedOffset(timeZoneId) => + // Spark's rules apply beyond chrono's date range and its last DST transition. + Unsupported( + Some("Native date-to-timestamp casts require a fixed whole-minute timezone offset")) case (DataTypes.DateType, toType) => canCastFromDate(toType, evalMode) case _ => unsupported(fromType, toType) } @@ -570,6 +577,35 @@ object CometCast case _ => Unsupported(Some(s"Cast from DateType to $toType is not supported")) } + private[comet] val requiredTimestampReason: String = + "TRY_CAST from date can overflow and produce a null in a non-nullable timestamp" + + // Spark counts date to timestamp as an upcast even though daysToMicros can overflow. Under + // TRY the result can then be null where Spark's nullability metadata says it cannot be. + private[comet] def mayNullRequiredTimestamp(expr: Expression): Boolean = expr match { + case cast: Cast if evalMode(cast) == CometEvalMode.TRY => + castsDateToRequiredTimestamp(cast.child.dataType, cast.dataType, cast.nullable) + case _ => false + } + + private def castsDateToRequiredTimestamp( + fromType: DataType, + toType: DataType, + nullable: Boolean): Boolean = + (fromType, toType) match { + case (DataTypes.DateType, TimestampType | TimestampNTZType) => !nullable + case (ArrayType(from, _), ArrayType(to, containsNull)) => + castsDateToRequiredTimestamp(from, to, containsNull) + case (MapType(fromKey, fromValue, _), MapType(toKey, toValue, valueContainsNull)) => + castsDateToRequiredTimestamp(fromKey, toKey, nullable = false) || + castsDateToRequiredTimestamp(fromValue, toValue, valueContainsNull) + case (StructType(from), StructType(to)) => + from.zip(to).exists { case (a, b) => + castsDateToRequiredTimestamp(a.dataType, b.dataType, b.nullable) + } + case _ => false + } + /** * Whether evaluating `cast` can produce a map with a null key. Under TRY a key cast that throws * becomes a null key, and Spark's `Cast.castMap` builds the result without rejecting it, so any diff --git a/spark/src/main/scala/org/apache/comet/serde/CometTimeZone.scala b/spark/src/main/scala/org/apache/comet/serde/CometTimeZone.scala index 1ed6967a116..64f7a763065 100644 --- a/spark/src/main/scala/org/apache/comet/serde/CometTimeZone.scala +++ b/spark/src/main/scala/org/apache/comet/serde/CometTimeZone.scala @@ -51,6 +51,9 @@ object CometTimeZone { } } + private[comet] def hasNativeFixedOffset(timeZoneId: Option[String]): Boolean = + nativeId(timeZoneId).exists(id => id == "UTC" || id.startsWith("+") || id.startsWith("-")) + /** Whether `timeZoneId` has a zero offset at every instant. */ def isUtc(timeZoneId: Option[String]): Boolean = nativeId(timeZoneId).contains("UTC") diff --git a/spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala b/spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala index 326b6da3380..fe2ac3b5254 100644 --- a/spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala +++ b/spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala @@ -91,6 +91,9 @@ trait ShimSparkErrorConverter { params("symbol").toString, params("value2").toString.toShort)) + case "LongOverflow" => + Some(new ArithmeticException("long overflow")) + case "ArithmeticOverflow" => val fromType = params("fromType").toString val functionName = params.get("functionName").map(_.toString).getOrElse("") diff --git a/spark/src/main/spark-4.x/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala b/spark/src/main/spark-4.x/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala index 53cd3f671ef..95f0bf1450d 100644 --- a/spark/src/main/spark-4.x/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala +++ b/spark/src/main/spark-4.x/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala @@ -111,6 +111,9 @@ trait ShimSparkErrorConverter { params("value2").toString.toShort, params("functionName").toString)) + case "LongOverflow" => + Some(new ArithmeticException("long overflow")) + case "ArithmeticOverflow" => val fromType = params("fromType").toString val functionName = params.get("functionName").map(_.toString).getOrElse("") diff --git a/spark/src/test/resources/sql-tests/expressions/aggregate/min_max_struct_timezone.sql b/spark/src/test/resources/sql-tests/expressions/aggregate/min_max_struct_timezone.sql index 1677acb61ae..5a4d99598ae 100644 --- a/spark/src/test/resources/sql-tests/expressions/aggregate/min_max_struct_timezone.sql +++ b/spark/src/test/resources/sql-tests/expressions/aggregate/min_max_struct_timezone.sql @@ -48,7 +48,7 @@ SELECT g, max(struct(coalesce(dt, DATE'2000-01-01') AS dt, coalesce(t, TIMESTAMP'2000-01-01 00:00:00') AS t)) FROM test_mm_struct_tz GROUP BY g ORDER BY g --- The timestamp field is computed natively from a date, which depends on the session time zone. +-- The timestamp field is computed from a date in the session time zone. query SELECT g, min(struct(CAST(coalesce(dt, DATE'2000-01-01') AS TIMESTAMP) AS t, coalesce(s, '') AS s)), diff --git a/spark/src/test/resources/sql-tests/expressions/cast/cast_date_to_timestamp.sql b/spark/src/test/resources/sql-tests/expressions/cast/cast_date_to_timestamp.sql new file mode 100644 index 00000000000..93de842d2db --- /dev/null +++ b/spark/src/test/resources/sql-tests/expressions/cast/cast_date_to_timestamp.sql @@ -0,0 +1,98 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +-- Config: spark.comet.exec.scalaUDF.codegen.enabled=true +-- Config: spark.sql.datetime.java8API.enabled=true +-- Config: spark.sql.session.timeZone=GMT+05:30 +-- Config: spark.sql.ansi.enabled=true + +statement +CREATE TABLE wide_dates(y INT, m INT, d INT) USING parquet + +-- The first two years exceed chrono's range but fit in timestamp microseconds. +-- July 2101 is beyond chrono-tz's last DST transition for Los Angeles. +statement +INSERT INTO wide_dates VALUES + (262143, 1, 1), (-262144, 1, 1), (2101, 7, 4), (1970, 1, 1), (NULL, 1, 1) + +query expect_native(cast, make_date) +SELECT CAST(make_date(y, m, d) AS TIMESTAMP), + CAST(array(named_struct('d', make_date(y, m, d))) AS ARRAY>) +FROM wide_dates + +statement +SET spark.sql.session.timeZone=America/Los_Angeles + +query expect_dispatch(cast) +SELECT CAST(make_date(y, m, d) AS TIMESTAMP) FROM wide_dates + +query expect_native(cast, make_date) +SELECT CAST(make_date(y, m, d) AS TIMESTAMP_NTZ) FROM wide_dates + +statement +SET spark.sql.session.timeZone=UTC + +statement +INSERT INTO wide_dates VALUES (300000, 6, 15), (-300000, 6, 15) + +query expect_native(try_cast) +SELECT TRY_CAST(make_date(y, m, d) AS TIMESTAMP), TRY_CAST(make_date(y, m, d) AS TIMESTAMP_NTZ) +FROM wide_dates + +-- Under ANSI, make_date with non-nullable inputs is non-nullable. Spark infers the +-- same for the TRY_CAST, even though 300000-06-15 overflows. +query expect_fallback(TRY_CAST from date can overflow) +SELECT TRY_CAST(make_date(coalesce(y, 300000), 6, 15) AS TIMESTAMP) FROM wide_dates + +statement +SET spark.sql.legacy.castComplexTypesToString.enabled=true + +-- The outer cast would otherwise send this subtree through the dispatcher. +query expect_fallback(TRY_CAST from date can overflow) +SELECT CAST(array(TRY_CAST(make_date(coalesce(y, 300000), 6, 15) AS TIMESTAMP_NTZ)) AS STRING) +FROM wide_dates + +statement +CREATE TABLE nested_dates USING parquet AS +SELECT y = 1970 AS keep, named_struct('d', dt) AS s, map(1, dt) AS m, map(dt, 1) AS k, + array(named_struct('d', dt)) AS a +FROM (SELECT y, make_date(y, m, d) AS dt FROM wide_dates WHERE y IN (1970, 300000)) + +-- IF can null a container without clearing the overflowing date in its child buffer. +-- Read each container twice to keep the optimizer from moving the cast inside the IF. +query expect_native(cast) +SELECT s, CAST(s AS STRUCT), m, CAST(m AS MAP), + k, CAST(k AS MAP), a, CAST(a AS ARRAY>) +FROM (SELECT IF(keep, s, NULL) AS s, IF(keep, m, NULL) AS m, IF(keep, k, NULL) AS k, + IF(keep, a, NULL) AS a FROM nested_dates) + +query expect_native(try_cast) +SELECT TRY_CAST(s AS STRUCT), TRY_CAST(m AS MAP), + TRY_CAST(a AS ARRAY>) +FROM nested_dates + +query expect_fallback(null map key) +SELECT TRY_CAST(k AS MAP) FROM nested_dates + +statement +SET spark.sql.session.timeZone=America/Los_Angeles + +statement +SET spark.comet.exec.scalaUDF.codegen.enabled=false + +query expect_fallback(cast: spark.comet.exec.scalaUDF.codegen.enabled=false) +SELECT CAST(make_date(y, m, d) AS TIMESTAMP) FROM wide_dates WHERE y = 262143 diff --git a/spark/src/test/resources/sql-tests/expressions/cast/timestamp_labels.sql b/spark/src/test/resources/sql-tests/expressions/cast/timestamp_labels.sql index be4fdd04a2c..096407ed706 100644 --- a/spark/src/test/resources/sql-tests/expressions/cast/timestamp_labels.sql +++ b/spark/src/test/resources/sql-tests/expressions/cast/timestamp_labels.sql @@ -46,8 +46,8 @@ SELECT CAST(CAST(s AS TIMESTAMP) AS STRING) FROM cast_timestamp_labels --- DATE → TIMESTAMP. -query expect_native(cast, equalto, lessthan, hour) +-- DATE → TIMESTAMP. The cast dispatches in Los Angeles (see cast_date_to_timestamp.sql). +query expect_native(equalto, lessthan, hour) SELECT CAST(dt AS TIMESTAMP) = ts, CAST(dt AS TIMESTAMP) < ts, diff --git a/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala b/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala index 894209204ac..197475121b3 100644 --- a/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala +++ b/spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala @@ -2323,6 +2323,44 @@ class CometNativeCastSuite } } + private def checkLongOverflow(query: String): Unit = { + val df = sql(query) + val plan = stripAQEPlan(df.queryExecution.executedPlan) + checkCometOperators(plan) + assertExpressionImpl(plan, native = Seq("cast"), dispatched = Seq.empty) + val (sparkError, cometError) = checkSparkAnswerMaybeThrows(df) + for ((engine, error) <- Seq("Spark" -> sparkError, "Comet" -> cometError)) { + val failure = error.getOrElse(fail(s"$engine should throw for $query")) + assert( + causeChain(failure).exists { cause => + cause.getClass == classOf[ArithmeticException] && cause.getMessage == "long overflow" + }, + s"Expected $engine to throw ArithmeticException: long overflow, got $failure") + } + } + + test("cast DateType to timestamps checks microsecond overflow") { + withParquetTable(Seq(Tuple1(300000)), "wide_year") { + for (ansi <- Seq("false", "true")) { + withSQLConf( + SQLConf.ANSI_ENABLED.key -> ansi, + SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") { + checkLongOverflow("SELECT CAST(make_date(_1, 6, 15) AS TIMESTAMP_NTZ) FROM wide_year") + checkLongOverflow("SELECT CAST(make_date(-_1, 6, 15) AS TIMESTAMP) FROM wide_year") + } + } + } + + // These days fit at UTC, but the offset pushes local midnight out of range. + for ((tz, day) <- Seq("+18:00" -> -106751991, "-18:00" -> 106751991)) { + withParquetTable(Seq(Tuple1(day)), "boundary_day") { + withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> tz) { + checkLongOverflow("SELECT CAST(date_from_unix_date(_1) AS TIMESTAMP) FROM boundary_day") + } + } + } + } + // CAST from TimestampType ignore("cast TimestampType to BooleanType") { diff --git a/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala b/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala index e44978c1d56..53be915299d 100644 --- a/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala +++ b/spark/src/test/scala/org/apache/comet/SparkErrorConverterSuite.scala @@ -29,6 +29,14 @@ import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus class SparkErrorConverterSuite extends AnyFunSuite { + test("LongOverflow converts to a plain ArithmeticException") { + val json = """{"errorType":"LongOverflow","errorClass":"","params":{}}""" + val ex = SparkErrorConverter.convertToSparkException( + new org.apache.comet.exceptions.CometQueryExecutionException(json)) + assert(ex.getClass == classOf[ArithmeticException]) + assert(ex.getMessage == "long overflow") + } + test("ArithmeticOverflow preserves the suggested function") { for (function <- Seq("try_add", "try_subtract", "try_multiply")) { val error = SparkErrorConverter diff --git a/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala b/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala index 10772988d3a..d9c96864ec9 100644 --- a/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala +++ b/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala @@ -358,6 +358,90 @@ abstract class ParquetReadSuite extends CometTestBase { } } + test("Parquet DATE read as TIMESTAMP_NTZ checks overflow in decoded row groups") { + val schema = MessageTypeParser.parseMessageType(""" + |message root { + | required int32 id; + | optional int32 d(DATE); + | optional group s { + | optional int32 d(DATE); + | } + | optional int64 stored(TIMESTAMP(MICROS,false)); + | optional int32 probe(DATE); + |} + |""".stripMargin) + val readSchema = + "id INT, d TIMESTAMP_NTZ, s STRUCT, " + + "stored TIMESTAMP_NTZ, probe TIMESTAMP_NTZ" + withTempDir { dir => + val path = new Path(dir.toURI.toString, "days.parquet") + // Keep the positive and negative overflows in distinct row groups, away from id = 1. + val writer = createParquetWriter(schema, path, rowGroupSize = 4096) + try { + (0 until 3000).foreach { id => + val day = id match { + case 1000 => 106751992 + case 2000 => -106751992 + case _ => id + } + val record = new SimpleGroup(schema) + record.add(0, id) + if (id % 3 != 0) record.add(1, day) + record.addGroup(2).add(0, day) + record.add(3, id.toLong) + if (id < 1500) record.add(4, id) + writer.write(record) + } + } finally { + writer.close() + } + + Seq(false, true).foreach { pushdown => + withSQLConf( + "spark.sql.parquet.datetimeRebaseModeInRead" -> "CORRECTED", + CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString) { + val df = spark.read.schema(readSchema).parquet(path.toString) + checkSparkAnswerAndOperator(df.where("id = 1000").select("stored")) + // Even Spark 3.5 skips the updater when statistics exclude every row group. + checkSparkAnswerAndOperator(df.where("id = 1 AND probe IS NULL").select("id")) + if (isSpark40Plus) { + checkSparkAnswerAndOperator(df.where("id = 1")) + if (!pushdown) { + // Null counts on probe must exclude the group at 1000 for IS NULL and the + // group at 2000 for IS NOT NULL before the reader converts the nested date. + checkSparkAnswerAndOperator(df.where("id <= 1500 AND probe IS NULL").select("s")) + checkSparkAnswerAndOperator( + df.where("id >= 1499 AND probe IS NOT NULL").select("s")) + // Both overflow groups contain nulls in d. The reader must convert d in + // those groups before the residual filter selects its nulls. + val error = + checkSparkError(df.where("d IS NULL").select("id"), "FAILED_READ_FILE.NO_HINT") + assert(error.getCause.getClass == classOf[ArithmeticException]) + assert(error.getCause.getMessage == "long overflow") + } + for (id <- Seq(1000, 2000); column <- Seq("d", "s")) { + val selected = df.where(s"id = $id").select(column) + val error = checkSparkError(selected, "FAILED_READ_FILE.NO_HINT") + assert(new java.net.URI(error.getMessageParameters.get("path")) == path.toUri) + assert(error.getCause.getClass == classOf[ArithmeticException]) + assert(error.getCause.getMessage == "long overflow") + } + } else { + // Spark 3.5 has no DateToTimestampNTZUpdater even when the value fits. + val selected = df.where("id = 1").select("d") + checkCometOperators(stripAQEPlan(selected.queryExecution.executedPlan)) + val (sparkError, cometError) = checkSparkAnswerMaybeThrows(selected) + for (error <- Seq(sparkError, cometError)) { + assert( + error.exists( + causeChain(_).exists(_.isInstanceOf[SchemaColumnConvertNotSupportedException]))) + } + } + } + } + } + } + test("TIMESTAMP_MILLIS overflow fails in native scan") { // Spark routes both TimestampType and TimestampNTZType through LongAsMicrosUpdater: // https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L140-L164