Skip to content

fix: reject negative decimal scales in native cast kernels - #7528

Open
ted-jenks wants to merge 8 commits into
apache:mainfrom
ted-jenks:negative-scale-cast-guard
Open

ted-jenks wants to merge 8 commits into
apache:mainfrom
ted-jenks:negative-scale-cast-guard

Conversation

@ted-jenks

@ted-jenks ted-jenks commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7022.

Rationale for this change

Since #5050 the serde keeps negative scales out of these kernels, but nothing in them rejects one, so another caller would panic in debug builds or return zeros in release builds.

What changes are included in this PR?

The four kernels share a decimal_scale_factor helper built on pow10_i128, and the cast compatibility page is updated.

How are these changes tested?

test_cast_kernels_reject_negative_scale covers all four kernels in Legacy and ANSI mode.

@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Nov 6, 2026
@ted-jenks
ted-jenks force-pushed the negative-scale-cast-guard branch from 0dfc904 to 5280143 Compare November 6, 2026 03:21
@ted-jenks
ted-jenks marked this pull request as ready for review November 6, 2026 21:23

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the guard matches what #7022 asked for and what #5050 did in decimal_div. One question on the doc comment of the new helper.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.

The guard itself looks right. decimal_scale_factor rejects anything outside 0..=38 through pow10_i128, and all four call sites (cast_decimal_to_int16_down, cast_decimal_to_int32_up, cast_int_to_decimal128, cast_decimal_to_timestamp) now use it. I also checked the Try path the test comment mentions and it does not reach these kernels.

The open question from the existing thread is whether a negative scale can still arrive from the Parquet schema adapter, and I think it can. In spark_has_updater in native/core/src/parquet/schema_adapter.rs, the integer to decimal check compares precision - scale against 10 or 20. A negative scale makes that difference larger, so an INT32 column read as DECIMAL(10,-2) passes the check and replace_with_spark_cast wraps it in a Spark Cast, which lands in cast_int_to_decimal128. Against the release branch the same query computes 10_i128.pow(scale as u32) with a wrapped exponent, which gives a multiplier of 0 in a release build, so the user silently got zeros. With this PR the user gets an INTERNAL_ERROR instead. That is better than wrong results, but it is still a query that fails where Spark returns values.

Since the author already offered, could we add the scan fallback in this PR rather than a follow-up? The type check in CometScanRule (isTypeSupported) could reject a DecimalType with a negative scale, with a fallback reason. A test that reads an integer Parquet column with spark.sql.legacy.allowNegativeScaleOfDecimal=true and a negative-scale read schema would then pin the behavior. If you would rather keep the scan change separate, a tracking issue linked from the doc comment on decimal_scale_factor would work. The comment should also say the path is currently reachable only through that scan case, so it does not read as "can never happen".

@ted-jenks

Copy link
Copy Markdown
Contributor Author

@andygrove added the scan fallback in ca2143a, with a test that reads an int column as DECIMAL(12,-2), and the comment now says the serde and scan both leave negative scales to Spark

@github-actions github-actions Bot added the area:scan Parquet scan / data reading label Nov 15, 2026

@coderfender coderfender left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

.ok()
.and_then(pow10_i128)
.ok_or_else(|| SparkError::Internal(format!("{cast}: unsupported decimal scale {scale}")))
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit : given that this is seldom reached given the Scala guards, wonder if we could just print scale and not the cast expression itself ?

.expect("Expected a Decimal128ArrayType");

let divisor = 10_i128.pow($scale as u32);
let divisor = decimal_scale_factor($scale, "cast_decimal_to_integral")?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit : wonder if we can take cast_decimal_to_integral into string

@coderfender coderfender added the run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue label Nov 15, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ted-jenks, the scan fallback is just what I was hoping for, and reading an INT column as DECIMAL(12,-2) with a null pins it end to end against Spark. Since Parquet can't store a negative scale, declining every negative-scale read type is the right scope. The last fork CI run I can find is on 8bcf2c290, before the fallback commit. Could you push again with Actions enabled on your fork so the new tests run?

# Conflicts:
#	native/spark-expr/src/conversion_funcs/numeric.rs
@coderfender

Copy link
Copy Markdown
Contributor

I'll merge this once the CI is green. Thank you @ted-jenks

@coderfender

Copy link
Copy Markdown
Contributor

Seems like there is a test failure

 an integer column read as a negative-scale decimal falls back to Spark *** FAILED *** (96 milliseconds)
    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 21.0 failed 1 times, most recent failure: Lost task 0.0 in stage 21.0 (TID 34) (localhost executor driver): org.apache.spark.SparkException: Parquet column cannot be converted in file file:///__w/datafusion-comet/datafusion-comet/spark/target/tmp/spark-32718606-1003-44c1-bcf8-d571100f829d/part-00000-4bc4fbde-8471-40b9-8e97-8ebf1de2c3e5-c000.snappy.parquet. Column: [a], Expected: decimal(12,-2), Found: INT32.

@andygrove

Copy link
Copy Markdown
Member

I think the Spark 3.5 failure is Spark's own reader rather than Comet. The fallback works, which is why the error comes from Spark's vectorized reader. But Spark 3.5 can't read an INT32 column as decimal(12,-2) at all, because widening an integer to a decimal on read arrived with Parquet type widening in Spark 4.0. So there's no Spark answer to compare against on 3.5.

Could the test check the fallback on every version and only compare answers where Spark can read it? For example, wrap the checkSparkAnswerAndFallbackReason call in if (isSpark40Plus), and on 3.5 assert that both Spark and Comet raise and that the plan falls back with the same reason.

@ted-jenks

Copy link
Copy Markdown
Contributor Author

@andygrove done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation area:scan Parquet scan / data reading backport-1.2 bug Something isn't working run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native decimal cast kernels compute 10^scale without checking for a negative scale

4 participants