Repository navigation
Description
Part of #271 and apache/datafusion-comet#7300.
What the function does
xpath_long(xml, path) evaluates an XPath 1.0 expression, converts the result with XPath number() into a
Java Double, and narrows it with Double.longValue(), giving a BIGINT. Both arguments are strings
(collation-aware from 4.0), and path must be foldable. The result is always nullable. Available since Spark
2.0.0.
The number() conversion is in the xpath_double issue. Parsing, DTD and entity handling, namespaces, NULL
handling and errors are in the xpath issue. This issue covers only the double-to-long step, which is Java's
(long) d:
NaNbecomes0;- values are truncated toward zero;
- out-of-range values saturate at
Long.MAX_VALUE/Long.MIN_VALUE.
This matches Rust's d as i64.
Examples
SELECT xpath_long('<a><b>1</b><b>2</b></a>', 'sum(a/b)');
-- 3
SELECT xpath_long('<a>this is not a number</a>', 'a');
-- 0
SELECT xpath_long('<a><b>2000000000</b><c>40000000000</c></a>', 'a/b * a/c');
-- 9223372036854775807
SELECT xpath_long('<a>try a boolean</a>', 'a = 10');
-- 0
SELECT xpath_long('<a><b class="odd">1</b><b class="even">2</b><b class="odd">4</b><c>8</c></a>', 'a/b');
-- 1
SELECT xpath_long('<a><b class="odd">1</b><b class="even">2</b><b class="odd">4</b><c>8</c></a>', 'sum(a/*)');
-- 15
SELECT xpath_long('<a><b class="odd">9000000000</b><b class="even">2</b><b class="odd">4</b><c>8</c></a>', 'sum(a/b[@class="odd"])');
-- 9000000004These come from the ExpressionDescription, the Hive udf_xpath_long golden files run by
HiveCompatibilitySuite, and XPathExpressionSuite.
Edge cases to test
- Precision. The value always goes through a double, so integers above 2^53 lose precision before the cast.
<a>9007199254740993</a>with'a'gives9007199254740992, by code reading (the nearest double). Spark
does not read the text directly as a long. - Saturation:
8.0E19gives9223372036854775807(above). By the same rule,-1 div 0gives
-9223372036854775808and1 div 0gives9223372036854775807. - Truncation toward zero (
<a>-2.7</a>gives-2), andNaNgives0. This covers non-numeric text,
0 div 0, and the empty node-set, which isNaNin XPath 1.0. NULLor''arguments give NULL. A non-literal path givesDATATYPE_MISMATCH.NON_FOLDABLE_INPUT.
Malformed XML and an invalid path raise runtime errors (see thexpathissue).
Differences across Spark versions
None found beyond the shared ones in the xpath issue.
Implementation notes
- Spark:
XPathLonginsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala.
It callsXPathLongEvaluator(ret.longValue()) in.../xml/XmlExpressionEvalUtils.scala. - DataFusion core has no equivalent.
- Reuse the shared
number()evaluator from thexpathandxpath_doubleissues in
crates/spark/src/function/xml/, and narrow withas i64. - Tests: add
crates/sqllogictest/test_files/xml/xpath_long.sltand document the function with#[user_doc].
This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.
Activity