Repository navigation
Description
Part of #271 and apache/datafusion-comet#7300.
What the function does
xpath_short(xml, path) evaluates an XPath 1.0 expression, converts the result with XPath number() into a
Java Double, and narrows it with Double.shortValue(), giving a SMALLINT. Both arguments are strings
(collation-aware from 4.0), and path must be foldable. The result is always nullable. Available since Spark
2.0.0.
The number() conversion is in the xpath_double issue. Parsing, DTD and entity handling, namespaces, NULL
handling and errors are in the xpath issue. This issue covers only the double-to-short step.
The narrowing is Java's (short) d, which does not saturate. JLS 5.1.3 first converts the double to int,
with saturation and NaN becoming 0, and then keeps the low 16 bits. In Rust this is (d as i32) as i16, not
d as i16, which would saturate at i16::MAX.
Examples
SELECT xpath_short('<a><b>1</b><b>2</b></a>', 'sum(a/b)');
-- 3
SELECT xpath_short('<a>this is not a number</a>', 'a');
-- 0
SELECT xpath_short('<a><b>2000000000</b><c>40000000000</c></a>', 'a/b * a/c');
-- -1
SELECT xpath_short('<a>try a boolean</a>', 'a = 10');
-- 0
SELECT xpath_short('<a><b class="odd">1</b><b class="even">2</b><b class="odd">4</b><c>8</c></a>', 'a/b');
-- 1
SELECT xpath_short('<a><b class="odd">1</b><b class="even">2</b><b class="odd">4</b><c>8</c></a>', 'sum(a/*)');
-- 15
SELECT xpath_short('<a><b class="odd">10000</b><b class="even">2</b><b class="odd">4</b><c>8</c></a>', 'sum(a/b[@class="odd"])');
-- 10004These come from the ExpressionDescription, the Hive udf_xpath_short golden files run by
HiveCompatibilitySuite, and XPathExpressionSuite. The -1 comes from
(short) (int) 8.0E19 = (short) 2147483647 = -1.
Edge cases to test
- Wrap-around, by code reading:
<a>40000</a>gives-25536;<a>32768</a>gives-32768;<a>65536</a>gives0;-1 div 0gives(short) Integer.MIN_VALUE, which is0;1 div 0gives-1.
- Truncation toward zero before wrapping:
<a>-2.7</a>gives-2. NaNgives0. This covers non-numeric text,0 div 0, and the empty node-set, which isNaNin
XPath 1.0.NULLor''arguments give NULL. A non-literal path givesDATATYPE_MISMATCH.NON_FOLDABLE_INPUT.
Malformed XML and an invalid path raise runtime errors (see thexpathissue).
Differences across Spark versions
None found beyond the shared ones in the xpath issue.
Implementation notes
- Spark:
XPathShortinsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala.
It callsXPathShortEvaluator(ret.shortValue()) in.../xml/XmlExpressionEvalUtils.scala. - DataFusion core has no equivalent.
- Reuse the shared
number()evaluator from thexpathandxpath_doubleissues in
crates/spark/src/function/xml/, and narrow with(d as i32) as i16. - Tests: add
crates/sqllogictest/test_files/xml/xpath_short.sltand document the function with#[user_doc].
This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.
Activity