Skip to content

Implement Spark-compatible xpath_short #288

Open

Description

@andygrove

Part of #271 and apache/datafusion-comet#7300.

What the function does

xpath_short(xml, path) evaluates an XPath 1.0 expression, converts the result with XPath number() into a
Java Double, and narrows it with Double.shortValue(), giving a SMALLINT. Both arguments are strings
(collation-aware from 4.0), and path must be foldable. The result is always nullable. Available since Spark
2.0.0.

The number() conversion is in the xpath_double issue. Parsing, DTD and entity handling, namespaces, NULL
handling and errors are in the xpath issue. This issue covers only the double-to-short step.

The narrowing is Java's (short) d, which does not saturate. JLS 5.1.3 first converts the double to int,
with saturation and NaN becoming 0, and then keeps the low 16 bits. In Rust this is (d as i32) as i16, not
d as i16, which would saturate at i16::MAX.

Examples

SELECT xpath_short('<a><b>1</b><b>2</b></a>', 'sum(a/b)');
-- 3
SELECT xpath_short('<a>this is not a number</a>', 'a');
-- 0
SELECT xpath_short('<a><b>2000000000</b><c>40000000000</c></a>', 'a/b * a/c');
-- -1
SELECT xpath_short('<a>try a boolean</a>', 'a = 10');
-- 0
SELECT xpath_short('<a><b class="odd">1</b><b class="even">2</b><b class="odd">4</b><c>8</c></a>', 'a/b');
-- 1
SELECT xpath_short('<a><b class="odd">1</b><b class="even">2</b><b class="odd">4</b><c>8</c></a>', 'sum(a/*)');
-- 15
SELECT xpath_short('<a><b class="odd">10000</b><b class="even">2</b><b class="odd">4</b><c>8</c></a>', 'sum(a/b[@class="odd"])');
-- 10004

These come from the ExpressionDescription, the Hive udf_xpath_short golden files run by
HiveCompatibilitySuite, and XPathExpressionSuite. The -1 comes from
(short) (int) 8.0E19 = (short) 2147483647 = -1.

Edge cases to test

  • Wrap-around, by code reading:
    • <a>40000</a> gives -25536;
    • <a>32768</a> gives -32768;
    • <a>65536</a> gives 0;
    • -1 div 0 gives (short) Integer.MIN_VALUE, which is 0;
    • 1 div 0 gives -1.
  • Truncation toward zero before wrapping: <a>-2.7</a> gives -2.
  • NaN gives 0. This covers non-numeric text, 0 div 0, and the empty node-set, which is NaN in
    XPath 1.0.
  • NULL or '' arguments give NULL. A non-literal path gives DATATYPE_MISMATCH.NON_FOLDABLE_INPUT.
    Malformed XML and an invalid path raise runtime errors (see the xpath issue).

Differences across Spark versions

None found beyond the shared ones in the xpath issue.

Implementation notes

  • Spark: XPathShort in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala.
    It calls XPathShortEvaluator (ret.shortValue()) in .../xml/XmlExpressionEvalUtils.scala.
  • DataFusion core has no equivalent.
  • Reuse the shared number() evaluator from the xpath and xpath_double issues in
    crates/spark/src/function/xml/, and narrow with (d as i32) as i16.
  • Tests: add crates/sqllogictest/test_files/xml/xpath_short.slt and document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions