Skip to content

Implement Spark-compatible xpath_boolean #282

Open

Description

@andygrove

Part of #271 and apache/datafusion-comet#7300.

What the function does

xpath_boolean(xml, path) evaluates an XPath 1.0 expression and converts the result to BOOLEAN using the
XPath boolean() rules (XPathConstants.BOOLEAN). Both arguments are strings (collation-aware from 4.0), and
path must be foldable. The result is always nullable. Available since Spark 2.0.0. The expression is a
Predicate, so it can be used directly in WHERE.

Parsing, DTD and entity handling, namespaces, NULL and empty-string handling, invalid-XML and invalid-path
errors, and lazy path compilation are shared with the rest of the family. They are specified in the xpath
issue, and the implementation should reuse that evaluator. This issue covers only the boolean conversion.

Examples

SELECT xpath_boolean('<a><b>1</b></a>','a/b');
-- true
SELECT xpath_boolean('<a><b>b</b></a>', 'a/c');
-- false
SELECT xpath_boolean('<a><b>b</b></a>', 'a/b = "b"');
-- true
SELECT xpath_boolean('<a><b>b</b></a>', 'a/b = "c"');
-- false
SELECT xpath_boolean('<a><b>10</b></a>', 'a/b < 10');
-- false
SELECT xpath_boolean('<a><b>10</b></a>', 'a/b = 10');
-- true

These come from the ExpressionDescription, XPathExpressionSuite and the Hive udf_xpath_boolean golden
files run by HiveCompatibilitySuite. In UDFXPathUtilSuite, a/b[1]/text() is true and a/b[4] (no match)
is false.

Edge cases to test

  • Conversion rules from XPath 1.0 boolean():
    • a node-set is true if it is non-empty, whatever the text. <b>false</b> with a/b/text() gives true;
    • a number is false for 0, -0 and NaN, and true otherwise;
    • a string is true if it is non-empty, so string(a/c) on a missing node gives false;
    • a boolean result is used as is.
  • Comparisons between a node-set and a value are existential: a/b = 2 is true if any b equals 2. Also test
    number versus string comparison (a/b = "10" versus a/b = 10 with <b>10.0</b>).
  • NULL or '' for either argument gives NULL. A non-literal path fails with
    DATATYPE_MISMATCH.NON_FOLDABLE_INPUT.
  • Malformed XML and an invalid path raise the runtime errors described in the xpath issue. There is no
    false fallback.
  • Use in a filter (WHERE xpath_boolean(xml, 'a/b')) with NULL rows. NULL rows are filtered out.

Differences across Spark versions

None found beyond the shared ones listed in the xpath issue (4.0 moved to RuntimeReplaceable and added
collation support).

Implementation notes

  • Spark: XPathBoolean in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala.
    It calls XPathBooleanEvaluator, which calls UDFXPathUtil.evalBoolean.
  • DataFusion core has no equivalent.
  • Put it in crates/spark/src/function/xml/ on top of the shared evaluator from the xpath issue.
    The existing crates/sqllogictest/test_files/xml/xpath_boolean.slt is a commented-out stub that should be
    enabled and extended. Document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions