Skip to content

feat: let contribs register expression serdes - #7686

Draft
dwsmith1983 wants to merge 2 commits into
apache:mainfrom
dwsmith1983:feat/expression-contrib-hook
Draft

dwsmith1983 wants to merge 2 commits into
apache:mainfrom
dwsmith1983:feat/expression-contrib-hook

Conversation

@dwsmith1983

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Refs #7573. This is the second of three PRs:

  1. feat: add a native kernel for Delta's InterleaveBits #7600: the native InterleaveBits kernel.
  2. This PR: a core hook that lets a contrib register an expression serde.
  3. feat(delta): run Delta InterleaveBits natively #7597: the Delta serde and expression suite. It uses both and closes Support Delta InterleaveBits for one or two INT inputs #7573.

1 and 2 can merge in either order; #7597 goes last.

Rationale for this change

Comet has no way to convert an expression class that only a contrib knows about. Delta's InterleaveBits lives in Delta, so core can't register a serde for it without naming a Delta class. The scan side already solves this with the CometScanContrib ServiceLoader interface. This adds the same kind of hook for expressions, so the Delta contrib can register its serde and core stays free of Delta classes. Matching the class name in core, the way icebergFunctions.scala handles Iceberg, was the other option raised in #7573.

What changes are included in this PR?

  • CometExpressionContrib, a trait a contrib implements to return serdes keyed by expression class. Implementations are found through ContribServices, so a default build with no service file sees an empty registry.
  • Core serdes always win. A contrib entry for a class core already handles is ignored with a warning, and a class two contribs both claim is dropped from both, since ServiceLoader order is unspecified. A contrib whose serde map throws, including a LinkageError from a jar built against other Comet internals, is skipped with a warning.
  • QueryPlanSerde.serdeFor looks up core first, then contribs. Expression conversion and the ScalaUDF guard check use it instead of exprSerdeMap directly.
  • A shared ContribServiceFiles test helper, now used by CometScanContribSuite as well.

How are these changes tested?

CometExpressionContribSuite covers the merge rules (an unclaimed class is kept, core wins, a class two contribs claim is dropped, a contrib that throws or hits a LinkageError is skipped), discovery from a service file through a child ClassLoader, the serdeFor lookup order, and the empty registry on a default build. The suite is added to the Linux and macOS PR workflows. CometScanContribSuite still passes on the shared helper. The end-to-end check, a real contrib serde converting InterleaveBits natively, is in #7597.

Adds CometExpressionContrib, a ServiceLoader hook beside CometScanContrib,
so a contrib can register a serde for an expression class core never
names. Delta's InterleaveBits is the first user: the class lives in
delta-spark, which core does not depend on.

QueryPlanSerde.serdeFor now looks in core's map first and the contrib
registry second, and both lookup sites use it. A contrib entry for a
class core already handles is ignored, a class claimed by two contribs
is dropped from both since ServiceLoader order is unspecified, and a
contrib that throws is skipped. Default builds ship no service file,
so nothing changes for them.

The scan suite's service-file test helper moves into a shared test
utility that both suites use.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Delta InterleaveBits for one or two INT inputs

1 participant