Repository navigation
feat: let contribs register expression serdes - #7686
Draft
dwsmith1983 wants to merge 2 commits into
Draft
dwsmith1983 wants to merge 2 commits into
dwsmith1983 wants to merge 2 commits into
Conversation
Adds CometExpressionContrib, a ServiceLoader hook beside CometScanContrib, so a contrib can register a serde for an expression class core never names. Delta's InterleaveBits is the first user: the class lives in delta-spark, which core does not depend on. QueryPlanSerde.serdeFor now looks in core's map first and the contrib registry second, and both lookup sites use it. A contrib entry for a class core already handles is ignored, a class claimed by two contribs is dropped from both since ServiceLoader order is unspecified, and a contrib that throws is skipped. Default builds ship no service file, so nothing changes for them. The scan suite's service-file test helper moves into a shared test utility that both suites use.
This was referenced Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Refs #7573. This is the second of three PRs:
InterleaveBitskernel.1 and 2 can merge in either order; #7597 goes last.
Rationale for this change
Comet has no way to convert an expression class that only a contrib knows about. Delta's
InterleaveBitslives in Delta, so core can't register a serde for it without naming a Delta class. The scan side already solves this with theCometScanContribServiceLoader interface. This adds the same kind of hook for expressions, so the Delta contrib can register its serde and core stays free of Delta classes. Matching the class name in core, the wayicebergFunctions.scalahandles Iceberg, was the other option raised in #7573.What changes are included in this PR?
CometExpressionContrib, a trait a contrib implements to return serdes keyed by expression class. Implementations are found throughContribServices, so a default build with no service file sees an empty registry.LinkageErrorfrom a jar built against other Comet internals, is skipped with a warning.QueryPlanSerde.serdeForlooks up core first, then contribs. Expression conversion and theScalaUDFguard check use it instead ofexprSerdeMapdirectly.ContribServiceFilestest helper, now used byCometScanContribSuiteas well.How are these changes tested?
CometExpressionContribSuitecovers the merge rules (an unclaimed class is kept, core wins, a class two contribs claim is dropped, a contrib that throws or hits aLinkageErroris skipped), discovery from a service file through a child ClassLoader, theserdeForlookup order, and the empty registry on a default build. The suite is added to the Linux and macOS PR workflows.CometScanContribSuitestill passes on the shared helper. The end-to-end check, a real contrib serde convertingInterleaveBitsnatively, is in #7597.