Skip to content

feat(substrait): add extension table consumer hook#23724

Draft
kadinrabo wants to merge 1 commit into
apache:mainfrom
kadinrabo:kadinrabo/substrait-extension-table-read
Draft

feat(substrait): add extension table consumer hook#23724
kadinrabo wants to merge 1 commit into
apache:mainfrom
kadinrabo:kadinrabo/substrait-extension-table-read

Conversation

@kadinrabo

@kadinrabo kadinrabo commented Jul 20, 2026

Copy link
Copy Markdown

Custom consumers can already override consume_read, but handling only ReadRel::ExtensionTable means copying the rest of from_read_rel.

This adds the same narrow hook used for extension relation cases. It doesn't add producer support or default table serialization.

Part of #13771

Tested with cargo test -p datafusion-substrait

@github-actions github-actions Bot added the substrait Changes to the substrait crate label Jul 20, 2026
consumer
.consume_extension_table(read, extension_table)
.await
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One bit of API that might be worth thinking about if you haven't already.

The NamedTable and LocalFiles read_types call the read_with_schema function at the end to also translate the projection, and filter fields of the ReadRel message. read_with_schema assumes that the translated read_type is vaguely TableReference shaped

With the addition of ExtensionTable, we have two read_types that aren't TableReference shaped (the other is VirtualTable) and don't correctly handle the projection and filter fields. Now might be a good time to try and figure out a better abstraction to handle those fields consistently for all read types.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants