Repository navigation
Conversation
|
@JingsongLi FYI, this PR would impact the work done in apache/paimon-rust#195 I've tried to avoid breaking the current implementation in Paimon. Happy to work with you or any of the other Paimon contributors if anything still needs to change. |
|
Does this lend support to iceberg too? |
|
Providing a first class way to query specific iceberg snapshots, branches, or tags is why I'm proposing this indeed. It gives the iceberg extension something to hook in to. I'm working on a sqlparser PR that goes hand in hand with this. The two combined should enable queries like https://iceberg.apache.org/docs/latest/spark-queries/#time-travel-queries-with-sql |
|
I will try and find time to read this one more carefully. It may be a few days |
Nice I was asking about Iceberg as I know it is of interest to many other DataFusion contributors so we may be able to get some more help review / buy in. @gabotechs do you have anyone you know that might be interested in this feature / helping review it from datafusion-iceberg land? |
|
Yes! I think @alexanderbianchi and/or @Perseus might be interested in this. If any of you guys can help with a review here that'd be awesome 🙏 |
|
I’m wondering whether this exposes an opportunity for a more general table-resolution extension point, with time travel as one implementation. The PR gets an important part right: resolving the selected provider before column binding. My concern is the amount of version-specific plumbing across Could we instead translate parsed modifiers into an extension-defined request, then pass that through a common resolution path? Illustratively: // Proposed preparation hook on the existing RelationPlanner.
// Runs before async table resolution.
fn prepare_table(
&self,
table: &TableFactor,
ctx: &mut dyn TablePreparationContext,
) -> Result<Option<TableResolutionRequest>>;The SQL extension would use the preparation context to evaluate expressions and produce a parser-independent request. The schema provider would consume it: // Proposed SchemaProvider method.
// An empty request delegates to ordinary table(name) lookup.
async fn resolve_table(
&self,
name: &str,
request: &TableResolutionRequest,
) -> Result<Option<ResolvedTable>>;
struct ResolvedTable {
provider: Arc<dyn TableProvider>,
identity: Option<ReadIdentity>,
}These are sketches rather than exact proposed types. For example: The SQL extension and catalog implementation share the request type. The existing We’d also need to preserve:
Would this be worth exploring before committing to the version-specific APIs? I’m motivated to help make it work, and have outlined related query-scoped catalog/provider limitations in apache/datafusion-iceberg#24. Thank you for working on this. |
I'll see what I can do. Certainly doesn't hurt to have a think about how we could generalise the concept introduced here. One benefit to the approach taken in the current shape of the PR is that the design is very straightforward to implement and use. Assuming you're not replacing any core components, only the I'm going to try parroting back your suggestion to make sure I understand it correctly. Could you confirm if I got the gist of it right?
To keep this ergonomic, I'm leaning towards the enum-with-extension approach. If we start from the current state of this PR, that would mean we rename In the current PR the resolution request and result representation have the same shape; both are |
|
Yes that summary is in line with what i'm thinking, open to this happening in various ways depending on how the code shakes out in practice. |
|
@alexanderbianchi I'm still reviewing and revising myself, but I've pushed the current state of things already to give you a chance to provide early feedback (if you have time to review a work-in-progress). Just want to give you a chance to nudge this in a different direction if you see the need. After another review round, the next step is to add the extension variant for I'm going to mark this PR as draft for now while I continue working on it. |
|
I took a quick look at the flow and went through it with the same agent that helped with my initial brainstorming. I think this is very much on the right track. The preparation hook and unified cache look like the key pieces we needed. The remaining extension support seems like a straightforward next step, following DataFusion’s existing extension patterns. Something like: pub enum TableSelector {
Version(ScalarValue),
Timestamp(i64),
Extension(Arc<dyn TableSelectorExtension>),
}With appropriate equality/hash and serialization behavior, that should let downstream implementations pass their own table-selection arguments through to the schema provider without adding more special cases to SessionState. Thanks for exploring this! Ping me when you want a deeper review! |
4f258c3 to
371f3ce
Compare
Plan time travel clauses such as `VERSION AS OF 3`, `TIMESTAMP AS OF '2024-01-01'`, `FOR SYSTEM_TIME AS OF ...` and `AT (VERSION => ...)`. All syntaxes are normalized to a `TableVersion`, either a `Version` holding any scalar value or a `Timestamp`. The version expression is evaluated during planning, so catalogs only see constants. - `SchemaProvider::table_at` returns a specific version of a table. The default returns "not implemented" rather than the current version. - `TableProvider::version` / `TableSource::version` report the version a table reads, which may differ from the requested version (e.g. a timestamp resolving to a snapshot). `ListingTable`, `MemTable`, `ViewTable` and `EmptyTable` gain `with_version`. - `TableScan` equality, ordering and hashing include the version, so subqueries over different versions of a table are not merged. EXPLAIN shows the version as `t@3`. - `RelationPlanner` extensions that handle time travel themselves keep working: the current version of versioned tables is still fetched when relation planners are registered. - Unparser, protobuf and Substrait support; parameters and execution time dependent functions (`now()`) are rejected in prepared statements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `TableReadSelector::Extension` for reads that are specific to a catalog or engine, defined by implementing `TableReadSelectorExtension`. Extensions are serialized with the new `LogicalExtensionCodec::try_encode_read_selector` and `try_decode_read_selector` methods, and unparsed to SQL with the new `Dialect::read_selector_extension_to_sql` hook. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I think this is at a point where the general shape of the PR is pretty clear. I feel like I'm stuck naming this thing. As I've been thinking through what concept we're introducing I've gone through:
None of these give me an "aha! that's it!" feeling. Other ideas were more in the direction of 'temporal', but that doesn't feel entirely right either. I feel like I'm going in circles at this point. Any extra opinions or suggestions would be welcome. PR description has been updated to reflect the current name. |
|
Sorry @pepijnve this has been on my list to review / weigh in on but I have been struggling to find time on it Basically I am trying to figure out what the core rationale is for including time travel queries in the core. I am worried that any feature we add may end up being too iceberg specific and thus not generalize if people want to support snowflake style snapshot queries, etc. Is if there is some way to implement this feature using an extension / rewrite on top of it, for example as described in https://datafusion.apache.org/blog/2026/01/12/extending-sql ? For example would it be possible to hook the table relation planner and intercept the AS OF times, and modify the TableProvider in a query with the appropriate timerange / snapshot time? If not, are there extension APIs we could add that would support this feature without having to thread it through the core |
I started from what Paimon's DataFusion integration does which can be seen at https://github.com/apache/paimon-rust/blob/main/crates/integrations/datafusion/src/relation_planner.rs#L121. What's done there is that the table name is first looked up using the bare table reference in the prefetched table map. Then a downcast is done, and a new TableProvider is derived that takes the options from the TableFactor into account. This is returned as a logical scan of the bare table reference. I think this is more or less the alternative solution you're talking about. I see a couple of possible issues with that approach.
What I'm experimenting with here is if there's some way to add an extension point that lets us improve upon the above without restricting ourselves to just time travel. What came out of a bunch of experimenting is two needs:
In @alexanderbianchi's suggestion the request and the identity values use two distinct types. Currently in this PR I'm using the same type as representation for both, but the request and identity values are allowed to be different. For lack of a better name this is currently called a 'table read selector'. The 'table read selector planner' is the bit that converts |
|
I think you've identified all the core design decisions to make here in that last message. IMO I agree that versioning seems like an abstract enough concept that it can be natively supported in DataFusion, so long as the same mechanism can be used. pub enum TableReadSelector {
AsOfVersion(ScalarValue),
AsOfTimestamp(i64),
Extension(Arc<dyn TableReadSelectorExtension>),
}Effectively that means removing the Enum and making |
Yes, that is basically what I was thinking
This is a good point. I think we could use the approach shown here for remote catalogs to mostly avoid this: https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/data_io/remote_catalog.rs#L38 However that API might need some changes to actually support AS OF times
I thought the model was that (effectively) there were different
Maybe this is the key thing we should push on -- if this feature is needed and could be used by more than one consumer then I think there is a strong case for including this in the core engine. For example, I would love to help ensure that https://github.com/apache/datafusion-iceberg ends up supporting time travel queries too So as a practical step forward, what do you think about making an example, in the spirit of https://github.com/apache/datafusion/blob/b7b014e731/datafusion-examples/examples/data_io/remote_catalog.rs#L38-L96 (or maybe extend it) to show how to connect in AS OF type queries? @alexanderbianchi I wonder if you could maybe try a POC integration of this API with datafusion-iceberg to verify this API would work for that project? |
I've been thinking about exactly that since for the most part DataFusion core treats that value as opaque anyway. Why bother with these two special cases? The only complexity that introduces is handling formatting in explain output and in the unparser. If we want to support that we would need to delegate the formatting/unparsing work to the extension point as well which would then have to deal with the dialect differences. Doesn't seem ideal either.
Yes, the table providers are indeed distinct, but logical plan equality doesn't and can't take that into account since
Our end goals are the same. I'm implementing this in function of our internal (I know, yet another one) fork of the alternative iceberg implementation. I'll see about extending the remote catalog example. |
That is a good point -- though I think we could add a PartialEq impl to the TableSource -- which would be a breaking change but that would not be too much of a breaking downstream change |
I totally emphathize -- we need to get something common that is actually possible to use in production and I don't think the datafusion-iceberg connector is that yet |
Very much agreed. We have our own catalog provider internally, which has some things that could improve datafusion-icebergo. Right now i'm working on apache/datafusion-iceberg#67 but can pickup work on apache/datafusion-iceberg#24 afterwards. |
|
I wonder if we could add some sort of general purpose API to the TableProvider to attach "user defined" extensions like there is to pass information down to a file scan / opener: datafusion/datafusion/datasource/src/mod.rs Lines 152 to 155 in 407bef6 We could then perhaps use the existing planner hooks to add the user defined extensions to the table provider / TableScan ? We would still have to expand TableScan to implement PartialEq, as well as sort out the async catalog access. However, the upside would be that the time travel specific parts ( |
Yes. |
Which issue does this PR close?
Rationale for this change
See #27209 description for full details. Adding first class support for time travel queries in the core library simplifies the implementation work for extensions that want to support this functionality.
Following review feedback, versions are generalized to a
TableReadSelector, which specifies which read of a table to perform. Time travel is the built-in case; extensions can define other reads.What changes are included in this PR?
TableReadSelectoris added:AsOfVersion,AsOfTimestampor anExtensiondefined outside DataFusion.SessionState::statement_to_plan, so the catalog can return the table for the selector. Time travel clauses are mapped by default; other syntax can be mapped by aTableReadSelectorPlanner.RelationPlannerbased version handlers like Paimon, the table without a selector is also prefetched when relation planners have been registered.SchemaProvider::table_with_read_selectorreturns a table for a selector. The default returns a "not implemented" error, so catalogs without time travel support fail instead of silently returning the current version.TableProvider::read_selector()/TableSource::read_selector()report the selector that will actually be read, e.g. the snapshot id a timestamp or tag resolves to.ListingTable,MemTable,ViewTableandEmptyTablegainwith_read_selector.TableScanequality, ordering and hashing include the selector reported by its source, which fixes the subquery described in the issue.Known limitations
clauses that depend on the time of execution, such as
now(), are rejectedin
PREPARE.What is the testing strategy for this PR?
time_travel.sltcovers all syntaxes, version and timestamp resolution, combining versions in one query, schema evolution,EXPLAIN, prepared statements and error cases. It includes the subquery queries from the issue as regression tests.table_read_selector_planner.rs/relation_planner.rs: extensions determining selectors, and handling time travel themselves.plan_to_sql.rs,resolve.rsandroundtrip_logical_plan.rscover unparsing, reference collection and protobuf round trips.Are there any user-facing changes?
Yes:
select.mdandcatalogs.mdgain time travel sections.AI Usage Disclosure
This PR was authored in collaboration with Claude Code. I've reviewed and understand the changes proposed here and will handle any review comments myself.