Skip to content

feat: support time travel queries and table read selectors - #27210

Draft
pepijnve wants to merge 3 commits into
apache:mainfrom
pepijnve:tt_queries
Draft

pepijnve wants to merge 3 commits into
apache:mainfrom
pepijnve:tt_queries

Conversation

@pepijnve

@pepijnve pepijnve commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

See #27209 description for full details. Adding first class support for time travel queries in the core library simplifies the implementation work for extensions that want to support this functionality.

Following review feedback, versions are generalized to a TableReadSelector, which specifies which read of a table to perform. Time travel is the built-in case; extensions can define other reads.

What changes are included in this PR?

  • TableReadSelector is added: AsOfVersion, AsOfTimestamp or an Extension defined outside DataFusion.
  • Table resolution becomes selector aware. Selectors are determined before tables are prefetched in SessionState::statement_to_plan, so the catalog can return the table for the selector. Time travel clauses are mapped by default; other syntax can be mapped by a TableReadSelectorPlanner.
  • In order to not break existing RelationPlanner based version handlers like Paimon, the table without a selector is also prefetched when relation planners have been registered.
  • SchemaProvider::table_with_read_selector returns a table for a selector. The default returns a "not implemented" error, so catalogs without time travel support fail instead of silently returning the current version.
  • TableProvider::read_selector() / TableSource::read_selector() report the selector that will actually be read, e.g. the snapshot id a timestamp or tag resolves to. ListingTable, MemTable, ViewTable and EmptyTable gain with_read_selector.
  • TableScan equality, ordering and hashing include the selector reported by its source, which fixes the subquery described in the issue.
  • Versions are intentionally lenient with respect to their representation type, to support tag and branch names as Iceberg does. It is up to implementations to reject unsupported values.
  • Selectors are supported by protobuf serialization and the unparser.

Known limitations

  • Substrait and FFI do not support selectors yet.
  • The DataFrame API has no way to read a table with a selector yet.
  • Prepared statements: parameters are not supported in time travel clauses, and
    clauses that depend on the time of execution, such as now(), are rejected
    in PREPARE.

What is the testing strategy for this PR?

  • time_travel.slt covers all syntaxes, version and timestamp resolution, combining versions in one query, schema evolution, EXPLAIN, prepared statements and error cases. It includes the subquery queries from the issue as regression tests.
  • table_read_selector_planner.rs / relation_planner.rs: extensions determining selectors, and handling time travel themselves.
  • plan_to_sql.rs, resolve.rs and roundtrip_logical_plan.rs cover unparsing, reference collection and protobuf round trips.

Are there any user-facing changes?

Yes:

  • New SQL feature: time travel queries, for the dialects whose sqlparser implementation parses these clauses (Snowflake, Databricks, BigQuery, MsSQL). The generic dialect needs a sqlparser change, which will follow upstream.
  • Behaviour change: time travel clauses that were silently ignored are now planned, or rejected if the catalog does not support time travel.
  • API changes are documented in the 57.0.0 upgrade guide; select.md and catalogs.md gain time travel sections.

AI Usage Disclosure

This PR was authored in collaboration with Claude Code. I've reviewed and understand the changes proposed here and will handle any review comments myself.

@github-actions github-actions Bot added documentation Improvements or additions to documentation sql SQL Planner logical-expr Logical plan and expressions core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate catalog Related to the catalog crate common Related to common crate proto Related to proto crate labels Nov 7, 2026
@pepijnve

pepijnve commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

@JingsongLi FYI, this PR would impact the work done in apache/paimon-rust#195 I've tried to avoid breaking the current implementation in Paimon. Happy to work with you or any of the other Paimon contributors if anything still needs to change.

@pepijnve
pepijnve marked this pull request as ready for review November 7, 2026 16:29
@alamb

alamb commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Does this lend support to iceberg too?

@pepijnve

pepijnve commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Providing a first class way to query specific iceberg snapshots, branches, or tags is why I'm proposing this indeed. It gives the iceberg extension something to hook in to.

I'm working on a sqlparser PR that goes hand in hand with this. The two combined should enable queries like https://iceberg.apache.org/docs/latest/spark-queries/#time-travel-queries-with-sql

@alamb

alamb commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

I will try and find time to read this one more carefully. It may be a few days

@alamb

alamb commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Providing a first class way to query specific iceberg snapshots, branches, or tags is why I'm proposing this indeed. It gives the iceberg extension something to hook in to.

I'm working on a sqlparser PR that goes hand in hand with this. The two combined should enable queries like https://iceberg.apache.org/docs/latest/spark-queries/#time-travel-queries-with-sql

Nice

I was asking about Iceberg as I know it is of interest to many other DataFusion contributors so we may be able to get some more help review / buy in.

@gabotechs do you have anyone you know that might be interested in this feature / helping review it from datafusion-iceberg land?

@gabotechs

Copy link
Copy Markdown
Contributor

Yes! I think @alexanderbianchi and/or @Perseus might be interested in this. If any of you guys can help with a review here that'd be awesome 🙏

@alexanderbianchi

Copy link
Copy Markdown
Contributor

I’m wondering whether this exposes an opportunity for a more general table-resolution extension point, with time travel as one implementation.

The PR gets an important part right: resolving the selected provider before column binding. My concern is the amount of version-specific plumbing across SessionState, ContextProvider, schema providers, and separate table caches.

Could we instead translate parsed modifiers into an extension-defined request, then pass that through a common resolution path? Illustratively:

// Proposed preparation hook on the existing RelationPlanner.
// Runs before async table resolution.
fn prepare_table(
    &self,
    table: &TableFactor,
    ctx: &mut dyn TablePreparationContext,
) -> Result<Option<TableResolutionRequest>>;

The SQL extension would use the preparation context to evaluate expressions and produce a parser-independent request. The schema provider would consume it:

// Proposed SchemaProvider method.
// An empty request delegates to ordinary table(name) lookup.
async fn resolve_table(
    &self,
    name: &str,
    request: &TableResolutionRequest,
) -> Result<Option<ResolvedTable>>;

struct ResolvedTable {
    provider: Arc<dyn TableProvider>,
    identity: Option<ReadIdentity>,
}

These are sketches rather than exact proposed types. TableResolutionRequest would carry typed extension arguments; ReadIdentity would distinguish resolved reads for logical-plan equality.

For example:

events AT(VERSION => 'release_tag')
  → SQL extension produces TimeTravelRequest::Version("release_tag")
  → core routes it to the schema provider
  → Iceberg resolves the tag and returns a snapshot-bound provider
  → normal planning binds columns using that provider’s schema

The SQL extension and catalog implementation share the request type. SessionState orchestrates resolution without interpreting version semantics, and the catalog never needs to depend on sqlparser’s AST.

The existing RelationPlanner runs after prefetching, so this would require a preparation hook or equivalent integration with the earlier resolution phase. Unsupported modifiers would still need to fail explicitly.

We’d also need to preserve:

  • Consistent expression evaluation using the query’s planning context.
  • Resolved-table storage that distinguishes requests while allowing reuse, without inherently requiring a separate versioned cache.
  • Read identity in scan equality/hashing; separate provider instances alone aren’t sufficient.
  • Consistency between the resolved schema and execution, including across serialization.

Would this be worth exploring before committing to the version-specific APIs? I’m motivated to help make it work, and have outlined related query-scoped catalog/provider limitations in apache/datafusion-iceberg#24. Thank you for working on this.

@pepijnve

Copy link
Copy Markdown
Contributor Author

Would this be worth exploring before committing to the version-specific APIs? I’m motivated to help make it work, and have outlined related query-scoped catalog/provider limitations

I'll see what I can do. Certainly doesn't hurt to have a think about how we could generalise the concept introduced here.

One benefit to the approach taken in the current shape of the PR is that the design is very straightforward to implement and use. Assuming you're not replacing any core components, only the SchemaProvider needs to actively participate. It would be nice if we could retain that ease-of-use.

I'm going to try parroting back your suggestion to make sure I understand it correctly. Could you confirm if I got the gist of it right?

  • Rather than going version/timestamp specific, we introduce a more general notion of a more specific request for a table (TableResolutionRequest)
  • To keep this extensible this will have to be an extensible type. Could be an enum with an extension variant a la LogicalPlan::Extension or alternatively it could be a trait object.
  • The conversion from SQL AST (TableFactor) to the resolution request is also an extension point. Suggestion is to add that to RelationPlanner
  • The various table_at functions are generalised to use the table request instead
  • The resolution result is also generalised. Same decision to be made here wrt a trait object vs an extensible enum

To keep this ergonomic, I'm leaning towards the enum-with-extension approach. If we start from the current state of this PR, that would mean we rename TableVersion to something more generic, keep the existing time and version variants, and add a third user-defined one.

In the current PR the resolution request and result representation have the same shape; both are TableVersion. That should probably be split in two.

@alexanderbianchi

Copy link
Copy Markdown
Contributor

Yes that summary is in line with what i'm thinking, open to this happening in various ways depending on how the code shakes out in practice.

@pepijnve

pepijnve commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

@alexanderbianchi I'm still reviewing and revising myself, but I've pushed the current state of things already to give you a chance to provide early feedback (if you have time to review a work-in-progress). Just want to give you a chance to nudge this in a different direction if you see the need.

After another review round, the next step is to add the extension variant for TableSelector. I'm open to suggestions for better names for this enum. That's the best alternative I came up with so far.

I'm going to mark this PR as draft for now while I continue working on it.

@pepijnve
pepijnve marked this pull request as draft November 12, 2026 20:47
@alexanderbianchi

alexanderbianchi commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

I took a quick look at the flow and went through it with the same agent that helped with my initial brainstorming. I think this is very much on the right track. The preparation hook and unified cache look like the key pieces we needed.

The remaining extension support seems like a straightforward next step, following DataFusion’s existing extension patterns. Something like:

pub enum TableSelector {
    Version(ScalarValue),
    Timestamp(i64),
    Extension(Arc<dyn TableSelectorExtension>),
}

With appropriate equality/hash and serialization behavior, that should let downstream implementations pass their own table-selection arguments through to the schema provider without adding more special cases to SessionState.

Thanks for exploring this! Ping me when you want a deeper review!

@pepijnve
pepijnve force-pushed the tt_queries branch 3 times, most recently from 4f258c3 to 371f3ce Compare November 13, 2026 15:28
Plan time travel clauses such as `VERSION AS OF 3`, `TIMESTAMP AS OF
'2024-01-01'`, `FOR SYSTEM_TIME AS OF ...` and `AT (VERSION => ...)`.
All syntaxes are normalized to a `TableVersion`, either a `Version`
holding any scalar value or a `Timestamp`. The version expression is
evaluated during planning, so catalogs only see constants.

- `SchemaProvider::table_at` returns a specific version of a table. The
  default returns "not implemented" rather than the current version.
- `TableProvider::version` / `TableSource::version` report the version a
  table reads, which may differ from the requested version (e.g. a
  timestamp resolving to a snapshot). `ListingTable`, `MemTable`,
  `ViewTable` and `EmptyTable` gain `with_version`.
- `TableScan` equality, ordering and hashing include the version, so
  subqueries over different versions of a table are not merged. EXPLAIN
  shows the version as `t@3`.
- `RelationPlanner` extensions that handle time travel themselves keep
  working: the current version of versioned tables is still fetched when
  relation planners are registered.
- Unparser, protobuf and Substrait support; parameters and execution
  time dependent functions (`now()`) are rejected in prepared statements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pepijnve and others added 2 commits November 15, 2026 22:48
Add `TableReadSelector::Extension` for reads that are specific to a catalog
or engine, defined by implementing `TableReadSelectorExtension`. Extensions
are serialized with the new
`LogicalExtensionCodec::try_encode_read_selector` and
`try_decode_read_selector` methods, and unparsed to SQL with the new
`Dialect::read_selector_extension_to_sql` hook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pepijnve

pepijnve commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor Author

I think this is at a point where the general shape of the PR is pretty clear. I feel like I'm stuck naming this thing. As I've been thinking through what concept we're introducing I've gone through:

  • table version
  • table selector
  • snapshot selector
  • read selector
  • table read selector

None of these give me an "aha! that's it!" feeling. Other ideas were more in the direction of 'temporal', but that doesn't feel entirely right either.

I feel like I'm going in circles at this point. Any extra opinions or suggestions would be welcome. PR description has been updated to reflect the current name.

@pepijnve pepijnve changed the title feat: support time travel queries with VERSION AS OF / TIMESTAMP AS OF feat: support time travel queries and table read selectors Nov 15, 2026
@alamb

alamb commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Sorry @pepijnve this has been on my list to review / weigh in on but I have been struggling to find time on it

Basically I am trying to figure out what the core rationale is for including time travel queries in the core. I am worried that any feature we add may end up being too iceberg specific and thus not generalize if people want to support snowflake style snapshot queries, etc.

Is if there is some way to implement this feature using an extension / rewrite on top of it, for example as described in https://datafusion.apache.org/blog/2026/01/12/extending-sql ?

For example would it be possible to hook the table relation planner and intercept the AS OF times, and modify the TableProvider in a query with the appropriate timerange / snapshot time?

If not, are there extension APIs we could add that would support this feature without having to thread it through the core

@pepijnve

Copy link
Copy Markdown
Contributor Author

Basically I am trying to figure out what the core rationale is for including time travel queries in the core.

I started from what Paimon's DataFusion integration does which can be seen at https://github.com/apache/paimon-rust/blob/main/crates/integrations/datafusion/src/relation_planner.rs#L121. What's done there is that the table name is first looked up using the bare table reference in the prefetched table map. Then a downcast is done, and a new TableProvider is derived that takes the options from the TableFactor into account. This is returned as a logical scan of the bare table reference.

I think this is more or less the alternative solution you're talking about. I see a couple of possible issues with that approach.

  • RelationPlanner is not async. As a consequence all metadata you might possible need must have been fetched already during prefetch and there's no clear path to optimising this. This means you pay the cost of fetching all historic metadata regardless of whether you'll actually need it or not.
  • The returned scan logical plan is in no way different from a scan of the same table without any or with different options. In other words, at the logical plan level from t version as of 1 and from t version as of 2 are identical. I'm not sure if this is currently the case, but something like select * from t version as of 1 union select * from t version as of 2 could end up being optimised incorrectly because of this.
  • Very practically, time travel queries are quite common across formats (not just Iceberg). It seems practical for the query planner to know how to handle these so that each extension doesn't have to redo this. The alternative is that every extension implements their own 'versioned scan' logical plan.

What I'm experimenting with here is if there's some way to add an extension point that lets us improve upon the above without restricting ourselves to just time travel. What came out of a bunch of experimenting is two needs:

  • During the async prefetch phase, you want some way to extract information out of the TableFactor which can then be passed to the SchemaProvider so that it knows which metadata is actually needed. It only needs to return a TableProvider that can read exactly what was asked for.
  • If I perform two different lookups like this I need some way to tell the two TableProvider instances apart. We could key this on the TableReference along with the request value (i.e. the info extracted from the TableFactor), but then it's no longer possible for multiple requests that resolve to the same thing to actually be identical. When using timestamp as of for instance it's quite possible that two distinct timestamps resolve to the same snapshot. If we want to be able to detect this situation, then we need something that identifies a particular version (or more generally aspect) of a table that may be different from the request parameters.

In @alexanderbianchi's suggestion the request and the identity values use two distinct types. Currently in this PR I'm using the same type as representation for both, but the request and identity values are allowed to be different. For lack of a better name this is currently called a 'table read selector'.

The 'table read selector planner' is the bit that converts TableFactor to something independent of the SQL AST. SchemaProvider::table_with_read_selector performs the lookup during prefetch. TableProvider::read_selector is the identity part.

@alexanderbianchi

alexanderbianchi commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

I think you've identified all the core design decisions to make here in that last message. IMO I agree that versioning seems like an abstract enough concept that it can be natively supported in DataFusion, so long as the same mechanism can be used.
If you think of Datafusion's UDF's - they all implement the same API's that user defined functions do. The code I think is close Andrew's suggestion of using an extension. There needs to be an extension point that simplifies TableProvider equality, passing information to SchemaProvider, and allows async metadata resolution like. Could versioning just be an implementation of TableReadSelector?

pub enum TableReadSelector {
    AsOfVersion(ScalarValue),
    AsOfTimestamp(i64),
    Extension(Arc<dyn TableReadSelectorExtension>),
}

Effectively that means removing the Enum and making AsOfVersion / AsOfTimestamp an implementation of TableReadSelectorExtension.

@alamb

alamb commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

I think this is more or less the alternative solution you're talking about. I see a couple of possible issues with that approach.

Yes, that is basically what I was thinking

https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/data_io/remote_catalog.rs#L38

  • RelationPlanner is not async. As a consequence all metadata you might possible need must have been fetched already during prefetch and there's no clear path to optimising this. This means you pay the cost of fetching all historic metadata regardless of whether you'll actually need it or not.

This is a good point. I think we could use the approach shown here for remote catalogs to mostly avoid this: https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/data_io/remote_catalog.rs#L38

However that API might need some changes to actually support AS OF times

  • The returned scan logical plan is in no way different from a scan of the same table without any or with different options. In other words, at the logical plan level from t version as of 1 and from t version as of 2 are identical. I'm not sure if this is currently the case, but something like select * from t version as of 1 union select * from t version as of 2 could end up being optimised incorrectly because of this.

I thought the model was that (effectively) there were different Arc<dyn TableProvider>s for each union input

  • Very practically, time travel queries are quite common across formats (not just Iceberg). It seems practical for the query planner to know how to handle these so that each extension doesn't have to redo this. The alternative is that every extension implements their own 'versioned scan' logical plan.

Maybe this is the key thing we should push on -- if this feature is needed and could be used by more than one consumer then I think there is a strong case for including this in the core engine.

For example, I would love to help ensure that https://github.com/apache/datafusion-iceberg ends up supporting time travel queries too

So as a practical step forward, what do you think about making an example, in the spirit of https://github.com/apache/datafusion/blob/b7b014e731/datafusion-examples/examples/data_io/remote_catalog.rs#L38-L96 (or maybe extend it) to show how to connect in AS OF type queries?

@alexanderbianchi I wonder if you could maybe try a POC integration of this API with datafusion-iceberg to verify this API would work for that project?

@pepijnve

pepijnve commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

@alexanderbianchi

Effectively that means removing the Enum and making AsOfVersion / AsOfTimestamp an implementation of TableReadSelectorExtension.

I've been thinking about exactly that since for the most part DataFusion core treats that value as opaque anyway. Why bother with these two special cases? The only complexity that introduces is handling formatting in explain output and in the unparser. If we want to support that we would need to delegate the formatting/unparsing work to the extension point as well which would then have to deal with the dialect differences. Doesn't seem ideal either.

@alamb

I thought the model was that (effectively) there were different Arcs for each union input

Yes, the table providers are indeed distinct, but logical plan equality doesn't and can't take that into account since TableSource is not required to have a PartialEq implementation. The only discriminator the logical plan has is the TableReference and that's the same for both. That's why it's being augmented here with an identity representing value that is PartialEq.

For example, I would love to help ensure that https://github.com/apache/datafusion-iceberg ends up supporting time travel queries too
So as a practical step forward, what do you think about making an example, in the spirit of https://github.com/apache/datafusion/blob/b7b014e731/datafusion-examples/examples/data_io/remote_catalog.rs#L38-L96 (or maybe extend it) to show how to connect in AS OF type queries?

Our end goals are the same. I'm implementing this in function of our internal (I know, yet another one) fork of the alternative iceberg implementation. I'll see about extending the remote catalog example.

@alamb

alamb commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Yes, the table providers are indeed distinct, but logical plan equality doesn't and can't take that into account since TableSource is not required to have a PartialEq implementation. The only discriminator the logical plan has is the TableReference and that's the same for both. That's why it's being augmented here with an identity representing value that is PartialEq.

That is a good point -- though I think we could add a PartialEq impl to the TableSource -- which would be a breaking change but that would not be too much of a breaking downstream change

@alamb

alamb commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Our end goals are the same. I'm implementing this in function of our internal (I know, yet another one) fork of the alternative iceberg implementation. I'll see about extending the remote catalog example.

I totally emphathize -- we need to get something common that is actually possible to use in production and I don't think the datafusion-iceberg connector is that yet

@alexanderbianchi

alexanderbianchi commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

I don't think the datafusion-iceberg connector is that yet

Very much agreed. We have our own catalog provider internally, which has some things that could improve datafusion-icebergo. Right now i'm working on apache/datafusion-iceberg#67 but can pickup work on apache/datafusion-iceberg#24 afterwards.

@alamb

alamb commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

I wonder if we could add some sort of general purpose API to the TableProvider to attach "user defined" extensions like there is to pass information down to a file scan / opener:

/// User-defined per-file metadata, keyed by Rust type. Multiple
/// independent components can each attach their own data here without
/// conflict — see [`FileExtensions`].
pub extensions: FileExtensions,

We could then perhaps use the existing planner hooks to add the user defined extensions to the table provider / TableScan ?

We would still have to expand TableScan to implement PartialEq, as well as sort out the async catalog access.

However, the upside would be that the time travel specific parts (AsOfVersion, the selector planner, etc) could live in an extension rather than the core, which I think is similar to what @alexanderbianchi is suggesting

@alexanderbianchi

alexanderbianchi commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

I think is similar to what @alexanderbianchi is suggesting

Yes.
It'll also need to be wired through the CatalogProvider/SchemaProvider api's since the TableProvider's schema could be different across versions. Unclear to me if it needs to be in the TableProvider trait or just part of constructing the TableProvider.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

catalog Related to the catalog crate common Related to common crate core Core DataFusion crate documentation Improvements or additions to documentation logical-expr Logical plan and expressions proto Related to proto crate sql SQL Planner sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add awareness of time travel queries to core (VERSION AS OF / TIMESTAMP AS OF)

4 participants