Skip to content

Session/ Request/ Auth Context Propagation #2774

Open

Description

@DerGut

Is your feature request related to a problem or challenge?

We need to pass request context from Datafusion to our Iceberg connector.

Today, the Datafusion Session (which is already available for each query) terminates at the Iceberg catalog boundary. Scans for example, use a shared catalog and then call the catalog.load_table(&self.table_ident) function on it, which doesn't support any way of context propagation.

Our downstream REST catalog requires this context to make authorization, rate limiting and shard routing decisions.

Describe the solution you'd like

SessionCatalog Trait

The Java implementation solves this with a separate SessionCatalog interface which coexists next to the regular Catalog interface.
It replicates all interface methods, but accepts a SessionContext for each. Compare the Java Catalog's:

Table loadTable(TableIdentifier ident);

to the SessionCatalog's:

Table loadTable(SessionContext context, TableIdentifier ident);

A Rust implementation could re-use this mechanism and provide a separate SessionCatalog trait, along with a SessionContext struct (which is a simple struct without any behavior). The Datafusion integration's IcebergTableProvider could then store an Arc<dyn SessionCatalog> instead, translate the Datafusion Session to an Iceberg SessionContext for each query, and pass it along to the Catalog.

REST Implementation

The RestCatalog then needs to materialize the query context into the HTTP request metadata. For our use case, we really only need to forward the same context that we received, but the full Java implementation allows for authentication handled by the client.

The Java implementation uses the AuthManager for this. It is a field of the RestSessionCatalog implementation, and is called on each catalog method to produce a "contextual" AuthSession from a SessionContext.
The AuthSession is then used by the Rest/ HTTP client to authenticate any request with headers.

Note that this flow doesn't support propagation of arbitrary metadata, but only authentication information supported by the AuthManager/ AuthContext. This still suits our current needs.


On my willingness to contribute: I'm happy to handle the implementation, but I need some feedback on the overall design (whether or not we want to follow the Java approach).

Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

Activity

  1. changed the title [-]Session/ Request/ Auth Context[/-] [+]Session/ Request/ Auth Context Propagation[/+] on Jul 6, 2026
  2. DerGut commented on Jul 16, 2026

    @DerGut
    ContributorAuthor

    I've sketched out a plan to implement this work in 3 phases:

    1. Add support for session contexts in the catalog
      1. Introduce a SessionCatalog following Java's SessionCatalog interface, have it accept a SessionContext for catalog methods: Add SessionCatalog trait #2836
      2. Introduce the RestSessionCatalog implementation: Add RestSessionCatalog #2920
    2. Add support for auth sessions that can carry over session context
      1. Introduce AuthManager and AuthSession. Then allow the RestCatalog+RestSessionCatalog to use a manager to inject metadata into requests: feat!(rest): introduce AuthManager/AuthSession and migrate OAuth2 #2838
      2. support "contextual sessions" by deriving an AuthSession from a SessionContext: feat(rest_catalog::auth) Contextual Sessions #3081
    3. Wire the SessionCatalog into the Datafusion integration crate. Translate datafusion::catalog::Sessions to iceberg::SessionContexts and propagate them to catalog functions: Datafusion session integration #3000

    Depending on PR size, I may break it down further than that of course!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions