Skip to content

feat(rest): support vended storage credentials with per-prefix FileIO - #2651

Open
plusplusjiajia wants to merge 27 commits into
apache:mainfrom
plusplusjiajia:feat/rest-vended-credentials
Open

plusplusjiajia wants to merge 27 commits into
apache:mainfrom
plusplusjiajia:feat/rest-vended-credentials

Conversation

@plusplusjiajia

@plusplusjiajia plusplusjiajia commented Jun 15, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

What changes are included in this PR?

LoadTableResult.storage_credentials is deserialized but never used, although the spec says clients should prefer it over config. A catalog that vends credentials only there leaves the table without storage access.

As @blackmwk suggested, the credentials travel in StorageConfig and the storage factory applies them when it builds the storage:

  • StorageConfig holds a list of StorageCredential { prefix, config }, set through StorageConfig::builder() and passed to FileIOBuilder::with_config, and serialized with the rest of the config. FileIO is unchanged.
  • Both OpenDAL factories build one storage per credential from the props overlaid by its config, and one from the props alone. The resolving factory does this for each scheme when it first resolves it, from the credentials whose prefix names that scheme. A path goes to the credential with the longest prefix it starts with, compared as plain strings as in Java's S3FileIO.clientForStoragePath, and to the props-only storage otherwise.
  • load_table, create_table and register_table pass the response's credentials. Delegation stays opt-in through the existing header.* config (header.X-Iceberg-Access-Delegation=vended-credentials), as in Java's RESTUtil.configHeaders.

Not obvious from the diff:

  • Breaking for the REST crate: its own StorageCredential is removed, and LoadTableResult.storage_credentials now holds iceberg::io::StorageCredential, which reads the same JSON but exposes prefix() and config() instead of public fields.
  • StorageCredential::new rejects an empty prefix, and so does deserializing one, so a loadTable response with an empty prefix fails to parse, as Java's Credential validation does.
  • StorageCredential matches the one in feat(io)!: [1/N] support refreshable storage credentials for S3 and GCS #2976 except that new returns a Result here, so whichever PR lands second drops its copy and adapts the new calls.
  • A custom S3 credential loader still applies to every path, including those under a credential's prefix, as Java prefers a configured credentials provider over static keys (AWS: Prefer custom credential provider if specified iceberg#15249).
  • StorageConfig keeps the later of two credentials that share a prefix, as Java's clientByPrefix does.
  • delete_prefix routes the path as a directory, so s3://b/t and s3://b/t/ both reach an s3://b/t/ credential. As in Java's S3FileIO.deletePrefix, only that storage deletes, even when a credential prefix is nested below the path. delete_stream flushes batches of 1000 paths per prefix, and the resolving storage now flushes 1000 paths per scheme instead of collecting the whole stream first, with or without credentials.
  • register_table now also applies the response config and user props, which it ignored before.
  • The table Transaction::commit returns still has no vended credentials, since CommitTableResponse carries none. I'll fix that in a follow-up so this PR stays within storage.
  • Out of scope, as tracked in Support refreshing vended storage credentials for REST catalog tables #2931: refreshing credentials before they expire, and parsing account-qualified ADLS keys such as adls.sas-token.<account>, which the ADLS backend ignores today whether they come from config or a credential.

@plusplusjiajia
plusplusjiajia force-pushed the feat/rest-vended-credentials branch from ac46430 to 894d5cf Compare June 15, 2026 12:01

@xanderbailey xanderbailey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, I had a quick look and left some comments / questions.

Comment thread crates/catalog/rest/src/catalog.rs Outdated
.client
.request(Method::GET, context.config.table_endpoint(table_ident))
// Opt in to vended storage credentials.
.header("X-Iceberg-Access-Delegation", "vended-credentials")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Java will read these headers off the catalog config, should we do the same?

header.X-Iceberg-Access-Delegation

Link to Java

@plusplusjiajia plusplusjiajia Jun 15, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xanderbailey Thanks for the catch. Reworked to match the Java client: dropped the dedicated prop and rely on the existing generic header.* config (extra_headers). Vended credentials are opt-in via header.X-Iceberg-Access-Delegation=vended-credentials, applied to all requests like Java's RESTUtil.configHeaders — no hardcoded default.

Comment thread crates/catalog/rest/src/catalog.rs Outdated
let file_io = self
.load_file_io(Some(&response.metadata_location), None)
.await?;
.load_table(commit.identifier())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A full load table seems heavy to me. Can we not wire in the pre-credentialed FileIO into this method somehow?

@plusplusjiajia plusplusjiajia Jun 15, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xanderbailey Good point — update_table no longer reloads. It builds FileIO from the commit response, and the transaction layer reuses the credentialed FileIO it already holds (do_commit + new Table::with_file_io). Back to one GET per commit.

Comment thread crates/catalog/rest/src/catalog.rs Outdated
}

/// FileIO props: server `config`, then vended `storage_credentials` (longest prefix wins), then user props.
fn table_file_io_config(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also notice that Java constructs clientByPrefix Link which keeps credentials separate per prefix. So we can't support multi-location tables here I think? Also it would silently fail in those cases which I don't think is ideal? WDYT?

@plusplusjiajia plusplusjiajia Jun 15, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xanderbailey Thanks, agreed. Each vended credential now maps to its own storage instead of merging all into one props map. FileIO routes each path to the longest-matching prefix(with_prefixed_props/get_storage); removed table_file_io_config.
On silent failures: this matches Java's S3FileIO.clientForStoragePath — a path matching no credential prefix falls back to the default storage from base properties (longest-prefix-match), so a real access problem surfaces as a normal I/O error, not a silent no-op.

@plusplusjiajia
plusplusjiajia force-pushed the feat/rest-vended-credentials branch from 61ed31d to 9591c53 Compare June 16, 2026 02:04
@plusplusjiajia plusplusjiajia changed the title feat(rest): use vended storage credentials from LoadTable response feat(rest): support vended storage credentials with per-prefix FileIO Jun 16, 2026
@plusplusjiajia
plusplusjiajia force-pushed the feat/rest-vended-credentials branch from 9591c53 to d5720b7 Compare June 16, 2026 02:33
huan233usc added a commit to huan233usc/iceberg-rust that referenced this pull request Jun 16, 2026
…redentials apache#2651)

Client implementation of the REST scan-planning protocol, reworked to build on
the vended-credential support from apache#2651 instead of duplicating it.

- Reuses `FileIOBuilder::with_prefixed_props` (from apache#2651) to build a plan-scoped
  FileIO from the `storage-credentials` the server vends in the plan response;
  no separate credential-resolution machinery.
- Narrow `ScanPlanner` capability trait + `Table` handle; `plan_files()`
  delegates and falls back to native planning on `FeatureUnsupported`.
- Endpoint negotiation, submit/poll/fetch state machine, wire→FileScanTask
  conversion. DataFusion unchanged.

Stacked on apache#2651 (vended storage credentials); will rebase to drop those commits
once it merges.
Comment thread crates/iceberg/src/io/file_io.rs Outdated
}

// Otherwise group paths by routed storage, then batch-delete each group.
let mut groups: HashMap<String, Vec<String>> = HashMap::new();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This collects the entire stream before deleting. Since delete_stream is used by purge/cleanup paths, large tables can make this unbounded in memory.

Can we keep routing per prefix but flush bounded batches as we iterate, instead of materializing all paths first?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@huan233usc Good catch, thanks. Reworked to flush bounded per-prefix batches as we iterate instead of materializing everything — memory's now bounded regardless of stream size, mirroring Java's S3FileIO.deleteFiles. Added a test crossing the batch boundary.

Comment thread crates/catalog/rest/src/catalog.rs Outdated
for cred in creds {
let mut prefixed = props.clone();
prefixed.extend(cred.config.clone());
builder = builder.with_prefixed_props(cred.prefix.clone(), prefixed);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qq: do we plan to handle credential refresh as follow up?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@huan233usc Good question!— not in this PR. FileIO is built once from the LoadTable creds, so no refresh yet. Good follow-up: the opendal S3 backend already has a ProvideCredential hook (reqsign re-invokes on expires_in), so a provider re-fetching from loadCredentials before s3.session-token-expires-at-ms — like Java's VendedCredentialsProvider — fits there. Will track as a separate PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey @huan233usc @plusplusjiajia, cred refresh is pretty important for a workflow I'm working on, so I've created a PR for it here: #2932

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zakariya-s Great to see this — credential refresh is the natural next step on top of the vended credentials here.

@plusplusjiajia
plusplusjiajia force-pushed the feat/rest-vended-credentials branch from 00a51e4 to 47748e3 Compare June 17, 2026 09:46
Comment thread crates/catalog/rest/src/catalog.rs Outdated

let file_io = self.load_file_io(Some(metadata_location), None).await?;
let file_io = self
.load_file_io(Some(metadata_location), None, None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth check if e2e for register table requires plumbing credentials. But it could be a separate pr if needed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@huan233usc Checked — register_table builds its FileIO without vended credentials, but that matches the spec: the register response doesn't carry storage-credentials. Callers needing them can load_table afterwards. No change needed as far as I can tell — happy to revisit if you had a specific case in mind.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking!

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction: I was wrong here. The register response is a LoadTableResult, so it can carry storage-credentials. register_table now passes them, along with the response config, to the table's FileIO.

@huan233usc huan233usc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

@huan233usc

Copy link
Copy Markdown
Contributor

Hi @kevinjqliu / @CTTY can I get some help from committers for reviewing this PR. The functionality introduced is quite important iiuc. Thanks

@plusplusjiajia

Copy link
Copy Markdown
Member Author

@CTTY Could you take a look? Complements #2660 (#2660) (control plane there, storage credentials here); they're independent.

@plusplusjiajia

Copy link
Copy Markdown
Member Author

@blackmwk @CTTY could one of you take another look? It also blocks @zakariya-s's credential-refresh series (#2976 (#2976), 1 of 6), which builds on the same FileIO routing.

// the transaction layer reuses the credentialed one it already holds.
let file_io = self
.load_file_io(Some(&response.metadata_location), None)
.load_file_io(Some(&response.metadata_location), None, None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder what the risks are here of not creating FileIo here without the credentials? Is that idea that after we have credential refresh we will just automatically refresh on the next read / write?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xanderbailey Good question. No risk on the Transaction::commit path: do_commit loads before committing and swaps that load's credentialed FileIO back in afterwards (with_file_io), reloading on SetLocation. Refresh isn't needed here. Only a direct Catalog::update_table caller sees the plain FileIO, and the commit response has no credentials to give it by spec — I've pointed the comment at do_commit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in Java they have TableOperations which carries the FileIo rather than having to build a new one after the update 🤔 . There are advantages to the way our commit path works so not saying we should look to move in that direction. Just trying to work out how we could make this more bullet-proof.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xanderbailey Good comparison — I think we're actually closer to it than it looks. UpdateTableRequest in Java doesn't carry a FileIO either; it stays on TableOperations and commit() leaves it alone. do_commit keeping the table's FileIO across update_table is the same idea, just spelled out because we build a fresh Table rather than mutating ops in place.

On bullet-proofing: TableCommit::build() is pub(crate), so do_commit ends up being the only caller update_table can have. I've added a note to the comment so that's easier to see

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: I've since taken the Transaction change out of this PR to keep it within storage, so the table Transaction::commit returns doesn't carry the vended credentials yet. I'll fix that in a follow-up.

@plusplusjiajia
plusplusjiajia force-pushed the feat/rest-vended-credentials branch from f96ace6 to 33cb651 Compare September 2, 2026 07:03
@plusplusjiajia
plusplusjiajia force-pushed the feat/rest-vended-credentials branch from 33cb651 to d5cfdcf Compare September 10, 2026 06:26
@plusplusjiajia
plusplusjiajia force-pushed the feat/rest-vended-credentials branch from d5cfdcf to 22f528d Compare September 20, 2026 18:37
@plusplusjiajia
plusplusjiajia force-pushed the feat/rest-vended-credentials branch from 9dd9d5d to 309792d Compare November 16, 2026 01:51
plusplusjiajia and others added 27 commits November 19, 2026 07:59
@plusplusjiajia
plusplusjiajia force-pushed the feat/rest-vended-credentials branch from 309792d to 5cb1bc6 Compare November 19, 2026 08:00

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants