Repository navigation
feat(rest): support vended storage credentials with per-prefix FileIO - #2651
plusplusjiajia wants to merge 27 commits into
Conversation
ac46430 to
894d5cf
Compare
xanderbailey
left a comment
There was a problem hiding this comment.
Thanks for the PR, I had a quick look and left some comments / questions.
| .client | ||
| .request(Method::GET, context.config.table_endpoint(table_ident)) | ||
| // Opt in to vended storage credentials. | ||
| .header("X-Iceberg-Access-Delegation", "vended-credentials") |
There was a problem hiding this comment.
Java will read these headers off the catalog config, should we do the same?
header.X-Iceberg-Access-Delegation
There was a problem hiding this comment.
@xanderbailey Thanks for the catch. Reworked to match the Java client: dropped the dedicated prop and rely on the existing generic header.* config (extra_headers). Vended credentials are opt-in via header.X-Iceberg-Access-Delegation=vended-credentials, applied to all requests like Java's RESTUtil.configHeaders — no hardcoded default.
| let file_io = self | ||
| .load_file_io(Some(&response.metadata_location), None) | ||
| .await?; | ||
| .load_table(commit.identifier()) |
There was a problem hiding this comment.
A full load table seems heavy to me. Can we not wire in the pre-credentialed FileIO into this method somehow?
There was a problem hiding this comment.
@xanderbailey Good point — update_table no longer reloads. It builds FileIO from the commit response, and the transaction layer reuses the credentialed FileIO it already holds (do_commit + new Table::with_file_io). Back to one GET per commit.
| } | ||
|
|
||
| /// FileIO props: server `config`, then vended `storage_credentials` (longest prefix wins), then user props. | ||
| fn table_file_io_config( |
There was a problem hiding this comment.
I also notice that Java constructs clientByPrefix Link which keeps credentials separate per prefix. So we can't support multi-location tables here I think? Also it would silently fail in those cases which I don't think is ideal? WDYT?
There was a problem hiding this comment.
@xanderbailey Thanks, agreed. Each vended credential now maps to its own storage instead of merging all into one props map. FileIO routes each path to the longest-matching prefix(with_prefixed_props/get_storage); removed table_file_io_config.
On silent failures: this matches Java's S3FileIO.clientForStoragePath — a path matching no credential prefix falls back to the default storage from base properties (longest-prefix-match), so a real access problem surfaces as a normal I/O error, not a silent no-op.
61ed31d to
9591c53
Compare
9591c53 to
d5720b7
Compare
…redentials apache#2651) Client implementation of the REST scan-planning protocol, reworked to build on the vended-credential support from apache#2651 instead of duplicating it. - Reuses `FileIOBuilder::with_prefixed_props` (from apache#2651) to build a plan-scoped FileIO from the `storage-credentials` the server vends in the plan response; no separate credential-resolution machinery. - Narrow `ScanPlanner` capability trait + `Table` handle; `plan_files()` delegates and falls back to native planning on `FeatureUnsupported`. - Endpoint negotiation, submit/poll/fetch state machine, wire→FileScanTask conversion. DataFusion unchanged. Stacked on apache#2651 (vended storage credentials); will rebase to drop those commits once it merges.
| } | ||
|
|
||
| // Otherwise group paths by routed storage, then batch-delete each group. | ||
| let mut groups: HashMap<String, Vec<String>> = HashMap::new(); |
There was a problem hiding this comment.
This collects the entire stream before deleting. Since delete_stream is used by purge/cleanup paths, large tables can make this unbounded in memory.
Can we keep routing per prefix but flush bounded batches as we iterate, instead of materializing all paths first?
There was a problem hiding this comment.
@huan233usc Good catch, thanks. Reworked to flush bounded per-prefix batches as we iterate instead of materializing everything — memory's now bounded regardless of stream size, mirroring Java's S3FileIO.deleteFiles. Added a test crossing the batch boundary.
| for cred in creds { | ||
| let mut prefixed = props.clone(); | ||
| prefixed.extend(cred.config.clone()); | ||
| builder = builder.with_prefixed_props(cred.prefix.clone(), prefixed); |
There was a problem hiding this comment.
qq: do we plan to handle credential refresh as follow up?
There was a problem hiding this comment.
@huan233usc Good question!— not in this PR. FileIO is built once from the LoadTable creds, so no refresh yet. Good follow-up: the opendal S3 backend already has a ProvideCredential hook (reqsign re-invokes on expires_in), so a provider re-fetching from loadCredentials before s3.session-token-expires-at-ms — like Java's VendedCredentialsProvider — fits there. Will track as a separate PR.
There was a problem hiding this comment.
hey @huan233usc @plusplusjiajia, cred refresh is pretty important for a workflow I'm working on, so I've created a PR for it here: #2932
There was a problem hiding this comment.
@zakariya-s Great to see this — credential refresh is the natural next step on top of the vended credentials here.
00a51e4 to
47748e3
Compare
|
|
||
| let file_io = self.load_file_io(Some(metadata_location), None).await?; | ||
| let file_io = self | ||
| .load_file_io(Some(metadata_location), None, None) |
There was a problem hiding this comment.
Worth check if e2e for register table requires plumbing credentials. But it could be a separate pr if needed.
There was a problem hiding this comment.
@huan233usc Checked — register_table builds its FileIO without vended credentials, but that matches the spec: the register response doesn't carry storage-credentials. Callers needing them can load_table afterwards. No change needed as far as I can tell — happy to revisit if you had a specific case in mind.
There was a problem hiding this comment.
Correction: I was wrong here. The register response is a LoadTableResult, so it can carry storage-credentials. register_table now passes them, along with the response config, to the table's FileIO.
090e7c8 to
ef3e2bf
Compare
ef3e2bf to
da370e8
Compare
|
Hi @kevinjqliu / @CTTY can I get some help from committers for reviewing this PR. The functionality introduced is quite important iiuc. Thanks |
da370e8 to
5ba6994
Compare
1379d0e to
b970488
Compare
|
@blackmwk @CTTY could one of you take another look? It also blocks @zakariya-s's credential-refresh series (#2976 (#2976), 1 of 6), which builds on the same FileIO routing. |
| // the transaction layer reuses the credentialed one it already holds. | ||
| let file_io = self | ||
| .load_file_io(Some(&response.metadata_location), None) | ||
| .load_file_io(Some(&response.metadata_location), None, None) |
There was a problem hiding this comment.
Wonder what the risks are here of not creating FileIo here without the credentials? Is that idea that after we have credential refresh we will just automatically refresh on the next read / write?
There was a problem hiding this comment.
@xanderbailey Good question. No risk on the Transaction::commit path: do_commit loads before committing and swaps that load's credentialed FileIO back in afterwards (with_file_io), reloading on SetLocation. Refresh isn't needed here. Only a direct Catalog::update_table caller sees the plain FileIO, and the commit response has no credentials to give it by spec — I've pointed the comment at do_commit.
There was a problem hiding this comment.
I guess in Java they have TableOperations which carries the FileIo rather than having to build a new one after the update 🤔 . There are advantages to the way our commit path works so not saying we should look to move in that direction. Just trying to work out how we could make this more bullet-proof.
There was a problem hiding this comment.
@xanderbailey Good comparison — I think we're actually closer to it than it looks. UpdateTableRequest in Java doesn't carry a FileIO either; it stays on TableOperations and commit() leaves it alone. do_commit keeping the table's FileIO across update_table is the same idea, just spelled out because we build a fresh Table rather than mutating ops in place.
On bullet-proofing: TableCommit::build() is pub(crate), so do_commit ends up being the only caller update_table can have. I've added a note to the comment so that's easier to see
There was a problem hiding this comment.
Update: I've since taken the Transaction change out of this PR to keep it within storage, so the table Transaction::commit returns doesn't carry the vended credentials yet. I'll fix that in a follow-up.
f96ace6 to
33cb651
Compare
33cb651 to
d5cfdcf
Compare
d5cfdcf to
22f528d
Compare
9dd9d5d to
309792d
Compare
…actories to apply
…Credential in REST
… validate credential prefixes
309792d to
5cb1bc6
Compare
Which issue does this PR close?
What changes are included in this PR?
LoadTableResult.storage_credentialsis deserialized but never used, although the spec says clients should prefer it overconfig. A catalog that vends credentials only there leaves the table without storage access.As @blackmwk suggested, the credentials travel in
StorageConfigand the storage factory applies them when it builds the storage:StorageConfigholds a list ofStorageCredential { prefix, config }, set throughStorageConfig::builder()and passed toFileIOBuilder::with_config, and serialized with the rest of the config.FileIOis unchanged.S3FileIO.clientForStoragePath, and to the props-only storage otherwise.load_table,create_tableandregister_tablepass the response's credentials. Delegation stays opt-in through the existingheader.*config (header.X-Iceberg-Access-Delegation=vended-credentials), as in Java'sRESTUtil.configHeaders.Not obvious from the diff:
StorageCredentialis removed, andLoadTableResult.storage_credentialsnow holdsiceberg::io::StorageCredential, which reads the same JSON but exposesprefix()andconfig()instead of public fields.StorageCredential::newrejects an empty prefix, and so does deserializing one, so aloadTableresponse with an empty prefix fails to parse, as Java'sCredentialvalidation does.StorageCredentialmatches the one in feat(io)!: [1/N] support refreshable storage credentials for S3 and GCS #2976 except thatnewreturns aResulthere, so whichever PR lands second drops its copy and adapts thenewcalls.StorageConfigkeeps the later of two credentials that share a prefix, as Java'sclientByPrefixdoes.delete_prefixroutes the path as a directory, sos3://b/tands3://b/t/both reach ans3://b/t/credential. As in Java'sS3FileIO.deletePrefix, only that storage deletes, even when a credential prefix is nested below the path.delete_streamflushes batches of 1000 paths per prefix, and the resolving storage now flushes 1000 paths per scheme instead of collecting the whole stream first, with or without credentials.register_tablenow also applies the responseconfigand user props, which it ignored before.Transaction::commitreturns still has no vended credentials, sinceCommitTableResponsecarries none. I'll fix that in a follow-up so this PR stays within storage.adls.sas-token.<account>, which the ADLS backend ignores today whether they come fromconfigor a credential.