Repository navigation
feat(rest): Support refreshing vended storage credentials - #2932
zakariya-s wants to merge 20 commits into
Conversation
| /// Disable header redaction in error logs (defaults to false for security) | ||
| pub const REST_CATALOG_PROP_DISABLE_HEADER_REDACTION: &str = "disable-header-redaction"; | ||
| /// Identifier for a server-side scan plan associated with credential requests. | ||
| pub const REST_CATALOG_PROP_SCAN_PLAN_ID: &str = "rest.scan.plan-id"; |
There was a problem hiding this comment.
Server-side planning isn't supported yet so this will always be empty anyway. I don't think it's a problem to keep it until it eventually is supported
| let config = response | ||
| .config | ||
| .into_iter() | ||
| .chain(self.user_config.props.clone()) | ||
| .collect(); | ||
| let file_io = self | ||
| .load_file_io(Some(metadata_location), Some(config)) | ||
| .await?; |
There was a problem hiding this comment.
Drive-by fix since this is inconsistent with create_table() and load_table()
| if let Some(no_auth) = m.remove(GCS_NO_AUTH) | ||
| && is_truthy(no_auth.to_lowercase().as_str()) | ||
| { |
There was a problem hiding this comment.
Drive-by fix since this looked pretty bad. AWS did this correctly, but GCS would disable this even if gcs.no-auth was set to true
| let url = url::Url::parse(path).map_err(|e| { | ||
| Error::new( | ||
| ErrorKind::DataInvalid, | ||
| format!("Invalid gcs url: {path}: {e}"), | ||
| ) | ||
| })?; |
There was a problem hiding this comment.
S3 had this validation before above but GCS didn't, so another drive-by fix
| /// `reqsign` [`Timestamp`](reqsign_core::time::Timestamp) used on backend | ||
| /// credential types (e.g. `AwsCredential::expires_in`, `google::Token::expires_at`). | ||
| #[cfg(any(feature = "opendal-s3", feature = "opendal-gcs"))] | ||
| pub(crate) fn system_time_to_timestamp( |
There was a problem hiding this comment.
Didn't want to rely on directly on reqsign's Timestamp, but I'm happy to hear different opinions
| /// It contains the location schemes it backs, the property keys it is configured | ||
| /// with, and how to parse its credential. The generic provider stays free of | ||
| /// any per-cloud knowledge. | ||
| struct CloudRefresh { |
There was a problem hiding this comment.
This could also be made into a trait I guess, and we could split aws and gcp support into different modules, but I thought it was small enough to keep it as a struct and make consts for the different cloud providers
|
Hi @mbutrovich! Would it be possible to get a first-round review of this PR when you have time please? |
Yep, it's in my queue! Just slammed with review requests :( Thanks for your patience! |
mbutrovich
left a comment
There was a problem hiding this comment.
First pass, thanks for tackling this @zakariya-s. I have specific feedback, and also some ideas how we might break this up for other reviewers since this is a lot to review in one pass. A split along the layers already present in the diff looks fairly clean and, other than one ordering constraint, each piece is independently testable rather than a bare stub:
- The
Debug-redaction hardening (RestCatalogConfig,HttpClient,StorageConfig,LoadTableResult,StorageCredential,OpenDalResolvingStorage, plus theis_sensitive_headerbroadening) and the two small unrelated fixes riding along (theregister_tableconfig merge, the GCSno-authtruthy parsing). None of this depends on credential refresh existing, and it already has its own tests. - The
StorageCredentialProvidertrait and credential types in theicebergcrate, plusStorageFactory::build_with_credentialswith its safe default (errors if a provider is supplied and the factory has not opted in). Covered by finding 5. No behavior change for existing backends. - OpenDAL S3 consuming the trait (the adapter, the anonymous-access guard, the
delete_streamchange). Testable on its own with a hand-rolled provider, same as the tests already in this diff, without needing anything from the REST side. - OpenDAL GCS consuming the trait, same shape as 3, independent of it.
Finding 1 (lost delete batching) sits in the shared uses_dynamic_credentials/delete_stream code in lib.rs rather than in s3.rs or gcs.rs specifically, so it isn't purely a 3-or-4 problem: whichever of the two lands first introduces that shared mechanism, and the other reuses it as-is, so the fix only needs to happen once.
- The REST catalog fetch/cache/jitter/backoff logic (
RestVendedCredentialProvider,CloudRefresh,resolve_endpoint, the table-scoped clientfor_table). Covered by findings 2, 3, 4, and the root cause of 6 (the fresh-HttpClient-per-table-scoped-provider behavior lives inclient.rs/credential.rs, both part of this PR). This can be written and tested against the trait directly withmockito, same as the tests already in this diff, without touching OpenDAL at all. - Wiring
RestCatalog::load_file_ioto actually attach the provider (catalog.rs:561-565). Covered by finding 7 (the inline path at that call site) and the rest of finding 6 (this is the call site that turns "everyload_file_iocall re-runs the OAuth handshake" from a latent property offor_tableinto something that fires on everyload_table/create_table/register_table).
This piece should land last on purpose, not just for tidiness: client.refresh-credentials-endpoint and gcs.oauth2.refresh-credentials-enabled are property keys the Java client and the REST spec already define, so a production catalog that's Java-interoperable may already be sending them today regardless of which client is asking. If the wiring lands before both the S3 and GCS sides can consume a provider, every existing rust client hitting such a catalog would start failing table loads on S3 or GCS the moment that PR merges, since the default build_with_credentials errors whenever a provider is supplied to a factory that hasn't opted in. So 3 and 4 both need to land before 6, even though 3, 4, and 5 are otherwise independent of each other and can be reviewed in any order.
#2931 is currently a single feature request rather than a tracking issue for a multi-PR stack. Worth turning it into one, or opening a separate tracking issue with a checklist for the pieces above, so reviewers can see the whole plan and where a given PR sits in it before reviewing any single piece.
| fn uses_dynamic_credentials(&self, path: &str) -> bool { | ||
| match self { | ||
| #[cfg(feature = "opendal-s3")] | ||
| OpenDalStorage::S3 { | ||
| credential_provider: Some(provider), | ||
| .. | ||
| } => provider.supports_path(path), | ||
| #[cfg(feature = "opendal-gcs")] | ||
| OpenDalStorage::Gcs { |
There was a problem hiding this comment.
crates/storage/opendal/src/lib.rs:423-431 (uses_dynamic_credentials) and :610-628 (delete_stream).
When a path is served by a credential provider, delete_stream skips the shared per-bucket Deleter and instead calls create_operator + a single op.delete(relative_path) per path, sequentially, inside the stream loop. The non-dynamic branch batches deletes through OpenDAL's Deleter (which can use bulk delete APIs); the dynamic branch does neither batching nor concurrency, and rebuilds the operator from scratch for every single file.
For expire_snapshots/purge on a table with vended-credential refresh enabled, this turns what would be a handful of batched multi-object delete calls into one HTTP round trip per file, plus an operator-construction cost per file. The code comment explains why deletes can't share a Deleter across different credential-prefix scopes (correctness: batch_key_for_path only groups by bucket, not by credential scope), but the fix taken forfeits batching entirely rather than partially, i.e. grouping deletes by (bucket, matched credential prefix) instead of just bucket would preserve batched delete within each credential-scope group. Was that considered?
There was a problem hiding this comment.
Yeah I left this in the initial commit for sake of simplicity, we now batch by DeleteBatchKey which includes the credential scope so batching behaviour is now restored
| let refreshed = self.fetch(configured).await.and_then(|entries| { | ||
| let credential = longest_prefix_match(&entries, path) | ||
| .filter(|entry| entry.is_unexpired(SystemTime::now())) | ||
| .map(|entry| entry.credential.clone()) | ||
| .ok_or_else(|| { | ||
| Error::new( | ||
| ErrorKind::Unexpected, | ||
| format!("no unexpired vended credential matches storage location: {path}"), | ||
| ) | ||
| })?; | ||
| Ok((entries, credential)) | ||
| }); | ||
|
|
||
| match refreshed { | ||
| Ok((entries, credential)) => { | ||
| let mut cache = configured.cache.lock().await; | ||
| cache.entries = entries; | ||
| cache.consecutive_failures = 0; | ||
| cache.retry_not_before = None; | ||
| Ok(credential) | ||
| } | ||
| Err(fetch_error) => { | ||
| let mut cache = configured.cache.lock().await; | ||
| cache.consecutive_failures = cache.consecutive_failures.saturating_add(1); | ||
| cache.retry_not_before = | ||
| Instant::now().checked_add(failure_backoff(cache.consecutive_failures)); | ||
|
|
||
| // Graceful degradation: while the cached credential remains | ||
| // usable, serve it and retry after jittered backoff. Expired | ||
| // credentials are never served. | ||
| fallback | ||
| .filter(|entry| entry.is_unexpired(SystemTime::now())) | ||
| .map(|entry| entry.credential) | ||
| .ok_or(fetch_error) | ||
| } | ||
| } |
There was a problem hiding this comment.
crates/catalog/rest/src/credential.rs:265-300 (refresh_credential), contrast with S3FileIO.refreshStorageCredentials()/GCSFileIO.refreshStorageCredentials() in Java.
Java's actual multi-prefix refresh (used by both S3FileIO and GCSFileIO, not the single-credential VendedCredentialsProvider/OAuth2RefreshCredentialsHandler used as an SDK credentials provider) is unconditional: on each scheduled refresh it fetches the credentials endpoint once, keeps every entry matching the cloud's root prefix, and replaces storageCredentials wholesale — no per-path filtering happens at refresh time at all.
The Rust refresh_credential instead does per-path filtering inline: it fetches all entries for a cloud, then immediately narrows to longest_prefix_match(&entries, path).filter(unexpired) for this specific call's path. If that narrowing yields nothing (no entry covers this path, or the covering entry happens to already be expired), the whole outcome is treated as Err and:
- the freshly-fetched
entriesare never written tocache.entries(only theOk((entries, credential))branch at line 279-284 updates the cache) — so if the response contained entries for other prefixes (as the existinglongest_prefix_match_ignores_freshnesstest exercises), they're thrown away even though a subsequent call for a different, valid path would have to fetch them all over again; cache.consecutive_failuresis incremented andretry_not_beforebackoff is armed (lines 288-290) even though the catalog responded successfully — it just didn't vend anything for this path.
Given Java's model of "cache everything the endpoint returns, unconditionally," was per-path filtering at refresh time (rather than only at cache-read time, where it already happens in cache_decision/longest_prefix_match) intentional here?
There was a problem hiding this comment.
Good point that I missed, now all results are cached (minus the malformed or expired ones) and then it's filtered
| parsed | ||
| .storage_credentials | ||
| .into_iter() | ||
| .filter(|sc| configured.cloud.matches_location(&sc.prefix)) | ||
| .map(|sc| { | ||
| (configured.cloud.parse_credential)(&sc.config, Some(sc.prefix)).map( | ||
| |credential| { | ||
| CachedEntry::new(credential, configured.cloud.jitter_prefetch) | ||
| }, | ||
| ) | ||
| }) | ||
| .collect() |
There was a problem hiding this comment.
collect() into Result<Vec<CachedEntry>> short-circuits on the first parse_credential error. If a server vends N credentials for one cloud and one entry is missing a required field, all N become unusable (and, per finding 2, this also counts as a "failure" for backoff purposes) rather than just the one bad entry. Java's equivalent (VendedCredentialsProvider.refreshCredential) only ever expects a single S3-prefixed entry and asserts on it directly, so there's no directly analogous "partial batch" behavior to compare against — but given this PR's own design supports N entries per cloud, is one bad entry meant to invalidate all the others?
There was a problem hiding this comment.
fetch() now returns a ParsedCredentials which splits between the successful creds and errors
| let enabled = props | ||
| .get(cloud.enabled_key) | ||
| .is_none_or(|value| value.parse().unwrap_or(false)); |
There was a problem hiding this comment.
str::parse::<bool>() only accepts the exact strings "true"/"false". Java's PropertyUtil.propertyAsBoolean uses Boolean.parseBoolean, which is case-insensitive for "true" ("True", "TRUE" all parse as true). A config value of client.refresh-credentials-enabled: "True" would enable refresh in Java but silently disable it in Rust (parse error → unwrap_or(false)). Low severity (fails closed either way), but worth a case-insensitive comparison to match Java's actual accepted input space.
| pub struct StorageCredential { | ||
| /// Storage-location prefix this credential is scoped to. `None` represents a | ||
| /// credential without a declared scope, sourced from flat storage properties. | ||
| pub prefix: Option<String>, | ||
| /// The backend-specific credential material. | ||
| pub kind: StorageCredentialKind, | ||
| /// When the credential expires, if known. `None` means non-expiring and | ||
| /// backends treat such a credential as always valid and never refresh it. | ||
| pub expires_at: Option<SystemTime>, | ||
| } | ||
|
|
||
| /// Backend-specific credential material. | ||
| #[derive(Clone, Debug)] | ||
| pub enum StorageCredentialKind { | ||
| /// Amazon S3 credentials. | ||
| S3(S3Credential), | ||
| /// Google Cloud Storage credentials. | ||
| Gcs(GcsCredential), | ||
| } | ||
|
|
||
| /// Temporary Amazon S3 credentials. | ||
| #[derive(Clone)] | ||
| pub struct S3Credential { | ||
| /// AWS access key ID. | ||
| pub access_key_id: String, | ||
| /// AWS secret access key. | ||
| pub secret_access_key: String, | ||
| /// AWS session token, set for temporary (STS/vended) credentials. | ||
| pub session_token: Option<String>, | ||
| } | ||
|
|
||
| impl Debug for S3Credential { | ||
| fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { | ||
| f.debug_struct("S3Credential").finish_non_exhaustive() | ||
| } | ||
| } | ||
|
|
||
| /// Temporary Google Cloud Storage credentials (an OAuth2 access token). | ||
| #[derive(Clone)] | ||
| pub struct GcsCredential { | ||
| /// OAuth2 bearer token used to access GCS. | ||
| pub token: String, | ||
| } |
There was a problem hiding this comment.
These are new public types (in public-api.txt) that third-party StorageCredentialProvider implementors must construct by hand. Every field is pub, with no constructor. That's inconsistent with StorageConfig in the same module (crates/iceberg/src/io/storage/config/mod.rs:55-58), which keeps props private and exposes with_prop/from_props instead. Was a constructor considered, or is direct struct-literal construction intentional here?
There was a problem hiding this comment.
Sorry this wasn't intentional, they're no longer public
…ential-refresh # Conflicts: # crates/storage/opendal/src/gcs.rs
|
Thanks @mbutrovich for the first review! I've made some changes to address your comments and pushed them to this PR for posterity. I will follow your suggestion and split this PR and use a tracker issue. |
|
First PR available here: #2976 |
Adapt table-scoped credential refresh to AuthManager and AuthSession.
|
Hey @CTTY, here's the full PR to see the broader picture. Happy if you wish to review the full PR if it's easier compared to splitting it. |
|
apache/opendal#8030 has been included as part of opendal-core 0.59.0, so Azure support is now possible. Unfortunately, 0.59.0 doesn't compile due to a packaging issue so we will have to wait for a patch to upgrade the dependency. |
|
@CTTY this now includes Azure support too, but as I mentioned before OpenDAL's latest release is broken so checks are failing. Hopefully 0.59.1 should release soon. |
Update OpenDAL to 0.59.1 now that the fixed release is available.
|
0.59.1 was released, so Azure support should now be complete. This PR should be fully ready for review |
|
Hey @mbutrovich @CTTY could I get a review here please? Thanks |
Rebuild the refresh provider after FileIO serialization and fix several correctness issues found in review. - Serialize the credential provider as a StorageCredentialProviderFactory that rebuilds it from the FileIO properties and connects to the catalog lazily, like Java's VendedCredentialsProvider. Catalogs with an injected AuthManager refuse to serialize; FileIO::without_credential_provider drops the provider instead. - Match credential prefixes on whole path segments and treat scheme aliases (s3a/s3n, gcs, abfs/wasb) as equal, via StorageCredential::covers. - Merge refreshed credentials prefix by prefix, keeping an unexpired cached credential when the response has no valid replacement. - Look up bulk-delete credentials by the batch scope, so a refresh during a delete can no longer abort it, and share one reqsign adapter across S3, GCS and ADLS. - Cap the refresh buffer at half the credential lifetime, so short-lived credentials are not re-fetched on every operation. - Ignore credential providers in storage factories that cannot use them instead of failing every operation, matching main and Java. - Accept ADLS SAS tokens keyed by host (adls.sas-token.<host>) as Java and Polaris send them, and accept explicit http endpoints for abfs/wasb. - Move auth manager selection to auth::load_auth_manager and auth property helpers onto RestCatalogConfig. - Mark StorageCredentialKind and OpenDalStorage #[non_exhaustive].
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks for the updates @zakariya-s, and for turning #2931 into a tracking issue.
Since my last review the PR has grown to about 4,800 added lines across 26 files. It now also includes ADLS, FileIO serialization of providers, a new AuthManager::table_session method, and the OpenDAL 0.58 to 0.59 bump. I don't think a change this size can get through review here as one PR, even with it split into commits. #2976 has also drifted from this branch. As far as I can tell it has no StorageCredentialProviderFactory and no AzdlsCredential. Could we keep #2932 as a draft reference and land the work as a stack of smaller PRs, with the #2931 task list updated to match? Here is one way to cut it along the layers already in the diff:
Debugredaction and the unrelated fixes:StorageConfig,LoadTableResult,StorageCredential,TokenResponse, theregister_tableconfig merge, andgcs.no-authparsing. None of these depend on refresh.- The core API in #2976, updated to match this branch:
StorageCredentialProvider,StorageCredential,StorageCredentialKindwith S3 and GCS,StorageFactory::build_with_credentials,FileIOBuilder::with_credential_provider, and the property constants.StorageCredentialKindis#[non_exhaustive], so the ADLS variant can be added later without a break. - Provider serialization:
StorageCredentialProvider::factory,StorageCredentialProviderFactory, andFileIO::without_credential_provider. This changes whatFileIO::serialize_alldoes, so it deserves its own discussion. - The OpenDAL S3 adapter and the shared pieces it introduces (
VendedCredentialSource,DeleteBatchKey), tested with a hand-written provider like the tests already in this diff. - The OpenDAL GCS adapter.
- The OpenDAL 0.59 bump as its own PR, then the ADLS storage pieces:
AzdlsCredential,AzdlsSasTokens, and the ADLS adapter. - REST auth:
AuthManager::table_session,HttpClient::for_table, and the move of auth-manager selection intoload_auth_manager. This adds a method to a public trait and doesn't touch storage. - The REST provider in
credential.rsfor S3 only: fetch, cache, prefix selection, and backoff, tested against the trait withmockito.credential.rsalone is 2,452 lines (about 1,030 of code and 1,420 of tests), so starting with one cloud keeps this piece reviewable. - GCS and ADLS in the REST provider, including the account-keyed seed strategy that ADLS needs.
- The wiring in
RestSessionCatalog::load_file_io.
Everything depends on 2. Pieces 5 and 6 reuse the shared code from 4, 8 needs 2 and 7, and the wiring goes last. Some neighbors could be combined if a reviewer prefers, but each piece can be tested on its own. The comments below apply to whichever piece they end up in.
| let credential_provider = build_vended_credential_provider( | ||
| &client.http_client, | ||
| client.auth_manager.as_ref(), | ||
| RestVendedCredentialProviderFactory::new( | ||
| &client.config.uri, | ||
| table.clone(), | ||
| table_config.unwrap_or_default(), | ||
| ), | ||
| &props, | ||
| self.auth_manager.is_none(), | ||
| ) |
There was a problem hiding this comment.
load_file_io seeds the provider only from the flat config properties. At the head commit nothing reads LoadTableResult::storage_credentials (grep -rn storage_credentials crates only finds types.rs). The spec says "Clients must first check whether the respective credentials exist in the storage-credentials field before checking the config for credentials" (LoadTableResult). The provider already caches credentials per prefix, so should the initial storage-credentials seed that cache? As I read it, for a catalog that vends only through storage-credentials, the first file access has to fetch from the refresh endpoint. With refresh disabled, the table gets no credentials at all.
#2651 also routes each path to the credential with the longest matching prefix, built from storage-credentials, but it uses a different mechanism (FileIOBuilder::with_prefixed_props). How do you see the two PRs fitting together? It would help to settle on one prefix-routing path before either lands.
There was a problem hiding this comment.
Indeed, this PR depends on #2651 for storage-credentials and won't consume them itself. After that other PR lands, its per-prefix storages will be passed the provider with optional seeding from storage-credentials.
| pub fn storage_prefix_covers(prefix: &str, location: &str) -> bool { | ||
| let Some((location_scheme, location_rest)) = location.split_once("://") else { | ||
| return false; | ||
| }; | ||
| let Some((prefix_scheme, prefix_rest)) = prefix.split_once("://") else { | ||
| return !prefix.is_empty() && canonical_scheme(prefix) == canonical_scheme(location_scheme); | ||
| }; | ||
|
|
||
| canonical_scheme(prefix_scheme) == canonical_scheme(location_scheme) | ||
| && location_rest | ||
| .strip_prefix(prefix_rest) | ||
| .is_some_and(|remainder| { | ||
| prefix_rest.is_empty() | ||
| || prefix_rest.ends_with('/') | ||
| || remainder.is_empty() | ||
| || remainder.starts_with('/') | ||
| }) | ||
| } |
There was a problem hiding this comment.
storage_prefix_covers only matches whole path segments, so a credential for s3://bucket/tab does not cover s3://bucket/table/file.parquet. The spec defines prefix as "a storage location prefix where the credential is relevant" and says clients should select the longest one (StorageCredential). It doesn't say whether a prefix has to end on a segment boundary. Java settles that with a plain string startsWith in S3FileIO.clientForStoragePath, and it doesn't treat s3a and s3 as the same scheme. With this code, a catalog that vends a prefix Java would accept gets a "does not cover" error here. Should this follow Java's matching? If the stricter rule is intentional, could the doc comment say why?
There was a problem hiding this comment.
This was intentional as whilst the spec doesn't mention segment boundaries, I thought it was in the spirit of it. If we want to follow Java's semantics (which is a good enough reason by itself), then I'd be happy to change this.
| Some(match credential.prefix() { | ||
| Some(prefix) => prefix.to_string(), | ||
| None => utils::storage_root(path)?, | ||
| }) |
There was a problem hiding this comment.
When the credential has no prefix, the batch looks up credentials for the storage root (s3://bucket/) for as long as its Deleter lives. What happens with a catalog that returns the initial credential in config, with no prefix, and returns prefix-scoped entries from the refresh endpoint? I tried this against RestVendedCredentialProvider at the head commit. I seeded an unscoped credential that expires in 2 seconds and a refresh endpoint that returns a credential for s3://bucket/table. After the seed expired, load_credential("s3://bucket/table/data/f.parquet") returned the refreshed credential, but load_credential("s3://bucket/") failed with no unexpired vended credential matches storage location: s3://bucket/. The deleters are closed at the end of delete_stream, so a stream that runs past the seed's expiry would fail on flush even though every path in it is covered. Before expiry, each root lookup inside the refresh window also fetches, finds no entry for s3://bucket/, and calls record_failure, which backs off refresh for the whole AWS cache.
One option is to flush the root-keyed deleter as soon as a path in the stream resolves to a prefixed credential, because that means the unscoped credential is being replaced while it is still valid. Could you also add a test for an unscoped seed followed by a scoped refresh?
… headers - Require StorageCredentialProvider::supports_path and rename StorageFactory::build_with_credentials to build_with_credential_provider. - Pass the whole table config to AuthManager::table_session in process; use the table `token` as a static bearer only for portable providers whose auth type was inferred, so an explicit `rest.auth.type=none` ignores it. - Report non-2xx credential and token responses without echoing the body. - Redact every HTTP header value in debug output unless redaction is disabled, matching Java's HTTPHeader. - Warn once per cause when FileIO serialization drops the provider. - GCS: disable the other credential sources when a provider is attached. - Add tests for ADLS SAS signing, GCS no-fallback, single-flight loads without a usable credential, register_table, and table-client redaction.
Resolve conflicts with the configurable OpenDAL IO timeout (apache#3263): storage variants carry both the credential provider and the client config.
laskoviymishka
left a comment
There was a problem hiding this comment.
+1 on keeping this as a draft reference and landing it as the stack @mbutrovich laid out. The comments below should carry over to whichever piece they end up in. The credential.rs tests are a good base to keep through the split, especially the mockito cases for single-flight, prefix masking, and error messages that don't leak secrets.
Before piece 8 goes up, I'd most like to settle how the cache picks a credential and when it gives up on one. Right now an expired narrow prefix quietly falls back to a broader credential that hasn't expired. A path that no prefix covers counts as a refresh failure for the whole cloud and backs off refresh for the paths that are covered. And fetch has no timeout while the single-flight guard is held, so a stalled catalog becomes a hung writer instead of an error. These three interact, so I'd fix them together. The tests need to be able to control time, since with Instant::now()/SystemTime::now() called directly the backoff-recovery paths can't be tested.
Two smaller things are about scope, not correctness. The abfs/wasb switch to https belongs in the ADLS piece with its own changelog line. Piece 2 is the cheap moment to settle the public shape: load_credential returning an owned StorageCredential, and LoadCredentialsResponse being exported even though only the private module uses it.
Port the vended credential provider to contextual sessions (apache#3081): credential requests are authenticated by a table session derived from the contextual session of the operation that loaded the table, and the table-scoped client no longer carries a session. Resolve conflicts with HDFS support (apache#3111) and ADLS relative paths (apache#3311).
Port the changes made while reviewing apache#2976: - `load_credential` returns an `Arc<StorageCredential>`; the REST provider caches shared credentials and warns once per failed refresh. - Add `StorageCredential::select`, `FileIO::has_credential_provider`, and a per-factory-type warning when a provider is ignored. - Test provider propagation, credential reloads, S3 fallback and bulk delete provider calls. - Keep the GCS fixes, refresh constants and ADLS support that apache#2976 leaves to later pieces.
- Select the longest covering credential even when it has expired, and refresh it, instead of serving a broader one. - Back off refresh only after a failed fetch or an invalid credential; a location the catalog does not vend waits briefly on its own. - Time out credential requests after 30 seconds. - Keep initial table credentials without a session token or expiry as seeds: like Java, the catalog is asked first, and they serve while refresh fails. Warn about ones that cannot be used. - Derive the contextual session only when a provider is built. - Keep `LoadCredentialsResponse` crate-private, and inject the clock so tests can control time. - Test ADLS no-fallback and SAS token reloads, sort `Debug` keys, and note this branch's user-visible changes in the changelog.
|
updated based on your comments @laskoviymishka, also pulled in the latest changes from #2976 so that this PR is still accurate |
laskoviymishka
left a comment
There was a problem hiding this comment.
Thanks @zakariya-s, the selection, backoff and timeout fixes all landed the way I expected.
I think we’re at the point where this is too big to keep designing through PR review. #2932 is already ~8.6k lines, and the implementation details are getting closer, but there are still a few pretty fundamental choices underneath it that we haven’t really agreed on.
The only architecture discussion I could find is @CTTY’s question on #2976 about whether we need the provider abstraction at all, and that stopped pretty early. I’d rather settle that kind of thing once before we keep cutting the stack around it.
Maybe the next step should be a small RFC under docs/rfcs/, probably as a follow-up to 0002 since this extends the storage layer. #2932 can stay as the reference implementation while we work through it.
The questions I think need a proper answer are:
- Do we actually want a
StorageCredentialProviderin core, or should refresh live inside eachStorageFactory? - Where does prefix routing belong? #2651 already does longest-prefix routing in
FileIO, while this PR also does it inside the provider. - Who owns refresh/caching? The provider cache and reqsign already have different refresh behavior.
- What do we serialize to workers? Right now that means catalog URI + table token in
FileIO, and workers linking the REST crate. - How is this enabled? If the catalog sends
client.refresh-credentials-endpoint, does that replace the normal backend credential chain automatically, or only when opted in? - How much of the REST auth API do we want to expose publicly, especially
AuthManager::table_sessionon top of #3081?
Most of this is already in pieces across the discussion, so I don’t think writing the RFC should be a huge amount of extra work. It would mostly make the choices explicit and give us one place to agree on them.
I’d keep #2976 open for review, but avoid merging the trait until at least the first few questions are settled. Could we also mark #2932 as draft for now?
|
thanks @laskoviymishka! i'll aim to get an RFC out soon |
Which issue does this PR close?
What changes are included in this PR?
This PR adds support for refreshing short-lived storage credentials vended by REST catalogs:
StorageCredentialProviderinterface toFileIOheader.*properties for refresh requestsDebugoutputAzure credential refresh is not included because the current OpenDAL Azure backend does not expose the credential-provider and expiry hooks required for safe refresh.ADLS refresh is now supported in OpenDAL 0.59.1 via the custom credential-provider.
Are these changes tested?
Yes