Repository navigation
fix(cache): add configurable refresh buffer to StorageCredentialCache - #5129
visit2rahul wants to merge 12 commits into
Conversation
dimas-b
left a comment
There was a problem hiding this comment.
Thanks for this enhancement, @visit2rahul !
| .put( | ||
| StorageAccessProperty.AWS_SESSION_TOKEN_EXPIRES_AT_MS, | ||
| String.valueOf(expiresAt)) | ||
| .put(StorageAccessProperty.EXPIRATION_TIME, String.valueOf(expiresAt)) |
There was a problem hiding this comment.
This will probably need to be reworked after #5070
be5f4a4 to
f06ec75
Compare
|
@dimas-b Morning. Just checking if anything else is needed on this one? |
flyingImer
left a comment
There was a problem hiding this comment.
I'm aligned with making the cache margin operator-configurable, but I don't think this is merge-ready yet. The positive-buffer contract needs to distinguish cache reuse from what getOrLoad may return, and values outside the supported range need deterministic rejection. Once those behaviors and boundary tests align, no other blocking concern from my side.
| realmConfigWithBuffer, | ||
| () -> nearExpiryConfig); | ||
|
|
||
| storageCredentialCache.getOrLoad(key); |
There was a problem hiding this comment.
IIUC, this test only verifies that a near-expiry credential is not retained but it does not verify the documented guarantee that clients receive at least the configured remaining validity.
StorageCredentialCache.getOrLoad directly returns Caffeine LoadingCache.get, and it returns the freshly loaded value even when expireAfterCreate gives it a zero TTL, so the first caller still receives nearExpiryConfig and only later lookups miss.
Could we ensure getOrLoad never returns an under-buffer credential and assert that behavior here? One option is to validate immediately after key.load(); if the intended contract is only to prevent later cache hits, the configuration description should be narrowed instead.
There was a problem hiding this comment.
fixed — it retries once if the fresh load is still too close to expiring, and fails loudly if it still cant clear the buffer after that
| private long refreshBufferMs(RealmConfig realmConfig) { | ||
| return realmConfig.getConfig(FeatureConfiguration.STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS) | ||
| * 1000L; |
There was a problem hiding this comment.
The advertised refresh-buffer range is not enforced. refreshBufferMs accepts any integer: a negative value silently selects the legacy half-life path, while a value at or above the configured credential duration can make every request reload from the provider when credentials are issued at that duration.
Could we reject values the public description declares invalid and define negative values explicitly, rather than silently changing cache behavior? One option is to accept only 0 <= buffer < STORAGE_CREDENTIAL_DURATION_SECONDS before key.load(), matching the adjacent cache-duration constraint, with negative/equal/greater boundary tests.
There was a problem hiding this comment.
yea fixed this too, though it moved since — bad values get caught at server startup now instead of at request time, and just fall back to the default if something's wrong
ayushtkn
left a comment
There was a problem hiding this comment.
Thanx @visit2rahul few suggestions
Missing CHANGELOG.md entry. This is a user-facing configuration flag. Maybe worth adding
Test coverage gaps:
- No test for the happy path: buffer=300, credential expires in 3600s → entry cached with ~3300s effective TTL
- No test that a second getOrLoad triggers reload when the first result was evicted due to buffer (similar to testCacheEvict)
- No boundary validation tests (blocked on implementation)
|
|
||
| ##### `polaris.features."STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS"` | ||
|
|
||
| Minimum remaining validity (in seconds) that a cached storage credential must have before it is considered stale and evicted. When non-zero, credentials are removed from the cache this many seconds before they expire, ensuring clients always receive credentials with at least this much validity left. When zero (the default), credentials are kept for half of their remaining lifetime, which is the legacy behavior. Must be less than STORAGE_CREDENTIAL_DURATION_SECONDS. |
There was a problem hiding this comment.
ensuring clients always receive credentials with at least this much validity lef
But getOrLoad delegates directly to LoadingCache.get():
public StorageAccessConfig getOrLoad(StorageCredentialCacheKey key) {
return cache.get(key).toAccessConfig();
}
When effectiveTtl clamps to 0, Caffeine still returns the freshly loaded entry on that same get() call.
The new test demonstrates this:
storageCredentialCache.getOrLoad(key); // returns cred with only 120s left
Assertions.assertThat(storageCredentialCache.getIfPresent(key)).isNull(); // not cached
So with buffer=300s and a credential expiring in 120s, the first caller still receives a nearly-expired credential — only subsequent cache lookups miss. That contradicts the documented contract.
There was a problem hiding this comment.
all added — changelog entry, happy path test, retry test, boundary tests
3cae09d to
c170ae0
Compare
|
Thanks both — good catch, the first load wasn't actually protected. Fixed |
flyingImer
left a comment
There was a problem hiding this comment.
Thanks for the update. The range check addresses my configuration concern. The retry also covers the case where the second credential is fresh, but I still see one gap before merge: getOrLoad returns the second credential after warning even when it is still below the buffer, while the config description and changelog still promise minimum validity. With a 300s buffer and both loads returning credentials with 120s left, the caller still receives one.
Keeping the retry bounded makes sense. Could we either fail after the retry cannot meet the buffer, or document this as best-effort refresh with that fallback? The repeated-near-expiry test should assert the chosen caller-visible outcome and exactly two loads.
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
orca-review-bot
left a comment
There was a problem hiding this comment.
Nice progress. The range check, happy-path test, and fresh-retry test cover most of the earlier asks, and default 0 keeps today's behavior.
The main open item is still flyingImer's from 2026-09-22: what callers see when the provider can't beat the buffer. Right now we retry once and return the stale credential anyway, while the CHANGELOG and the flag description say "always". Can we pick one (fail, or best effort) and make the docs, code, and tests agree?
Smaller things:
- Can we fold the two rejection tests into a @ParameterizedTest?
- Can we pull out a realmConfigWithBuffer(int) helper?
- Can we add a no-expiresAt + positive-buffer test?
- Can we refresh the PR description? It still says two tests.
Verified fine: the WARN logs no credential material, and the cache key and realm isolation are untouched.
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
| StorageCredentialCacheEntry entry = cache.get(key); | ||
| if (isUnderRefreshBuffer(entry)) { | ||
| cache.invalidate(key); | ||
| entry = cache.get(key); |
There was a problem hiding this comment.
If the provider keeps issuing tokens shorter than the buffer, this reload never helps. Both loads land inside the buffer, the entry gets a TTL of 0, and the next request does it all again. So that setup pays two provider calls plus a WARN on every request, with no caching. Can we either fail here, or return the first credential as best effort and skip the second load (maybe with a rate-limited warning)?
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
| - Python CLI: added `--scheme` to specify URL scheme | ||
| - Added opt-in idempotency for `createTable` and `updateTable` in the Iceberg REST catalog. When enabled via `polaris.idempotency.enabled=true` (default `false`), a client-supplied `Idempotency-Key` header is embedded into the table entity and committed in the same transaction as the operation; a retry carrying the same key within the TTL window (`polaris.idempotency.ttl`, default `PT5M`) replays the original success instead of failing — with `AlreadyExists` for `createTable`, or with `CommitFailedException` for `updateTable` when the request's requirements no longer match the already-advanced table. When idempotency is enabled, the reuse window is advertised to clients through the `idempotency-key-lifetime` field of the `GET /v1/config` response. | ||
| - Python CLI: Added views and generic-tables support. | ||
| - Added the `STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS` feature flag (default `0`). When non-zero, a cached storage credential is evicted this many seconds before it actually expires, guaranteeing callers always receive credentials with at least this much validity left; a freshly loaded credential that is already within the buffer is reloaded once rather than handed back as-is. Must be `>= 0` and less than `STORAGE_CREDENTIAL_DURATION_SECONDS`. When `0` (the default), behavior is unchanged: credentials are kept for half of their remaining lifetime. |
There was a problem hiding this comment.
"guaranteeing callers always receive" isn't what happens. getOrLoad returns an under-buffer credential after one failed retry (same gap flyingImer pointed out). Can we soften this to best effort and mention the fallback?
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
| "Minimum remaining validity (in seconds) that a cached storage credential must " | ||
| + "have before it is considered stale and evicted. When non-zero, credentials " | ||
| + "are removed from the cache this many seconds before they expire, ensuring " | ||
| + "clients always receive credentials with at least this much validity left. " |
There was a problem hiding this comment.
Same as the CHANGELOG: "ensuring clients always receive" overpromises. Can we update it, and also mention that a positive buffer replaces the half-lifetime rule instead of combining with it? Then regenerate flags-polaris_features.md.
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
| realmConfigWithBuffer, | ||
| () -> nearExpiryConfig); | ||
|
|
||
| storageCredentialCache.getOrLoad(key); |
There was a problem hiding this comment.
The comment says the retry happens, but nothing checks it. Can we count loads, assert exactly 2, and assert whatever the caller is supposed to get back?
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
| } | ||
|
|
||
| @Test | ||
| public void testRefreshBufferRejectsNegativeValue() { |
There was a problem hiding this comment.
This and the at/above-duration test differ only in the buffer value. Can we merge them into one @ParameterizedTest (-1, 3600, 4000)?
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
c170ae0 to
0f9e2ae
Compare
|
Went with throwing instead of a fallback — if a provider can't clear its own buffer even after a retry, that feels like something the caller should know about, not quietly swallow. Updated the test to check for the exception and that it's exactly two loads. |
0f9e2ae to
693c872
Compare
dimas-b
left a comment
There was a problem hiding this comment.
Sorry about the delayed review, @visit2rahul , and thanks for the update!
| + "are removed from the cache this many seconds before they expire, ensuring " | ||
| + "clients always receive credentials with at least this much validity left. " | ||
| + "When zero (the default), credentials are kept for half of their remaining " | ||
| + "lifetime, which is the legacy behavior. Must be less than " |
There was a problem hiding this comment.
I do not think the alternative behaviour is "legacy". It may still be preferable over a fixed duration, especially when actual expiry timeouts cannot be estimated upfront.
| var credentialDurationSeconds = | ||
| realmConfig.getConfig(FeatureConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS); | ||
| if (refreshBufferSeconds < 0 || refreshBufferSeconds >= credentialDurationSeconds) { | ||
| throw new IllegalArgumentException( |
There was a problem hiding this comment.
This check probably belongs in a ProductionReadinessCheck.
Failing cache put operations on invalid configuration is too late.
I believe here, ignoring invalid refreshBufferSeconds would be preferable.
There was a problem hiding this comment.
okay..understood and makes sense after more analysis. incorporated.
| return cache.get(key).toAccessConfig(); | ||
| StorageCredentialCacheEntry entry = cache.get(key); | ||
| if (isUnderRefreshBuffer(entry)) { | ||
| cache.invalidate(key); |
There was a problem hiding this comment.
Is Caffeine's native time-based expiry not robust enough?
There was a problem hiding this comment.
i believe it controls how long something sits in the cache, and not about rejecting a value thats already bad the at the load time. thoughts?
|
Heads-up: #5967 has been merged, so Could you rebase onto |
Adds STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS (default 0, preserving existing half-lifetime behavior). When non-zero, credentials are evicted before their remaining validity drops below the threshold. Closes apache#2046
- Remove EXPIRATION_TIME usage (removed in apache#5070) - Rename testRefreshBufferZeroPreservesHalfLifetimeBehavior to testRefreshBufferZeroKeepsEntryInCache (assert only proves entry is cached, not the specific TTL formula)
getOrLoad() previously returned a freshly-loaded credential as-is even when it was already within the configured refresh buffer of expiring, because Caffeine's expireAfter only governs how long an entry is kept -- not what a synchronous get() returns on the load that populates it. This violated the documented guarantee that clients always receive credentials with at least STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS of validity left. getOrLoad() now checks the loaded entry against the buffer and, if under it, invalidates and reloads once before returning -- bypassing the stale copy without retrying indefinitely against a misconfigured or short-lived credential source. Also validates STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS is >= 0 and less than STORAGE_CREDENTIAL_DURATION_SECONDS, matching the existing validation pattern for STORAGE_CREDENTIAL_CACHE_DURATION_SECONDS. Addresses review from flyingImer and ayushtkn.
… validation Addresses ayushtkn's coverage gaps and flyingImer's ask to assert getOrLoad() never returns an under-buffer credential: - testRefreshBufferHappyPathCachesWithEffectiveTtl: buffer=300s, credential expires in 3600s -- stays cached with a comfortably positive effective TTL. - testGetOrLoadRetriesWhenFreshLoadIsUnderBuffer: first load is near-expiry, second (retried) load is fresh -- getOrLoad() must return the fresh one, proving the buffer is enforced on the initial load, not just on cache hits. - testRefreshBufferRejectsNegativeValue / testRefreshBufferRejectsValueAtOrAboveCredentialDuration: boundary validation on the new range check. Also clarifies the existing testRefreshBufferEvictsNearExpiryCredentials comment to note why its fixed-value loader can't exercise the new retry path (see testGetOrLoadRetriesWhenFreshLoadIsUnderBuffer for that).
Addresses ayushtkn's ask for a CHANGELOG line -- this is a user-facing configuration flag.
…sh buffer getOrLoad's retry (added for the first-load gap) could still return a credential under the configured buffer if the provider issues another near-expiry one, silently violating the documented guarantee. Throw UnprocessableEntityException instead, per flyingImer's review. Test asserts the exact caller-visible outcome and exactly two loads.
Extract a realmConfigWithBuffer(int) helper used by all the buffer tests instead of repeating the RealmConfigImpl construction inline. Fold the negative and at-the-duration-limit rejection tests into one @ParameterizedTest. Add a case verifying a credential with no expiry at all is never considered under the buffer (getExpirationTime() falls back to Long.MAX_VALUE).
apache#5967 replaced direct CHANGELOG.md edits with fragment files under changelog.d/, to avoid merge conflicts between unrelated PRs editing the same section. Moves our entry there unchanged. Per adutra's note on this PR.
693c872 to
429f2d0
Compare
…throwing Addresses dimas-b's review: - Reworded the buffer=0 (half-lifetime) description to not call it "legacy" -- it's a reasonable choice on its own, not just a fallback being phased out. - Moved range validation for STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS out of the cache's request path and into a new ProductionReadinessChecks.checkStorageCredentialRefreshBuffer, modeled on the existing checkCatalogConfigAuthorization pattern (checks both the global default and every realm override). An out-of-range value is now surfaced loudly at server startup instead of failing a live credential request the first time a realm happens to need one. - StorageCredentialCache.refreshBufferMs() no longer throws on a bad value; it logs a warning and falls back to the default half-lifetime behavior (buffer=0). Failing a request over a config typo was worse than a safe fallback, now that the real signal lives at startup. Updated StorageCredentialCacheTest's out-of-range test to assert the fallback behavior instead of an exception, and added ProductionReadinessChecksStorageCredentialRefreshBufferTest covering unset/valid/negative/at-duration defaults plus realm-override cases.
Missed this mirrored doc page when rewording the 'legacy' framing in FeatureConfiguration.java's description string earlier -- same text, kept in sync.
Closes #2046
The cache currently evicts credentials after half their remaining
lifetime. That works, but there's no way to tune it -- if your storage
provider returns short-lived tokens, you can end up serving credentials
that are nearly expired by the time the client uses them.
This adds STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS (default 0). When
set to a positive value, credentials are evicted from the cache once
their remaining validity drops below that threshold. With the default of
0, behavior is unchanged -- the cache still uses the half-lifetime
approach.
The buffer is enforced on every call, not just cache hits: a freshly
loaded credential that's already under the buffer is discarded and
reloaded once. If the reload is still under the buffer, getOrLoad throws
rather than silently handing back a credential that violates the
guarantee -- a provider that can't clear its own configured buffer is a
configuration problem the caller should see.
Changes:
default, validated to be
>= 0and less thanSTORAGE_CREDENTIAL_DURATION_SECONDS
maxCacheDurationMs so the Caffeine expiry function can use both
happy-path caching with a positive buffer and comfortable effective
TTL; a near-expiry load retried once that comes back fresh; a
persistently near-expiry load that still violates the buffer after
retry throws; out-of-range buffer values (negative, or at/above the
credential duration) are rejected; a credential with no expiry at all
is never considered under the buffer.