Skip to content

fix(cache): add configurable refresh buffer to StorageCredentialCache - #5129

Open
visit2rahul wants to merge 12 commits into
apache:mainfrom
visit2rahul:fix/storage-credential-refresh-buffer
Open

visit2rahul wants to merge 12 commits into
apache:mainfrom
visit2rahul:fix/storage-credential-refresh-buffer

Conversation

@visit2rahul

@visit2rahul visit2rahul commented Jul 23, 2026 •

Copy link
Copy Markdown
Contributor

Closes #2046

The cache currently evicts credentials after half their remaining
lifetime. That works, but there's no way to tune it -- if your storage
provider returns short-lived tokens, you can end up serving credentials
that are nearly expired by the time the client uses them.

This adds STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS (default 0). When
set to a positive value, credentials are evicted from the cache once
their remaining validity drops below that threshold. With the default of
0, behavior is unchanged -- the cache still uses the half-lifetime
approach.

The buffer is enforced on every call, not just cache hits: a freshly
loaded credential that's already under the buffer is discarded and
reloaded once. If the reload is still under the buffer, getOrLoad throws
rather than silently handing back a credential that violates the
guarantee -- a provider that can't clear its own configured buffer is a
configuration problem the caller should see.

Changes:

  • New FeatureConfiguration constant with a clear description and a safe
    default, validated to be >= 0 and less than
    STORAGE_CREDENTIAL_DURATION_SECONDS
  • StorageCredentialCacheEntry carries the buffer alongside
    maxCacheDurationMs so the Caffeine expiry function can use both
  • Test coverage: zero-buffer preserves existing half-lifetime behavior;
    happy-path caching with a positive buffer and comfortable effective
    TTL; a near-expiry load retried once that comes back fresh; a
    persistently near-expiry load that still violates the buffer after
    retry throws; out-of-range buffer values (negative, or at/above the
    credential duration) are rejected; a credential with no expiry at all
    is never considered under the buffer.

@dimas-b dimas-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this enhancement, @visit2rahul !

.put(
StorageAccessProperty.AWS_SESSION_TOKEN_EXPIRES_AT_MS,
String.valueOf(expiresAt))
.put(StorageAccessProperty.EXPIRATION_TIME, String.valueOf(expiresAt))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will probably need to be reworked after #5070

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done...

@visit2rahul
visit2rahul force-pushed the fix/storage-credential-refresh-buffer branch from be5f4a4 to f06ec75 Compare August 1, 2026 21:07
@visit2rahul

Copy link
Copy Markdown
Contributor Author

@dimas-b Morning. Just checking if anything else is needed on this one?

@visit2rahul
visit2rahul requested a review from dimas-b August 14, 2026 12:45

@flyingImer flyingImer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm aligned with making the cache margin operator-configurable, but I don't think this is merge-ready yet. The positive-buffer contract needs to distinguish cache reuse from what getOrLoad may return, and values outside the supported range need deterministic rejection. Once those behaviors and boundary tests align, no other blocking concern from my side.

realmConfigWithBuffer,
() -> nearExpiryConfig);

storageCredentialCache.getOrLoad(key);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, this test only verifies that a near-expiry credential is not retained but it does not verify the documented guarantee that clients receive at least the configured remaining validity.

StorageCredentialCache.getOrLoad directly returns Caffeine LoadingCache.get, and it returns the freshly loaded value even when expireAfterCreate gives it a zero TTL, so the first caller still receives nearExpiryConfig and only later lookups miss.

Could we ensure getOrLoad never returns an under-buffer credential and assert that behavior here? One option is to validate immediately after key.load(); if the intended contract is only to prevent later cache hits, the configuration description should be narrowed instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed — it retries once if the fresh load is still too close to expiring, and fails loudly if it still cant clear the buffer after that

Comment on lines +91 to +93
private long refreshBufferMs(RealmConfig realmConfig) {
return realmConfig.getConfig(FeatureConfiguration.STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS)
* 1000L;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The advertised refresh-buffer range is not enforced. refreshBufferMs accepts any integer: a negative value silently selects the legacy half-life path, while a value at or above the configured credential duration can make every request reload from the provider when credentials are issued at that duration.

Could we reject values the public description declares invalid and define negative values explicitly, rather than silently changing cache behavior? One option is to accept only 0 <= buffer < STORAGE_CREDENTIAL_DURATION_SECONDS before key.load(), matching the adjacent cache-duration constraint, with negative/equal/greater boundary tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea fixed this too, though it moved since — bad values get caught at server startup now instead of at request time, and just fall back to the default if something's wrong

@ayushtkn ayushtkn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanx @visit2rahul few suggestions

Missing CHANGELOG.md entry. This is a user-facing configuration flag. Maybe worth adding

Test coverage gaps:

  • No test for the happy path: buffer=300, credential expires in 3600s → entry cached with ~3300s effective TTL
  • No test that a second getOrLoad triggers reload when the first result was evicted due to buffer (similar to testCacheEvict)
  • No boundary validation tests (blocked on implementation)


##### `polaris.features."STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS"`

Minimum remaining validity (in seconds) that a cached storage credential must have before it is considered stale and evicted. When non-zero, credentials are removed from the cache this many seconds before they expire, ensuring clients always receive credentials with at least this much validity left. When zero (the default), credentials are kept for half of their remaining lifetime, which is the legacy behavior. Must be less than STORAGE_CREDENTIAL_DURATION_SECONDS.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ensuring clients always receive credentials with at least this much validity lef

But getOrLoad delegates directly to LoadingCache.get():

public StorageAccessConfig getOrLoad(StorageCredentialCacheKey key) {
  return cache.get(key).toAccessConfig();
}

When effectiveTtl clamps to 0, Caffeine still returns the freshly loaded entry on that same get() call.

The new test demonstrates this:
storageCredentialCache.getOrLoad(key); // returns cred with only 120s left
Assertions.assertThat(storageCredentialCache.getIfPresent(key)).isNull(); // not cached
So with buffer=300s and a credential expiring in 120s, the first caller still receives a nearly-expired credential — only subsequent cache lookups miss. That contradicts the documented contract.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all added — changelog entry, happy path test, retry test, boundary tests

@visit2rahul
visit2rahul force-pushed the fix/storage-credential-refresh-buffer branch from 3cae09d to c170ae0 Compare September 15, 2026 02:29
@visit2rahul

Copy link
Copy Markdown
Contributor Author

Thanks both — good catch, the first load wasn't actually protected. Fixed getOrLoad to retry once if the fresh load is still under the buffer, and added the range check you both mentioned. Added tests for the happy path, the retry, and the boundary cases, plus the CHANGELOG entry. Also rebased since the branch was pretty far behind main. Let me know if this isn't what you had in mind.

@flyingImer flyingImer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. The range check addresses my configuration concern. The retry also covers the case where the second credential is fresh, but I still see one gap before merge: getOrLoad returns the second credential after warning even when it is still below the buffer, while the config description and changelog still promise minimum validity. With a 300s buffer and both loads returning credentials with 120s left, the caller still receives one.

Keeping the retry bounded makes sense. Could we either fail after the retry cannot meet the buffer, or document this as best-effort refresh with that fallback? The repeated-near-expiry test should assert the chosen caller-visible outcome and exactly two loads.

@github-actions

Copy link
Copy Markdown

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions Bot added the stale label Oct 25, 2026

@orca-review-bot orca-review-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice progress. The range check, happy-path test, and fresh-retry test cover most of the earlier asks, and default 0 keeps today's behavior.

The main open item is still flyingImer's from 2026-09-22: what callers see when the provider can't beat the buffer. Right now we retry once and return the stale credential anyway, while the CHANGELOG and the flag description say "always". Can we pick one (fail, or best effort) and make the docs, code, and tests agree?

Smaller things:

  • Can we fold the two rejection tests into a @ParameterizedTest?
  • Can we pull out a realmConfigWithBuffer(int) helper?
  • Can we add a no-expiresAt + positive-buffer test?
  • Can we refresh the PR description? It still says two tests.

Verified fine: the WARN logs no credential material, and the cache key and realm isolation are untouched.

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

StorageCredentialCacheEntry entry = cache.get(key);
if (isUnderRefreshBuffer(entry)) {
cache.invalidate(key);
entry = cache.get(key);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the provider keeps issuing tokens shorter than the buffer, this reload never helps. Both loads land inside the buffer, the entry gets a TTL of 0, and the next request does it all again. So that setup pays two provider calls plus a WARN on every request, with no caching. Can we either fail here, or return the first credential as best effort and skip the second load (maybe with a rate-limited warning)?

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

Comment thread CHANGELOG.md Outdated
- Python CLI: added `--scheme` to specify URL scheme
- Added opt-in idempotency for `createTable` and `updateTable` in the Iceberg REST catalog. When enabled via `polaris.idempotency.enabled=true` (default `false`), a client-supplied `Idempotency-Key` header is embedded into the table entity and committed in the same transaction as the operation; a retry carrying the same key within the TTL window (`polaris.idempotency.ttl`, default `PT5M`) replays the original success instead of failing — with `AlreadyExists` for `createTable`, or with `CommitFailedException` for `updateTable` when the request's requirements no longer match the already-advanced table. When idempotency is enabled, the reuse window is advertised to clients through the `idempotency-key-lifetime` field of the `GET /v1/config` response.
- Python CLI: Added views and generic-tables support.
- Added the `STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS` feature flag (default `0`). When non-zero, a cached storage credential is evicted this many seconds before it actually expires, guaranteeing callers always receive credentials with at least this much validity left; a freshly loaded credential that is already within the buffer is reloaded once rather than handed back as-is. Must be `>= 0` and less than `STORAGE_CREDENTIAL_DURATION_SECONDS`. When `0` (the default), behavior is unchanged: credentials are kept for half of their remaining lifetime.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"guaranteeing callers always receive" isn't what happens. getOrLoad returns an under-buffer credential after one failed retry (same gap flyingImer pointed out). Can we soften this to best effort and mention the fallback?

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

"Minimum remaining validity (in seconds) that a cached storage credential must "
+ "have before it is considered stale and evicted. When non-zero, credentials "
+ "are removed from the cache this many seconds before they expire, ensuring "
+ "clients always receive credentials with at least this much validity left. "

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the CHANGELOG: "ensuring clients always receive" overpromises. Can we update it, and also mention that a positive buffer replaces the half-lifetime rule instead of combining with it? Then regenerate flags-polaris_features.md.

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

realmConfigWithBuffer,
() -> nearExpiryConfig);

storageCredentialCache.getOrLoad(key);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says the retry happens, but nothing checks it. Can we count loads, assert exactly 2, and assert whatever the caller is supposed to get back?

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

}

@Test
public void testRefreshBufferRejectsNegativeValue() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and the at/above-duration test differ only in the buffer value. Can we merge them into one @ParameterizedTest (-1, 3600, 4000)?

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

@visit2rahul
visit2rahul force-pushed the fix/storage-credential-refresh-buffer branch from c170ae0 to 0f9e2ae Compare October 29, 2026 02:03
@visit2rahul

Copy link
Copy Markdown
Contributor Author

Went with throwing instead of a fallback — if a provider can't clear its own buffer even after a retry, that feels like something the caller should know about, not quietly swallow. Updated the test to check for the exception and that it's exactly two loads.

@visit2rahul
visit2rahul force-pushed the fix/storage-credential-refresh-buffer branch from 0f9e2ae to 693c872 Compare October 30, 2026 02:05

@dimas-b dimas-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about the delayed review, @visit2rahul , and thanks for the update!

+ "are removed from the cache this many seconds before they expire, ensuring "
+ "clients always receive credentials with at least this much validity left. "
+ "When zero (the default), credentials are kept for half of their remaining "
+ "lifetime, which is the legacy behavior. Must be less than "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think the alternative behaviour is "legacy". It may still be preferable over a fixed duration, especially when actual expiry timeouts cannot be estimated upfront.

@visit2rahul visit2rahul Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm..refined

var credentialDurationSeconds =
realmConfig.getConfig(FeatureConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS);
if (refreshBufferSeconds < 0 || refreshBufferSeconds >= credentialDurationSeconds) {
throw new IllegalArgumentException(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check probably belongs in a ProductionReadinessCheck.

Failing cache put operations on invalid configuration is too late.

I believe here, ignoring invalid refreshBufferSeconds would be preferable.

@visit2rahul visit2rahul Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay..understood and makes sense after more analysis. incorporated.

return cache.get(key).toAccessConfig();
StorageCredentialCacheEntry entry = cache.get(key);
if (isUnderRefreshBuffer(entry)) {
cache.invalidate(key);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is Caffeine's native time-based expiry not robust enough?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i believe it controls how long something sits in the cache, and not about rejecting a value thats already bad the at the load time. thoughts?

@adutra

adutra commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Heads-up: #5967 has been merged, so CHANGELOG.md is no longer edited directly. Changelog entries now go in a fragment file under changelog.d/, which avoids merge conflicts on CHANGELOG.md. See changelog.d/README.md for the format.

Could you rebase onto main, revert your changes to CHANGELOG.md, and add a fragment file instead if your change needs a changelog entry? A fragment is only needed for things worth calling out in release notes, such as breaking changes, metastore schema changes, or changes that require manual steps during an upgrade. Thanks!

Adds STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS (default 0, preserving
existing half-lifetime behavior). When non-zero, credentials are evicted
before their remaining validity drops below the threshold.

Closes apache#2046
- Remove EXPIRATION_TIME usage (removed in apache#5070)
- Rename testRefreshBufferZeroPreservesHalfLifetimeBehavior to
  testRefreshBufferZeroKeepsEntryInCache (assert only proves entry
  is cached, not the specific TTL formula)
getOrLoad() previously returned a freshly-loaded credential as-is even
when it was already within the configured refresh buffer of expiring,
because Caffeine's expireAfter only governs how long an entry is kept
-- not what a synchronous get() returns on the load that populates it.
This violated the documented guarantee that clients always receive
credentials with at least STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS of
validity left.

getOrLoad() now checks the loaded entry against the buffer and, if
under it, invalidates and reloads once before returning -- bypassing
the stale copy without retrying indefinitely against a misconfigured
or short-lived credential source.

Also validates STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS is >= 0 and
less than STORAGE_CREDENTIAL_DURATION_SECONDS, matching the existing
validation pattern for STORAGE_CREDENTIAL_CACHE_DURATION_SECONDS.

Addresses review from flyingImer and ayushtkn.
… validation

Addresses ayushtkn's coverage gaps and flyingImer's ask to assert
getOrLoad() never returns an under-buffer credential:

- testRefreshBufferHappyPathCachesWithEffectiveTtl: buffer=300s,
  credential expires in 3600s -- stays cached with a comfortably
  positive effective TTL.
- testGetOrLoadRetriesWhenFreshLoadIsUnderBuffer: first load is
  near-expiry, second (retried) load is fresh -- getOrLoad() must
  return the fresh one, proving the buffer is enforced on the initial
  load, not just on cache hits.
- testRefreshBufferRejectsNegativeValue /
  testRefreshBufferRejectsValueAtOrAboveCredentialDuration: boundary
  validation on the new range check.

Also clarifies the existing testRefreshBufferEvictsNearExpiryCredentials
comment to note why its fixed-value loader can't exercise the new
retry path (see testGetOrLoadRetriesWhenFreshLoadIsUnderBuffer for that).
Addresses ayushtkn's ask for a CHANGELOG line -- this is a
user-facing configuration flag.
…sh buffer

getOrLoad's retry (added for the first-load gap) could still return a
credential under the configured buffer if the provider issues another
near-expiry one, silently violating the documented guarantee. Throw
UnprocessableEntityException instead, per flyingImer's review. Test
asserts the exact caller-visible outcome and exactly two loads.
Extract a realmConfigWithBuffer(int) helper used by all the buffer tests
instead of repeating the RealmConfigImpl construction inline. Fold the
negative and at-the-duration-limit rejection tests into one
@ParameterizedTest. Add a case verifying a credential with no expiry at
all is never considered under the buffer (getExpirationTime() falls
back to Long.MAX_VALUE).
apache#5967 replaced direct CHANGELOG.md edits with fragment
files under changelog.d/, to avoid merge conflicts between unrelated
PRs editing the same section. Moves our entry there unchanged.

Per adutra's note on this PR.
@visit2rahul
visit2rahul force-pushed the fix/storage-credential-refresh-buffer branch from 693c872 to 429f2d0 Compare November 19, 2026 03:52
…throwing

Addresses dimas-b's review:

- Reworded the buffer=0 (half-lifetime) description to not call it
  "legacy" -- it's a reasonable choice on its own, not just a fallback
  being phased out.

- Moved range validation for STORAGE_CREDENTIAL_REFRESH_BUFFER_SECONDS
  out of the cache's request path and into a new
  ProductionReadinessChecks.checkStorageCredentialRefreshBuffer, modeled
  on the existing checkCatalogConfigAuthorization pattern (checks both
  the global default and every realm override). An out-of-range value
  is now surfaced loudly at server startup instead of failing a live
  credential request the first time a realm happens to need one.

- StorageCredentialCache.refreshBufferMs() no longer throws on a bad
  value; it logs a warning and falls back to the default half-lifetime
  behavior (buffer=0). Failing a request over a config typo was worse
  than a safe fallback, now that the real signal lives at startup.

Updated StorageCredentialCacheTest's out-of-range test to assert the
fallback behavior instead of an exception, and added
ProductionReadinessChecksStorageCredentialRefreshBufferTest covering
unset/valid/negative/at-duration defaults plus realm-override cases.
Missed this mirrored doc page when rewording the 'legacy' framing in
FeatureConfiguration.java's description string earlier -- same text,
kept in sync.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StorageCredentialCache can provide unusable credentials

6 participants