Skip to content

Bound Idempotency-Key retries by the advertised idempotency-key-lifetime - #4429

Open
linzch3 wants to merge 1 commit into
apache:mainfrom
linzch3:fix/4428-idempotency-key-lifetime
Open

linzch3 wants to merge 1 commit into
apache:mainfrom
linzch3:fix/4428-idempotency-key-lifetime

Conversation

@linzch3

@linzch3 linzch3 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Closes #4428

Rationale for this change

#4310 sends Idempotency-Key on mutations and lets rest.client.* retry keyed POSTs, but it never looks at the idempotency-key-lifetime the server returns from /v1/config beyond checking it's there. urllib3 sleeps for the full Retry-After (and backoff can add up too), so a retry can go out after the key has expired, and the server may then apply the same mutation twice. The spec says clients SHOULD NOT reuse a key after that window.

This follows what the Java client does (HTTPClient.parseKeyLifetime + ExponentialHttpRequestRetryStrategy, apache/iceberg#18070):

  • idempotency-key-lifetime is parsed as an ISO-8601 duration (TypeAdapter(timedelta)).
    • malformed: log a warning and don't bound retries; the key is still sent (same as Java)
    • zero or negative: keyed requests are never retried
  • _RetryTimeoutHTTPAdapter.send records a deadline (now + lifetime, monotonic clock) in a ContextVar when the request carries an Idempotency-Key.
  • A small Retry subclass checks it in increment: if now + next_sleep >= deadline the retry is declined. next_sleep is what Retry.sleep would wait: Retry-After when urllib3 would honor it, otherwise the backoff plus the max jitter.
  • A declined retry raises MaxRetryError, i.e. it behaves exactly like running out of retries: with raise_on_status=False urllib3 hands back the last response, so _handle_non_200_response still raises the typed error (e.g. ServiceUnavailableError); a connection error is re-raised as before.

Requests without a key (all GET/HEAD, and everything when the server doesn't advertise a lifetime) are untouched.

Are these changes tested?

Yes, in tests/catalog/test_rest.py, using the existing loopback server (now with an optional Retry-After header):

  • Retry-After: 60 with PT30S lifetime: no retry, ServiceUnavailableError, 1 call
  • backoff alone exceeding the lifetime (PT5S, factor 10): stops before the second retry
  • Retry-After: 0 within PT30M: still retried and succeeds
  • PT0S and -PT5M: keyed POST never retried, key still sent
  • PT0S doesn't affect unkeyed GETs, they still retry
  • malformed lifetime: warning logged, keyed POST still retried with the same key
  • _parse_idempotency_key_lifetime unit cases

make lint passes. pytest tests/catalog -m unit passes except 4 kerberos tests in test_hive.py that fail the same way on main here (kerberos extra not installed locally).

Are there any user-facing changes?

Only for catalogs whose server advertises idempotency-key-lifetime and that set rest.client.max-retries: a keyed mutation is no longer retried once the next attempt would land outside the key's lifetime; the error that would have been raised after exhausting retries is raised earlier instead.

Limitations

  • The deadline starts when the adapter sends the first attempt, not when the caller started the operation (e.g. the commit retry loop in table/update creates a new key per attempt, so that's fine).
  • The SigV4 adapter mounted on the catalog URI has its own plain integer retries and doesn't retry POST on status codes, so it isn't changed here.
  • The lifetime is checked before sleeping, so a slow next attempt itself (e.g. a long socket timeout) can still finish after the window. Java has the same behavior.

#4310 cc @YuangGao. Happy to adjust if u'd prefer a different shape (e.g. doing this in the adapter instead of a Retry subclass). thx!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

REST catalog: bound Idempotency-Key retries to the advertised idempotency-key-lifetime

1 participant