feat(io): add retry support for OpenDAL stores

[ddupg]

Problem

OpenDAL-backed object stores currently propagate temporary backend errors directly. In distributed writes, transient throttling such as HTTP 429 can fail the entire job even though retrying the request would succeed.

Changes

• Add a shared OpenDAL RetryLayer to S3, Azure Blob/ADLS, GCS, TOS, OSS, COS, Hugging Face, and GooseFS stores.
• Reuse client_max_retries (including OBJECT_STORE_CLIENT_MAX_RETRIES); the existing default is 3 and 0 disables retries.
• Retry only errors marked temporary by OpenDAL, with exponential backoff starting at 100 ms, factor 2, a 15-second maximum base delay, and jitter.
• Preserve the retry configuration when dynamic TOS, OSS, or Hugging Face stores are rebuilt.
• Add fault-injection coverage for retry success, persistent errors, disabled/exhausted retries, writer payload replay, and writer close.

Testing

• cargo fmt --all --check
• cargo test -p lance-io --lib --all-features -- --test-threads=1 --skip uring::tests
• cargo clippy -p lance-io --all-features --tests --benches -- -D warnings

[lance-gatekeeper]

❌ Gate recommendation: request changes.

Retrying at the OpenDAL operator boundary is the right direction, but this revision does not yet satisfy the cross-provider client retry contract. A viable revision should pair the shared layer with provider-level throttle classification and tests, preferably fixed upstream, and enforce the existing client_retry_timeout as an elapsed-time budget.

[lance-gatekeeper]

OpenDAL 0.58.1 RetryLayer only retries errors that the service marks temporary. The pinned Azblob, Azdls, OSS, COS, and Hugging Face parsers classify HTTP 429 as persistent, so this layer makes only one request for the motivating throttle failure on five advertised HTTP backends; OSS, COS, and Hugging Face also have no outer AIMD fallback. Please fix or upgrade those provider classifiers, or add equivalent classification before this layer, and cover each affected provider.

Reproducer

I pointed an Azblob operator wrapped with finish_opendal_operator(operator, 3) at a local endpoint that always returns 429 and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate-8363-target-92801910180 cargo test -p lance-io reproducer_azblob_429_is_not_retried --lib -- --nocapture

Expected four requests before exhaustion; observed one request and a persistent, non-temporary error. The disposable test passed those assertions.

[lance-gatekeeper]

client_retry_timeout is a documented general object-store retry bound and is honored by native S3, Azure, and GCS, but this helper receives only the retry count. An OpenDAL request can therefore keep backing off long after its configured timeout. Please pass the timeout into the shared construction and enforce an elapsed-time budget, with a paused-time regression.

Reproducer

With paused time, client_max_retries=10, client_retry_timeout=1, and four injected temporary stat failures, I ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate-8363-target-92801910180 cargo test -p lance-io reproducer_client_retry_timeout_is_ignored --lib -- --nocapture

Expected the one-second budget to stop before attempt five; observed attempt five succeed, stat_attempts == 5, and virtual elapsed time greater than one second. The disposable test passed those assertions.

[westonpace]

This looks pretty cool for opendal. I wonder if we can add support for AIMD (feel free to do this in a future PR). The way non-dal object stores work is that we first retry 3 times (dependent on client_max_retries) and then we trigger the aimd throttle.

I think all we need to do is wrap the object store with AimdThrottledStore and then update the logic on is_throttle_error to recognize opendal temporary errors.

This will add an additional 3 outer retries (configurable via LANCE_AIMD_MAX_RETRIES) for a total of 9 retries. The outer AIMD retries will both:

• Cut the rate in half (it will grow back eventually)
• Have a different delay (on the order of 100-300ms)

Anyways, all of this is fine for future PRs. I think we can merge this once gatekeeper is happy.

[westonpace]

It would be nice if these were configurable, at least via environment variables. But that can be done in a future PR.

[Xuanwo]

I'm thinking of add AIMD in opendal directly