Skip to content

Shared s3.retry.* properties for the S3 FileIO backends #3604

Open

Description

@emanuilov

Apache Iceberg Rust version

0.10.1

Describe the bug

Is your feature request related to a problem or challenge?

We run a high-throughput writer that streams Kafka topics into Amazon S3 Tables using iceberg, iceberg-catalog-s3tables and iceberg-storage-opendal. When many data files are uploaded at once (for example a backfill flushing hundreds of partitions, or dozens of writers creating files in new tables in one table bucket), S3 answers 503 SlowDown while it scales the prefix up. That is expected S3 behaviour, and AWS's guidance is to retry with exponential backoff and jitter.

The opendal S3 operator is always wrapped in RetryLayer::new(): 3 retries starting at 1 s, factor 2, no jitter. There is no property or builder option to change it (the per-operation timeout became configurable in #3263, the retry policy did not). So:

  • the error reaches the caller after about 7 s, while S3's scaling usually needs longer, so a whole commit cycle fails and has to be redone, which adds to the load that caused the throttling;
  • without jitter, all uploads throttled in the same burst retry at the same moments and get throttled again together.

The only workaround today is a custom StorageFactory that re-implements the opendal storage just to change one layer.

Describe the solution you'd like

Read Iceberg Java's existing S3FileIO properties, with the same meaning, so one catalog configuration behaves the same from Java and Rust:

  • s3.retry.num-retries
  • s3.retry.min-wait-ms
  • s3.retry.max-wait-ms

Define the keys once in iceberg::io (next to the other S3_* constants), so any S3 backend can read them: the opendal backend now, and the object_store backend proposed in #3165 later (it maps naturally onto object_store's RetryConfig).

For the opendal backend: map them onto the S3 operator's RetryLayer (with_max_times, with_min_delay, with_max_delay) and enable jitter whenever any of them is set. Java has no jitter key, and the AWS SDK always jitters. With none set, the layer stays RetryLayer::new(), so existing users see no change. Other backends are not affected.

Willingness to contribute

I've prepared a PR implementing this for the opendal backend (unit tests included).

AI Notice

Drafted with AI

To Reproduce

  1. Create an AWS S3 Tables catalog with S3TablesCatalogBuilder (default OpenDalStorageFactory::S3) and one or more new tables in one table bucket.
  2. Upload many data files at once: for example a few hundred concurrent DataFileWriters / FanoutWriter partitions flushing together, or several processes writing into new tables in the same table bucket. In our case this was dozens of concurrent writers flushing hundreds of partition files into fresh tables.
  3. S3 starts answering 503 SlowDown while it scales the new prefixes up.
  4. Each throttled request is retried 3 times (about 1 s, 2 s, 4 s, no jitter). Requests from the same burst retry in lockstep, are throttled again, and after about 7 s the write fails with an opendal RateLimited/Unexpected error. The writer's commit fails, and the whole cycle has to be redone.

There is no property or builder option to change the retry policy: RetryLayer::new() is hard-coded where the operator is built in iceberg-storage-opendal (crates/storage/opendal/src/lib.rs).

Expected behavior

The retry policy for S3 can be configured, the way Iceberg Java's S3FileIO allows with s3.retry.num-retries, s3.retry.min-wait-ms and s3.retry.max-wait-ms. Writers that burst many uploads into S3 / S3 Tables can then retry SlowDown for long enough (with jitter, so throttled requests don't retry in lockstep) for S3 to scale, instead of failing the commit after about 7 s. With none of the properties set, behaviour stays as today.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions