Skip to content

Commit b61cab2

Browse files
jey1987-cmdclaude
andcommitted
Add concurrent writes guide to write documentation
Explain how append, overwrite and delete behave on a concurrent commit, how commit retries work, how isolation levels decide conflicts, and how to handle CommitFailedException and ValidationException. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1 parent e3564a5 commit b61cab2

1 file changed

Lines changed: 61 additions & 0 deletions

File tree

‎mkdocs/docs/api.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -578,6 +578,67 @@ assert upd.rows_inserted == 1
578578

579579
PyIceberg will automatically detect which rows need to be updated, inserted or can simply be ignored.
580580

581+
### Concurrent writes
582+
583+
Several writers can commit to the same table at the same time. Iceberg uses optimistic concurrency: each writer prepares its changes against the table state it loaded, and the catalog only accepts the commit if the table has not changed in the meantime. If another writer committed first, the catalog rejects the commit with a `CommitFailedException`.
584+
585+
What happens next depends on the operation:
586+
587+
| Operation | When a concurrent commit is detected |
588+
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
589+
| `append` | Retried automatically. Appends are not validated against concurrent changes, so the retry succeeds once the commit lands on the latest table state. |
590+
| `overwrite`, `delete` | Retried automatically, but first validated against the commits made since the table was loaded. A real conflict raises `ValidationException` and is not retried. |
591+
| Metadata-only changes | Not retried. Transactions that do not write data, for example a schema update on its own, raise `CommitFailedException` on the first conflict. |
592+
593+
#### How a retry works
594+
595+
After a `CommitFailedException`, PyIceberg waits, refreshes the table metadata, rebuilds the snapshot against the latest table state, and tries again:
596+
597+
- The wait before retry `n` (counting from 0) is `min-wait-ms * 2^n`, capped at `max-wait-ms`, plus up to 10% random jitter.
598+
- Retrying stops after `commit.retry.num-retries` retries or once `commit.retry.total-timeout-ms` has elapsed, whichever comes first. The last `CommitFailedException` is then raised.
599+
- The snapshot ID stays the same across attempts. If an earlier attempt actually landed but its response was lost, the refreshed metadata contains that snapshot and PyIceberg stops instead of writing the data twice.
600+
- When the commit fails with `CommitFailedException` or `ValidationException`, the files written for it are deleted. The transaction is then marked as failed and cannot be reused; create a new one to try again.
601+
602+
The retry settings are described in [Commit retry options](configuration.md#commit-retry-options).
603+
604+
#### Isolation levels for `overwrite` and `delete`
605+
606+
Before retrying an `overwrite` or `delete`, PyIceberg checks the commits made since the table was loaded. Which conflicts count depends on [`write.update.isolation-level` and `write.delete.isolation-level`](configuration.md#isolation-level-options):
607+
608+
- Under `serializable` (the default), data files added concurrently that match the operation's filter cause a `ValidationException`.
609+
- Under `snapshot`, concurrent appends are accepted and only conflicting deletes are rejected.
610+
- Under either level, a concurrent delete of the same data files, or delete files added concurrently, is a conflict. Currently, any delete file committed concurrently counts as a conflict for a filtered `overwrite` or `delete`, even in an unrelated partition.
611+
612+
#### Handling failures
613+
614+
```python
615+
from pyiceberg.exceptions import CommitFailedException, ValidationException
616+
from pyiceberg.expressions import EqualTo
617+
618+
try:
619+
tbl.overwrite(df, overwrite_filter=EqualTo("city", "Paris"))
620+
except ValidationException:
621+
# A concurrent commit conflicts with this overwrite. Reload the table and decide
622+
# whether the overwrite is still valid before running it again.
623+
tbl = catalog.load_table("default.cities")
624+
except CommitFailedException:
625+
# The table kept changing and the retries were exhausted. Retrying later, or
626+
# raising `commit.retry.num-retries`, may help.
627+
tbl = catalog.load_table("default.cities")
628+
```
629+
630+
To change the retry behavior for a table, set the properties on it:
631+
632+
```python
633+
tbl = tbl.transaction().set_properties(**{"commit.retry.num-retries": "8"}).commit_transaction()
634+
```
635+
636+
#### Reducing conflicts
637+
638+
- Commit fewer, larger appends instead of many small ones. Every commit is another chance for writers to collide, and each writer has to refresh the table before retrying.
639+
- Avoid having several writers overwrite or delete in the same partitions. Appends from many writers are cheap to retry; overwrites are validated and can fail.
640+
- Use `snapshot` isolation if concurrent appends to the partitions you overwrite are acceptable for your workload.
641+
581642
## Inspecting tables
582643

583644
To explore the table metadata, tables can be inspected.

0 commit comments

Comments
 (0)