You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b61cab2
Browse filesBrowse the repository at this point in the historyBrowse files
Add concurrent writes guide to write documentation
Explain how append, overwrite and delete behave on a concurrent commit, how
commit retries work, how isolation levels decide conflicts, and how to handle
CommitFailedException and ValidationException.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
PyIceberg will automatically detect which rows need to be updated, inserted or can simply be ignored.
580
580
581
+
### Concurrent writes
582
+
583
+
Several writers can commit to the same table at the same time. Iceberg uses optimistic concurrency: each writer prepares its changes against the table state it loaded, and the catalog only accepts the commit if the table has not changed in the meantime. If another writer committed first, the catalog rejects the commit with a `CommitFailedException`.
584
+
585
+
What happens next depends on the operation:
586
+
587
+
| Operation | When a concurrent commit is detected |
| `append` | Retried automatically. Appends are not validated against concurrent changes, so the retry succeeds once the commit lands on the latest table state. |
590
+
| `overwrite`, `delete` | Retried automatically, but first validated against the commits made since the table was loaded. A real conflict raises `ValidationException` and is not retried. |
591
+
| Metadata-only changes | Not retried. Transactions that do not write data, for example a schema update on its own, raise `CommitFailedException` on the first conflict. |
592
+
593
+
#### How a retry works
594
+
595
+
After a `CommitFailedException`, PyIceberg waits, refreshes the table metadata, rebuilds the snapshot against the latest table state, and tries again:
596
+
597
+
- The wait before retry `n` (counting from 0) is `min-wait-ms * 2^n`, capped at `max-wait-ms`, plus up to 10% random jitter.
598
+
- Retrying stops after `commit.retry.num-retries` retries or once `commit.retry.total-timeout-ms` has elapsed, whichever comes first. The last `CommitFailedException` is then raised.
599
+
- The snapshot ID stays the same across attempts. If an earlier attempt actually landed but its response was lost, the refreshed metadata contains that snapshot and PyIceberg stops instead of writing the data twice.
600
+
- When the commit fails with `CommitFailedException` or `ValidationException`, the files written for it are deleted. The transaction is then marked as failed and cannot be reused; create a new one to try again.
601
+
602
+
The retry settings are described in [Commit retry options](configuration.md#commit-retry-options).
603
+
604
+
#### Isolation levels for `overwrite` and `delete`
605
+
606
+
Before retrying an `overwrite` or `delete`, PyIceberg checks the commits made since the table was loaded. Which conflicts count depends on [`write.update.isolation-level` and `write.delete.isolation-level`](configuration.md#isolation-level-options):
607
+
608
+
- Under `serializable` (the default), data files added concurrently that match the operation's filter cause a `ValidationException`.
609
+
- Under `snapshot`, concurrent appends are accepted and only conflicting deletes are rejected.
610
+
- Under either level, a concurrent delete of the same data files, or delete files added concurrently, is a conflict. Currently, any delete file committed concurrently counts as a conflict for a filtered `overwrite` or `delete`, even in an unrelated partition.
611
+
612
+
#### Handling failures
613
+
614
+
```python
615
+
from pyiceberg.exceptions import CommitFailedException, ValidationException
- Commit fewer, larger appends instead of many small ones. Every commit is another chance for writers to collide, and each writer has to refresh the table before retrying.
639
+
- Avoid having several writers overwrite or delete in the same partitions. Appends from many writers are cheap to retry; overwrites are validated and can fail.
640
+
- Use `snapshot` isolation if concurrent appends to the partitions you overwrite are acceptable for your workload.
641
+
581
642
## Inspecting tables
582
643
583
644
To explore the table metadata, tables can be inspected.
0 commit comments