Repository navigation
Description
Goal
Add support for reading and writing tables using Iceberg's encryption model, interoperable with other engines.
Deliver this incrementally in PyIceberg. Reuse Rust implementations through pyiceberg-core when the necessary bindings are available, without making complete Rust-backed readers and writers a prerequisite.
This tracks Iceberg table encryption, not object-store server-side encryption such as S3 SSE-KMS.
Approach
- Keep Python's public APIs, catalog commits, and provider integrations.
- Define narrow interfaces so individual Python implementations can later delegate to Rust.
- Keep key management and encrypted I/O reusable across manifests, manifest lists, and data files.
- Deliver encrypted reads before requiring the full encrypted-write path.
1. Models, configuration, and key-metadata codec
- Add encryption fields to table and snapshot metadata: apache/iceberg-python#1972.
- Add
StandardKeyMetadata: apache/iceberg-python#3948. - Complete configuration validation and supported-version checks.
- Delegate the key-metadata codec to Rust while retaining the Python class.
Rust prerequisite: merge and release apache/iceberg-rust#3206, then establish the required pyiceberg-core version or capability check. This step does not need file I/O, KMS, or manifest bindings.
2. Key management
- Define a KMS provider interface for wrapping, unwrapping, and optional server-side key generation.
- Add provider integration and a deterministic test provider.
- Add an encryption manager for per-file keys/AAD, key resolution, wrapping, reuse, and caching.
Depends on: step 1.
Rust option: bind EncryptionManager and add a Python KMS provider adapter. Rust can own Iceberg's key-management mechanics while Python keeps cloud-provider integration. Define callback, concurrency, and error behavior before switching implementations.
3. Encrypted streams and file access
- Add encrypted input/output compatible with PyIceberg's file interfaces.
- Support the required sequential and range reads, length handling, buffering, and write finalization.
- Define ownership, close, cancellation, and authentication-failure behavior.
Depends on: steps 1 and 2.
Rust option: expose Rust's encrypted input/output through a storage adapter. Python's seekable streams must be adapted to Rust's asynchronous range-read/write interfaces. Complete native manifest or data-file bindings are not required.
4. Data files and delete files
- Configure PyArrow's Parquet encryption/decryption using the selected per-file key and AAD.
- Carry key metadata between file operations and
DataFilerecords, including newly rolled output files. - Cover the delete-file formats PyIceberg supports, including deletion-vector/Puffin handling where applicable. Track unsupported write formats separately.
Depends on: steps 1 and 2; encrypted-stream consumers also need step 3. Parquet integration can proceed alongside step 3 because Parquet uses its own encryption mechanism.
Rust option: reuse the codec and encryption manager while retaining PyArrow. Replacing entire data-file readers/writers is separate work requiring native bindings, Arrow batch transfer, and equivalent schema, writer-option, and output-metadata behavior.
5. Manifests and manifest lists
- Read encrypted manifests through the existing Python manifest APIs.
- Read encrypted manifest lists by resolving the snapshot's key reference.
- Add encrypted writes, carrying manifest key metadata in manifest-list entries and wrapped manifest-list key metadata in table key entries.
Depends on: steps 1-3. Read support can land independently; encrypted V3 writes also require the writer/version support tracked in step 6.
Rust option: Python readers/writers can use Rust-backed encrypted streams first. Moving complete operations into Rust additionally requires complete manifest and manifest-list bindings, version support, metadata conversion, and stable Python errors. Today's byte-parsing helpers are not that full integration.
6. Table writes, snapshots, and commits
- Complete the V3 write support needed for encrypted tables: apache/iceberg-python#1551.
- Wire encrypted file operations into table reads and writes.
- Commit new key entries together with their snapshot and file references.
- Handle retries, failed commits, and orphaned-file cleanup without losing key references.
Depends on: the relevant file integrations from steps 4 and 5. Start V3 write prerequisites in parallel with the earlier steps.
Boundary: keep catalog transactions in Python. Rust-backed operations must return the file information and key entries Python needs to commit; completing a file write is not a table commit.
Completion criteria
- PyIceberg reads encrypted tables produced by another Iceberg implementation.
- Another implementation reads encrypted tables written by PyIceberg.
- Supported data/delete files, manifests, and manifest lists work together end to end.
- Existing unencrypted behavior remains unchanged.
- Missing keys, wrong keys/AAD, unsupported capabilities, and interrupted operations fail clearly without silently falling back to plaintext.
Add focused coverage with each step, rather than leaving all integration work until the end.
Optional follow-up: complete native readers and writers
Full Rust-backed manifest and data-file I/O can follow independently. Where Rust performs the next operation, keep decoded keys and encryption state native rather than round-tripping through Python.
This is not a prerequisite for shipping Python encryption support.
Related work
- Existing encryption tracking: apache/iceberg-python#3222.
- Python V3 tracking: apache/iceberg-python#1818.
- Rust encryption tracking: apache/iceberg-rust#2034.
Activity