Skip to content

Commit c688f1e

Browse files
damansingh1313kevinjqliuCopilot
authored
docs: Document missing write table properties in configuration (#3879)
* docs: Document missing write table properties in configuration The write-options table omitted several properties that PyIceberg reads when writing, including write.target-file-size-bytes, write.format.default, write.avro.compression-codec, the Parquet row-group/bloom-filter sizes, the column metrics-mode properties and write.summary.partition-limit. Add rows for them and fix a typo ("coddec" -> "codec") in the existing compression-codec entry. * docs: Remove unimplemented Parquet write properties PyIceberg ignores write.parquet.row-group-size-bytes and the bloom-filter properties, emitting a "not implemented" warning when they are set. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs: Clarify write table property descriptions Clarify how write.target-file-size-bytes is measured, what each metrics mode collects, which files the Avro codec applies to, and that the default write.summary.partition-limit of 0 disables partition summaries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs: Correct metrics mode description The none mode still records column sizes, and nested fields are capped at counts regardless of the configured mode. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent 35bb9bb commit c688f1e

1 file changed

Lines changed: 7 additions & 1 deletion

File tree

‎mkdocs/docs/configuration.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,12 +79,18 @@ Iceberg tables support table properties to configure table behavior.
7979

8080
| Key | Options | Default | Description |
8181
|------------------------------------------|------------------------------------|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
82-
| `write.parquet.compression-codec` | `{uncompressed,zstd,gzip,snappy}` | zstd | Sets the Parquet compression coddec. |
82+
| `write.parquet.compression-codec` | `{uncompressed,zstd,gzip,snappy}` | zstd | Sets the Parquet compression codec. |
8383
| `write.parquet.compression-level` | Integer | null | Parquet compression level for the codec. If not set, it is up to PyIceberg |
8484
| `write.parquet.row-group-limit` | Number of rows | 1048576 | The upper bound of the number of entries within a single row group |
8585
| `write.parquet.page-size-bytes` | Size in bytes | 1MB | Set a target threshold for the approximate encoded size of data pages within a column chunk |
8686
| `write.parquet.page-row-limit` | Number of rows | 20000 | Set a target threshold for the maximum number of rows within a column chunk |
8787
| `write.parquet.dict-size-bytes` | Size in bytes | 2MB | Set the dictionary page size limit per row group |
88+
| `write.target-file-size-bytes` | Size in bytes | 536870912 (512MB) | Target size of generated data files. Measured in uncompressed in-memory Arrow bytes, so files on disk are typically smaller after compression |
89+
| `write.format.default` | `{parquet}` | parquet | File format for data files written by PyIceberg. Only Parquet is supported |
90+
| `write.avro.compression-codec` | `{gzip,zstd,snappy}` | gzip | Compression codec for manifest files and manifest lists. For `snappy`, install with `pip install pyiceberg[snappy]` |
91+
| `write.metadata.metrics.default` | `{none,counts,truncate(N),full}` | truncate(16) | Default column metrics for data files: `none` records only column sizes, `counts` adds value and null counts, `full` adds lower/upper bounds, and `truncate(N)` truncates string and binary bounds to N characters or bytes. Nested fields collect at most counts |
92+
| `write.metadata.metrics.column.{column-name}` | `{none,counts,truncate(N),full}` | (value of `write.metadata.metrics.default`) | Overrides `write.metadata.metrics.default` for a single column. Use the dot-separated name for nested fields |
93+
| `write.summary.partition-limit` | Integer | 0 | Maximum number of changed partitions for which per-partition summaries are included in the snapshot summary. The default of 0 disables them |
8894
| `write.metadata.previous-versions-max` | Integer | 100 | The max number of previous version metadata files to keep before deleting after commit. |
8995
| `write.metadata.delete-after-commit.enabled` | Boolean | False | Whether to automatically delete old *tracked* metadata files after each table commit. It will retain a number of the most recent metadata files, which can be set using property `write.metadata.previous-versions-max`. |
9096
| `write.object-storage.enabled` | Boolean | False | Enables the [`ObjectStoreLocationProvider`](configuration.md#object-store-location-provider) that adds a hash component to file paths. |

0 commit comments

Comments
 (0)