Repository navigation
feat(databases): new Delta tables name their statistics columns (ITL-1187) #418
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
eywalker
merged 5 commits into
main
from
eywalker/itl-1187-delta-tables-name-the-statistics-columns
Nov 17, 2026
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
da3fffb
feat(databases): new Delta tables name their statistics columns (ITL-…
02b1337
docs(dev): ITL-1187 alter experiment script docstring
7b8b66b
fix(databases): leave out a statistics column named like a nested fie…
fcd3491
Merge remote-tracking branch 'origin/main' into eywalker/itl-1187-del…
95c5709
Merge remote-tracking branch 'origin/main' into eywalker/itl-1187-del…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| ### Changed | ||
|
|
||
| - **New Delta tables name their statistics columns.** A table that `DeltaTableDatabase` creates now sets the `delta.dataSkippingStatsColumns` table property to the columns deltalake can keep minimum and maximum statistics for (top-level numbers, booleans, strings, dates, timestamps and decimals, except a name that a nested field also has; never binary, list, map or struct columns), so writes to it no longer log `Skipping column ... because it's a binary field`. A table with no such column sets `delta.dataSkippingNumIndexedCols` to `0` instead. The property is set once, at creation: a column added later by schema evolution has no statistics. Existing tables are not altered and keep logging the line (see Troubleshooting for the `RUST_LOG` filter). Stored rows, hashes, paths and layouts are unchanged. (ITL-1187) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| ## `src/orcapod/databases/delta_lake_databases.py` — Delta statistics columns | ||
|
|
||
| ### DS2 — delta-rs warns on every write for each binary column; lookups cannot use file statistics | ||
| **Status:** resolved | ||
| **Severity:** low | ||
| **Issue:** ITL-1187 | ||
|
|
||
| By default delta-rs collects minimum and maximum statistics for the first 32 columns of each | ||
| file it writes and logs `WARN deltalake_core::writer::stats ... Skipping column ... because | ||
| it's a binary field` for each binary column on every write. Orcapod stores record ids and | ||
| hashes as binary, so every commit of a `DeltaTableDatabase` logged several such lines, which | ||
| workshop attendees would see on every table they create (developer decision, November 16, | ||
| 2026: fix it at table creation, not only document it). | ||
|
|
||
| Also found (for the performance project, not changed here): no Orcapod lookup uses these | ||
| statistics. The columns lookups filter on are `__record_id`, `__input_data_hash`, | ||
| `__node_content_hash` and `__pipeline_base_entry_id`, all binary, for which delta-rs makes no | ||
| statistics; the fifth, `__data_context`, is a string but matches every row. Lookups do not | ||
| push a filter into Delta anyway: `DeltaTableDatabase` reads the table into memory and | ||
| answers from its index (ITL-919). The MERGE predicate is on `__record_id`, so MERGE cannot | ||
| prune files by statistics either. File skipping for lookups would need the looked-up | ||
| columns stored in a type delta-rs keeps statistics for, and a lookup that pushes its | ||
| filter into the Delta read. | ||
|
|
||
| Seen while testing (delta-rs 1.0.2, not changed here): MERGE into a table that has a column | ||
| whose name holds a backtick fails (`Expected close delimiter`), with or without the property. | ||
|
|
||
| **Fix:** `DeltaTableDatabase` creates every new table with `delta.dataSkippingStatsColumns`, | ||
| from one rule, `delta_statistics.statistics_configuration`: the top-level columns whose type | ||
| (or extension storage type) is an integer, `float32`/`float64`, boolean, string, `date32`, | ||
| timestamp or `decimal128`, whose name holds no comma or backtick, and whose name is not | ||
| also a nested field's name in the schema (delta-rs matches a listed name against nested | ||
| leaf names too, so `v` would also select a binary `rec.v` and warn). When none qualifies, | ||
| the table gets `delta.dataSkippingNumIndexedCols = 0`, since delta-rs refuses an empty list. | ||
| The list is an allowlist because delta-rs 1.0.2 refuses a write whose list names a list, map, | ||
| struct with a binary field or missing column. The property is set once and never changed: a | ||
| column added by schema evolution gets no statistics and logs nothing. Existing tables are | ||
| not altered and keep warning (`docs/troubleshooting.md`, the `RUST_LOG` filter). Tests | ||
| capture the delta-rs log on create, append, MERGE, schema evolution and a job run, write a | ||
| column of every kind, check file pruning on a listed column, and check that an older table is | ||
| left as it is and that a plain `deltalake.write_deltalake` appends to a new one. | ||
|
|
||
| A property can be changed on an existing table later (`DeltaTable.alter.set_table_properties`, | ||
| one metadata commit); it applies only to files written after the change, and earlier files | ||
| keep the statistics they were written with. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| """ITL-1187: does changing delta.dataSkippingStatsColumns on an existing table apply to old files? | ||
|
|
||
| Creates a table without the property (deltalake keeps statistics for ``v`` and ``s`` by | ||
| default), sets the property to name only ``s``, appends once more, and prints for each data | ||
| file the columns it has a minimum for. Seen with deltalake 1.0.2: the first file keeps ``v`` | ||
| and ``s``; the file appended after the change has ``s`` only. A change applies to files | ||
| written afterwards; earlier files are not rewritten. | ||
|
|
||
| Usage: uv run python docs/dev/scripts/itl_1187/alter_property.py <empty or missing directory> | ||
| """ | ||
|
|
||
| import os | ||
| import sys | ||
|
|
||
| import deltalake | ||
| import pyarrow as pa | ||
|
|
||
|
|
||
| def rows(lo: int, hi: int) -> pa.Table: | ||
| """Rows with an int ``v``, a string ``s`` and a binary ``h``.""" | ||
| ids = list(range(lo, hi)) | ||
| return pa.Table.from_arrays( | ||
| [ | ||
| pa.array(ids, pa.int64()), | ||
| pa.array([f"s{i}" for i in ids], pa.large_string()), | ||
| pa.array([b"h"] * len(ids), pa.large_binary()), | ||
| ], | ||
| names=["v", "s", "h"], | ||
| ) | ||
|
|
||
|
|
||
| def main(directory: str) -> None: | ||
| """Run the experiment in ``directory`` and print the statistics of each file.""" | ||
| if os.path.exists(directory) and os.listdir(directory): | ||
| sys.exit(f"{directory} is not empty: give an empty or missing directory") | ||
| path = os.path.join(directory, "t") | ||
| deltalake.write_deltalake(path, rows(0, 5)) | ||
| deltalake.DeltaTable(path).alter.set_table_properties( | ||
| {"delta.dataSkippingStatsColumns": "`s`"} | ||
| ) | ||
| deltalake.write_deltalake(path, rows(5, 10), mode="append") | ||
|
|
||
| table = deltalake.DeltaTable(path) | ||
| print("configuration:", table.metadata().configuration) | ||
| adds = pa.table(table.get_add_actions(flatten=True)).to_pylist() | ||
| for add in sorted(adds, key=lambda a: str(a["modification_time"])): | ||
| with_min = sorted( | ||
| str(k)[len("min.") :] | ||
| for k, v in add.items() | ||
| if str(k).startswith("min.") and v is not None | ||
| ) | ||
| print(f"{add['path']}: statistics for {with_min}") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main(sys.argv[1]) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,156 @@ | ||
| """Name the columns delta-rs keeps file statistics for, when a Delta table is created (ITL-1187). | ||
|
|
||
| By default delta-rs collects minimum and maximum statistics for the first 32 columns of | ||
| every file it writes, and logs a warning (``deltalake_core::writer::stats``, "Skipping | ||
| column ... because it's a binary field") for each binary column on every write. Orcapod | ||
| stores hashes and record ids as binary columns, so every commit logged a few of these | ||
| lines. | ||
|
|
||
| A table that ``DeltaTableDatabase`` creates therefore names its statistics columns in a | ||
| table property, ``delta.dataSkippingStatsColumns``, set by ``statistics_configuration``: | ||
| the one rule for that list. A column is listed when it is a top-level column whose type, | ||
| or the storage type of its extension type, is one delta-rs makes statistics for: an | ||
| integer, ``float32`` or ``float64``, a boolean, a string, ``date32``, a timestamp or | ||
| ``decimal128``. Every other column is left out: binary of any kind, lists, maps, | ||
| structs, ``null`` and the rest. A column whose name holds a comma or a backtick is left | ||
| out too: delta-rs splits the property on commas before it reads quoting, and it cannot | ||
| MERGE into a table with a backtick in a column name whether or not the column is listed | ||
| (a delta-rs limitation, seen with 1.0.2). Each listed name is put in backticks, so a | ||
| dot, a space or ``::`` in it is part of the name. delta-rs 1.0.2 matches each listed name | ||
| against the last name of every column path in the file, nested ones included: listing | ||
| ``v`` also selects ``rec.v`` of a struct ``rec``, and warns when that field is binary. So a | ||
| top-level column is left out when its name is also the name of a nested field anywhere in | ||
| the schema, or of a part delta-rs gives a nested column in Parquet (``element`` and | ||
| ``list`` when the schema has a list, ``key_value``, ``key`` and ``value`` when it has a | ||
| map). The list is | ||
| deliberately an allowlist: delta-rs 1.0.2 refuses the whole write when the property | ||
| names a list, a map, a struct with a binary field or a missing column, and still warns | ||
| for a named binary column. | ||
|
|
||
| When no column qualifies, the property would be empty, which delta-rs refuses; such a | ||
| table is created with ``delta.dataSkippingNumIndexedCols`` set to ``0`` (no statistics) | ||
| instead. | ||
|
|
||
| The property is set once, when the table is created, and Orcapod never changes it. A | ||
| column added later by schema evolution is not in the list: delta-rs makes no statistics | ||
| for it and logs nothing for it, unless it has a nested binary field named like a listed | ||
| column (then delta-rs warns for that field). A table created before this change has no property and is | ||
| read and appended to as before, unaltered; delta-rs keeps logging its warnings (see | ||
| ``docs/troubleshooting.md``). The property is Delta table metadata only: it changes no | ||
| stored value, column, hash, path or layout. | ||
|
|
||
| Orcapod's own lookups do not use these statistics: they read the table into memory and | ||
| answer from an index, and the columns they filter on (record ids and hashes) are binary. | ||
| The statistics serve other Delta readers that filter a table on its listed columns. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from typing import TYPE_CHECKING | ||
|
|
||
| from orcapod.utils.lazy_module import LazyModule | ||
|
|
||
| if TYPE_CHECKING: | ||
| import pyarrow as pa | ||
| else: | ||
| pa = LazyModule("pyarrow") | ||
|
|
||
| STATS_COLUMNS_PROPERTY = "delta.dataSkippingStatsColumns" | ||
| NUM_INDEXED_COLUMNS_PROPERTY = "delta.dataSkippingNumIndexedCols" | ||
|
|
||
|
|
||
| def _has_statistics(arrow_type: pa.DataType) -> bool: | ||
| """Return whether delta-rs makes min/max statistics for a column of ``arrow_type``. | ||
|
|
||
| Args: | ||
| arrow_type: The column's type as it is written to Delta. | ||
|
|
||
| Returns: | ||
| True for an integer, ``float32``/``float64``, boolean, string, ``date32``, | ||
| timestamp or ``decimal128`` column, or an extension type stored as one of | ||
| those; False for every other type. | ||
| """ | ||
| if isinstance(arrow_type, pa.ExtensionType): | ||
| arrow_type = arrow_type.storage_type | ||
| t = pa.types | ||
| return ( | ||
| t.is_integer(arrow_type) | ||
| or (t.is_floating(arrow_type) and arrow_type.bit_width in (32, 64)) | ||
| or t.is_boolean(arrow_type) | ||
| or t.is_string(arrow_type) | ||
| or t.is_large_string(arrow_type) | ||
| or t.is_date32(arrow_type) | ||
| or t.is_timestamp(arrow_type) | ||
| or t.is_decimal128(arrow_type) | ||
| ) | ||
|
|
||
|
|
||
| def _nested_names(arrow_type: pa.DataType) -> set[str]: | ||
| """Return every name a nested part of a column of ``arrow_type`` has, at any depth. | ||
|
|
||
| Args: | ||
| arrow_type: A column's type as it is written to Delta. | ||
|
|
||
| Returns: | ||
| The names of the fields inside the type (struct fields, list items, map keys | ||
| and values) and the names delta-rs gives those parts in Parquet: ``list`` and | ||
| ``element`` for a list, ``key_value``, ``key`` and ``value`` for a map. Empty | ||
| for a type with no nested part. | ||
| """ | ||
| if isinstance(arrow_type, pa.ExtensionType): | ||
| arrow_type = arrow_type.storage_type | ||
| names: set[str] = set() | ||
| t = pa.types | ||
| if ( | ||
| t.is_list(arrow_type) | ||
| or t.is_large_list(arrow_type) | ||
| or t.is_fixed_size_list(arrow_type) | ||
| ): | ||
| names |= {"list", "element"} | ||
| if t.is_map(arrow_type): | ||
| names |= {"key_value", "key", "value"} | ||
| for i in range(arrow_type.num_fields): | ||
| child = arrow_type.field(i) | ||
| names.add(child.name) | ||
| names |= _nested_names(child.type) | ||
| return names | ||
|
|
||
|
|
||
| def statistics_columns(schema: pa.Schema) -> list[str]: | ||
| """Return the names of the columns a new table keeps statistics for, in schema order. | ||
|
|
||
| Args: | ||
| schema: The schema of the rows that create the table, as written to Delta. | ||
|
|
||
| Returns: | ||
| The top-level columns whose type delta-rs makes statistics for, whose name holds | ||
| no comma and no backtick, and whose name is no nested part's name | ||
| (``_nested_names``) in any column of ``schema``. | ||
| """ | ||
| nested: set[str] = set() | ||
| for field in schema: | ||
| nested |= _nested_names(field.type) | ||
| return [ | ||
| field.name | ||
| for field in schema | ||
| if _has_statistics(field.type) | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| and "," not in field.name | ||
| and "`" not in field.name | ||
| and field.name not in nested | ||
| ] | ||
|
|
||
|
|
||
| def statistics_configuration(schema: pa.Schema) -> dict[str, str]: | ||
| """Return the table properties that name a new Delta table's statistics columns. | ||
|
|
||
| Args: | ||
| schema: The schema of the rows that create the table, as written to Delta. | ||
|
|
||
| Returns: | ||
| ``{"delta.dataSkippingStatsColumns": <names in backticks, comma-separated>}``, or | ||
| ``{"delta.dataSkippingNumIndexedCols": "0"}`` when no column qualifies. | ||
| """ | ||
| names = statistics_columns(schema) | ||
| if not names: | ||
| return {NUM_INDEXED_COLUMNS_PROPERTY: "0"} | ||
| return {STATS_COLUMNS_PROPERTY: ",".join(f"`{name}`" for name in names)} | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Recheck nested-name collisions after schema evolution.
Create a table with an integer column
x, then add a struct with a binary child also namedx. The retained property still namesx. In deltalake 1.0.2, MERGE uses that property and matches Parquet leaf names, so it also selects the binary child and emits the warning this change must prevent. Recompute collision exclusions when the schema changes. (raw.githubusercontent.com)Preserve statistics when the schema evolves.
A table created with only binary columns gets
delta.dataSkippingNumIndexedCols=0. If a later MERGE adds an integer column, the property remains zero, so later files cannot be skipped using that column. This also turns statistics off, which ITL-1187 excludes. Update the selection for evolved schemas without changing pre-existing tables. (raw.githubusercontent.com)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed that this case exists; declined by design. The developer-approved design (decision (c) for ITL-1187) sets the property once, at creation, and Orcapod never changes it: recomputing the list on schema evolution would mean altering the table's properties (
set_table_properties, an extra metadata commit) on every evolving write. The case needs a later struct with a binary child that has exactly the name of a listed top-level column. That schema is rare, and its only effect is the log line this PR removes for every other table: no write fails and no stored value changes. It is stated in the module docstring ofdelta_statistics.py: a column added later "has a nested binary field named like a listed column (then delta-rs warns for that field)". If it shows up in practice, an explicit alter on evolution can follow as its own issue.Posted by a Claude Code agent on behalf of @eywalker.