diff --git a/changelog.d/ITL-1187.md b/changelog.d/ITL-1187.md new file mode 100644 index 000000000..abf17c0f6 --- /dev/null +++ b/changelog.d/ITL-1187.md @@ -0,0 +1,3 @@ +### Changed + +- **New Delta tables name their statistics columns.** A table that `DeltaTableDatabase` creates now sets the `delta.dataSkippingStatsColumns` table property to the columns deltalake can keep minimum and maximum statistics for (top-level numbers, booleans, strings, dates, timestamps and decimals, except a name that a nested field also has; never binary, list, map or struct columns), so writes to it no longer log `Skipping column ... because it's a binary field`. A table with no such column sets `delta.dataSkippingNumIndexedCols` to `0` instead. The property is set once, at creation: a column added later by schema evolution has no statistics. Existing tables are not altered and keep logging the line (see Troubleshooting for the `RUST_LOG` filter). Stored rows, hashes, paths and layouts are unchanged. (ITL-1187) diff --git a/docs/dev/design_issues/ITL-1173-DS1.md b/docs/dev/design_issues/ITL-1173-DS1.md index f48bd0ba8..93565b984 100644 --- a/docs/dev/design_issues/ITL-1173-DS1.md +++ b/docs/dev/design_issues/ITL-1173-DS1.md @@ -26,7 +26,8 @@ took 346 s. Three costs grew with the rows stored: Also seen: deltalake logs one "Skipping column ... binary field" line per binary column per write (1,000 lines at 10k rows). Changing that means setting a Delta table property, -which is a developer decision (F4, open). +which is a developer decision (F4: decided November 16, 2026, done in ITL-1187, entry +ITL-1187-DS2). **Fix:** - `matching_rows` starts from the condition with the fewest rows and checks the others on @@ -53,7 +54,8 @@ At 100k rows (one run each): 159 s at the default interval (346 s before) and 11 per-row work (7.5 s on `NoOpArrowDatabase`) is the largest remaining cost. The deltalake "binary field" log line is explained in `docs/troubleshooting.md`, with the -`RUST_LOG` filter that hides it. No table property was set; that decision (F4) is open. +`RUST_LOG` filter that hides it. No table property was set here; ITL-1187 sets one on new +tables (F4). Follow-ups: ITL-1182 (`InMemoryArrowDatabase` lookups over the pending batch), ITL-1183 (peak memory at 100k rows), ITL-1184 (compaction: the append cost of deltalake itself diff --git a/docs/dev/design_issues/ITL-1187-DS2.md b/docs/dev/design_issues/ITL-1187-DS2.md new file mode 100644 index 000000000..3b9415a4e --- /dev/null +++ b/docs/dev/design_issues/ITL-1187-DS2.md @@ -0,0 +1,45 @@ +## `src/orcapod/databases/delta_lake_databases.py` — Delta statistics columns + +### DS2 — delta-rs warns on every write for each binary column; lookups cannot use file statistics +**Status:** resolved +**Severity:** low +**Issue:** ITL-1187 + +By default delta-rs collects minimum and maximum statistics for the first 32 columns of each +file it writes and logs `WARN deltalake_core::writer::stats ... Skipping column ... because +it's a binary field` for each binary column on every write. Orcapod stores record ids and +hashes as binary, so every commit of a `DeltaTableDatabase` logged several such lines, which +workshop attendees would see on every table they create (developer decision, November 16, +2026: fix it at table creation, not only document it). + +Also found (for the performance project, not changed here): no Orcapod lookup uses these +statistics. The columns lookups filter on are `__record_id`, `__input_data_hash`, +`__node_content_hash` and `__pipeline_base_entry_id`, all binary, for which delta-rs makes no +statistics; the fifth, `__data_context`, is a string but matches every row. Lookups do not +push a filter into Delta anyway: `DeltaTableDatabase` reads the table into memory and +answers from its index (ITL-919). The MERGE predicate is on `__record_id`, so MERGE cannot +prune files by statistics either. File skipping for lookups would need the looked-up +columns stored in a type delta-rs keeps statistics for, and a lookup that pushes its +filter into the Delta read. + +Seen while testing (delta-rs 1.0.2, not changed here): MERGE into a table that has a column +whose name holds a backtick fails (`Expected close delimiter`), with or without the property. + +**Fix:** `DeltaTableDatabase` creates every new table with `delta.dataSkippingStatsColumns`, +from one rule, `delta_statistics.statistics_configuration`: the top-level columns whose type +(or extension storage type) is an integer, `float32`/`float64`, boolean, string, `date32`, +timestamp or `decimal128`, whose name holds no comma or backtick, and whose name is not +also a nested field's name in the schema (delta-rs matches a listed name against nested +leaf names too, so `v` would also select a binary `rec.v` and warn). When none qualifies, +the table gets `delta.dataSkippingNumIndexedCols = 0`, since delta-rs refuses an empty list. +The list is an allowlist because delta-rs 1.0.2 refuses a write whose list names a list, map, +struct with a binary field or missing column. The property is set once and never changed: a +column added by schema evolution gets no statistics and logs nothing. Existing tables are +not altered and keep warning (`docs/troubleshooting.md`, the `RUST_LOG` filter). Tests +capture the delta-rs log on create, append, MERGE, schema evolution and a job run, write a +column of every kind, check file pruning on a listed column, and check that an older table is +left as it is and that a plain `deltalake.write_deltalake` appends to a new one. + +A property can be changed on an existing table later (`DeltaTable.alter.set_table_properties`, +one metadata commit); it applies only to files written after the change, and earlier files +keep the statistics they were written with. diff --git a/docs/dev/scripts/itl_1187/alter_property.py b/docs/dev/scripts/itl_1187/alter_property.py new file mode 100644 index 000000000..11c91cb4b --- /dev/null +++ b/docs/dev/scripts/itl_1187/alter_property.py @@ -0,0 +1,56 @@ +"""ITL-1187: does changing delta.dataSkippingStatsColumns on an existing table apply to old files? + +Creates a table without the property (deltalake keeps statistics for ``v`` and ``s`` by +default), sets the property to name only ``s``, appends once more, and prints for each data +file the columns it has a minimum for. Seen with deltalake 1.0.2: the first file keeps ``v`` +and ``s``; the file appended after the change has ``s`` only. A change applies to files +written afterwards; earlier files are not rewritten. + +Usage: uv run python docs/dev/scripts/itl_1187/alter_property.py +""" + +import os +import sys + +import deltalake +import pyarrow as pa + + +def rows(lo: int, hi: int) -> pa.Table: + """Rows with an int ``v``, a string ``s`` and a binary ``h``.""" + ids = list(range(lo, hi)) + return pa.Table.from_arrays( + [ + pa.array(ids, pa.int64()), + pa.array([f"s{i}" for i in ids], pa.large_string()), + pa.array([b"h"] * len(ids), pa.large_binary()), + ], + names=["v", "s", "h"], + ) + + +def main(directory: str) -> None: + """Run the experiment in ``directory`` and print the statistics of each file.""" + if os.path.exists(directory) and os.listdir(directory): + sys.exit(f"{directory} is not empty: give an empty or missing directory") + path = os.path.join(directory, "t") + deltalake.write_deltalake(path, rows(0, 5)) + deltalake.DeltaTable(path).alter.set_table_properties( + {"delta.dataSkippingStatsColumns": "`s`"} + ) + deltalake.write_deltalake(path, rows(5, 10), mode="append") + + table = deltalake.DeltaTable(path) + print("configuration:", table.metadata().configuration) + adds = pa.table(table.get_add_actions(flatten=True)).to_pylist() + for add in sorted(adds, key=lambda a: str(a["modification_time"])): + with_min = sorted( + str(k)[len("min.") :] + for k, v in add.items() + if str(k).startswith("min.") and v is not None + ) + print(f"{add['path']}: statistics for {with_min}") + + +if __name__ == "__main__": + main(sys.argv[1]) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 26d81aeab..fc9bdd801 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -238,10 +238,14 @@ version to regenerate them. ## Messages you can ignore - `WARN deltalake_core::writer::stats] Skipping column ... because it's a binary field.` - whenever a `DeltaTableDatabase` is written to: one line per binary column (record ids and - hashes) per commit. The deltalake library keeps no minimum and maximum statistics for binary - columns and says so on every write; nothing is wrong with the table. To hide only this line, - set `RUST_LOG` in the environment before Python starts, keeping deltalake's other warnings: + whenever a `DeltaTableDatabase` writes to a table that an earlier version of Orcapod + created: one line per binary column (record ids and hashes) per commit. The deltalake + library keeps no minimum and maximum statistics for binary columns and says so on every + write; nothing is wrong with the table. A table that `DeltaTableDatabase` creates now names + the columns it keeps statistics for (the `delta.dataSkippingStatsColumns` table property), + and writes to it log nothing. Orcapod never changes the property of an existing table. To + hide only this line for an older table, set `RUST_LOG` in the environment before Python + starts, keeping deltalake's other warnings: ``` RUST_LOG="warn,deltalake_core::writer::stats=error" python my_pipeline.py diff --git a/src/orcapod/databases/delta_lake_databases.py b/src/orcapod/databases/delta_lake_databases.py index 980b04e84..b2ba444dc 100644 --- a/src/orcapod/databases/delta_lake_databases.py +++ b/src/orcapod/databases/delta_lake_databases.py @@ -29,6 +29,7 @@ recorded_type, to_delta_layout, ) +from orcapod.databases.delta_statistics import statistics_configuration from orcapod.databases.storage_utils import ( delta_to_pyarrow_dataset, is_cloud_uri, @@ -1774,6 +1775,8 @@ def _flush_by_key(self, record_key: str) -> None: mode="overwrite", schema_mode="merge" if self.allow_schema_evolution else "overwrite", storage_options=self._storage_options or None, + # Name the statistics columns once, at creation (ITL-1187). + configuration=statistics_configuration(combined_table.schema), ) base = _CommittedSnapshot(version=-1, ids=set()) inserted_ids = self._non_null_ids(combined_table) diff --git a/src/orcapod/databases/delta_statistics.py b/src/orcapod/databases/delta_statistics.py new file mode 100644 index 000000000..f713cc7de --- /dev/null +++ b/src/orcapod/databases/delta_statistics.py @@ -0,0 +1,156 @@ +"""Name the columns delta-rs keeps file statistics for, when a Delta table is created (ITL-1187). + +By default delta-rs collects minimum and maximum statistics for the first 32 columns of +every file it writes, and logs a warning (``deltalake_core::writer::stats``, "Skipping +column ... because it's a binary field") for each binary column on every write. Orcapod +stores hashes and record ids as binary columns, so every commit logged a few of these +lines. + +A table that ``DeltaTableDatabase`` creates therefore names its statistics columns in a +table property, ``delta.dataSkippingStatsColumns``, set by ``statistics_configuration``: +the one rule for that list. A column is listed when it is a top-level column whose type, +or the storage type of its extension type, is one delta-rs makes statistics for: an +integer, ``float32`` or ``float64``, a boolean, a string, ``date32``, a timestamp or +``decimal128``. Every other column is left out: binary of any kind, lists, maps, +structs, ``null`` and the rest. A column whose name holds a comma or a backtick is left +out too: delta-rs splits the property on commas before it reads quoting, and it cannot +MERGE into a table with a backtick in a column name whether or not the column is listed +(a delta-rs limitation, seen with 1.0.2). Each listed name is put in backticks, so a +dot, a space or ``::`` in it is part of the name. delta-rs 1.0.2 matches each listed name +against the last name of every column path in the file, nested ones included: listing +``v`` also selects ``rec.v`` of a struct ``rec``, and warns when that field is binary. So a +top-level column is left out when its name is also the name of a nested field anywhere in +the schema, or of a part delta-rs gives a nested column in Parquet (``element`` and +``list`` when the schema has a list, ``key_value``, ``key`` and ``value`` when it has a +map). The list is +deliberately an allowlist: delta-rs 1.0.2 refuses the whole write when the property +names a list, a map, a struct with a binary field or a missing column, and still warns +for a named binary column. + +When no column qualifies, the property would be empty, which delta-rs refuses; such a +table is created with ``delta.dataSkippingNumIndexedCols`` set to ``0`` (no statistics) +instead. + +The property is set once, when the table is created, and Orcapod never changes it. A +column added later by schema evolution is not in the list: delta-rs makes no statistics +for it and logs nothing for it, unless it has a nested binary field named like a listed +column (then delta-rs warns for that field). A table created before this change has no property and is +read and appended to as before, unaltered; delta-rs keeps logging its warnings (see +``docs/troubleshooting.md``). The property is Delta table metadata only: it changes no +stored value, column, hash, path or layout. + +Orcapod's own lookups do not use these statistics: they read the table into memory and +answer from an index, and the columns they filter on (record ids and hashes) are binary. +The statistics serve other Delta readers that filter a table on its listed columns. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from orcapod.utils.lazy_module import LazyModule + +if TYPE_CHECKING: + import pyarrow as pa +else: + pa = LazyModule("pyarrow") + +STATS_COLUMNS_PROPERTY = "delta.dataSkippingStatsColumns" +NUM_INDEXED_COLUMNS_PROPERTY = "delta.dataSkippingNumIndexedCols" + + +def _has_statistics(arrow_type: pa.DataType) -> bool: + """Return whether delta-rs makes min/max statistics for a column of ``arrow_type``. + + Args: + arrow_type: The column's type as it is written to Delta. + + Returns: + True for an integer, ``float32``/``float64``, boolean, string, ``date32``, + timestamp or ``decimal128`` column, or an extension type stored as one of + those; False for every other type. + """ + if isinstance(arrow_type, pa.ExtensionType): + arrow_type = arrow_type.storage_type + t = pa.types + return ( + t.is_integer(arrow_type) + or (t.is_floating(arrow_type) and arrow_type.bit_width in (32, 64)) + or t.is_boolean(arrow_type) + or t.is_string(arrow_type) + or t.is_large_string(arrow_type) + or t.is_date32(arrow_type) + or t.is_timestamp(arrow_type) + or t.is_decimal128(arrow_type) + ) + + +def _nested_names(arrow_type: pa.DataType) -> set[str]: + """Return every name a nested part of a column of ``arrow_type`` has, at any depth. + + Args: + arrow_type: A column's type as it is written to Delta. + + Returns: + The names of the fields inside the type (struct fields, list items, map keys + and values) and the names delta-rs gives those parts in Parquet: ``list`` and + ``element`` for a list, ``key_value``, ``key`` and ``value`` for a map. Empty + for a type with no nested part. + """ + if isinstance(arrow_type, pa.ExtensionType): + arrow_type = arrow_type.storage_type + names: set[str] = set() + t = pa.types + if ( + t.is_list(arrow_type) + or t.is_large_list(arrow_type) + or t.is_fixed_size_list(arrow_type) + ): + names |= {"list", "element"} + if t.is_map(arrow_type): + names |= {"key_value", "key", "value"} + for i in range(arrow_type.num_fields): + child = arrow_type.field(i) + names.add(child.name) + names |= _nested_names(child.type) + return names + + +def statistics_columns(schema: pa.Schema) -> list[str]: + """Return the names of the columns a new table keeps statistics for, in schema order. + + Args: + schema: The schema of the rows that create the table, as written to Delta. + + Returns: + The top-level columns whose type delta-rs makes statistics for, whose name holds + no comma and no backtick, and whose name is no nested part's name + (``_nested_names``) in any column of ``schema``. + """ + nested: set[str] = set() + for field in schema: + nested |= _nested_names(field.type) + return [ + field.name + for field in schema + if _has_statistics(field.type) + and "," not in field.name + and "`" not in field.name + and field.name not in nested + ] + + +def statistics_configuration(schema: pa.Schema) -> dict[str, str]: + """Return the table properties that name a new Delta table's statistics columns. + + Args: + schema: The schema of the rows that create the table, as written to Delta. + + Returns: + ``{"delta.dataSkippingStatsColumns": }``, or + ``{"delta.dataSkippingNumIndexedCols": "0"}`` when no column qualifies. + """ + names = statistics_columns(schema) + if not names: + return {NUM_INDEXED_COLUMNS_PROPERTY: "0"} + return {STATS_COLUMNS_PROPERTY: ",".join(f"`{name}`" for name in names)} diff --git a/tests/test_databases/test_delta_statistics_columns.py b/tests/test_databases/test_delta_statistics_columns.py new file mode 100644 index 000000000..87d368bd3 --- /dev/null +++ b/tests/test_databases/test_delta_statistics_columns.py @@ -0,0 +1,489 @@ +"""A Delta table created by ``DeltaTableDatabase`` names its statistics columns (ITL-1187). + +delta-rs logs ``WARN deltalake_core::writer::stats ... Skipping column ... because it's a +binary field`` on every write for each binary column, unless the table names the columns +it keeps statistics for. These tests capture the Rust log (written to the process's +stderr) and fail on that warning, check that the list never makes a write fail, that file +skipping works on a listed column, and that a table created without the property is left +as it is. +""" + +from __future__ import annotations + +import datetime +import decimal +from pathlib import Path +from typing import Any, cast + +import deltalake +import pyarrow as pa +import pytest + +from orcapod.databases import DeltaTableDatabase +from orcapod.databases.delta_statistics import ( + NUM_INDEXED_COLUMNS_PROPERTY, + STATS_COLUMNS_PROPERTY, + statistics_columns, + statistics_configuration, +) + +STATS_WARNING = "deltalake_core::writer::stats" +PATH = ("t",) + + +class _StrExt(pa.ExtensionType): + """An extension type stored as a string (as ``orcapod.path`` is).""" + + def __init__(self) -> None: + super().__init__(pa.large_string(), "itl1187.test.str") + + def __arrow_ext_serialize__(self) -> bytes: + return b"" + + @classmethod + def __arrow_ext_deserialize__(cls, storage_type, serialized) -> _StrExt: + return cls() + + +class _BinExt(pa.ExtensionType): + """An extension type stored as binary (as ``orcapod.numpy.ndarray`` is).""" + + def __init__(self) -> None: + super().__init__(pa.large_binary(), "itl1187.test.bin") + + def __arrow_ext_serialize__(self) -> bytes: + return b"" + + @classmethod + def __arrow_ext_deserialize__(cls, storage_type, serialized) -> _BinExt: + return cls() + + +def rows(ids: range, extra: bool = False) -> pa.Table: + """Rows keyed by a binary record id, with a binary hash and an int column. + + ``extra`` adds a binary and an int column, for schema evolution. + """ + columns: dict[str, pa.Array[Any]] = { + DeltaTableDatabase.RECORD_ID_COLUMN: pa.array( + [f"id{i}".encode() for i in ids], pa.large_binary() + ), + "h": pa.array([f"h{i}".encode() for i in ids], pa.large_binary()), + "v": pa.array(list(ids), pa.int64()), + "__data_context": pa.array(["std:v1:default"] * len(ids), pa.large_string()), + } + if extra: + columns["new_bin"] = pa.array([b"x"] * len(ids), pa.large_binary()) + columns["new_int"] = pa.array(list(ids), pa.int64()) + return pa.Table.from_arrays(list(columns.values()), names=list(columns)) + + +def stats_warnings(capfd: pytest.CaptureFixture[str]) -> list[str]: + """Return the delta-rs statistics warnings logged since the last call.""" + return [ln for ln in capfd.readouterr().err.splitlines() if STATS_WARNING in ln] + + +def configuration(table_path: Path) -> dict[str, str]: + """Return the table properties of the Delta table at ``table_path``.""" + return dict(deltalake.DeltaTable(str(table_path)).metadata().configuration) + + +def columns_with_statistics(table_path: Path) -> list[set[str]]: + """Return, per data file, the columns that file has a minimum for.""" + adds = pa.table( + deltalake.DeltaTable(str(table_path)).get_add_actions(flatten=True) + ).to_pylist() + return [ + { + k[len("min.") :] + for k, v in a.items() + if k.startswith("min.") and v is not None + } + for a in adds + ] + + +def stored_ids(db: DeltaTableDatabase) -> list[bytes]: + """Return the record ids the database reads back, sorted.""" + table = db.get_all_records(PATH, record_id_column="rid") + assert table is not None + return sorted(cast("list[bytes]", table["rid"].to_pylist())) + + +class TestStatisticsRule: + """The rule that picks the statistics columns.""" + + @pytest.mark.parametrize( + "arrow_type", + [ + pa.int8(), + pa.int16(), + pa.int32(), + pa.int64(), + pa.uint8(), + pa.uint64(), + pa.float32(), + pa.float64(), + pa.bool_(), + pa.string(), + pa.large_string(), + pa.date32(), + pa.timestamp("us"), + pa.timestamp("us", tz="UTC"), + pa.decimal128(10, 2), + _StrExt(), + ], + ids=str, + ) + def test_listed_types(self, arrow_type: pa.DataType) -> None: + assert statistics_columns(pa.schema([("c", arrow_type)])) == ["c"] + + @pytest.mark.parametrize( + "arrow_type", + [ + pa.binary(), + pa.large_binary(), + pa.binary(4), + pa.float16(), + pa.null(), + pa.large_list(pa.int64()), + pa.map_(pa.large_string(), pa.int64()), + pa.struct([("x", pa.int64())]), + pa.struct([("x", pa.int64()), ("y", pa.large_binary())]), + pa.dictionary(pa.int32(), pa.large_string()), + _BinExt(), + ], + ids=str, + ) + def test_left_out_types(self, arrow_type: pa.DataType) -> None: + assert statistics_columns(pa.schema([("c", arrow_type)])) == [] + + def test_name_with_a_comma_or_backtick_is_left_out(self) -> None: + schema = pa.schema( + [("a,b", pa.int64()), ("x`y", pa.int64()), ("c", pa.int64())] + ) + assert statistics_columns(schema) == ["c"] + + @pytest.mark.parametrize( + ("name", "other"), + [ + ("v", pa.struct([("v", pa.large_binary())])), + ("v", pa.struct([("v", pa.int64())])), + ("v", pa.large_list(pa.struct([("v", pa.large_binary())]))), + ("item", pa.large_list(pa.int64())), + ("element", pa.large_list(pa.large_binary())), + ("list", pa.large_list(pa.int64())), + ("key", pa.map_(pa.large_string(), pa.int64())), + ("value", pa.map_(pa.large_string(), pa.large_binary())), + ("key_value", pa.map_(pa.large_string(), pa.int64())), + ], + ids=lambda x: str(x), + ) + def test_name_of_a_nested_part_is_left_out( + self, name: str, other: pa.DataType + ) -> None: + """delta-rs matches a listed name against nested leaf names too.""" + schema = pa.schema([(name, pa.int64()), ("rec", other), ("c", pa.int64())]) + assert statistics_columns(schema) == ["c"] + + def test_names_are_quoted(self) -> None: + schema = pa.schema([("a.b", pa.int64()), ("s::t", pa.int64())]) + assert statistics_configuration(schema) == { + STATS_COLUMNS_PROPERTY: "`a.b`,`s::t`" + } + + def test_no_column_qualifies(self) -> None: + schema = pa.schema([("h", pa.large_binary())]) + assert statistics_configuration(schema) == {NUM_INDEXED_COLUMNS_PROPERTY: "0"} + + +class TestNewTableWritePaths: + """Every write path on a new table logs no statistics warning.""" + + def test_create_append_merge_and_schema_evolution( + self, tmp_path: Path, capfd: pytest.CaptureFixture[str] + ) -> None: + db = DeltaTableDatabase(tmp_path) + capfd.readouterr() + + db.add_records(PATH, rows(range(0, 10)), flush=True) + assert stats_warnings(capfd) == [], "create" + assert configuration(tmp_path / "t") == { + STATS_COLUMNS_PROPERTY: "`v`,`__data_context`" + } + + db.add_records(PATH, rows(range(10, 20)), flush=True) + assert stats_warnings(capfd) == [], "append" + + # One stored id among the new ones forces the MERGE path. + db.add_records(PATH, rows(range(15, 30)), skip_duplicates=False, flush=True) + assert stats_warnings(capfd) == [], "merge" + + # New columns (one binary, one int): schema evolution, through MERGE. + db.add_records(PATH, rows(range(30, 40), extra=True), flush=True) + assert stats_warnings(capfd) == [], "schema evolution" + + db.add_records(PATH, rows(range(40, 50), extra=True), flush=True) + assert stats_warnings(capfd) == [], "append after schema evolution" + + # The property is never changed: a column added later has no statistics. + assert configuration(tmp_path / "t") == { + STATS_COLUMNS_PROPERTY: "`v`,`__data_context`" + } + assert all( + stats == {"v", "__data_context"} + for stats in columns_with_statistics(tmp_path / "t") + ) + assert stored_ids(db) == sorted(f"id{i}".encode() for i in range(50)) + + def test_job_writes_log_no_statistics_warning( + self, tmp_path: Path, capfd: pytest.CaptureFixture[str] + ) -> None: + """A function node's pipeline and result tables, written by a job.""" + from orcapod import PipelineJob, function_pod + from orcapod.core.sources import ArrowTableSource + + @function_pod(output_keys="d") + def double(v: int | None) -> int: + """Return twice ``v``.""" + return (v or 0) * 2 + + source = ArrowTableSource( + pa.table({"k": [1, 2, 3], "v": [1, 2, 3]}), + tag_columns=["k"], + tag_only_columns=True, + ) + job = PipelineJob(name="stats", database=DeltaTableDatabase(tmp_path)) + with job: + double.pod(source, label="double") + capfd.readouterr() + job.run() + assert stats_warnings(capfd) == [] + + tables = [log.parent for log in tmp_path.rglob("_delta_log")] + assert len(tables) == 2 # the pipeline table and the result table + for table_path in tables: + listed = configuration(table_path)[STATS_COLUMNS_PROPERTY] + assert "`__data_context`" in listed + assert "`__record_id`" not in listed + + def test_table_with_no_listable_column( + self, tmp_path: Path, capfd: pytest.CaptureFixture[str] + ) -> None: + db = DeltaTableDatabase(tmp_path) + only_binary = rows(range(0, 5)).select( + [DeltaTableDatabase.RECORD_ID_COLUMN, "h"] + ) + capfd.readouterr() + db.add_records(PATH, only_binary, flush=True) + more = rows(range(5, 10)).select([DeltaTableDatabase.RECORD_ID_COLUMN, "h"]) + db.add_records(PATH, more, flush=True) + assert stats_warnings(capfd) == [] + assert configuration(tmp_path / "t") == {NUM_INDEXED_COLUMNS_PROPERTY: "0"} + assert len(stored_ids(db)) == 10 + + +def _every_kind(ids: range) -> pa.Table: + """Rows with a column of every kind the rule lists or leaves out.""" + n = len(ids) + columns: dict[str, pa.Array[Any]] = { + DeltaTableDatabase.RECORD_ID_COLUMN: pa.array( + [f"id{i}".encode() for i in ids], pa.large_binary() + ), + # Listed. + "i8": pa.array([1] * n, pa.int8()), + "i32": pa.array([1] * n, pa.int32()), + "u64": pa.array([2**63 + 1] * n, pa.uint64()), + "f32": pa.array([1.5] * n, pa.float32()), + "f64": pa.array([1.5] * n, pa.float64()), + "flag": pa.array([True] * n, pa.bool_()), + "s": pa.array(["a"] * n, pa.large_string()), + "day": pa.array([datetime.date(2026, 11, 20)] * n, pa.date32()), + "ts": pa.array([1] * n, pa.timestamp("us", tz="UTC")), + "ts_ns": pa.array([1] * n, pa.timestamp("ns")), + "amount": pa.array([decimal.Decimal("1.25")] * n, pa.decimal128(10, 2)), + "ext_s": pa.ExtensionArray.from_storage( + _StrExt(), pa.array(["p"] * n, pa.large_string()) + ), + "a.dot": pa.array([1] * n, pa.int64()), + "sp ace": pa.array([1] * n, pa.int64()), + "_tag_source_id::ab12": pa.array(["src"] * n, pa.large_string()), + "ünï": pa.array([1] * n, pa.int64()), + # Left out. + "b": pa.array([b"x"] * n, pa.large_binary()), + "fsb": pa.array([b"abcd"] * n, pa.binary(4)), + "ext_b": pa.ExtensionArray.from_storage( + _BinExt(), pa.array([b"x"] * n, pa.large_binary()) + ), + "ints": pa.array([[1, 2]] * n, pa.large_list(pa.int64())), + "hashes": pa.array([[b"x"]] * n, pa.large_list(pa.large_binary())), + "pairs": pa.array([[("k", 1)]] * n, pa.map_(pa.large_string(), pa.int64())), + "rec": pa.array( + [{"x": 1, "y": b"z"}] * n, + pa.struct([("x", pa.int64()), ("y", pa.large_binary())]), + ), + "com,ma": pa.array([1] * n, pa.int64()), + } + return pa.Table.from_arrays(list(columns.values()), names=list(columns)) + + +LISTED = { + "i8", + "i32", + "u64", + "f32", + "f64", + "flag", + "s", + "day", + "ts", + "ts_ns", + "amount", + "ext_s", + "a.dot", + "sp ace", + "_tag_source_id::ab12", + "ünï", +} + + +class TestEveryColumnKind: + """A wrong list makes delta-rs refuse the write, so every kind is written for real.""" + + def test_every_kind_is_written_without_warning( + self, tmp_path: Path, capfd: pytest.CaptureFixture[str] + ) -> None: + db = DeltaTableDatabase(tmp_path) + capfd.readouterr() + db.add_records(PATH, _every_kind(range(0, 3)), flush=True) + db.add_records(PATH, _every_kind(range(3, 6)), flush=True) + db.add_records( + PATH, _every_kind(range(5, 9)), skip_duplicates=False, flush=True + ) + assert stats_warnings(capfd) == [] + + assert set(statistics_columns(_delta_schema(tmp_path / "t"))) == LISTED + assert all(stats == LISTED for stats in columns_with_statistics(tmp_path / "t")) + assert len(stored_ids(db)) == 9 + # The stored values come back as written. + fresh = DeltaTableDatabase(tmp_path).get_all_records( + PATH, record_id_column="rid" + ) + assert fresh is not None + compared = sorted(LISTED - {"ext_s"}) + assert ( + fresh.sort_by("rid").select(compared).slice(0, 1).to_pylist() + == _every_kind(range(0, 1)).select(compared).to_pylist() + ) + + def test_backtick_name_is_left_out_and_written( + self, tmp_path: Path, capfd: pytest.CaptureFixture[str] + ) -> None: + """Create and append work; delta-rs cannot MERGE such a table either way.""" + db = DeltaTableDatabase(tmp_path) + + def table(ids: range) -> pa.Table: + return rows(ids).append_column("x`y", pa.array(list(ids), pa.int64())) + + capfd.readouterr() + db.add_records(PATH, table(range(0, 3)), flush=True) + db.add_records(PATH, table(range(3, 6)), flush=True) + assert stats_warnings(capfd) == [] + assert configuration(tmp_path / "t") == { + STATS_COLUMNS_PROPERTY: "`v`,`__data_context`" + } + assert len(stored_ids(db)) == 6 + + def test_names_of_nested_parts_are_left_out_and_written( + self, tmp_path: Path, capfd: pytest.CaptureFixture[str] + ) -> None: + """A top-level name that is also a nested binary field's name logs nothing.""" + db = DeltaTableDatabase(tmp_path) + + def table(ids: range) -> pa.Table: + n = len(ids) + columns: dict[str, pa.Array[Any]] = { + DeltaTableDatabase.RECORD_ID_COLUMN: pa.array( + [f"id{i}".encode() for i in ids], pa.large_binary() + ), + "v": pa.array(list(ids), pa.int64()), + "rec": pa.array( + [{"v": b"x"}] * n, pa.struct([("v", pa.large_binary())]) + ), + "element": pa.array([1] * n, pa.int64()), + "hashes": pa.array([[b"x"]] * n, pa.large_list(pa.large_binary())), + "value": pa.array([1] * n, pa.int64()), + "pairs": pa.array( + [[("k", b"x")]] * n, pa.map_(pa.large_string(), pa.large_binary()) + ), + "c": pa.array([1] * n, pa.int64()), + } + return pa.Table.from_arrays(list(columns.values()), names=list(columns)) + + capfd.readouterr() + db.add_records(PATH, table(range(0, 3)), flush=True) + db.add_records(PATH, table(range(3, 6)), flush=True) + db.add_records(PATH, table(range(5, 9)), skip_duplicates=False, flush=True) + assert stats_warnings(capfd) == [] + assert configuration(tmp_path / "t") == {STATS_COLUMNS_PROPERTY: "`c`"} + assert len(stored_ids(db)) == 9 + + +def _delta_schema(table_path: Path) -> pa.Schema: + """Return the Arrow schema delta-rs reports for the table at ``table_path``.""" + return pa.schema(deltalake.DeltaTable(str(table_path)).schema().to_arrow()) # pyright: ignore[reportArgumentType] # an arro3 schema, read through the Arrow C interface + + +class TestFileSkipping: + def test_a_listed_column_prunes_files(self, tmp_path: Path) -> None: + """A filter on a listed column reads only the file that can hold the value.""" + db = DeltaTableDatabase(tmp_path) + for lo in (0, 100, 200): + db.add_records(PATH, rows(range(lo, lo + 5)), flush=True) + + table = deltalake.DeltaTable(str(tmp_path / "t")) + plan = pa.table( + deltalake.QueryBuilder() + .register("t", table) + .execute("EXPLAIN ANALYZE SELECT * FROM t WHERE v = 101") + .read_all() + ) + text = "\n".join(str(row) for row in plan.to_pylist()) + assert "files_pruned=2, files_scanned=1" in text, text + + +class TestTablesCreatedElsewhere: + def test_existing_table_without_the_property_is_unaltered( + self, tmp_path: Path, capfd: pytest.CaptureFixture[str] + ) -> None: + """A table created before ITL-1187 keeps no property and keeps warning.""" + deltalake.write_deltalake(str(tmp_path / "t"), rows(range(0, 10))) + db = DeltaTableDatabase(tmp_path) + capfd.readouterr() + + db.add_records(PATH, rows(range(10, 20)), flush=True) + db.add_records(PATH, rows(range(15, 30)), skip_duplicates=False, flush=True) + + # The capture works: this table has no property, so delta-rs warns. + assert stats_warnings(capfd) != [] + assert configuration(tmp_path / "t") == {} + assert stored_ids(db) == sorted(f"id{i}".encode() for i in range(30)) + + def test_plain_deltalake_appends_to_a_table_with_the_property( + self, tmp_path: Path, capfd: pytest.CaptureFixture[str] + ) -> None: + """A writer that knows nothing of the property (main before ITL-1187) still appends.""" + DeltaTableDatabase(tmp_path).add_records(PATH, rows(range(0, 10)), flush=True) + capfd.readouterr() + + deltalake.write_deltalake( + str(tmp_path / "t"), rows(range(10, 20)), mode="append" + ) + assert stats_warnings(capfd) == [] + + assert stored_ids(DeltaTableDatabase(tmp_path)) == sorted( + f"id{i}".encode() for i in range(20) + ) + assert configuration(tmp_path / "t") == { + STATS_COLUMNS_PROPERTY: "`v`,`__data_context`" + }