diff --git a/changelog.d/ITL-1173.md b/changelog.d/ITL-1173.md new file mode 100644 index 000000000..93c875807 --- /dev/null +++ b/changelog.d/ITL-1173.md @@ -0,0 +1,3 @@ +### Changed + +- **Faster job runs into `DeltaTableDatabase`.** An append commit now loads its table once and reads only the data files it added, instead of rebuilding the whole table's dataset (deltalake's own append cost still grows with the number of files). A lookup no longer walks every stored row. A table that does not exist yet is not reopened on every lookup. Stored rows, hashes, paths and table properties are unchanged. A table that another process creates during a run is seen from this database's next commit, as rows another process appends to an already-loaded table are. (ITL-1173) diff --git a/docs/dev/design_issues/ITL-1173-DS1.md b/docs/dev/design_issues/ITL-1173-DS1.md new file mode 100644 index 000000000..f48bd0ba8 --- /dev/null +++ b/docs/dev/design_issues/ITL-1173-DS1.md @@ -0,0 +1,60 @@ +## `src/orcapod/databases/` — PipelineJob run time quadratic in rows + +### DS1 — A job into a local `DeltaTableDatabase` spends most of its time on store bookkeeping +**Status:** resolved +**Severity:** medium +**Issue:** ITL-1173 + +Measured on `1531b8d5`, one function node `double(value)`, local disk, min of 3. A +10,000-row job took 16.25 s with the default `commit_every_rows=100`, against 1.13 s for +the pod without a job and 7.53 s for the job on `NoOpArrowDatabase`. At 100,000 rows it +took 346 s. Three costs grew with the rows stored: + +- **Each commit rebuilt the table's dataset.** A commit loaded the table three times (the + existence check, the writer's own reload when given a URI, the reload after the commit). + `_extend_snapshot` also built `to_pyarrow_dataset` over every data file to find the one + file the commit added. Per-commit cost rose from 8 ms to 37 ms over a 10k-row job. At + 100k rows commits took 192 of the 346 s, 131 s of it in the snapshot rebuild. +- **Each lookup walked every stored row.** Every lookup filters on the record id and on + `__data_context`. The data-context position list holds every row, and + `_row_index.matching_rows` intersected the position sets of all conditions. A lookup + cost 100 µs at 10k rows and 250 µs at 100k (50 s). +- **A missing table was reopened on every lookup.** Until a table exists, every lookup + tried to open it again, about 7 failed `DeltaTable()` opens per row. That cost 10.8 s + of a 10k-row job committing once at node end, which is why one commit at node end + (20.95 s) was slower than the default. + +Also seen: deltalake logs one "Skipping column ... binary field" line per binary column +per write (1,000 lines at 10k rows). Changing that means setting a Delta table property, +which is a developer decision (F4, open). + +**Fix:** +- `matching_rows` starts from the condition with the fewest rows and checks the others on + those rows only. +- The database family remembers tables it found absent until its next flush; + `table_exists` always checks again. +- An append commit is written through the table loaded for it. The snapshot is extended + by reading only the commit's added files, found from the table's add actions and read + with the format and filesystem of the dataset its rows came from. A partitioned table + reads its dataset as before. + +Stored rows, ids, hashes, paths, schemas and table properties are unchanged, and so is +the default commit interval (developer decision: it stays 100). The table is still loaded +afresh before each commit (developer decision): a handle refreshed in place could commit a +version gap into a store directory that was deleted and recreated. + +Measured with the fixes, min of 3 at 10k rows against `main` in the same session: +- default interval: 12.96 s, against 15.81 s; +- `commit_every_rows=1000`: 11.02 s, against 12.33 s; +- one commit at node end: 10.58 s, against 20.95 s before. + +At 100k rows (one run each): 159 s at the default interval (346 s before) and 117 s at +1000 (154 s before). Commits now cost 1.7 s of a 10k run (4.3 s before). The job's own +per-row work (7.5 s on `NoOpArrowDatabase`) is the largest remaining cost. + +The deltalake "binary field" log line is explained in `docs/troubleshooting.md`, with the +`RUST_LOG` filter that hides it. No table property was set; that decision (F4) is open. + +Follow-ups: ITL-1182 (`InMemoryArrowDatabase` lookups over the pending batch), ITL-1183 +(peak memory at 100k rows), ITL-1184 (compaction: the append cost of deltalake itself +still grows with the file count), ITL-1177 / ITL-1179 (per-row hashing cost). diff --git a/docs/dev/scripts/itl_1173/README.md b/docs/dev/scripts/itl_1173/README.md new file mode 100644 index 000000000..c7a850d85 --- /dev/null +++ b/docs/dev/scripts/itl_1173/README.md @@ -0,0 +1,36 @@ +# ITL-1173 measurement scripts + +These scripts measured why a `PipelineJob` over 10,000 rows into a local `DeltaTableDatabase` +is slow. The findings are in `docs/dev/design_issues/ITL-1173-DS1.md`. Each run starts a fresh +interpreter. Each measurement was repeated at least 3 times, and the minimum and spread are +reported. Run them from the repository root on an otherwise idle machine: Arrow is +multithreaded, and a loaded machine moves the numbers. + +| Script | What it measures | +|---|---| +| `bench_one.py ` | Wall time of one job run over N rows of `double(value)`. `commit_every` 0 means one commit at node end. Prints the run time, the same job run again in the same process, and for Delta the versions, files and bytes of each table. `--profile out.prof` writes a cProfile. | +| `instrument.py ` | Inclusive time and call counts of the named code paths (lookups, inserts, commits, hashing, deltalake calls), plus mean commit time per decile of the commits. In `memory` mode it also times a second run by a fresh job object over the same database. | +| `commit_curve.py ` | Raw deltalake cost per commit against the table version: load, append by URI, dataset build, `update_incremental`, and append through a kept handle. | +| `warn_test.py all ` | Counts deltalake's "Skipping column ... binary field" log lines on Orcapod's create, append, MERGE and schema-evolution paths, with and without a statistics table property, and for an existing table after the property is set. | + +The matrix used for the report: + +``` +for rep in 1 2 3; do + for n in 1000 10000; do + uv run python docs/dev/scripts/itl_1173/bench_one.py nojob $n 0 x + for every in 100 1000 0; do + rm -rf /tmp/itl1173_db + uv run python docs/dev/scripts/itl_1173/bench_one.py delta $n $every /tmp/itl1173_db + uv run python docs/dev/scripts/itl_1173/bench_one.py delta $n $every /tmp/itl1173_db # rerun, all stored + done + done +done +``` + +The in-memory rows of the report came from `instrument.py memory` at N = 1,000, 3,000 and +10,000 (commit every 100), and at 10,000 with 1,000 and node end. The run at 10,000 with node +end takes about 37 minutes. + +A raw `deltalake` dataset read in a script can hang the interpreter at exit (ITL-892). +`commit_curve.py` and `warn_test.py` end with `os._exit(0)` for that reason. diff --git a/docs/dev/scripts/itl_1173/bench_one.py b/docs/dev/scripts/itl_1173/bench_one.py new file mode 100644 index 000000000..b2328fdb6 --- /dev/null +++ b/docs/dev/scripts/itl_1173/bench_one.py @@ -0,0 +1,100 @@ +"""One ITL-1173 measurement in a fresh interpreter; prints one JSON line. + +Usage: uv run python docs/dev/scripts/itl_1173/bench_one.py [--rerun] [--profile out.prof] + mode: delta | memory | noop | nojob + commit_every: rows per commit, or 0 for "one commit at node end" +""" + +import json +import os +import sys +import time +from typing import Any + +import orcapod as op +from orcapod import sources + + +def table_stats(root: str) -> dict[str, dict[str, int]]: + """Commits (versions), data files and bytes per Delta table under root.""" + out = {} + for dirpath, dirnames, filenames in os.walk(root): + if "_delta_log" in dirnames: + log = os.path.join(dirpath, "_delta_log") + versions = [f for f in os.listdir(log) if f.endswith(".json")] + parquet = [f for f in filenames if f.endswith(".parquet")] + size = sum(os.path.getsize(os.path.join(dirpath, f)) for f in parquet) + rel = os.path.relpath(dirpath, root) + out[rel] = { + "versions": len(versions), + "files": len(parquet), + "bytes": size, + } + dirnames.remove("_delta_log") + return out + + +def main() -> None: + mode, n, every, dbdir = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), sys.argv[4] + rerun = "--rerun" in sys.argv + prof = ( + sys.argv[sys.argv.index("--profile") + 1] if "--profile" in sys.argv else None + ) + + source = sources.DictSource( + [{"key": i, "value": i} for i in range(n)], tag_columns="key" + ) + + @op.function_pod("result") + def double(value: int) -> int: + return 2 * value + + result: dict[str, Any] = {"mode": mode, "n": n, "every": every, "rerun": rerun} + if mode == "nojob": + t0 = time.perf_counter() + double.pod(source).run() + result["run_s"] = time.perf_counter() - t0 + print(json.dumps(result)) + return + + if mode == "delta": + # commit_every_rows must be < batch_size; 0 = one commit at node end. + rows_per_commit = every if every else n + 1 + database = op.databases.DeltaTableDatabase( + dbdir, batch_size=max(1000, rows_per_commit + 1) + ) + else: + rows_per_commit = every if every else n + 1 + database = ( + op.databases.NoOpArrowDatabase() + if mode == "noop" + else op.databases.InMemoryArrowDatabase() + ) + + job = op.PipelineJob("pipeline_v1", database=database) + with job: + double.pod(source) + + profiler = None + if prof: + import cProfile + + profiler = cProfile.Profile() + profiler.enable() + t0 = time.perf_counter() + job.run(commit_every_rows=rows_per_commit) + result["run_s"] = time.perf_counter() - t0 + if profiler and prof: + profiler.disable() + profiler.dump_stats(prof) + # Same job again in the same process (state rule: held, nothing runs). + t0 = time.perf_counter() + job.run(commit_every_rows=rows_per_commit) + result["same_job_second_run_s"] = time.perf_counter() - t0 + if mode == "delta": + result["tables"] = table_stats(dbdir) + print(json.dumps(result)) + + +if __name__ == "__main__": + main() diff --git a/docs/dev/scripts/itl_1173/commit_curve.py b/docs/dev/scripts/itl_1173/commit_curve.py new file mode 100644 index 000000000..779dc8acf --- /dev/null +++ b/docs/dev/scripts/itl_1173/commit_curve.py @@ -0,0 +1,95 @@ +"""ITL-1173: raw deltalake cost per commit against the table version. + +Appends 300 batches of 100 rows (the shape of a function node's pipeline table) and +times, at selected versions: a fresh table load, an append given the URI, a load plus +``to_pyarrow_dataset``, and ``update_incremental`` of a kept handle. Then times appends +written through a kept handle. + +Usage: uv run python docs/dev/scripts/itl_1173/commit_curve.py +""" + +from __future__ import annotations + +import json +import os +import sys +import time + +import deltalake +import pyarrow as pa + +SAMPLED = (1, 2, 10, 50, 99, 100, 101, 150, 200, 250, 299, 300) + + +def batch(k: int) -> pa.Table: + ids = range(k * 100, k * 100 + 100) + return pa.table( + { + "__record_id": pa.array( + [i.to_bytes(32, "big") for i in ids], pa.large_binary() + ), + "h1": pa.array([i.to_bytes(40, "big") for i in ids], pa.large_binary()), + "h2": pa.array([i.to_bytes(40, "big") for i in ids], pa.large_binary()), + "v": pa.array(ids, pa.int64()), + "ctx": pa.array(["std:v1:default"] * 100, pa.large_string()), + } + ) + + +def ms(seconds: float) -> float: + return round(1e3 * seconds, 1) + + +def main() -> None: + d = sys.argv[1] + if os.path.exists(d) and os.listdir(d): + sys.exit(f"{d} is not empty: give an empty or missing directory") + deltalake.write_deltalake(d, batch(0), mode="overwrite") + handle = deltalake.DeltaTable(d) + for k in range(1, 301): + t0 = time.perf_counter() + loaded = deltalake.DeltaTable(d) + t_load = time.perf_counter() - t0 + t0 = time.perf_counter() + deltalake.write_deltalake(d, batch(k), mode="append") + t_write = time.perf_counter() - t0 + t0 = time.perf_counter() + _ = deltalake.DeltaTable(d).to_pyarrow_dataset() + t_dataset = time.perf_counter() - t0 + t0 = time.perf_counter() + handle.update_incremental() + t_incremental = time.perf_counter() - t0 + if k in SAMPLED: + print( + json.dumps( + { + "version": loaded.version() + 1, + "load_ms": ms(t_load), + "write_append_uri_ms": ms(t_write), + "load+to_pyarrow_dataset_ms": ms(t_dataset), + "update_incremental_ms": ms(t_incremental), + } + ), + flush=True, + ) + kept = deltalake.DeltaTable(d) + times: list[float] = [] + for k in range(301, 321): + t0 = time.perf_counter() + deltalake.write_deltalake(kept, batch(k), mode="append") + times.append(time.perf_counter() - t0) + print( + "append through a kept handle, min / mean ms:", + ms(min(times)), + ms(sum(times) / len(times)), + ) + print( + "checkpoints:", + sorted(f for f in os.listdir(f"{d}/_delta_log") if not f.endswith(".json")), + flush=True, + ) + os._exit(0) # the dataset reads above can hang interpreter exit (ITL-892) + + +if __name__ == "__main__": + main() diff --git a/docs/dev/scripts/itl_1173/instrument.py b/docs/dev/scripts/itl_1173/instrument.py new file mode 100644 index 000000000..a38b8f2a7 --- /dev/null +++ b/docs/dev/scripts/itl_1173/instrument.py @@ -0,0 +1,214 @@ +"""ITL-1173: inclusive wall time of named code paths during one job run (no cProfile). + +Usage: uv run python docs/dev/scripts/itl_1173/instrument.py +Prints a JSON object: {name: [calls, inclusive seconds]} plus run_s. +""" + +import json +import sys +import time +from typing import Any +from collections import defaultdict + +import deltalake +import deltalake.writer.writer as dl_writer + +import orcapod as op +from orcapod import sources +from orcapod.core import cached_function_pod, data_function, result_cache +from orcapod.core.nodes import function_node +from orcapod.databases import delta_lake_databases as dldb +from orcapod.databases import in_memory_databases as imdb +from orcapod.hashing import arrow_hashers + +STATS: dict[str, list[float]] = defaultdict(lambda: [0, 0.0]) +_depth: dict[str, int] = defaultdict(int) + + +def wrap(owner, attr, name): + orig = getattr(owner, attr) + is_static = isinstance(owner.__dict__.get(attr), staticmethod) + is_class = isinstance(owner.__dict__.get(attr), classmethod) + func = owner.__dict__[attr].__func__ if (is_static or is_class) else orig + + def timed(*a, **k): + # Count only the outermost call of a name (recursion / re-entry). + _depth[name] += 1 + t0 = time.perf_counter() + try: + return func(*a, **k) + finally: + _depth[name] -= 1 + if _depth[name] == 0: + s = STATS[name] + s[0] += 1 + s[1] += time.perf_counter() - t0 + + if is_static: + setattr(owner, attr, staticmethod(timed)) + elif is_class: + setattr(owner, attr, classmethod(timed)) + else: + setattr(owner, attr, timed) + + +FJN = function_node.FunctionJobNode +D = dldb.DeltaTableDatabase +M = imdb.InMemoryArrowDatabase +for owner, attr, name in [ + (FJN, "execute", "node.execute"), + (FJN, "compute_base_entry_id", "hash.base_entry_id"), + (FJN, "compute_pipeline_entry_id", "hash.pipeline_entry_id"), + (FJN, "_process_data_internal", "row.process_data_internal"), + (FJN, "_add_pipeline_record", "row.add_pipeline_record"), + (FJN, "_commit_pending_writes", "commit.node_commit"), + (FJN, "get_cached_results", "node.get_cached_results"), + ( + cached_function_pod.CachedFunctionPod, + "process_data", + "row.cached_pod.process_data", + ), + (result_cache.ResultCache, "lookup", "row.result_cache.lookup"), + (result_cache.ResultCache, "store", "row.result_cache.store"), + (data_function.PythonDataFunction, "call", "row.user_function_call"), + (arrow_hashers.StarfixArrowHasher, "hash_table", "hash.arrow_hash_table(all)"), + (D, "get_records_with_column_value", "db.get_records_with_column_value"), + (D, "get_record_by_id", "db.get_record_by_id"), + (D, "get_records_by_ids", "db.get_records_by_ids"), + (D, "add_records", "db.add_records"), + (D, "_flush_by_key", "commit.flush_by_key"), + (D, "_extend_snapshot", "commit.extend_snapshot"), + (D, "_snapshot_for_commit", "commit.snapshot_for_commit"), + (D, "_scan_ids", "db.scan_ids"), + (D, "_scan_rows", "db.scan_rows"), + (M, "get_records_with_column_value", "db.get_records_with_column_value"), + (M, "add_records", "db.add_records"), + (M, "flush", "commit.memory_flush"), + (M, "_filter_committed", "mem.filter_committed(index)"), + (M, "_filter_existing_records", "mem.filter_existing_records"), +]: + wrap(owner, attr, name) + +# deltalake internals, as called from Orcapod's module. +_orig_write = deltalake.write_deltalake + + +def _timed_write(*a, **k): + t0 = time.perf_counter() + try: + return _orig_write(*a, **k) + finally: + s = STATS["commit.write_deltalake"] + s[0] += 1 + s[1] += time.perf_counter() - t0 + + +dldb.deltalake.write_deltalake = _timed_write # module attribute used by dldb +_orig_dt_init = deltalake.DeltaTable.__init__ + + +def _timed_dt_init(self, *a, **k): + t0 = time.perf_counter() + try: + return _orig_dt_init(self, *a, **k) + finally: + s = STATS["delta.DeltaTable()"] + s[0] += 1 + s[1] += time.perf_counter() - t0 + + +deltalake.DeltaTable.__init__ = _timed_dt_init +_orig_merge = deltalake.DeltaTable.merge + + +def _timed_merge(self, *a, **k): + t0 = time.perf_counter() + try: + return _orig_merge(self, *a, **k) + finally: + s = STATS["commit.merge(builder)"] + s[0] += 1 + s[1] += time.perf_counter() - t0 + + +deltalake.DeltaTable.merge = _timed_merge +_ = dl_writer + +# Per-commit sequence: (table kind, seconds) for every _flush_by_key call, in order. +SEQ: list[tuple[str, float]] = [] +_inner_flush = D._flush_by_key + + +def _seq_flush(self, record_key): + t0 = time.perf_counter() + try: + return _inner_flush(self, record_key) + finally: + kind = "result" if "/_result/" in record_key else "pipeline" + SEQ.append((kind, time.perf_counter() - t0)) + + +D._flush_by_key = _seq_flush + + +def main() -> None: + mode, n, every, dbdir = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), sys.argv[4] + source = sources.DictSource( + [{"key": i, "value": i} for i in range(n)], tag_columns="key" + ) + + @op.function_pod("result") + def double(value: int) -> int: + return 2 * value + + rows = every if every else n + 1 + if mode == "delta": + database = op.databases.DeltaTableDatabase( + dbdir, batch_size=max(1000, rows + 1) + ) + else: + database = ( + op.databases.NoOpArrowDatabase() + if mode == "noop" + else op.databases.InMemoryArrowDatabase() + ) + job = op.PipelineJob("pipeline_v1", database=database) + with job: + double.pod(source) + STATS.clear() + t0 = time.perf_counter() + job.run(commit_every_rows=rows) + out: dict[str, Any] = { + "run_s": time.perf_counter() - t0, + "mode": mode, + "n": n, + "every": every, + } + if mode == "memory": + out["stats_first"] = { + k: [int(v[0]), round(v[1], 4)] for k, v in sorted(STATS.items()) + } + STATS.clear() + job2 = op.PipelineJob("pipeline_v1", database=database) + with job2: + double.pod(source) + t0 = time.perf_counter() + job2.run(commit_every_rows=rows) + out["rerun_fresh_job_same_db_s"] = time.perf_counter() - t0 + out["stats"] = {k: [int(v[0]), round(v[1], 4)] for k, v in sorted(STATS.items())} + for kind in ("result", "pipeline"): + seq = [s for k, s in SEQ if k == kind] + if seq: + # Mean commit time per decile of the commit sequence. + dec = max(1, len(seq) // 10) + out[f"commit_curve_{kind}_ms"] = [ + round(1000 * sum(seq[i : i + dec]) / len(seq[i : i + dec]), 1) + for i in range(0, len(seq), dec) + ] + out[f"commits_{kind}"] = len(seq) + SEQ.clear() + print(json.dumps(out)) + + +if __name__ == "__main__": + main() diff --git a/docs/dev/scripts/itl_1173/warn_test.py b/docs/dev/scripts/itl_1173/warn_test.py new file mode 100644 index 000000000..3783c60fc --- /dev/null +++ b/docs/dev/scripts/itl_1173/warn_test.py @@ -0,0 +1,171 @@ +"""ITL-1173: does delta.dataSkippingStatsColumns stop the binary-field warnings on +Orcapod's own create / append / merge / schema-evolution paths, and for an +EXISTING table? Each scenario runs in a subprocess so stderr can be counted. + +Usage: uv run python docs/dev/scripts/itl_1173/warn_test.py (driver: warn_test.py all ) +""" + +import os +import subprocess +import sys +from typing import Any + +import pyarrow as pa + + +def patch_create(property_on: bool) -> None: + """Make Orcapod's create path pass the statistics property (experiment only).""" + if not property_on: + return + from orcapod.databases import delta_lake_databases as dldb + + orig = dldb.deltalake.write_deltalake + + def write(table_uri, data, *a, **k): + """Add the statistics property to a create (``mode="overwrite"``) write. + + Replaces ``write_deltalake`` in Orcapod's module for the whole process, so + every table this scenario creates gets the property; appends are unchanged. + """ + if k.get("mode") == "overwrite": + mode = os.environ.get("ITL1173_PROP", "stats_cols") + if mode == "num0": + k["configuration"] = {"delta.dataSkippingNumIndexedCols": "0"} + else: + + def stat_ok(t): + """Whether deltalake keeps min/max for a column of type ``t``. + + Experiment only: a column name with a comma breaks the + property even when quoted, and is not handled here. + """ + t = t.storage_type if isinstance(t, pa.ExtensionType) else t + return ( + pa.types.is_integer(t) + or pa.types.is_floating(t) + or pa.types.is_string(t) + or pa.types.is_large_string(t) + or pa.types.is_boolean(t) + or pa.types.is_date(t) + or pa.types.is_timestamp(t) + or pa.types.is_decimal(t) + ) + + cols = [ + "`" + f.name.replace("`", "``") + "`" + for f in data.schema + if stat_ok(f.type) + ] + k["configuration"] = {"delta.dataSkippingStatsColumns": ",".join(cols)} + return orig(table_uri, data, *a, **k) + + dldb.deltalake.write_deltalake = write + + +def rows(ids, extra=False): + """Rows keyed by ``__record_id``: a binary hash ``h``, an int ``v``, and with + ``extra`` one more binary and one more int column (schema evolution).""" + cols: dict[str, pa.Array[Any]] = { + "__record_id": pa.array([f"id{i}".encode() for i in ids], pa.large_binary()), + "h": pa.array([f"h{i}".encode() for i in ids], pa.large_binary()), + "v": pa.array(ids, pa.int64()), + } + if extra: + cols["new_bin"] = pa.array([b"x"] * len(ids), pa.large_binary()) + cols["new_int"] = pa.array(ids, pa.int64()) + return pa.Table.from_arrays(list(cols.values()), names=list(cols)) + + +def scenario(name: str, d: str) -> None: + """Run one step on the table under ``d`` and print its version and properties. + + The steps build on each other and must run in the driver's order (create, + append, merge, evolve, append_after_evolve, then set_property_on_existing and + append_new on the table created without the property). A ``_prop`` suffix + creates the table with the statistics property. + """ + from orcapod.databases import DeltaTableDatabase + + prop = name.endswith("_prop") + patch_create(prop) + db = DeltaTableDatabase(d) + p = ("t",) + step = name.removesuffix("_prop") + if step == "create": + db.add_records(p, rows(range(0, 10)), flush=True) + elif step == "append": + db.add_records(p, rows(range(10, 20)), flush=True) + elif step == "merge": + # One stored id among the pending ones forces the MERGE path. + db.add_records(p, rows(range(15, 30)), skip_duplicates=False, flush=True) + elif step == "evolve": + # A new column (one binary, one int): schema evolution goes through MERGE. + db.add_records(p, rows(range(30, 40), extra=True), flush=True) + elif step == "append_after_evolve": + db.add_records(p, rows(range(40, 50), extra=True), flush=True) + elif step == "append_new": + db.add_records(p, rows(range(50, 60), extra=True), flush=True) + elif step == "set_property_on_existing": + import deltalake + + dt = deltalake.DeltaTable(os.path.join(d, "t")) + cols = [ + f.name + for f in pa.schema(dt.schema().to_arrow()) # pyright: ignore[reportArgumentType] # an arro3 schema, read through the Arrow C interface + if not (pa.types.is_binary(f.type) or pa.types.is_large_binary(f.type)) + ] + dt.alter.set_table_properties( + {"delta.dataSkippingStatsColumns": ",".join(cols)} + ) + import deltalake + + dt = deltalake.DeltaTable(os.path.join(d, "t")) + print( + f"version={dt.version()} config={dt.metadata().configuration} rows={dt.to_pyarrow_table().num_rows}" + ) + + +def run(step: str, d: str) -> str: + """Run ``step`` in a subprocess and report its warning lines. + + Returns one line: the step, the number of "Skipping column" lines and the + columns they name, the step's output, and the tail of stderr if it failed + (or ``TIMEOUT``). Failures are reported, never raised. + """ + try: + r = subprocess.run( + [sys.executable, __file__, step, d], + capture_output=True, + text=True, + timeout=60, + ) + except subprocess.TimeoutExpired: + return f"{step:28s} TIMEOUT" + warns = [ln for ln in r.stderr.splitlines() if "Skipping column" in ln] + cols = sorted({ln.split("Skipping column ")[1].split(" ")[0] for ln in warns}) + return f"{step:28s} warnings={len(warns)} {cols} | {r.stdout.strip()} {r.stderr.strip()[-300:] if r.returncode else ''}" + + +if __name__ == "__main__": + if sys.argv[1] == "all": + base = sys.argv[2] + for name in ("t_none", "t_prop"): + if os.path.exists(os.path.join(base, name)): + sys.exit(f"{os.path.join(base, name)} exists: give a fresh directory") + for variant in ("", "_prop"): + d = os.path.join(base, f"t{variant or '_none'}") + print(f"--- table created {'WITH' if variant else 'WITHOUT'} the property") + for step in ("create", "append", "merge", "evolve", "append_after_evolve"): + print(run(step + variant, d)) + d = os.path.join(base, "t_none") + print("--- existing table (created without): set the property, then append") + print(run("set_property_on_existing", d)) + print(run("append_new", d)) + else: + scenario(sys.argv[1], sys.argv[2]) + if os.environ.get("ITL1173_HARD_EXIT", "1") == "1": + # Interpreter exit after a MERGE hangs in Arrow's ThreadPool shutdown + # (seen with deltalake 1.0.2 / pyarrow 23.0.1); skip it here. + sys.stdout.flush() + sys.stderr.flush() + os._exit(0) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 3c577d79c..26d81aeab 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -238,7 +238,17 @@ version to regenerate them. ## Messages you can ignore - `WARN deltalake_core::writer::stats] Skipping column ... because it's a binary field.` - whenever a `DeltaTableDatabase` is written to. + whenever a `DeltaTableDatabase` is written to: one line per binary column (record ids and + hashes) per commit. The deltalake library keeps no minimum and maximum statistics for binary + columns and says so on every write; nothing is wrong with the table. To hide only this line, + set `RUST_LOG` in the environment before Python starts, keeping deltalake's other warnings: + + ``` + RUST_LOG="warn,deltalake_core::writer::stats=error" python my_pipeline.py + ``` + + deltalake reads `RUST_LOG` once, when it is first imported, so setting it later in the + same process (for example with `os.environ`) has no effect. ## A script finishes its work but the process never exits diff --git a/src/orcapod/databases/_row_index.py b/src/orcapod/databases/_row_index.py index b9da1f6e8..a8223514d 100644 --- a/src/orcapod/databases/_row_index.py +++ b/src/orcapod/databases/_row_index.py @@ -138,9 +138,12 @@ def matching_rows( """Return the rows of ``key_index.table`` where every column equals its value. The pairs that can use a hash index select the rows at the intersection of their - indexes' positions. When every pair can, those rows are the result; when only some - can, just those rows are filtered by ``expression``; when none can, the whole table - is. Rows come back in table order. + indexes' positions: the positions of the pair with the fewest rows, kept where the + other indexed pairs' columns hold their values, so a lookup costs time in the + smallest match, not in the rows a common value occupies. When every pair can use + an index, those rows are the result; when only some can, just those rows are + filtered by ``expression``; when none can, the whole table is. Rows come back in + table order. Args: key_index: The indexes of the table to search. @@ -160,11 +163,23 @@ def matching_rows( if not indexed: return table.filter(expression) - matches: set[int] | None = None - for column, value in indexed: - found = key_index.column(column).get(value, []) - matches = set(found) if matches is None else matches.intersection(found) - candidates = rows_at(table, sorted(matches or ())) + # Start from the condition with the fewest rows and check the others on those rows + # only: a condition most rows meet (the data context column holds one value in + # every row) must not cost a walk over its positions on every lookup (ITL-1173). + found = sorted( + ( + (column, value, key_index.column(column).get(value, [])) + for column, value in indexed + ), + key=lambda condition: len(condition[2]), + ) + positions: Sequence[int] = found[0][2] + for column, value, _ in found[1:]: + if not positions: + break + values = rows_at(table.select([column]), positions).column(column).to_pylist() + positions = [p for p, v in zip(positions, values, strict=True) if v == value] + candidates = rows_at(table, positions) if len(indexed) == len(pairs): return candidates return candidates.filter(expression) diff --git a/src/orcapod/databases/delta_lake_databases.py b/src/orcapod/databases/delta_lake_databases.py index 19dd15fea..980b04e84 100644 --- a/src/orcapod/databases/delta_lake_databases.py +++ b/src/orcapod/databases/delta_lake_databases.py @@ -43,12 +43,14 @@ import pyarrow as pa import pyarrow.compute as pc import pyarrow.dataset as pa_ds + import pyarrow.fs as pa_fs from upath import UPath else: deltalake = LazyModule("deltalake") pa = LazyModule("pyarrow") pl = LazyModule("polars") pc = LazyModule("pyarrow.compute") + pa_ds = LazyModule("pyarrow.dataset") # Module-level logger logger = logging.getLogger(__name__) @@ -438,6 +440,9 @@ class _CommittedSnapshot: reads them, or ``None`` until first needed. files: The data files ``rows`` was read from (set together with ``rows``). dataset_schema: The schema of the dataset ``rows`` was read from. + dataset_format: The file format of that dataset, which reads a data file + the way the whole table is read (ITL-1173). + dataset_filesystem: The filesystem the dataset's ``files`` are paths in. """ version: int @@ -445,6 +450,8 @@ class _CommittedSnapshot: rows: KeyIndex | None = None files: frozenset[str] = frozenset() dataset_schema: "pa.Schema | None" = None + dataset_format: "pa_ds.FileFormat | None" = None + dataset_filesystem: "pa_fs.FileSystem | None" = None class DeltaTableDatabase: @@ -480,6 +487,7 @@ def __init__( _shared_delta_table_cache: "dict[str, deltalake.DeltaTable] | None" = None, _shared_snapshots: "dict[str, _CommittedSnapshot] | None" = None, _shared_pending_indexes: "dict[str, KeyIndex] | None" = None, + _shared_absent_tables: "set[str] | None" = None, ): self._root_uri, self._storage_options = parse_base_path( base_path, storage_options @@ -531,6 +539,14 @@ def __init__( self._pending_indexes: dict[str, KeyIndex] = ( _shared_pending_indexes if _shared_pending_indexes is not None else {} ) + # Record keys whose table was found absent (ITL-1173), so lookups do not try + # to open it again on every row. Family-wide like the loaded tables, and + # forgotten on any flush of the family and on a flush of the key: a table + # another process creates is seen after this database's next flush, as rows + # it appends to a loaded table are. ``table_exists`` always checks again. + self._absent_tables: set[str] = ( + _shared_absent_tables if _shared_absent_tables is not None else set() + ) def _clear_pending(self): """Clear all pending state.""" @@ -637,6 +653,8 @@ def _get_delta_table( # Check cache first if dt := self._delta_table_cache.get(record_key): return dt + if record_key in self._absent_tables: + return None try: # Try to load existing table @@ -647,7 +665,8 @@ def _get_delta_table( logger.debug(f"Loaded existing Delta table for {record_key}") return delta_table except deltalake.exceptions.TableNotFoundError: - # Table doesn't exist + # Table doesn't exist: remembered until the next flush (ITL-1173). + self._absent_tables.add(record_key) return None except Exception as e: # The table exists but could not be loaded: never report it as absent. @@ -802,6 +821,8 @@ def table_exists(self, record_path: tuple[str, ...]) -> bool: Returns: ``True`` if a table exists at the given path, ``False`` otherwise. """ + # Checks again even when the table was found absent before (ITL-1173). + self._absent_tables.discard(self._get_record_key(record_path)) return self._get_delta_table(record_path) is not None def _ensure_record_id_column( @@ -898,7 +919,7 @@ def _scan_ids(cls, delta_table: deltalake.DeltaTable) -> set[bytes]: @staticmethod def _scan_rows( delta_table: deltalake.DeltaTable, - ) -> tuple[pa.Table, frozenset[str], pa.Schema]: + ) -> tuple[pa.Table, "pa_ds.FileSystemDataset"]: """Read every row of ``delta_table`` as ``get_all_records`` reads it. Args: @@ -906,8 +927,7 @@ def _scan_rows( Returns: The rows (view types normalized: PyArrow has no comparison kernels for - ``string_view``, ENG-601), the data - files they were read from and the dataset's schema. + ``string_view``, ENG-601) and the dataset they were read from. Raises: Exception: The backend's error when the table's data cannot be read. @@ -920,7 +940,17 @@ def _scan_rows( table = from_delta_layout( arrow_utils.normalize_table_view_types(dataset.to_table()) ) - return table.combine_chunks(), frozenset(dataset.files), dataset.schema + return table.combine_chunks(), dataset + + @staticmethod + def _describe_read( + snapshot: _CommittedSnapshot, dataset: "pa_ds.FileSystemDataset" + ) -> None: + """Record in ``snapshot`` the dataset its rows were read from.""" + snapshot.files = frozenset(dataset.files) + snapshot.dataset_schema = dataset.schema + snapshot.dataset_format = dataset.format + snapshot.dataset_filesystem = dataset.filesystem def _stored_ids(self, record_path: tuple[str, ...]) -> set[bytes]: """Return the record ids stored at ``record_path`` (in the loaded version). @@ -954,15 +984,14 @@ def _stored_rows(self, record_path: tuple[str, ...]) -> KeyIndex | None: snapshot = self._snapshot(self._get_record_key(record_path), delta_table) if snapshot.rows is None: try: - table, files, schema = self._scan_rows(delta_table) + table, dataset = self._scan_rows(delta_table) except TypeIdentityError: # The table is intact; the registered class differs (ITL-881). raise except Exception as e: raise self._read_failed(record_path) from e snapshot.rows = KeyIndex(table) - snapshot.files = files - snapshot.dataset_schema = schema + self._describe_read(snapshot, dataset) return snapshot.rows def add_record( @@ -1637,6 +1666,8 @@ def flush(self) -> None: RuntimeError: If batches failed with other errors only, naming the failed batches, with the first failure as its ``__cause__``. """ + # A flush is where this database looks at the store again (ITL-1173). + self._absent_tables.clear() errors: list[tuple[str, Exception]] = [] for record_key in list(self._pending_batches.keys()): try: @@ -1678,6 +1709,7 @@ def _flush_by_key(self, record_key: str) -> None: without applying any instance-level prefix. """ logger.debug("Flushing triggered!!") + self._absent_tables.discard(record_key) if ( record_key not in self._pending_batches @@ -1731,6 +1763,9 @@ def _flush_by_key(self, record_key: str) -> None: table_uri, abs_path_components ) from load_error + # Set when the rows were appended through ``delta_table``, which the + # writer then moves to the version it committed (ITL-1173). + appended = False if delta_table is None: # TODO: reconsider mode="overwrite" here deltalake.write_deltalake( @@ -1788,12 +1823,12 @@ def _flush_by_key(self, record_key: str) -> None: # Every pending id is new in the version just loaded and the rows # have the stored schema: append (ITL-919). A MERGE reads the whole # stored table on each commit; it remains for every other batch. + # Written through the table just loaded: given a URI, the writer + # would load the table again (ITL-1173). deltalake.write_deltalake( - table_uri, - combined_table, - mode="append", - storage_options=self._storage_options or None, + delta_table, combined_table, mode="append" ) + appended = True else: # Some pending ids are stored already: insert only the others. delta_table.merge( @@ -1819,8 +1854,12 @@ def _flush_by_key(self, record_key: str) -> None: # The rows are committed: nothing below may put them back in the queue. try: - committed = deltalake.DeltaTable( - table_uri, storage_options=self._storage_options or None + committed = ( + delta_table + if appended and delta_table is not None + else deltalake.DeltaTable( + table_uri, storage_options=self._storage_options or None + ) ) self._delta_table_cache[record_key] = committed self._snapshots[record_key] = self._extend_snapshot( @@ -1955,9 +1994,12 @@ def _extend_snapshot( When ``committed`` is the version right after ``base`` (no other writer committed in between), its ids are ``base``'s plus the inserted ones, and its rows, if ``base`` has them, are ``base``'s plus the rows of the data - files the commit added, read as ``get_all_records`` reads them. Anything - else (a skipped version, a removed file, a changed schema) leaves that - part to be read again on first use. + files the commit added, read as ``get_all_records`` reads them. Only those + files are read, with the format and filesystem of the dataset ``base``'s + rows came from (ITL-1173); a partitioned table, whose partition values are + not in its files, reads its dataset instead. Anything else (a skipped + version, a removed file, a changed schema) leaves that part to be read + again on first use. Args: base: The snapshot of the version the commit built on. @@ -1972,8 +2014,66 @@ def _extend_snapshot( return _CommittedSnapshot(version) assert base.ids is not None snapshot = _CommittedSnapshot(version, ids=base.ids | inserted_ids) - if base.rows is None: + if ( + base.rows is None + or base.dataset_schema is None + or base.dataset_format is None + or base.dataset_filesystem is None + ): return snapshot + if committed.metadata().partition_columns: + # A partition value is not in the data file: read the table's dataset. + return self._extend_snapshot_from_dataset(base, snapshot, committed) + # Read only the files this commit added, as the table's dataset reads them, + # never building that dataset over every file (ITL-1173). + paths = cast( + list[str], + # A record batch in deltalake 1.0.2, a table from 1.5.0: pa.table takes both. + pa.table(committed.get_add_actions(flatten=False)) + .column("path") + .to_pylist(), + ) + files = frozenset(paths) + # The schema deltalake gives the table's dataset (``to_pyarrow_dataset``). + schema = pa.schema( + committed.schema().to_arrow(as_large_types=True) # pyright: ignore[reportArgumentType] # an arro3 schema; pyarrow reads it through the Arrow C interface + ) + if not base.files <= files or not schema.equals(base.dataset_schema): + return snapshot + added = [path for path in paths if path not in base.files] + dataset = pa_ds.FileSystemDataset( + [ + base.dataset_format.make_fragment( + path, filesystem=base.dataset_filesystem + ) + for path in added + ], + base.dataset_schema, + base.dataset_format, + base.dataset_filesystem, + ) + new_rows = from_delta_layout( + arrow_utils.normalize_table_view_types(dataset.to_table()) + ) + return self._append_rows(base, snapshot, new_rows, files) + + def _extend_snapshot_from_dataset( + self, + base: _CommittedSnapshot, + snapshot: _CommittedSnapshot, + committed: deltalake.DeltaTable, + ) -> _CommittedSnapshot: + """Extend ``base``'s rows into ``snapshot`` by reading ``committed``'s dataset. + + Args: + base: The snapshot of the version the commit built on, with its rows. + snapshot: The new snapshot, with its ids. + committed: The table reloaded after the commit. + + Returns: + ``snapshot``, with the rows when they could be extended. + """ + assert base.rows is not None dataset = cast( "pa_ds.FileSystemDataset", delta_to_pyarrow_dataset(committed, as_large_types=True), @@ -1998,13 +2098,40 @@ def _extend_snapshot( else dataset.schema.empty_table() ) new_rows = from_delta_layout(new_rows) + snapshot = self._append_rows(base, snapshot, new_rows, files) + if snapshot.rows is not None: + self._describe_read(snapshot, dataset) + return snapshot + + @staticmethod + def _append_rows( + base: _CommittedSnapshot, + snapshot: _CommittedSnapshot, + new_rows: pa.Table, + files: frozenset[str], + ) -> _CommittedSnapshot: + """Give ``snapshot`` ``base``'s rows followed by ``new_rows``, read from ``files``. + + Args: + base: The snapshot of the version the commit built on, with its rows. + snapshot: The new snapshot, with its ids. + new_rows: The rows of the data files the commit added. + files: Every data file of the new version. + + Returns: + ``snapshot``, without rows when ``new_rows`` has another schema than + ``base``'s rows. + """ + assert base.rows is not None if not new_rows.schema.equals(base.rows.table.schema): return snapshot rows = base.rows rows.append(new_rows, compact_tail(pa.concat_tables([rows.table, new_rows]))) snapshot.rows = rows snapshot.files = files - snapshot.dataset_schema = dataset.schema + snapshot.dataset_schema = base.dataset_schema + snapshot.dataset_format = base.dataset_format + snapshot.dataset_filesystem = base.dataset_filesystem return snapshot @staticmethod @@ -2140,6 +2267,7 @@ def at(self, *path_components: str) -> "DeltaTableDatabase": _shared_delta_table_cache=root_instance._delta_table_cache, _shared_snapshots=root_instance._snapshots, _shared_pending_indexes=root_instance._pending_indexes, + _shared_absent_tables=root_instance._absent_tables, ) def list_sources(self) -> list[tuple[str, ...]]: diff --git a/tests/test_databases/test_delta_absent_tables.py b/tests/test_databases/test_delta_absent_tables.py new file mode 100644 index 000000000..7cdb6ee8d --- /dev/null +++ b/tests/test_databases/test_delta_absent_tables.py @@ -0,0 +1,151 @@ +"""A ``DeltaTableDatabase`` remembers a table it found absent until its next flush (ITL-1173). + +Until a table exists, a job's per-row lookups would each try to open it again. The family +of a database (every handle from ``at()``) keeps the record keys it found absent and +answers lookups for them without opening the table, until any flush of the family (a +commit) or a flush of that table. ``table_exists`` always checks again. A table another +process creates is therefore seen after this database's next flush, as rows another +process appends to an already-loaded table are; a flush to a table believed absent +loads it first and never replaces it. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import cast + +import deltalake +import pyarrow as pa +import pytest + +from orcapod.databases import DeltaTableDatabase + +PATH = ("table",) + + +def _rows(ids: list[str], values: list[int]) -> pa.Table: + return pa.table( + { + "__record_id": pa.array([i.encode() for i in ids], type=pa.large_binary()), + "v": pa.array(values, type=pa.int64()), + } + ) + + +@pytest.fixture +def table_opens(monkeypatch: pytest.MonkeyPatch) -> list[str]: + """Record every attempt to open a Delta table (its URI).""" + opens: list[str] = [] + original = deltalake.DeltaTable.__init__ + + def counting_init(self, table_uri, *args, **kwargs): + opens.append(str(table_uri)) + original(self, table_uri, *args, **kwargs) + + monkeypatch.setattr(deltalake.DeltaTable, "__init__", counting_init) + return opens + + +def _values(table: pa.Table | None) -> list[int]: + return [] if table is None else sorted(cast(list[int], table["v"].to_pylist())) + + +def test_lookups_on_an_absent_table_open_it_once( + tmp_path: Path, table_opens: list[str] +) -> None: + db = DeltaTableDatabase(tmp_path) + for i in range(50): + assert db.get_record_by_id(PATH, f"r{i}".encode()) is None + assert db.get_records_with_column_value(PATH, {"v": i}) is None + assert db.get_all_records(PATH) is None + assert len(table_opens) == 1 + + +def test_lookups_on_an_absent_table_open_it_once_across_the_family( + tmp_path: Path, table_opens: list[str] +) -> None: + root = DeltaTableDatabase(tmp_path) + scoped = root.at("scope") + for i in range(20): + assert scoped.get_record_by_id(PATH, f"r{i}".encode()) is None + assert root.get_record_by_id(("scope", *PATH), f"r{i}".encode()) is None + assert len(table_opens) == 1 + + +def test_a_family_flush_forgets_absent_tables( + tmp_path: Path, table_opens: list[str] +) -> None: + db = DeltaTableDatabase(tmp_path) + assert db.get_all_records(PATH) is None + db.flush() + assert db.get_all_records(PATH) is None + assert len(table_opens) == 2 + + +def test_create_after_miss_is_read_back(tmp_path: Path) -> None: + """The family's own create replaces the absent entry, through any handle.""" + root = DeltaTableDatabase(tmp_path) + scoped = root.at("scope") + assert root.get_all_records(("scope", *PATH)) is None + assert scoped.get_all_records(PATH) is None + + scoped.add_records(PATH, _rows(["a", "b"], [1, 2]), flush=True) + + assert _values(scoped.get_all_records(PATH)) == [1, 2] + assert _values(root.get_all_records(("scope", *PATH))) == [1, 2] + assert _values(root.get_record_by_id(("scope", *PATH), b"a")) == [1] + + +def test_pending_rows_of_an_absent_table_are_read(tmp_path: Path) -> None: + db = DeltaTableDatabase(tmp_path) + assert db.get_all_records(PATH) is None + db.add_records(PATH, _rows(["a"], [1])) + assert _values(db.get_record_by_id(PATH, b"a")) == [1] + assert _values(db.get_all_records(PATH)) == [1] + + +def test_table_another_database_creates_is_seen_after_the_next_flush( + tmp_path: Path, +) -> None: + ours = DeltaTableDatabase(tmp_path) + assert ours.get_all_records(PATH) is None + + other = DeltaTableDatabase(tmp_path) # stands in for another process + other.add_records(PATH, _rows(["x"], [10]), flush=True) + + # Remembered as absent until this database flushes. + assert ours.get_all_records(PATH) is None + ours.flush() + assert _values(ours.get_all_records(PATH)) == [10] + + +def test_table_exists_checks_again(tmp_path: Path) -> None: + ours = DeltaTableDatabase(tmp_path) + assert not ours.table_exists(PATH) + + DeltaTableDatabase(tmp_path).add_records(PATH, _rows(["x"], [10]), flush=True) + + assert ours.table_exists(PATH) + assert _values(ours.get_all_records(PATH)) == [10] + + +def test_flush_to_a_table_believed_absent_keeps_the_other_rows( + tmp_path: Path, +) -> None: + """A flush loads the table first: rows another database committed are kept.""" + ours = DeltaTableDatabase(tmp_path) + assert ours.get_all_records(PATH) is None + + DeltaTableDatabase(tmp_path).add_records( + PATH, _rows(["x", "shared"], [10, 11]), flush=True + ) + + ours.add_records(PATH, _rows(["a", "shared"], [1, 2]), skip_duplicates=True) + ours.flush() + + stored = deltalake.DeltaTable(str(tmp_path / "table")).to_pyarrow_table() + by_id = dict( + zip(stored["__record_id"].to_pylist(), stored["v"].to_pylist(), strict=True) + ) + assert by_id == {b"x": 10, b"shared": 11, b"a": 1} + assert _values(ours.get_all_records(PATH)) == [1, 10, 11] diff --git a/tests/test_databases/test_delta_commit_cost.py b/tests/test_databases/test_delta_commit_cost.py new file mode 100644 index 000000000..c2314a502 --- /dev/null +++ b/tests/test_databases/test_delta_commit_cost.py @@ -0,0 +1,220 @@ +"""What a ``DeltaTableDatabase`` commit costs, and that its snapshot stays exact (ITL-1173). + +After the rows of a table are loaded for lookups (ITL-919), each commit of this database +extends them with the rows of the data files the commit added. These tests pin: + +- the cost: an append commit loads the table once and never builds the table's dataset + (which lists every data file); counted, not timed; +- the result: after any sequence of commits (appends, MERGEs, schema evolution, another + writer's commits, a partitioned table) the extended rows equal a full read of the + table, and lookups through this database answer as a fresh database's do. + +The snapshot's rows are compared directly (``_stored_rows``): they are the subject. +""" + +from __future__ import annotations + +import random +import types +from datetime import datetime, timezone +from pathlib import Path +from typing import cast + +import deltalake +import pyarrow as pa +import pyarrow.dataset as pa_dataset +import pytest + +from orcapod.databases import DeltaTableDatabase +from orcapod.databases import delta_lake_databases + +PATH = ("table",) + + +def _rows(ids: list[str], *, extra: bool = False) -> pa.Table: + n = len(ids) + table = pa.table( + { + "__record_id": pa.array([i.encode() for i in ids], pa.large_binary()), + "hash": pa.array([f"h{i}".encode() for i in ids], pa.large_binary()), + "context": pa.array(["ctx"] * n, pa.large_string()), + "value": pa.array( + [None if k % 5 == 0 else k for k in range(n)], pa.int64() + ), + "items": pa.array( + [[k, k + 1] for k in range(n)], pa.large_list(pa.int64()) + ), + "point": pa.array( + [{"x": float(k), "label": f"p{k}"} for k in range(n)], + pa.struct([("x", pa.float64()), ("label", pa.large_string())]), + ), + "at": pa.array( + [datetime(2026, 1, 1, tzinfo=timezone.utc)] * n, + pa.timestamp("us", "UTC"), + ), + } + ) + if extra: + table = table.append_column( + "added", pa.array([f"a{i}" for i in ids], pa.large_string()) + ) + return table + + +def _data_files(table_dir: str) -> list[str]: + """The data files of the table's current version, as its add actions name them.""" + actions = pa.table(deltalake.DeltaTable(table_dir).get_add_actions(flatten=False)) + return cast(list[str], actions.column("path").to_pylist()) + + +def _sorted(table: pa.Table) -> pa.Table: + return table.sort_by("__record_id") + + +def _assert_snapshot_exact(db: DeltaTableDatabase, root: Path) -> None: + """The extended rows equal a full read, and lookups agree with a fresh database.""" + ours = db._stored_rows(PATH) + fresh_db = DeltaTableDatabase(root) + fresh = fresh_db._stored_rows(PATH) + assert ours is not None and fresh is not None + assert ours.table.schema == fresh.table.schema + assert _sorted(ours.table).equals(_sorted(fresh.table)) + for record_id in cast(list[bytes], fresh.table["__record_id"].to_pylist()): + got = db.get_record_by_id(PATH, record_id) + want = fresh_db.get_record_by_id(PATH, record_id) + assert got is not None and want is not None + assert got.equals(want) + by_value = db.get_records_with_column_value( + PATH, {"hash": b"h" + record_id, "context": "ctx"} + ) + assert by_value is not None and by_value.num_rows == 1 + + +@pytest.fixture +def files_read(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]: + """Record the data files of every dataset the snapshot extension reads directly.""" + read: list[list[str]] = [] + real = pa_dataset.FileSystemDataset + + def recording(fragments, *args, **kwargs): + fragments = list(fragments) + read.append([fragment.path for fragment in fragments]) + return real(fragments, *args, **kwargs) + + monkeypatch.setattr( + delta_lake_databases, + "pa_ds", + types.SimpleNamespace(FileSystemDataset=recording), + ) + return read + + +@pytest.fixture +def counts(monkeypatch: pytest.MonkeyPatch) -> dict[str, int]: + """Count table loads and whole-table dataset builds by the Delta database.""" + counted = {"table_loads": 0, "datasets": 0} + original_init = deltalake.DeltaTable.__init__ + original_dataset = delta_lake_databases.delta_to_pyarrow_dataset + + def counting_init(self, *args, **kwargs): + counted["table_loads"] += 1 + original_init(self, *args, **kwargs) + + def counting_dataset(*args, **kwargs): + counted["datasets"] += 1 + return original_dataset(*args, **kwargs) + + monkeypatch.setattr(deltalake.DeltaTable, "__init__", counting_init) + monkeypatch.setattr( + delta_lake_databases, "delta_to_pyarrow_dataset", counting_dataset + ) + return counted + + +def test_append_commit_loads_once_and_reads_only_its_files( + tmp_path: Path, counts: dict[str, int], files_read: list[list[str]] +) -> None: + db = DeltaTableDatabase(tmp_path) + db.add_records(PATH, _rows(["r0", "r1"]), flush=True) + assert db.get_record_by_id(PATH, b"r0") is not None # loads the rows once + + table_dir = str(tmp_path / "table") + for commit in range(1, 21): + files_before = set(_data_files(table_dir)) + counts["table_loads"] = counts["datasets"] = 0 + files_read.clear() + db.add_records(PATH, _rows([f"r{commit}_{k}" for k in range(3)])) + db.flush() + # Lookups between commits are answered from the extended rows. + assert db.get_record_by_id(PATH, f"r{commit}_1".encode()) is not None + assert db.get_record_by_id(PATH, b"r0") is not None + assert counts == {"table_loads": 1, "datasets": 0}, commit + added = set(_data_files(table_dir)) - files_before + assert len(added) == 1 + assert files_read == [sorted(added)], commit + + _assert_snapshot_exact(db, tmp_path) + + +def test_snapshot_equals_a_full_read_after_mixed_commits(tmp_path: Path) -> None: + """Appends, MERGEs (a stored id), schema evolution and another writer's commits.""" + rng = random.Random(1173) + db = DeltaTableDatabase(tmp_path) + other = DeltaTableDatabase(tmp_path) + db.add_records(PATH, _rows(["s0", "s1", "s2"]), flush=True) + db._stored_rows(PATH) + stored = ["s0", "s1", "s2"] + for step in range(30): + new = [f"n{step}_{k}" for k in range(rng.randint(1, 4))] + kind = rng.choice(["append", "append", "merge", "other"]) + if kind == "merge": + db.add_records( + PATH, _rows([rng.choice(stored), *new]), skip_duplicates=True + ) + db.flush() + elif kind == "other": + other.add_records(PATH, _rows(new), flush=True) + db.flush() # sees the other commit at its next commit + db.add_records(PATH, _rows([f"{n}_ours" for n in new]), flush=True) + new += [f"{n}_ours" for n in new] + else: + db.add_records(PATH, _rows(new), flush=True) + stored += new + _assert_snapshot_exact(db, tmp_path) + + # Schema evolution: the rows are read again, then extended as before. + db.add_records(PATH, _rows(["e0", "e1"], extra=True), flush=True) + _assert_snapshot_exact(db, tmp_path) + db.add_records(PATH, _rows(["e2"], extra=True), flush=True) + _assert_snapshot_exact(db, tmp_path) + + +def test_partitioned_table_reads_its_dataset( + tmp_path: Path, counts: dict[str, int] +) -> None: + """Partition values are not in the data files: the table's dataset is read. + + Orcapod's own commits to a partitioned table fail before this point (its id read + uses a one-column schema), so the extension is driven directly, after a commit + by another writer. + """ + table_dir = str(tmp_path / "table") + deltalake.write_deltalake(table_dir, _rows(["p0", "p1"]), partition_by=["context"]) + db = DeltaTableDatabase(tmp_path) + assert db.get_record_by_id(PATH, b"p0") is not None + base = db._snapshots["table"] + base.ids = {b"p0", b"p1"} + deltalake.write_deltalake( + table_dir, _rows(["q0", "q1"]), mode="append", partition_by=["context"] + ) + counts["datasets"] = 0 + + extended = db._extend_snapshot( + base, deltalake.DeltaTable(table_dir), {b"q0", b"q1"} + ) + + assert counts["datasets"] == 1 + assert extended.rows is not None + fresh = DeltaTableDatabase(tmp_path)._stored_rows(PATH) + assert fresh is not None + assert _sorted(extended.rows.table).equals(_sorted(fresh.table)) diff --git a/tests/test_databases/test_row_index.py b/tests/test_databases/test_row_index.py new file mode 100644 index 000000000..f6e0a62ce --- /dev/null +++ b/tests/test_databases/test_row_index.py @@ -0,0 +1,137 @@ +"""Tests for the hash-index helpers shared by the Arrow databases (ITL-919, ITL-1173). + +``matching_rows`` is tested directly: the cost of a lookup is the subject (ITL-1173), and +the databases' own lookup tests cover the results through their public methods. +""" + +from __future__ import annotations + +import random +from collections.abc import Iterator + +import pyarrow as pa +import pyarrow.compute as pc +import pytest + +from orcapod.databases._row_index import KeyIndex, matching_rows + + +class _CountingPositions(list[int]): + """A position list that counts how often it is walked.""" + + walks = 0 + + def __iter__(self) -> Iterator[int]: + _CountingPositions.walks += 1 + return super().__iter__() + + +def _expression(pairs: list[tuple[str, object]]) -> pc.Expression: + expression = pc.field(pairs[0][0]) == pairs[0][1] + for column, value in pairs[1:]: + expression = expression & (pc.field(column) == value) + return expression + + +def _table(n: int) -> pa.Table: + return pa.table( + { + "id": pa.array([f"id{i}".encode() for i in range(n)], pa.large_binary()), + "context": pa.array(["ctx"] * n, pa.large_string()), + "value": pa.array(range(n), pa.int64()), + } + ) + + +@pytest.mark.parametrize("context_first", [False, True]) +@pytest.mark.parametrize("n", [10, 1_000, 10_000]) +def test_lookup_walks_only_the_shortest_position_list( + n: int, context_first: bool +) -> None: + """A lookup on (id, context) does not walk the context list, which holds every row. + + Before ITL-1173 the positions of every condition were intersected as sets, so a + condition every row meets (the data context) made each lookup walk all N rows. + """ + key_index = KeyIndex(_table(n)) + context_positions = _CountingPositions(key_index.column("context")["ctx"]) + key_index.columns["context"] = {"ctx": context_positions} + _CountingPositions.walks = 0 + + pairs: list[tuple[str, object]] = [("id", b"id3"), ("context", "ctx")] + if context_first: + pairs.reverse() + result = matching_rows(key_index, pairs, _expression(pairs)) + + assert result["value"].to_pylist() == [3] + assert _CountingPositions.walks == 0 + + +def test_lookup_with_no_match_in_one_condition_is_empty() -> None: + key_index = KeyIndex(_table(5)) + pairs: list[tuple[str, object]] = [("id", b"id3"), ("context", "other")] + result = matching_rows(key_index, pairs, _expression(pairs)) + assert result.num_rows == 0 + assert result.schema == key_index.table.schema + + +def test_lookup_equals_the_arrow_filter() -> None: + """Indexed results equal ``table.filter`` for random conditions, rows in table order.""" + rng = random.Random(1173) + n = 400 + table = pa.Table.from_batches( + pa.table( + { + "a": pa.array( + [rng.choice([b"x", b"y", b"z", None]) for _ in range(n)], + pa.large_binary(), + ), + "b": pa.array( + [rng.choice(["p", "q", None]) for _ in range(n)], pa.large_string() + ), + "c": pa.array([rng.choice(["u", "v"]) for _ in range(n)], pa.string()), + "n": pa.array([rng.randrange(3) for _ in range(n)], pa.int64()), + "row": pa.array(range(n), pa.int64()), + } + ).to_batches(max_chunksize=37) + ) + key_index = KeyIndex(table) + for _ in range(200): + columns = rng.sample(["a", "b", "c", "n"], rng.randint(1, 4)) + pairs: list[tuple[str, object]] = [] + for column in columns: + if column == "a": + pairs.append((column, rng.choice([b"x", b"y", b"z", b"w"]))) + elif column == "b": + pairs.append((column, rng.choice(["p", "q", "r"]))) + elif column == "c": + pairs.append((column, rng.choice(["u", "v"]))) + else: + pairs.append((column, rng.randrange(3))) + expression = _expression(pairs) + got = matching_rows(key_index, pairs, expression) + assert got["row"].to_pylist() == table.filter(expression)["row"].to_pylist() + + +def test_lookup_after_append_uses_the_extended_index() -> None: + first = _table(4) + key_index = KeyIndex(first) + key_index.column("id") + key_index.column("context") + more = pa.table( + { + "id": pa.array([b"id9", b"id2"], pa.large_binary()), + "context": pa.array(["ctx", "new"], pa.large_string()), + "value": pa.array([90, 20], pa.int64()), + } + ) + key_index.append(more, pa.concat_tables([first, more])) + + cases: list[tuple[list[tuple[str, object]], list[int]]] = [ + ([("id", b"id2"), ("context", "ctx")], [2]), + ([("id", b"id2"), ("context", "new")], [20]), + ([("id", b"id9"), ("context", "ctx")], [90]), + ] + for pairs, expected in cases: + got = matching_rows(key_index, pairs, _expression(pairs)) + assert got["value"].to_pylist() == expected