Skip to content
3 changes: 3 additions & 0 deletions changelog.d/ITL-1173.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
### Changed

- **Faster job runs into `DeltaTableDatabase`.** An append commit now loads its table once and reads only the data files it added, instead of rebuilding the whole table's dataset (deltalake's own append cost still grows with the number of files). A lookup no longer walks every stored row. A table that does not exist yet is not reopened on every lookup. Stored rows, hashes, paths and table properties are unchanged. A table that another process creates during a run is seen from this database's next commit, as rows another process appends to an already-loaded table are. (ITL-1173)
60 changes: 60 additions & 0 deletions docs/dev/design_issues/ITL-1173-DS1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
## `src/orcapod/databases/` — PipelineJob run time quadratic in rows

### DS1 — A job into a local `DeltaTableDatabase` spends most of its time on store bookkeeping
**Status:** resolved
**Severity:** medium
**Issue:** ITL-1173

Measured on `1531b8d5`, one function node `double(value)`, local disk, min of 3. A
10,000-row job took 16.25 s with the default `commit_every_rows=100`, against 1.13 s for
the pod without a job and 7.53 s for the job on `NoOpArrowDatabase`. At 100,000 rows it
took 346 s. Three costs grew with the rows stored:

- **Each commit rebuilt the table's dataset.** A commit loaded the table three times (the
existence check, the writer's own reload when given a URI, the reload after the commit).
`_extend_snapshot` also built `to_pyarrow_dataset` over every data file to find the one
file the commit added. Per-commit cost rose from 8 ms to 37 ms over a 10k-row job. At
100k rows commits took 192 of the 346 s, 131 s of it in the snapshot rebuild.
- **Each lookup walked every stored row.** Every lookup filters on the record id and on
`__data_context`. The data-context position list holds every row, and
`_row_index.matching_rows` intersected the position sets of all conditions. A lookup
cost 100 µs at 10k rows and 250 µs at 100k (50 s).
- **A missing table was reopened on every lookup.** Until a table exists, every lookup
tried to open it again, about 7 failed `DeltaTable()` opens per row. That cost 10.8 s
of a 10k-row job committing once at node end, which is why one commit at node end
(20.95 s) was slower than the default.

Also seen: deltalake logs one "Skipping column ... binary field" line per binary column
per write (1,000 lines at 10k rows). Changing that means setting a Delta table property,
which is a developer decision (F4, open).

**Fix:**
- `matching_rows` starts from the condition with the fewest rows and checks the others on
those rows only.
- The database family remembers tables it found absent until its next flush;
`table_exists` always checks again.
- An append commit is written through the table loaded for it. The snapshot is extended
by reading only the commit's added files, found from the table's add actions and read
with the format and filesystem of the dataset its rows came from. A partitioned table
reads its dataset as before.

Stored rows, ids, hashes, paths, schemas and table properties are unchanged, and so is
the default commit interval (developer decision: it stays 100). The table is still loaded
afresh before each commit (developer decision): a handle refreshed in place could commit a
version gap into a store directory that was deleted and recreated.

Measured with the fixes, min of 3 at 10k rows against `main` in the same session:
- default interval: 12.96 s, against 15.81 s;
- `commit_every_rows=1000`: 11.02 s, against 12.33 s;
- one commit at node end: 10.58 s, against 20.95 s before.

At 100k rows (one run each): 159 s at the default interval (346 s before) and 117 s at
1000 (154 s before). Commits now cost 1.7 s of a 10k run (4.3 s before). The job's own
per-row work (7.5 s on `NoOpArrowDatabase`) is the largest remaining cost.

The deltalake "binary field" log line is explained in `docs/troubleshooting.md`, with the
`RUST_LOG` filter that hides it. No table property was set; that decision (F4) is open.

Follow-ups: ITL-1182 (`InMemoryArrowDatabase` lookups over the pending batch), ITL-1183
(peak memory at 100k rows), ITL-1184 (compaction: the append cost of deltalake itself
still grows with the file count), ITL-1177 / ITL-1179 (per-row hashing cost).
36 changes: 36 additions & 0 deletions docs/dev/scripts/itl_1173/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# ITL-1173 measurement scripts

These scripts measured why a `PipelineJob` over 10,000 rows into a local `DeltaTableDatabase`
is slow. The findings are in `docs/dev/design_issues/ITL-1173-DS1.md`. Each run starts a fresh
interpreter. Each measurement was repeated at least 3 times, and the minimum and spread are
reported. Run them from the repository root on an otherwise idle machine: Arrow is
multithreaded, and a loaded machine moves the numbers.

| Script | What it measures |
|---|---|
| `bench_one.py <delta\|memory\|noop\|nojob> <N> <commit_every> <dbdir>` | Wall time of one job run over N rows of `double(value)`. `commit_every` 0 means one commit at node end. Prints the run time, the same job run again in the same process, and for Delta the versions, files and bytes of each table. `--profile out.prof` writes a cProfile. |
| `instrument.py <delta\|memory\|noop> <N> <commit_every> <dbdir>` | Inclusive time and call counts of the named code paths (lookups, inserts, commits, hashing, deltalake calls), plus mean commit time per decile of the commits. In `memory` mode it also times a second run by a fresh job object over the same database. |
| `commit_curve.py <empty dir>` | Raw deltalake cost per commit against the table version: load, append by URI, dataset build, `update_incremental`, and append through a kept handle. |
| `warn_test.py all <dir>` | Counts deltalake's "Skipping column ... binary field" log lines on Orcapod's create, append, MERGE and schema-evolution paths, with and without a statistics table property, and for an existing table after the property is set. |

The matrix used for the report:

```
for rep in 1 2 3; do
for n in 1000 10000; do
uv run python docs/dev/scripts/itl_1173/bench_one.py nojob $n 0 x
for every in 100 1000 0; do
rm -rf /tmp/itl1173_db
uv run python docs/dev/scripts/itl_1173/bench_one.py delta $n $every /tmp/itl1173_db
uv run python docs/dev/scripts/itl_1173/bench_one.py delta $n $every /tmp/itl1173_db # rerun, all stored
done
done
done
```

The in-memory rows of the report came from `instrument.py memory` at N = 1,000, 3,000 and
10,000 (commit every 100), and at 10,000 with 1,000 and node end. The run at 10,000 with node
end takes about 37 minutes.

A raw `deltalake` dataset read in a script can hang the interpreter at exit (ITL-892).
`commit_curve.py` and `warn_test.py` end with `os._exit(0)` for that reason.
100 changes: 100 additions & 0 deletions docs/dev/scripts/itl_1173/bench_one.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
"""One ITL-1173 measurement in a fresh interpreter; prints one JSON line.

Usage: uv run python docs/dev/scripts/itl_1173/bench_one.py <mode> <N> <commit_every> <dbdir> [--rerun] [--profile out.prof]
mode: delta | memory | noop | nojob
commit_every: rows per commit, or 0 for "one commit at node end"
"""

import json
import os
import sys
import time
from typing import Any

import orcapod as op
from orcapod import sources


def table_stats(root: str) -> dict[str, dict[str, int]]:
"""Commits (versions), data files and bytes per Delta table under root."""
out = {}
for dirpath, dirnames, filenames in os.walk(root):
if "_delta_log" in dirnames:
log = os.path.join(dirpath, "_delta_log")
versions = [f for f in os.listdir(log) if f.endswith(".json")]
parquet = [f for f in filenames if f.endswith(".parquet")]
size = sum(os.path.getsize(os.path.join(dirpath, f)) for f in parquet)
rel = os.path.relpath(dirpath, root)
out[rel] = {
"versions": len(versions),
"files": len(parquet),
"bytes": size,
}
dirnames.remove("_delta_log")
return out


def main() -> None:
mode, n, every, dbdir = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), sys.argv[4]
rerun = "--rerun" in sys.argv
prof = (
sys.argv[sys.argv.index("--profile") + 1] if "--profile" in sys.argv else None
)

source = sources.DictSource(
[{"key": i, "value": i} for i in range(n)], tag_columns="key"
)

@op.function_pod("result")
def double(value: int) -> int:
return 2 * value

result: dict[str, Any] = {"mode": mode, "n": n, "every": every, "rerun": rerun}
if mode == "nojob":
t0 = time.perf_counter()
double.pod(source).run()
result["run_s"] = time.perf_counter() - t0
print(json.dumps(result))
return

if mode == "delta":
# commit_every_rows must be < batch_size; 0 = one commit at node end.
rows_per_commit = every if every else n + 1
database = op.databases.DeltaTableDatabase(
dbdir, batch_size=max(1000, rows_per_commit + 1)
)
else:
rows_per_commit = every if every else n + 1
database = (
op.databases.NoOpArrowDatabase()
if mode == "noop"
else op.databases.InMemoryArrowDatabase()
)

job = op.PipelineJob("pipeline_v1", database=database)
with job:
double.pod(source)

profiler = None
if prof:
import cProfile

profiler = cProfile.Profile()
profiler.enable()
t0 = time.perf_counter()
job.run(commit_every_rows=rows_per_commit)
result["run_s"] = time.perf_counter() - t0
if profiler and prof:
profiler.disable()
profiler.dump_stats(prof)
# Same job again in the same process (state rule: held, nothing runs).
t0 = time.perf_counter()
job.run(commit_every_rows=rows_per_commit)
result["same_job_second_run_s"] = time.perf_counter() - t0
if mode == "delta":
result["tables"] = table_stats(dbdir)
print(json.dumps(result))


if __name__ == "__main__":
main()
95 changes: 95 additions & 0 deletions docs/dev/scripts/itl_1173/commit_curve.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
"""ITL-1173: raw deltalake cost per commit against the table version.

Appends 300 batches of 100 rows (the shape of a function node's pipeline table) and
times, at selected versions: a fresh table load, an append given the URI, a load plus
``to_pyarrow_dataset``, and ``update_incremental`` of a kept handle. Then times appends
written through a kept handle.

Usage: uv run python docs/dev/scripts/itl_1173/commit_curve.py <empty dir>
"""

from __future__ import annotations

import json
import os
import sys
import time

import deltalake
import pyarrow as pa

SAMPLED = (1, 2, 10, 50, 99, 100, 101, 150, 200, 250, 299, 300)


def batch(k: int) -> pa.Table:
ids = range(k * 100, k * 100 + 100)
return pa.table(
{
"__record_id": pa.array(
[i.to_bytes(32, "big") for i in ids], pa.large_binary()
),
"h1": pa.array([i.to_bytes(40, "big") for i in ids], pa.large_binary()),
"h2": pa.array([i.to_bytes(40, "big") for i in ids], pa.large_binary()),
"v": pa.array(ids, pa.int64()),
"ctx": pa.array(["std:v1:default"] * 100, pa.large_string()),
}
)


def ms(seconds: float) -> float:
return round(1e3 * seconds, 1)


def main() -> None:
d = sys.argv[1]
if os.path.exists(d) and os.listdir(d):
sys.exit(f"{d} is not empty: give an empty or missing directory")
deltalake.write_deltalake(d, batch(0), mode="overwrite")
handle = deltalake.DeltaTable(d)
for k in range(1, 301):
t0 = time.perf_counter()
loaded = deltalake.DeltaTable(d)
t_load = time.perf_counter() - t0
t0 = time.perf_counter()
deltalake.write_deltalake(d, batch(k), mode="append")
t_write = time.perf_counter() - t0
t0 = time.perf_counter()
_ = deltalake.DeltaTable(d).to_pyarrow_dataset()
t_dataset = time.perf_counter() - t0
t0 = time.perf_counter()
handle.update_incremental()
t_incremental = time.perf_counter() - t0
if k in SAMPLED:
print(
json.dumps(
{
"version": loaded.version() + 1,
"load_ms": ms(t_load),
"write_append_uri_ms": ms(t_write),
"load+to_pyarrow_dataset_ms": ms(t_dataset),
"update_incremental_ms": ms(t_incremental),
}
),
flush=True,
)
kept = deltalake.DeltaTable(d)
times: list[float] = []
for k in range(301, 321):
t0 = time.perf_counter()
deltalake.write_deltalake(kept, batch(k), mode="append")
times.append(time.perf_counter() - t0)
print(
"append through a kept handle, min / mean ms:",
ms(min(times)),
ms(sum(times) / len(times)),
)
print(
"checkpoints:",
sorted(f for f in os.listdir(f"{d}/_delta_log") if not f.endswith(".json")),
flush=True,
)
os._exit(0) # the dataset reads above can hang interpreter exit (ITL-892)


if __name__ == "__main__":
main()
Loading
Loading