Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,25 @@ jobs:
- name: Compare starfix.legacy.v0_4 with starfix==0.4.0 from PyPI
run: uv run --python 3.12 python scripts/legacy_differential.py

release-differential:
# Every hash (one-shot and through a prepared schema) must stay byte-identical to the
# released starfix-python 0.5.0 (PyPI), under every option combination.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4

- name: Install uv
uses: astral-sh/setup-uv@e58605a9b6da7c637471fab8847a5e5a6b8df081 # v5

- name: Set up Python 3.12
run: uv python install 3.12

- name: Install dependencies
run: uv sync --dev --python 3.12

- name: Compare this checkout with starfix==0.5.0 from PyPI
run: uv run --python 3.12 python scripts/release_differential.py

license-check:
runs-on: ubuntu-latest
steps:
Expand Down
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,36 @@
# Changelog

## 0.6.0 (unreleased)

No hash changes: every hash is byte-identical to 0.5.0 (hash format 2), for every input and
option combination.

### Added — prepared schemas (ITL-1177)

- `ArrowDigester.prepare(schema, *, include_metadata=False, ordered_columns=False,
ordered_struct_fields=False)` analyses a schema once and returns a `PreparedSchema`
(exported from `starfix`), with `schema_digest` (equal to `hash_schema`),
`hash_record_batch(batch)`, `hash_table(table)` and `digester()` (a fresh streaming
`ArrowDigester`). Its hashes equal the one-shot calls; hashing a one-row table of a
six-column schema takes about 35 µs instead of about 150 µs (0.5.0, without
`include_metadata`; about 530 µs with it). See the README section "Hashing many inputs of
one schema" and `scripts/bench_prepared.py`.
- A `PreparedSchema` is immutable and owned by the caller; starfix keeps no cache. A table or
batch is accepted exactly when its schema hashes like the prepared one under the prepared
options (the rule streamed batches already follow); otherwise `ValueError`. Identical
schemas are recognized by comparing their Arrow IPC serialization (about 1.5 µs).
- `scripts/release_differential.py` (and a CI job): this checkout, one-shot and prepared,
against the released `starfix==0.5.0`, under all option combinations.

### Changed — performance

- One-shot calls (`hash_table`, `hash_record_batch`, `ArrowDigester(...)`) analyse the
schema once instead of three times (plus once per streamed batch): a one-row `hash_table`
takes about 85 µs instead of about 150 µs (about 175 µs instead of about 530 µs with
`include_metadata`).
- `ArrowDigester.update` checks a batch whose schema is identical to the digester's in about
1.5 µs instead of re-analysing it.

## 0.5.0 (2026-11-11)

### Changed — hash format 2 (every hash changes)
Expand Down
50 changes: 45 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,11 +55,48 @@ ArrowDigester.hash_record_batch(table.to_batches()[0])
ArrowDigester.hash_array(pa.array([1, 2, 3], pa.int32()))
```

### Hashing many inputs of one schema

Every call above first analyses the schema (walks its nodes, builds its canonical text and
digest). To hash many small tables or batches of one schema, analyse it once:

```python
prepared = ArrowDigester.prepare(schema, include_metadata=False) # options as below

for row_table in tables_of_this_schema:
key = prepared.hash_table(row_table) # == ArrowDigester.hash_table(row_table)
prepared.hash_record_batch(batch) # == ArrowDigester.hash_record_batch(batch)
prepared.schema_digest # == ArrowDigester.hash_schema(schema)
digester = prepared.digester() # a fresh streaming ArrowDigester
```

- Every hash equals the one-shot call with the same options, byte for byte.
- A `PreparedSchema` is immutable and belongs to the caller: starfix keeps no cache or other
state between calls. It can be shared between threads and pickled (it is prepared again when
loaded).
- **Same-schema rule.** A table or batch is accepted exactly when its schema hashes like the
prepared one under the prepared options (`hash_schema(its_schema, **options) ==
prepared.schema_digest`): the same column names, logical types and nullability at every
depth, the same column order with `ordered_columns`, the same struct field order with
`ordered_struct_fields`, and with `include_metadata` the same field metadata at every node,
schema metadata and extension types. It is the rule `update` applies to streamed batches.
Anything else raises `ValueError` ("Table schema does not match the prepared schema" /
"Record batch schema does not match the prepared schema"); nothing is ever hashed with
another schema's analysis.
- The check costs about 1.5 µs when the input's schema is identical to the prepared one (equal
Arrow IPC serialization, compared as bytes). Any other accepted schema (for example `string`
data for a prepared `large_string` column, or metadata that differs while `include_metadata`
is off) is checked by computing its canonical schema text, as expensive as a one-shot call's
analysis.
- `prepare` raises what `hash_schema` raises. A schema whose data cannot be hashed (see
[Rejected inputs](#rejected-inputs)) still prepares, for its `schema_digest`; hashing data
with it raises `ValueError`.

## Options

Keyword-only, all `False` by default, the same names as the Rust `HasherConfig` fields, accepted by
`ArrowDigester(...)`, `hash_schema`, `hash_record_batch`, `hash_table` and `hash_array`
(`hash_array` has no `include_metadata`).
`ArrowDigester(...)`, `ArrowDigester.prepare`, `hash_schema`, `hash_record_batch`, `hash_table` and
`hash_array` (`hash_array` has no `include_metadata`).

| Option | Effect |
|---|---|
Expand Down Expand Up @@ -132,7 +169,8 @@ rebuilds the type from them, so starfix-python hashes what pyarrow can observe:
- a non-nullable field holding a null that no enclosing struct null explains;
- with `include_metadata`: a duplicate metadata key on a field or the schema, or explicit
`ARROW:extension:*` keys that conflict with the field's registered extension type;
- streaming: a batch whose schema differs from the digester's. Physically different but
- streaming, and hashing through a prepared schema: a batch or table whose schema differs from
the digester's or the prepared one (see the same-schema rule above). Physically different but
equivalent types (`Utf8`/`LargeUtf8`, `Binary`/`LargeBinary`, `List`/`LargeList`, dictionary
and plain values) are accepted; reordered columns or struct fields are accepted unless the
matching order option is on.
Expand All @@ -158,8 +196,10 @@ are materialized (decoded with `take`) before they are traversed. A fixed-size l
without null slots is hashed from one zero-copy slice of its child; with null slots, its items
are selected in chunks of about one million items. Fixed-width, Boolean and list columns hash at tens of millions of rows per second.
String and binary columns hash at about 3 to 5 million rows per second, because each value is
visited in Python. Each `update` call also costs a few microseconds per column, to check the
batch's schema. starfix-python is a pure-Python implementation; the Rust crate is faster on
visited in Python. Each call also analyses the schema, which dominates for small inputs (tens to
hundreds of microseconds, more with `include_metadata`); `ArrowDigester.prepare` does it once
for many inputs. A streamed batch whose schema is identical to the digester's is checked in
about 1.5 µs; `scripts/bench_prepared.py` measures both paths. starfix-python is a pure-Python implementation; the Rust crate is faster on
string-heavy data.

## Hash format
Expand Down
107 changes: 107 additions & 0 deletions docs/metamorphic/plans/2026-11-16-prepared-schema.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Prepared Schema Implementation Plan (ITL-1177)

> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Let a caller analyse an Arrow schema once (`ArrowDigester.prepare`) and hash many tables or record
batches of that schema without redoing the analysis, with every digest byte-identical to the one-shot calls.

**Architecture:** `_format2.analyse_schema` runs the schema analysis once and returns an immutable
`SchemaAnalysis` (node list, canonical schema key, schema digest, entry specs, the data-support verdict, and
the schema's Arrow IPC bytes for the same-schema check). `Digester` is built from an analysis; the one-shot
entry points build one analysis per call and never re-check their own input. The public
`starfix.PreparedSchema` (in `arrow_digester.py`) wraps one analysis and its schema, owned by the caller.
No module-level, global or cached state is added.

**Tech Stack:** Python >= 3.10, pyarrow >= 19, pytest, uv.

---

## Interface

```python
ArrowDigester.prepare(schema, *, include_metadata=False, ordered_columns=False,
ordered_struct_fields=False) -> PreparedSchema

class PreparedSchema: # built only by prepare; immutable; pickles by re-preparing
schema: pa.Schema # read-only properties
include_metadata: bool
ordered_columns: bool
ordered_struct_fields: bool
schema_digest: bytes # == ArrowDigester.hash_schema(schema, same options)
def hash_record_batch(self, record_batch) -> bytes
def hash_table(self, table) -> bytes
def digester(self) -> ArrowDigester
```

* `prepare` raises exactly what `hash_schema` raises. Types whose schema hashes but whose data does not
(Null, Map, Interval, view types) do not make `prepare` fail; `hash_record_batch`, `hash_table` and
`digester()` raise the existing `ValueError` for them.
* Same-schema rule: accepted iff the input schema's canonical schema key under the prepared options equals
the prepared one (Phase 1 text; plus Phase 2 text with `include_metadata`), i.e. iff
`hash_schema(input.schema, options) == prepared.schema_digest`. This is today's rule of
`ArrowDigester.update`.
* Check: (1) `input.schema.serialize()` bytes equal the prepared schema's IPC bytes -> identical schema ->
accept; (2) otherwise compute the input's key (today's analysis) and compare; mismatch -> `ValueError`
("Record batch schema does not match the prepared schema" / "Table schema does not match the prepared
schema"; `digester().update` keeps "Record batch schema does not match ArrowDigester schema").
`pa.Schema.equals(check_metadata=True)` is not used: it ignores nested metadata of dictionary value types
and the parameters of Python-defined extension types.

## Files

* Modify `src/starfix/_format2.py`: `SchemaAnalysis`, `analyse_schema`, `phase2_json(schema, nodes)`,
`_Entry(kind, nullable)`, `Digester.from_analysis`, unchecked ingestion for one-shot calls.
* Modify `src/starfix/arrow_digester.py`: `ArrowDigester.prepare`, `PreparedSchema`.
* Modify `src/starfix/__init__.py`: export `PreparedSchema`.
* Create `tests/random_arrow.py`: seeded random schemas and data (nested, dictionary, fixed-size list,
extension types, metadata, empty and sliced inputs).
* Create `tests/test_prepared_schema.py`: API, immutability, no module state, golden vectors x all option
combinations, the type matrix, accept/refuse rule, seeded differential, analysis-count tests.
* Create `scripts/release_differential.py`: this checkout (one-shot and prepared) against the released
`starfix==0.5.0` from PyPI, byte-identical. Add a CI job for it beside `legacy-differential`.
* Create `scripts/bench_prepared.py`: time per one-row hash, one-shot and prepared.
* Modify `README.md` (usage + same-schema rule), `CHANGELOG.md` (Unreleased, proposed 0.6.0).

## Tasks

### Task 1: failing tests for the public interface
- [ ] Write `tests/test_prepared_schema.py` tests: `prepare` returns a `PreparedSchema` whose properties
echo the options; `schema_digest == hash_schema` for all 8 option combinations; `PreparedSchema()` raises
`TypeError`; setting/deleting attributes raises `AttributeError`; pickle round trip hashes the same;
`hash_record_batch` / `hash_table` / `digester()` equal the one-shot results.
- [ ] Run `uv run pytest tests/test_prepared_schema.py -q`; expect failures (no `prepare`).

### Task 2: analysis once (refactor, no behaviour change)
- [ ] `analyse_schema` computes nodes once, Phase 1 once, Phase 2 once (from the same nodes), the digest
from that key, sorted entry specs, the data-support verdict and the IPC bytes.
- [ ] `Digester.__init__` = `analyse_schema` + `from_analysis`; `update` checks via the analysis;
`hash_record_batch` / `hash_table` analyse once and ingest without re-checking (a table's batches carry
the table's schema).
- [ ] Test: counting wrappers on `schema_nodes`, `phase1_json`, `phase2_json` show one call each per
one-shot `hash_table` / `hash_record_batch`, and none for a prepared hash of an identical schema.
- [ ] Run the full suite: `uv run pytest tests/ -q` (golden vectors pin the bytes).

### Task 3: `PreparedSchema` and `ArrowDigester.prepare`
- [ ] Implement in `arrow_digester.py`; export from `starfix`. Run Task 1 tests: pass.

### Task 4: same-schema rule
- [ ] Refusal cases (each with every relevant option combination): column name, type, nullability (top
and nested), column order with `ordered_columns`, struct field order with `ordered_struct_fields`, field
metadata at top level, in a struct child, a list / fixed-size-list element, the struct inside a dictionary
value, schema metadata, a Python extension type's parameter, extension vs storage type, all with
`include_metadata`. Acceptance cases: string vs large_string, dictionary vs plain, reordered columns or
struct fields when unordered, metadata differences without `include_metadata`; accepted results equal the
one-shot hash of the input.
- [ ] Seeded pairs: accepted iff `hash_schema` digests are equal.

### Task 5: differential proofs
- [ ] Golden vectors (`tests/golden/format2_v0.5.json`) x 8 option combinations: prepared == one-shot;
for the vector's own options == expected.
- [ ] Type matrix (`tests/test_format2_type_matrix.py::_hashes`) also through a prepared schema.
- [ ] Seeded random inputs (`tests/random_arrow.py`), all option combinations, batch / table / stream.
- [ ] `scripts/release_differential.py` against `starfix==0.5.0`; CI job.

### Task 6: benchmark, docs, changelog, PR
- [ ] `scripts/bench_prepared.py`; numbers before (origin/main) and after.
- [ ] README section "Hashing many inputs of one schema"; CHANGELOG Unreleased.
- [ ] `uv run pytest tests/ -q`; commit (conventional commits); PR against `main`.
94 changes: 94 additions & 0 deletions scripts/bench_prepared.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
"""Time per hash of small tables: one-shot calls against a prepared schema (ITL-1177).

Usage (from the repository root):

uv run python scripts/bench_prepared.py [--rows 1] [--repeat 7]

Hashes tables of a few-column schema (ints, strings, a list, a struct, field and schema
metadata), each built anew as a caller hashing row by row would (so every table carries a new
``pa.Schema`` object). Prints the best-of-``--repeat`` mean time per call in microseconds.
Against a checkout without ``ArrowDigester.prepare`` (0.5.0 and earlier) only the one-shot rows
are printed, so the same script measures before and after: for example

git worktree add /tmp/starfix-main origin/main
PYTHONPATH=/tmp/starfix-main/src uv run python scripts/bench_prepared.py
"""

from __future__ import annotations

import argparse
import platform
import timeit

import pyarrow as pa

import starfix
from starfix import ArrowDigester

SCHEMA = pa.schema(
[
pa.field("id", pa.int64(), nullable=False),
pa.field("name", pa.large_string()),
pa.field("score", pa.float64(), metadata={"unit": "points"}),
pa.field("tags", pa.list_(pa.large_string())),
pa.field("pos", pa.struct([pa.field("x", pa.float32()), pa.field("y", pa.float32())])),
pa.field("path", pa.large_string(), metadata={"semantic": "path"}),
],
metadata={"producer": "bench"},
)


def _rows(n: int) -> list[dict]:
return [
{"id": i, "name": f"n{i}", "score": i / 3, "tags": ["a", "b"][: i % 3], "pos": {"x": 1.0, "y": 2.0},
"path": f"/data/{i}.bin"}
for i in range(n)
]


def _bench_options(bench, tables, batches, options: dict[str, bool], has_prepare: bool) -> None:
bench("one-shot ArrowDigester.hash_table", lambda i: ArrowDigester.hash_table(tables[i], **options))
bench("one-shot ArrowDigester.hash_record_batch", lambda i: ArrowDigester.hash_record_batch(batches[i], **options))
if not has_prepare:
return
prepared = ArrowDigester.prepare(SCHEMA, **options)
bench("prepared.hash_table", lambda i: prepared.hash_table(tables[i]))
bench("prepared.hash_record_batch", lambda i: prepared.hash_record_batch(batches[i]))
bench(" of which: ArrowDigester.prepare (once per schema)", lambda i: ArrowDigester.prepare(SCHEMA, **options))
analysis = prepared._analysis
bench(" of which: same-schema check (identical schema)", lambda i: analysis.matches(tables[i].schema))
string_schema = pa.schema([f.with_type(pa.string()) if f.type == pa.large_string() else f for f in SCHEMA],
metadata=SCHEMA.metadata)
bench(" same-schema check, hash-equivalent (string) schema", lambda i: analysis.matches(string_schema))


def main() -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("--rows", type=int, default=1)
parser.add_argument("--repeat", type=int, default=7)
parser.add_argument("--number", type=int, default=2000)
args = parser.parse_args()
rows = _rows(args.rows)
tables = [pa.Table.from_pylist(rows, schema=SCHEMA) for _ in range(64)] # distinct Schema objects
batches = [t.to_batches()[0] for t in tables]
has_prepare = hasattr(ArrowDigester, "prepare")
print(f"starfix from {starfix.__file__}; pyarrow {pa.__version__}; Python {platform.python_version()}; "
f"{platform.machine()}; {args.rows} row(s), {len(SCHEMA)} columns")

def bench(label: str, call) -> None:
it = iter(range(1 << 62))

def step() -> None:
call(next(it) % len(tables))

best = min(timeit.repeat(step, number=args.number, repeat=args.repeat)) / args.number
print(f" {label:<52} {best * 1e6:8.1f} us")

for include_metadata in (False, True):
print(f"include_metadata={include_metadata}")
_bench_options(bench, tables, batches, {"include_metadata": include_metadata}, has_prepare)
return 0


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading