Skip to content

feat: ArrowDigester.prepare / PreparedSchema — analyse a schema once, hash many inputs (ITL-1177) - #13

Closed
eywalker wants to merge 4 commits into
mainfrom
eywalker/itl-1177-starfix-and-starfix-python-a-prepared
Closed

eywalker wants to merge 4 commits into
mainfrom
eywalker/itl-1177-starfix-and-starfix-python-a-prepared

Conversation

@eywalker

Copy link
Copy Markdown
Contributor

Summary

  • New: ArrowDigester.prepare(schema, *, include_metadata=False, ordered_columns=False, ordered_struct_fields=False) -> PreparedSchema (exported from starfix). PreparedSchema has schema, the three options, schema_digest (35 bytes, equal to hash_schema), hash_record_batch(batch), hash_table(table) and digester() (a fresh streaming ArrowDigester). It is immutable and owned by the caller. starfix keeps no cache or module state (there is a test for this). Pickling prepares the schema again on load.
  • Same-schema rule: an input is accepted exactly when its schema has the prepared canonical schema key under the prepared options, i.e. hash_schema(input.schema, **options) == prepared.schema_digest. That means the same names, logical types and nullability at every depth, column order with ordered_columns, struct field order with ordered_struct_fields, and with include_metadata the same field, schema and extension metadata. This is the rule ArrowDigester.update already applies. Anything else raises ValueError("Record batch schema does not match the prepared schema") ("Table schema ..." for tables).
    • Fast path (about 1.5 µs): identical Arrow IPC serialization of the schema.
    • Otherwise the input's key is computed and compared.
    • pa.Schema.equals(check_metadata=True) is not used. It ignores nested field metadata of dictionary value types and the parameters of Python-defined extension types; both cases are regression tests.
  • One-shot calls analyse the schema once instead of three times, plus once per streamed batch. No hash changes. Schemas whose data cannot be hashed (Map, Null, …) still prepare, for schema_digest; hashing data with them raises the existing error.

Numbers

uv run python scripts/bench_prepared.py: a one-row table, 6 columns (int, strings, list, struct, metadata), a new table and Schema object per call. Apple Silicon, pyarrow 26, Python 3.12. The machine was under load, so values vary by about ±15%.

per call 0.5.0 this PR, one-shot this PR, prepared
hash_table ~150 µs ~85 µs ~35 µs
hash_table, include_metadata=True ~530 µs ~175 µs ~35 µs
  • prepare itself: ~42 µs, or ~110 µs with metadata.
  • Same-schema check for an identical schema: ~1.5 µs.
  • Before: PYTHONPATH=<checkout of main>/src uv run python scripts/bench_prepared.py.

Test plan

  • uv run pytest tests/: 3683 passed. New in tests/test_prepared_schema.py:
    • every record and schema vector of the Rust golden file, under all 8 option combinations: prepared == one-shot, and == the Rust hash for the vector's own options;
    • the type matrix, also through a prepared schema;
    • seeded random tables (nested, dictionary, fixed-size list, extension types, metadata at every depth, empty and multi-batch) under all option combinations;
    • refuse and accept cases for every hashed aspect;
    • schema pairs: accepted iff the schema digests are equal;
    • the analysis runs once (call counters);
    • immutability, pickle, no module state.
  • uv run python scripts/release_differential.py: 29,360 one-shot results byte-identical to the released starfix==0.5.0 from PyPI (136 raise the same exception in both). No prepared result differs from its one-shot counterpart. Added as a CI job next to legacy-differential; no permission changes.
  • No hypothesis property tests: the repository has none, and its license check does not allow MPL-2.0. The seeded random generators follow the pattern of scripts/legacy_differential.py.

Version

Proposed 0.6.0 (new API, no hash change). The version comes from the release tag (hatch-vcs), so this PR changes no version. The CHANGELOG has a 0.6.0 (unreleased) section. Releasing is the maintainers' call.

Linear

ITL-1177

🤖 Generated with Claude Code

Edgar Y. Walker and others added 3 commits November 16, 2026 17:27
…hash many inputs (ITL-1177)

PreparedSchema holds the schema analysis (nodes, canonical key, schema
digest, entry specs, data-support verdict) and hashes tables, record
batches and streams of that schema without analysing it again. Its hashes
equal the one-shot calls byte for byte. An input is accepted exactly when
its schema has the prepared canonical schema key under the prepared
options (the streaming rule); identical schemas are recognised by their
Arrow IPC bytes, others by computing their key. No cache or module state.

One-shot calls now analyse the schema once instead of three times, and
ArrowDigester.update no longer re-analyses a batch of an identical schema.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a benchmark (ITL-1177)

scripts/release_differential.py hashes seeded random tables under all
option combinations, one-shot and prepared, and requires byte-identical
results to the released starfix==0.5.0 (run in CI like the legacy
differential). scripts/bench_prepared.py times one-row hashes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@eywalker
eywalker requested a balanced review from Copilot November 17, 2026 02:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The test resets counters before checking prepared operations, so it would still pass if every prepared hash reanalysed the schema, leaving the central performance guarantee unprotected.

1 open issue

Review effort: Balanced

Comment thread tests/test_prepared_schema.py Resolved
…1177)

The prepared-schema analysis test reset its counters before checking them,
so it passed even if every prepared hash re-analysed the schema. It now
asserts zero analysis calls after each prepared operation (digester(),
update, finalize, hash_table, hash_record_batch, schema_digest) without
resetting, and checks that prepare itself analyses once, so the zero
counts cannot pass because the counters observe nothing. The streaming
digester's update check moves to its own test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

With metadata checking enabled, the IPC shortcut can accept conflicting extension metadata that one-shot hashing rejects, undermining schema validation for prepared hashes and streaming updates.

3 open issues

Review effort: Balanced

Comment thread src/starfix/_format2.py
Comment on lines +886 to +888
if self._ipc is not None and _ipc_bytes(schema) == self._ipc:
return True
return schema_key(schema, self.include_metadata, self.flags) == self.key
Comment thread src/starfix/_format2.py
(path, _entry_kind(field.type), field.nullable) for path, field in sorted(nodes, key=lambda n: n[0])
)
self._data_error = _data_support_error(nodes)
self._ipc = _ipc_bytes(schema)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The IPC fast path can accept conflicting explicit extension metadata that one-shot hashing rejects, bypassing validation in prepared hashing and streaming updates.

2 open issues
1 resolved since last review

Review effort: Balanced

@eywalker

Copy link
Copy Markdown
Contributor Author

Closing without merging. The maintainer decided that starfix 0.6.0 (including the prepared-schema API of ITL-1177) is published from the Rust repository, https://github.com/metamorphic-labs/starfix (see PR 24 there), and that this pure-Python implementation stays frozen at 0.5.x. The branch is kept for reference. The two open review comments here are not being fixed in this repository; the extension-metadata case from the first one is carried over as a test case to the Rust implementation.

Posted by a Claude Code agent on behalf of @eywalker.

@eywalker eywalker closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants