Repository navigation
feat: ArrowDigester.prepare / PreparedSchema — analyse a schema once, hash many inputs (ITL-1177) - #13
feat: ArrowDigester.prepare / PreparedSchema — analyse a schema once, hash many inputs (ITL-1177)#13eywalker wants to merge 4 commits into
Conversation
…hash many inputs (ITL-1177) PreparedSchema holds the schema analysis (nodes, canonical key, schema digest, entry specs, data-support verdict) and hashes tables, record batches and streams of that schema without analysing it again. Its hashes equal the one-shot calls byte for byte. An input is accepted exactly when its schema has the prepared canonical schema key under the prepared options (the streaming rule); identical schemas are recognised by their Arrow IPC bytes, others by computing their key. No cache or module state. One-shot calls now analyse the schema once instead of three times, and ArrowDigester.update no longer re-analyses a batch of an identical schema. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a benchmark (ITL-1177) scripts/release_differential.py hashes seeded random tables under all option combinations, one-shot and prepared, and requires byte-identical results to the released starfix==0.5.0 (run in CI like the legacy differential). scripts/bench_prepared.py times one-row hashes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
…1177) The prepared-schema analysis test reset its counters before checking them, so it passed even if every prepared hash re-analysed the schema. It now asserts zero analysis calls after each prepared operation (digester(), update, finalize, hash_table, hash_record_batch, schema_digest) without resetting, and checks that prepare itself analyses once, so the zero counts cannot pass because the counters observe nothing. The streaming digester's update check moves to its own test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| if self._ipc is not None and _ipc_bytes(schema) == self._ipc: | ||
| return True | ||
| return schema_key(schema, self.include_metadata, self.flags) == self.key |
| (path, _entry_kind(field.type), field.nullable) for path, field in sorted(nodes, key=lambda n: n[0]) | ||
| ) | ||
| self._data_error = _data_support_error(nodes) | ||
| self._ipc = _ipc_bytes(schema) |
|
Closing without merging. The maintainer decided that starfix 0.6.0 (including the prepared-schema API of ITL-1177) is published from the Rust repository, https://github.com/metamorphic-labs/starfix (see PR 24 there), and that this pure-Python implementation stays frozen at 0.5.x. The branch is kept for reference. The two open review comments here are not being fixed in this repository; the extension-metadata case from the first one is carried over as a test case to the Rust implementation. Posted by a Claude Code agent on behalf of @eywalker. |



Summary
ArrowDigester.prepare(schema, *, include_metadata=False, ordered_columns=False, ordered_struct_fields=False) -> PreparedSchema(exported fromstarfix).PreparedSchemahasschema, the three options,schema_digest(35 bytes, equal tohash_schema),hash_record_batch(batch),hash_table(table)anddigester()(a fresh streamingArrowDigester). It is immutable and owned by the caller. starfix keeps no cache or module state (there is a test for this). Pickling prepares the schema again on load.hash_schema(input.schema, **options) == prepared.schema_digest. That means the same names, logical types and nullability at every depth, column order withordered_columns, struct field order withordered_struct_fields, and withinclude_metadatathe same field, schema and extension metadata. This is the ruleArrowDigester.updatealready applies. Anything else raisesValueError("Record batch schema does not match the prepared schema")("Table schema ..."for tables).pa.Schema.equals(check_metadata=True)is not used. It ignores nested field metadata of dictionary value types and the parameters of Python-defined extension types; both cases are regression tests.schema_digest; hashing data with them raises the existing error.Numbers
uv run python scripts/bench_prepared.py: a one-row table, 6 columns (int, strings, list, struct, metadata), a new table andSchemaobject per call. Apple Silicon, pyarrow 26, Python 3.12. The machine was under load, so values vary by about ±15%.hash_tablehash_table,include_metadata=Trueprepareitself: ~42 µs, or ~110 µs with metadata.PYTHONPATH=<checkout of main>/src uv run python scripts/bench_prepared.py.Test plan
uv run pytest tests/: 3683 passed. New intests/test_prepared_schema.py:uv run python scripts/release_differential.py: 29,360 one-shot results byte-identical to the releasedstarfix==0.5.0from PyPI (136 raise the same exception in both). No prepared result differs from its one-shot counterpart. Added as a CI job next tolegacy-differential; no permission changes.scripts/legacy_differential.py.Version
Proposed 0.6.0 (new API, no hash change). The version comes from the release tag (hatch-vcs), so this PR changes no version. The CHANGELOG has a
0.6.0 (unreleased)section. Releasing is the maintainers' call.Linear
ITL-1177
🤖 Generated with Claude Code