Skip to content

feat(golden, operators): provisional operators outside the v1 freeze; PolarsFilter is the first (ITL-1266) - #450

Open
eywalker wants to merge 2 commits into
mainfrom
eywalker/itl-1266-experimental-operators-a-group-whose
Open

eywalker wants to merge 2 commits into
mainfrom
eywalker/itl-1266-experimental-operators-a-group-whose

Conversation

@eywalker

@eywalker eywalker commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes ITL-1266. Related: ITL-1247, ITL-1265.

What this does

Data context v1 freezes every hash at 0.4.0. Provisional operators are outside that freeze. PolarsFilter is the first one.

  • The list. orcapod.operators.PROVISIONAL_OPERATORS (core/operators/provisional.py) maps each operator's name to the issue that decides its identity: {"PolarsFilter": "ITL-1247"}. Each listed operator also sets PROVISIONAL = True on its class. A test checks that the list and the markers agree.

  • The warning. Building a provisional operator issues ProvisionalHashWarning once per operator per process, at construction (this covers stream.polars_filter). The text says what is not guaranteed: the operator's identity, and so the stored paths and record ids of its results and of everything downstream of it.

  • The golden store. Provisional files live in data/v1/provisional/<group>.json. The manifest lists them, each with its sha256, in golden.provisional, outside golden.history. So no release revision lists them and --release never freezes them. The manifest pin excludes golden, and scripts/check_golden_tags.py does not compare these files. Their values are still computed and checked on every run.

  • The case declaration. A golden case declares the provisional operators it depends on: @case(group, name, provisional=("PolarsFilter",)). compute_group_kinds runs each case while the library records which provisional operators it builds. It refuses a case that builds one without declaring it, and a case that declares a name that is not in the list. A golden test checks that every entry sits in the kind of file its case declares.

  • The tool. generate_golden_hashes.py --relocate moves pinned entries, values copied unchanged, to the kind of file their case declares:

    • frozen to provisional. It refuses an entry in a released file, because a frozen value never leaves its file.
    • provisional to the open frozen file. This is how an operator leaves the group before a release.

    --add, --check and --remove handle provisional files too.

The moved list: 86 entries, no value changed

From To Cases Entries
pipeline-01.json provisional/pipeline.json polars_filter (9), polars_filter_expr (18), polars_filter_expr_job (49) 76
semantic-01.json provisional/semantic.json polars_expr (10: polars_expr_v1; only PolarsFilter feeds it) 10

Proof (docs/dev/scripts/itl_1266_moved_entries.py origin/main):

entries before: 1493, after: 1493
values changed: 0
moved 76: pipeline-01.json -> provisional/pipeline.json; cases ['polars_filter', 'polars_filter_expr', 'polars_filter_expr_job']
moved 10: semantic-01.json -> provisional/semantic.json; cases ['polars_expr']
moved in all: 86

The diff of pipeline-01.json and semantic-01.json only deletes lines. The 21 entries of polars_filter_expr_job that do not depend on the filter (node[subjects], node[scans]) each have the identical value pinned in the frozen job case, so no frozen value is lost.

Docs

  • docs/concepts/operators.md has a new section, Provisional operators: the list, what is not guaranteed, and the warning. The PolarsFilter section now carries a warning admonition.
  • docs/reference/golden-hashes.md describes the provisional files, the placement check, and how an operator leaves the group.
  • hashing.md, RELEASING.md, the golden README, the generator header, CLAUDE.md and .zed/rules are updated.
  • The changelog fragment is changelog.d/ITL-1266.md.

Open decision

The concept word ("provisional" or "experimental") is still with the developer. The word sits in few places. If "experimental" is chosen, everything is renamed in one mechanical commit, with no aliases. Follow-up, before 0.4.0: the six provisional logical types (119 golden entries) move to the same provisional files, using the same mechanism.

Merging with other golden PRs

This PR changes pipeline-01.json, semantic-01.json and revision 0 of v1.json, and so the previous and digest of revision 1. A PR that also edits v1.json (#434, #441, #448, #449) will conflict there. Whichever PR lands second takes main's golden files and v1.json, then re-runs its own tool steps (--relocate for this one, --remove / --add for the others). Never hand-merge digests. #448/#449 regenerate entries inside polars_filter_expr_job; --remove / --add work in the provisional files.

🤖 Generated with Claude Code

… PolarsFilter is the first (ITL-1266)

- PROVISIONAL_OPERATORS (core/operators/provisional.py) lists the operators whose
  identity is not frozen with data context v1; each also sets PROVISIONAL = True.
  Constructing one issues ProvisionalHashWarning once per operator per process and
  notes the use (utils/provisional.recording_provisional_uses).
- The golden store gains provisional files, data/v1/provisional/<group>.json, listed in
  golden.provisional of the manifest, outside the history: no release freezes them.
- A golden case declares the provisional operators it depends on; compute_group_kinds
  refuses a case that uses one without declaring it, and the golden test checks that
  every entry is in the kind of file its case declares.
- generate_golden_hashes.py --relocate moved the 86 entries of the four PolarsFilter
  cases (polars_filter, polars_filter_expr, polars_filter_expr_job, polars_expr) from
  pipeline-01.json and semantic-01.json into the provisional files. No value changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

  • Run on-demand review

This review includes 29 billable files and costs up to $7.25.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 53 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 123 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between d81eec8 and 2ff2064.

Files selected for processing: 29

  • .zed/rules
  • CLAUDE.md
  • RELEASING.md
  • changelog.d/ITL-1266.md
  • docs/concepts/operators.md
  • docs/dev/scripts/itl_1266_moved_entries.py
  • docs/reference/golden-hashes.md
  • docs/reference/hashing.md
  • scripts/check_golden_tags.py
  • src/orcapod/contexts/data/schemas/context_schema.json
  • src/orcapod/contexts/data/v1.json
  • src/orcapod/core/operators/__init__.py
  • src/orcapod/core/operators/filters.py
  • src/orcapod/core/operators/provisional.py
  • src/orcapod/core/operators/static_output_pod.py
  • src/orcapod/errors.py
  • src/orcapod/utils/provisional.py
  • tests/test_check_golden_tags.py
  • tests/test_core/operators/test_provisional_operators.py
  • tests/test_hashing/golden/README.md
  • tests/test_hashing/golden/cases.py
  • tests/test_hashing/golden/data/v1/pipeline-01.json
  • tests/test_hashing/golden/data/v1/provisional/pipeline.json
  • tests/test_hashing/golden/data/v1/provisional/semantic.json
  • tests/test_hashing/golden/data/v1/semantic-01.json
  • tests/test_hashing/golden/generate_golden_hashes.py
  • tests/test_hashing/golden/store.py
  • tests/test_hashing/golden/test_golden_hashes.py
  • tests/test_hashing/golden/test_golden_store.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

The operators page lists the provisional operators and what is not guaranteed; the
golden hash reference describes the provisional files, the placement check and how an
operator leaves the group; hashing.md, RELEASING.md, the golden README, the generator
header, CLAUDE.md and .zed/rules state the rule. Changelog fragment ITL-1266.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eywalker
eywalker marked this pull request as ready for review November 18, 2026 22:27
@codecov

codecov Bot commented Nov 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.61017% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/orcapod/core/operators/provisional.py 93.54% 2 Missing ⚠️
Files with missing lines Coverage Δ
src/orcapod/core/operators/__init__.py 100.00% <100.00%> (ø)
src/orcapod/core/operators/filters.py 84.28% <100.00%> (+0.22%) ⬆️
src/orcapod/core/operators/static_output_pod.py 92.22% <100.00%> (+0.16%) ⬆️
src/orcapod/errors.py 100.00% <ø> (ø)
src/orcapod/utils/provisional.py 100.00% <100.00%> (ø)
src/orcapod/core/operators/provisional.py 93.54% <93.54%> (ø)

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant