feat(devtools): add lab schema commit, the real full-corpus persist path

[Sinity]

Summary

devtools lab schema generate --full-corpus is documented across this repo as the entry point for regenerating committed provider schema packages, but it only ever previews a generation and never writes to polylogue/schemas/providers/. This adds devtools lab schema commit, the actual persisting entry point.

Problem

generate_provider_schema() (what devtools lab schema generate calls, via infer_schema()) only returns a GenerationResult -- it never calls persist_generated_provider_bundle, the function that writes polylogue/schemas/providers/<provider>/versions/... via SchemaRegistry.replace_provider_packages. That write path only existed inside generate_all_schemas, which had zero CLI/devtools wiring: its only callers were polylogue/demo/workspace.py (demo seeding) and a unit test. A full-corpus regenerate against the live archive therefore silently no-ops on the committed package files while printing plausible-looking generation output (sample_count, versions, suggested corpus specs) -- confirmed live: a 138-day-old package showed zero git diff after a 30M-sample "full-corpus regenerate".

Solution

• New devtools lab schema commit command (devtools/schema_commit.py), backed by polylogue.schemas.operator.commit.commit_provider_schema. It calls generate_all_schemas for real and reports a before/after diff per package version: new / changed / unchanged, sample counts.
• generate_all_schemas gained a full_corpus passthrough parameter -- it never threaded the flag to _build_provider_bundle before, so full-corpus semantics were unreachable even from the one caller that did persist.
• Safety reporting: previously test-only _types_by_path (from tests/unit/schemas/test_promotion_monotonicity.py) is extracted to polylogue/schemas/type_narrowing.py so the production commit path and its test coverage share one implementation. commit reports (and exits non-zero on) any previously-committed leaf type that was lost or narrowed. In practice SchemaRegistry.replace_provider_packages already enforces monotonic merging end-to-end, so this is defense-in-depth, verified directly in test_thin_regeneration_window_cannot_narrow_committed_union.
• --dry-run runs the same real generation against a scratch copy of the committed provider directory (via shutil.copytree) so the report reflects real registry merge/carry-forward behavior without ever touching the committed tree.
• Checked whether devtools lab schema promote (promote_schema_cluster) makes this redundant: it does not. promote takes one reviewed evidence cluster (from generate --cluster mode) into a single registered package version -- a narrow, single-version operation. commit performs a full-corpus, potentially multi-version replace across every version generate_all_schemas produces for a provider. Documented this distinction in both modules' docstrings and in docs/internals.md.
• Updated docs/internals.md and docs/providers/index.md, which repeated the stale "generate is the entry point for committed packages" claim.

Verification

• devtools test tests/unit/schemas/test_operator_commit.py tests/unit/schemas/test_promotion_monotonicity.py tests/unit/devtools/test_schema_commit_command.py tests/unit/devtools/test_schema_lab_commands.py tests/unit/core/test_schema_generation.py -- 64 passed, 1 skipped (unrelated slow test).
• devtools verify --quick -- exit 0 (ruff format, ruff check, mypy --strict, render all --check, layering, closure-matrix, schema-versioning policy, schema promotion audit).
• devtools render devtools-reference and devtools render topology-projection regenerated for the new command/module and committed.
• Not run: the full non-quick devtools verify (testmon-affected pytest) -- this worktree had no seeded testmon baseline and seeding it runs close to the full non-slow suite, which is out of scope for a foreground-only verification pass on this change; the explicit file-selection run above covers every touched module and its existing neighbors.

Coordinator follow-up

Once merged, the real per-provider commit invocation is:

devtools lab schema commit --provider <provider> --full-corpus --dry-run   # preview first
devtools lab schema commit --provider <provider> --full-corpus            # persist for real

Per the task instructions, do not run this against the live archive until polylogue-u19l's quarantine-pruning pass lands.

Ref polylogue-k45pq
Ref polylogue-2qx.3

Summary by CodeRabbit

• New Features

  • Added lab schema commit to generate and persist provider schemas from the full archive.
  • Added dry-run, JSON output, sampling, privacy, and custom output options.
  • Reports per-version changes and flags narrowed schemas.

• Documentation

  • Clarified the distinction between preview generation, full-corpus commits, and schema promotion.

• Maintenance

  • Updated archive classification handling and required index replay for the latest schema version.

[coderabbitai]

Warning

Review limit reached

@Sinity, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 34 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 5bddbcba-52a9-4872-9477-55a6efc75928

📥 Commits

Reviewing files that changed from the base of the PR and between 8f921dc and cdc6a01.

📒 Files selected for processing (5)

• devtools/command_catalog.py
• devtools/schema_commit.py
• docs/devtools.md
• polylogue/schemas/type_narrowing.py
• tests/unit/schemas/test_promotion_monotonicity.py

📝 Walkthrough

Walkthrough

Adds full-corpus provider schema committing with dry-run support, schema change reporting, CLI registration, documentation, shared narrowing helpers, persistence tests, and index reparse metadata updates.

Changes

Schema commit workflow

Layer / File(s)	Summary
Schema commit contracts and comparison helpers polylogue/schemas/operator/models.py, polylogue/schemas/operator/inference.py, polylogue/schemas/generation/workflow.py, polylogue/schemas/type_narrowing.py, tests/unit/schemas/test_promotion_monotonicity.py	Adds request/result models, privacy payload conversion, full-corpus generation forwarding, and shared schema path comparison helpers.
Provider schema commit operator polylogue/schemas/operator/commit.py, polylogue/schemas/operator/workflow.py, tests/unit/schemas/test_operator_commit.py	Generates, compares, persists, and dry-runs provider schema packages. Tests cover new, changed, unchanged, narrowed, dry-run, and failed commits.
CLI command and workflow documentation devtools/command_catalog.py, devtools/schema_commit.py, devtools/schema_generate.py, docs/devtools.md, docs/internals.md, docs/providers/index.md, tests/unit/devtools/test_schema_commit_command.py	Registers and documents lab schema commit. The CLI supports full-corpus, dry-run, sampling, privacy, output, and JSON options. Tests cover request forwarding and exit behavior.
Semantic reparse and topology metadata polylogue/storage/sqlite/archive_tiers/index.py, polylogue/storage/sqlite/lifecycle.py, docs/plans/classifier-fingerprints.json, docs/plans/topology-target.yaml	Records index schema version 54 as a semantic reparse and updates classifier and topology metadata.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI as schema_commit CLI
  participant Operator as commit_provider_schema
  participant Generator as generate_all_schemas
  participant Packages as Provider schema packages
  CLI->>Operator: Submit SchemaCommitRequest
  Operator->>Generator: Generate full-corpus schemas
  Generator->>Packages: Write or stage schema packages
  Operator->>Packages: Compare previous and generated schemas
  Operator-->>CLI: Return SchemaCommitResult

Loading

Possibly related PRs

• Sinity/polylogue#2137: Introduced the related unified schema command namespace.
• Sinity/polylogue#3502: Covers related schema commit persistence and monotonicity behavior.
• Sinity/polylogue#3390: Modifies related schema generation, merging, and promotion workflows.

Suggested labels: area:schema, area:qa

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 31.25% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the new lab schema commit command and its full-corpus persistence purpose.
Description check	✅ Passed	The description covers the summary, problem, solution, verification, and follow-up details with specific commands and risks.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/devtools/schema-commit-full-corpus

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@devtools/command_catalog.py`:
- Around line 1571-1584: Escape or wrap the provider placeholder in inline code
within the use_when text of the “lab schema commit” CommandSpec so generated
Markdown displays it literally; then regenerate the command catalog to update
docs/devtools.md at line 68, with no direct manual change needed there beyond
the generated output.

In `@devtools/schema_commit.py`:
- Around line 74-90: Update main() around build_schema_privacy_config() so
ValueError from an invalid --privacy-config is caught, reported to stderr with a
“schema-commit:” prefixed message, and causes main() to return 1 instead of
exposing a traceback; preserve the existing schema commit flow for valid
configurations.

In `@polylogue/schemas/type_narrowing.py`:
- Around line 34-41: Update types_by_path to merge type sets when accumulating
results from multiple schema branches, preserving all types for duplicate paths
instead of allowing later found.update() calls to replace earlier values. Ensure
composed anyOf/oneOf branches retain both string and number types, and add a
regression test covering this behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a4622fc5-a892-4c2a-a330-053ceab16c63

📥 Commits

Reviewing files that changed from the base of the PR and between b751cf0 and 8f921dc.

📒 Files selected for processing (19)

• devtools/command_catalog.py
• devtools/schema_commit.py
• devtools/schema_generate.py
• docs/devtools.md
• docs/internals.md
• docs/plans/classifier-fingerprints.json
• docs/plans/topology-target.yaml
• docs/providers/index.md
• polylogue/schemas/generation/workflow.py
• polylogue/schemas/operator/commit.py
• polylogue/schemas/operator/inference.py
• polylogue/schemas/operator/models.py
• polylogue/schemas/operator/workflow.py
• polylogue/schemas/type_narrowing.py
• polylogue/storage/sqlite/archive_tiers/index.py
• polylogue/storage/sqlite/lifecycle.py
• tests/unit/devtools/test_schema_commit_command.py
• tests/unit/schemas/test_operator_commit.py
• tests/unit/schemas/test_promotion_monotonicity.py