Skip to content

fix(crane): block incomplete parity completion#100

Merged
mrjf merged 1 commit into
mainfrom
codex/fix-crane-completion-gate
Jun 2, 2026
Merged

fix(crane): block incomplete parity completion#100
mrjf merged 1 commit into
mainfrom
codex/fix-crane-completion-gate

Conversation

@mrjf
Copy link
Copy Markdown
Contributor

@mrjf mrjf commented Jun 2, 2026

fix(crane): block incomplete parity completion

TL;DR

This PR fixes the false-completion path for the Python-to-Go Crane migration. An intentionally-incomplete behavior contract manifest can still produce progress reports, but it can no longer satisfy the completion gate or let the scorer reach deletion-grade readiness.

It also restores #78 to active migration state by ensuring the code now treats the current missing coverage as incomplete instead of complete.

Problem (WHY)

  • Crane Migration: Python to Go -- Full APM CLI Rewrite #78 was marked crane-completed even though tests/parity/python_contract_coverage.yml explicitly says the manifest is intentionally-incomplete.
  • The checker returned exit 0 for that status, so TestParityCompletionPythonBehaviorContracts could pass without any command or Python-test mappings.
  • The scorer could infer the behavior-contract gate from a passing test name, which made the completion path too trusting.
  • Current inventory evidence still reports 69 missing command mappings and 24,156 missing Python test mappings.

Approach (WHAT)

Area Change
Contract checker Make check strict by default; add --allow-intentionally-incomplete only for report-only summaries.
Go completion test Emit an explicit python_behavior_contracts Crane gate event with passing: 0/1 or 1/1; hard-fail only when enforcement is requested.
Scorer Parse explicit gate events from Go test output and stop inferring the behavior-contract gate from test names.
Migration CI Keep uploading/reporting incomplete coverage summaries without letting that report mode imply completion.
Tests/docs Add scorer regression tests and document that status: intentionally-incomplete is never completion evidence.

Implementation (HOW)

  • .crane/scripts/score.go: factors gate parsing/application, reads gate events embedded in go test -json output, and defaults python_behavior_contracts to failing when no explicit gate event appears.
  • cmd/apm/python_behavior_contracts_test.go: emits a Crane ratio gate before reporting or passing the behavior-contract completion test; APM_ENFORCE_PYTHON_BEHAVIOR_CONTRACTS=1 turns incomplete progress into a hard failure.
  • scripts/ci/python_behavior_contracts.py: keeps report-only behavior behind --allow-intentionally-incomplete; strict mode now fails if the manifest still declares status: intentionally-incomplete.
  • .github/workflows/migration-ci.yml: uses report-only mode only for the uploaded coverage summary.
  • tests/unit/test_crane_score.py: verifies the scorer cannot reach 1.0 from completion test names alone and blocks explicit incomplete behavior gates.
  • tests/parity/test_python_behavior_contracts.py and tests/parity/README.md: make incomplete coverage an expected tracking result by default and a hard failure when enforcement is requested.

Flow

Completion now needs strict checker success plus an explicit scorer gate event.

flowchart TD
    A["Extract Python behavior inventory"] --> B["Check coverage manifest"]
    B --> C{"status: intentionally-incomplete?"}
    C -->|yes, strict mode| D["Fail behavior-contract gate"]
    C -->|yes, report-only flag| E["Write summary only"]
    C -->|no missing findings| F["Emit passing gate event"]
    D --> G["migration_score < 1.0"]
    F --> H["Scorer may consider deletion-grade readiness"]
Loading

Trade-offs

  • Report-only mode remains available so CI can upload useful missing-coverage artifacts while the migration is in progress.
  • Default PR tracking stays mergeable while the emitted score blocks completion; explicit enforcement is available for final cutover checks.
  • Other completion gates still have legacy inference paths. This PR tightens the newly added exhaustive behavior-contract gate, which is the false-positive path that caused this issue.

Benefits

  1. Prevents migration_score: 1.0 while the behavior manifest is declared incomplete.
  2. Preserves actionable progress reporting with the full missing-coverage summary.
  3. Gives Crane a machine-readable python_behavior_contracts gate result instead of relying on test-name inference.
  4. Restores Crane Migration: Python to Go -- Full APM CLI Rewrite #78 as an active migration target for scheduled Crane runs.

Validation

Scenario Evidence

Scenario Evidence Result
Scorer cannot infer completion from test names alone tests/unit/test_crane_score.py 19 passed
Strict checker fails on incomplete manifest python_behavior_contracts.py check exit 1 with 24,225 findings
Report-only summaries still work python_behavior_contracts.py check --allow-intentionally-incomplete exit 0 with same findings
Normal Go CI does not fail without migration inventory go test ./cmd/apm -run TestParityCompletionPythonBehaviorContracts pass/skip path succeeds
Tracking mode reports incomplete behavior coverage without failing PR CI go test ./cmd/apm -run TestParityCompletionPythonBehaviorContracts with APM_PYTHON_CONTRACT_INVENTORY pass, emits passing:0,total:1 gate
Enforcement mode hard-fails incomplete behavior coverage same Go test with APM_ENFORCE_PYTHON_BEHAVIOR_CONTRACTS=1 expected failure
Command output
PATH=/opt/homebrew/bin:/Users/mrjf/.local/bin:$PATH UV_CACHE_DIR=/private/tmp/apm-uv-cache /Users/mrjf/.local/bin/uv run --extra dev pytest tests/unit/test_crane_score.py -q
19 passed in 61.71s (0:01:01)

UV_CACHE_DIR=/private/tmp/apm-uv-cache /Users/mrjf/.local/bin/uv run ruff check scripts/ci/python_behavior_contracts.py tests/parity/test_python_behavior_contracts.py tests/unit/test_crane_score.py
All checks passed!

UV_CACHE_DIR=/private/tmp/apm-uv-cache /Users/mrjf/.local/bin/uv run python scripts/ci/python_behavior_contracts.py check --summary /private/tmp/apm-strict-contract-summary.md
exit 1
Commands: 69
Python tests: 24156
Coverage findings: 24225
coverage manifest declares status: intentionally-incomplete; remove that status only after all findings are resolved

UV_CACHE_DIR=/private/tmp/apm-uv-cache /Users/mrjf/.local/bin/uv run python scripts/ci/python_behavior_contracts.py check --allow-intentionally-incomplete --summary /private/tmp/apm-report-contract-summary.md
exit 0
Commands: 69
Python tests: 24156
Coverage findings: 24225

GOCACHE=/private/tmp/apm-go-cache /opt/homebrew/bin/go test ./cmd/apm -run TestParityCompletionPythonBehaviorContracts -count=1
ok github.com/githubnext/apm/cmd/apm 0.780s

APM_PYTHON_BIN=/Users/mrjf/github/apm/.venv/bin/apm APM_PYTHON_CONTRACT_INVENTORY=/private/tmp/apm-behavior-contracts.json GOCACHE=/private/tmp/apm-go-cache /opt/homebrew/bin/go test ./cmd/apm -run TestParityCompletionPythonBehaviorContracts -count=1
ok github.com/githubnext/apm/cmd/apm 1.080s

APM_ENFORCE_PYTHON_BEHAVIOR_CONTRACTS=1 APM_PYTHON_BIN=/Users/mrjf/github/apm/.venv/bin/apm APM_PYTHON_CONTRACT_INVENTORY=/private/tmp/apm-behavior-contracts.json GOCACHE=/private/tmp/apm-go-cache /opt/homebrew/bin/go test ./cmd/apm -run TestParityCompletionPythonBehaviorContracts -count=1
{"crane":"gate","name":"python_behavior_contracts","passing":0,"total":1}
FAIL github.com/githubnext/apm/cmd/apm

How to test

  • Run UV_CACHE_DIR=/private/tmp/apm-uv-cache /Users/mrjf/.local/bin/uv run --extra dev pytest tests/unit/test_crane_score.py -q.
  • Run UV_CACHE_DIR=/private/tmp/apm-uv-cache /Users/mrjf/.local/bin/uv run ruff check scripts/ci/python_behavior_contracts.py tests/parity/test_python_behavior_contracts.py tests/unit/test_crane_score.py.
  • Run strict python_behavior_contracts.py check and confirm it fails while the manifest is incomplete.
  • Run the same checker with --allow-intentionally-incomplete and confirm it exits 0 for report generation only.
  • Run the behavior-contract Go test with and without APM_ENFORCE_PYTHON_BEHAVIOR_CONTRACTS=1 to confirm tracking passes and enforcement fails.
  • Verify Crane Migration: Python to Go -- Full APM CLI Rewrite #78 has crane-migration and not crane-completed.

@mrjf mrjf force-pushed the codex/fix-crane-completion-gate branch from 26d36c4 to f5c780b Compare June 2, 2026 05:35
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 2, 2026

Migration Benchmark Results

Migration CLI Benchmark

Max allowed Go/Python median ratio: 5.00

Command Python median Go median Go/Python Result Return codes
--help 0.4295s 0.0012s 0.00x 361.57x faster {'python': [0], 'go': [0]}
--version 0.4327s 0.0012s 0.00x 365.93x faster {'python': [0], 'go': [0]}
compile --help 0.4224s 0.0012s 0.00x 354.90x faster {'python': [0], 'go': [0]}
install --help 0.4245s 0.0012s 0.00x 361.33x faster {'python': [0], 'go': [0]}
pack --help 0.4283s 0.0012s 0.00x 348.04x faster {'python': [0], 'go': [0]}
audit --help 0.4295s 0.0012s 0.00x 357.21x faster {'python': [0], 'go': [0]}
init --yes 0.4301s 0.0012s 0.00x 348.13x faster {'python': [0], 'go': [0]}

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 2, 2026

Migration Benchmark Results

Migration CLI Benchmark

Max allowed Go/Python median ratio: 5.00

Command Python median Go median Go/Python Result Return codes
--help 0.3820s 0.0011s 0.00x 333.87x faster {'python': [0], 'go': [0]}
--version 0.3790s 0.0011s 0.00x 337.16x faster {'python': [0], 'go': [0]}
compile --help 0.3847s 0.0012s 0.00x 333.12x faster {'python': [0], 'go': [0]}
install --help 0.3863s 0.0012s 0.00x 321.04x faster {'python': [0], 'go': [0]}
pack --help 0.3822s 0.0012s 0.00x 318.40x faster {'python': [0], 'go': [0]}
audit --help 0.3797s 0.0011s 0.00x 344.12x faster {'python': [0], 'go': [0]}
init --yes 0.3887s 0.0012s 0.00x 326.87x faster {'python': [0], 'go': [0]}

@mrjf mrjf merged commit e564750 into main Jun 2, 2026
13 checks passed
@mrjf mrjf deleted the codex/fix-crane-completion-gate branch June 2, 2026 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant