diff --git a/ISA.md b/ISA.md index 772902fe..b4f82e19 100644 --- a/ISA.md +++ b/ISA.md @@ -2,7 +2,7 @@ project: cogant phase: current-readiness mode: source-grounded -updated: 2026-06-07 +updated: 2026-09-08 --- # COGANT Ideal State Artifact @@ -54,7 +54,7 @@ uv run --directory cogant python ../tools/regenerate_metrics.py uv run python scripts/z_generate_manuscript_variables.py --strict uv run python tools/check_metrics_fresh.py uv run python tools/audit_docs_constants.py -uv run python tools/audit_stage_list.py +uv run --directory cogant python ../tools/audit_stage_list.py uv run python tools/audit_manuscript_numbers.py --output /tmp/cogant_number_audit.md uv run python tools/audit_manuscript_markdown_links.py uv run python tools/audit_manuscript_crossrefs.py @@ -107,6 +107,15 @@ target. ## Changelog +- **2026-09-08 — Review-and-improvement pass.** Property-law tests re-pinned + to the fail-closed renderer contract (laws 4/5/6 + matrix stochasticity); + `run_all` fail-fast now persists the partial manifest/summary, capture + steps are timeout-bounded, interrupted git clones self-repair; math-adjacency + and coverage-table gates fail loud on unavailable inputs; cog-p1-02 closed + (parser capability registry + fallback contract tests); METRICS.yaml + regenerated against commit-bound HEAD and the manuscript chain re-rendered + (publication readiness: ready). Full suite 9612 passed / 0 failed / 47 + skipped, coverage 94.61%. See `REVIEW_LOG_2026-09-08.md`. - **2026-06-08 — Post-review hardening + held-out eval (round 2).** - **Guard audit (durable):** added `tools/audit_manuscript_math_adjacency.py` + `tests/test_audit_manuscript_math_adjacency.py` and wired it into the gate diff --git a/README.md b/README.md index 0ce6b3ca..7f1843af 100644 --- a/README.md +++ b/README.md @@ -51,8 +51,7 @@ Audit drift-sensitive docs and stubs: ```bash uv run python tools/audit_docs_constants.py uv run python tools/audit_folder_docs.py -uv run python tools/audit_pyi_exports.py -uv run python tools/audit_stage_list.py +uv run --directory cogant python ../tools/audit_stage_list.py uv run python tools/audit_manuscript_crossrefs.py uv run python tools/audit_manuscript_citations.py uv run python tools/audit_manuscript_numbers.py diff --git a/REVIEW_LOG_2026-09-08.md b/REVIEW_LOG_2026-09-08.md new file mode 100644 index 00000000..6c127b53 --- /dev/null +++ b/REVIEW_LOG_2026-09-08.md @@ -0,0 +1,134 @@ +# COGANT review-and-improvement pass log — 2026-09-08 + +Fleet review-and-improvement pass on `ActiveInferenceInstitute/COGANT` +(default branch `main`, base `39b961b` "chore(metrics): regenerate METRICS +against commit-bound HEAD"). Clean worktree at start; no pre-existing dirty +files. + +## Phase 0 — Preflight + +- Baseline gates verified at HEAD: manuscript audit battery green + (crossrefs 126 ids / 515 refs, citations 155/155, numbers 0 mismatches, + markdown links 0, math adjacency 0, claim scope 0, robustness 8/8, + formalisms, synthetic surfaces, module refs 88/0 unresolved, figure + renderers 14+3, coverage table, metrics freshness, folder docs, docs + constants, `.pyi` exports, publication readiness `verdict=ready`). +- Full package suite baseline: `uv run pytest tests/ -q` → 1 failed with + `-x` (property law 4), and `.pytest_cache/lastfailed` listed 39 stale + node-ids (most already fixed; cache had not been cleared). + +## Phase 1 — Review findings (7 parallel dimension scouts) + +Ranked outcomes per dimension: + +1. **Package tests** — 6 genuinely failing property tests + (`test_correctness_laws.py` laws 4/5/6, `test_law_matrix_stochasticity.py` + 5a/5b/5c) pinned a removed contract: they built source models with empty + A/B/C/D and expected `render_matrices_module` to invent defaults. The + renderer is now deliberately fail-closed ("no matrix is invented when the + source is incomplete"), so the tests contradicted the implemented + contract. The other lastfailed clusters (config loaders, language + fallbacks, package-init) were already green — stale cache. +2. **run_all orchestration** — `--fail-fast` returned mid-loop before + `run_manifest.json`/`summary.*` were written, discarding all completed + targets' evidence; `run_cmd_capture` had no timeout (a hung child blocks + the batch forever); an interrupted git clone (dest without `.git`) + poisoned every re-run; `export-gnn:{tid}` check label did not match the + recorded `export_gnn:{tid}` step name. +3. **Tools/scripts hygiene** — two silent-fallback paths weakened gates: + `audit_manuscript_math_adjacency._resolve` swallowed resolution failures + (auditing unsubstituted text = silent pass) and + `check_coverage_table._load_metrics_yaml` returned `{}` on missing or + broken METRICS.yaml. Entry points are otherwise thin with `__main__` + guards; no dead tools found; duplication findings noted as taste, not + fixed. +4. **Manuscript** — prose quality high, cross-refs/figures/citations clean; + the injected copy under `output/manuscript/` was stale relative to + METRICS.yaml (9560/53/48 vs 9606/6/47 and older timestamps). Fixed by + re-running the documented generator (regenerated outputs are disposable). + The audit battery contradicts the scout's "7 uncited bib entries" + observation: 155 keys defined, 155 used, 0 orphans. +5. **Plans vs implemented state** — cog-p1-02 (parser capability registry) + is fully implemented and accepted → closed; cog-p1-01 substantially + implemented but the schema-version migration mechanism is partial → kept + active; tasks.yaml has overdue `in_progress` rows (server-managed file, + left untouched, flagged to the owner); TODO.md cited a regression test at + a path that does not exist (historical section, left as dated history). +6. **REVIEW_LOG follow-through** — sole log `REVIEW_LOG_2026-08-02.md`; of + its three open items, the audit-test failure was already fixed + (5/5 green) and METRICS had been regenerated on 2026-08-19, but the + TODO checkboxes were never ticked. Notebook stubs remain deliberately + open. Both resolved items are now recorded; METRICS regenerated again + against this pass's commit-bound HEAD. +7. **Figures** — 18/18 registered figures pass strict copy + visual QA; no + orphaned manuscript references; only cosmetic extras (unregistered audit + PNG; missing source sidecars for eval fig1–fig4, tolerated by the copier). + +## Phase 2 — Implementation (commits) + +1. `9ac186d` — test(property): laws 4/5/6 + matrix-stochasticity strategies + supply complete valid source matrices; law4/law5 exercise the degenerate + aggregate broadcast (D `[1.0]` → uniform over declared cardinality, + single-column A → column-stochastic expansion). Template-contract test + accepts a sidecar clone named `COGANT`. +2. `68ce23e` — fix(runner): `FailFastExit` aborts after persisting the + partial manifest + summary; 1800s timeouts on capture steps; incomplete + git clones repaired; `export-gnn` label aligned. +3. `c14e1d9` — fix(tools): math-adjacency and coverage-table gates fail + loud when their inputs are unavailable. +4. (this pass) — TODO.md reconciliation (cog-p1-02 closed, dated pass + section, open items ticked), REVIEW_LOG (this file), README + `audit_stage_list` invocation corrected to the inner-package env form + matching CI, METRICS.yaml + manuscript/figure/report artifacts + regenerated against commit-bound HEAD. + +## Phase 3 — Verification + +- Full package suite: **9612 passed / 0 failed / 47 skipped**, coverage + 94.61% against the 89% line gate (12m09s). +- Root tools lane (`uv run pytest tests/` at the project root, minus the + three tests that require the inner package env — those pass 15/15 under + `uv run --directory cogant`): the template-contract test is fixed; 10 + remaining failures are pre-existing at HEAD (verified by stash) and + environmental: the root compatibility-shell venv intentionally installs + only `pyyaml`+`pytest`, while four test files invoke audit tools that + import `cogant.*` internals (pydantic). CI never runs this lane — every + audit gate runs via `uv run --directory cogant` (`.github/workflows/ci.yml`), + which is green. +- `./run_all.sh --dry-run` → exit 0 (24 targets); batch dashboard + regeneration → exit 0. +- Manuscript generator re-run → publication readiness `verdict=ready`, + blockers=0; visual QA 18/18. +- Release gate (`tools/release_gate.py`) lane matrix, root venv + (Python 3.14.6): package-tests **passed**, ruff **passed**, mypy + **passed**, rust-format **passed**; rust lanes require + `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` (PyO3 0.22.6 caps at 3.13) and + then rust-check / rust-test / rust-clippy all **pass**; wheel-smoke + **fails** only because no pydantic_core binary supports 3.14. Under the + inner 3.12 env the wheel toolchain is supported but the gate's + package-tests lane exceeds its 1800s cap (inner addopts add coverage + + verbose output and the gate runs rust lanes concurrently). All failures + are environmental (interpreter/toolchain), not code regressions: this + pass touches no Rust, wheel, or dependency code, and the identical + pre-pass HEAD fails the same lanes under the same root venv. + +## Open / deferred + +- **Root-lane environmental test failures** (10, pre-existing at HEAD): + `tests/test_audit_stage_list.py` (4), `tests/test_audit_synthetic_surfaces.py` (2), + `tests/test_check_metrics_fresh.py` (3), `tests/test_manuscript_figures.py` (1). + Fix options: skip-if-`cogant.config`-unavailable markers, or run the tools + under the inner env. Not caused by this pass; out of every configured CI gate. +- **tasks.yaml date slips** (owner/taskboard server): six `in_progress` rows + with past end dates and `cog-p4-01` carrying a future `start` while active. +- **Notebook stubs** (`cogant/docs/notebooks/*.md`, 12 pages) — unchanged, + deliberate placeholders. +- **cog-p1-01** schema-version migration mechanism remains partial (field + exists; no version-to-version migration code). +- **Pre-existing ruff I001/SIM115 in `tools/run_all_runner.py`** — outside + the configured lint scope (CI lints `py/cogant/ tests/` in the inner + package only); left untouched. +- **Release gate wheel-smoke under Python 3.14** — needs a PyO3/pydantic + toolchain that supports 3.14, or running the gate from a ≤3.13 + interpreter with a package-tests timeout budget that accommodates the + inner env's coverage addopts. Environmental; out of scope for this pass. diff --git a/TODO.md b/TODO.md index 0ed2d6e0..20aca0ba 100644 --- a/TODO.md +++ b/TODO.md @@ -2,7 +2,10 @@ - Status: Active (P0–P5 roadmap below). - Owner: DAF. -- Last reviewed: 2026-08-02 (docs-deep review pass: Minor/Medium findings implemented; see "Completed/Closed — 2026-08-02 docs-deep review pass"). +- Last reviewed: 2026-09-08 (review-and-improvement pass: property-law tests + re-pinned to the fail-closed renderer contract, runner fail-fast evidence + persistence, gate honesty fixes, METRICS regenerated against commit-bound + HEAD; see "Completed/Closed — 2026-09-08 review pass"). This is the future-only execution backlog. Completed work belongs in Git history, generated evidence, audit reports, or the taskboard; it must not be @@ -87,15 +90,6 @@ Source-of-truth rules: - Acceptance: compatibility fixtures prove precedence `defaults < preset < file < environment < CLI`; no parallel registry remains. -- [~] **cog-p1-02 — Explicit parser capabilities.** - - Priority: P1. - - Depends on: cog-p0-01, cog-p0-03. - - Deliverable: registry-only parser selection with language, extensions, - implementation, optional dependency, fallback/degraded status, and source - provenance. - - Acceptance: tree-sitter is selected when installed; fallback is observable; - unavailable-parser errors and capability tests cover every registered language. - - [~] **cog-p1-03 — Fail-closed pipeline contracts.** - Priority: P1. - Depends on: cog-p1-01, cog-p1-02. @@ -509,11 +503,48 @@ layout, and METRICS.yaml before editing. Commits: `d86528c`, `d2d7156`. "(planned)" placeholders with one-line descriptions; filling them needs the Jupyter toolchain and is a major effort. Left as-is (they are honestly labeled in the nav). -- [ ] **Pre-existing audit test failure** — - `tests/test_audit_docs_constants.py::test_roundtrip_claim_audit_accepts_current_ledger_claim` - fails at HEAD without this pass's changes (verified by stash); not caused by the - docs pass. Owner should reconcile the test fixture with the current qualifier - regex. -- [ ] **METRICS regeneration** — `METRICS.yaml` must be regenerated against the - new HEAD (`tools/regenerate_metrics.py`) before the metrics-fresh / release - gates can claim commit-bound freshness (see `cog-p0-04`). +- [x] **Pre-existing audit test failure** — resolved: the 2026-08-19 fleet + review pass reconciled the qualifier regex; the targeted test now passes + 5/5 (`uv run pytest -q --no-cov tests/test_audit_docs_constants.py`). +- [x] **METRICS regeneration** — done in the 2026-09-08 pass: + `tools/regenerate_metrics.py` ran against commit-bound HEAD and + `check_metrics_fresh.py --fail-on-dirty` is green. + +## Completed/Closed — 2026-09-08 review-and-improvement pass + +Items implemented and closed this pass (verified by targeted runs; the full +package suite is green — 9612 passed / 0 failed / 47 skipped, coverage +94.61% against the 89% line gate): + +- [x] **Property-law tests re-pinned to the fail-closed renderer contract** + (`9ac186d`): laws 4/5/6 and the matrix-stochasticity laws built source + models with empty A/B/C/D and expected the renderer to invent defaults — + a contract `render_matrices_module` removed when matrix synthesis went + fail-closed. The strategies now supply complete valid matrices, and + law4/law5 exercise the degenerate aggregate broadcast (D `[1.0]` → uniform + over the declared cardinality; single-column A → column-stochastic + expansion). +- [x] **Runner fail-fast evidence persistence** (`68ce23e`): `--fail-fast` + previously returned mid-loop before `run_manifest.json`/summary were + written, discarding every completed target's evidence. `FailFastExit` + now aborts after persisting the partial manifest and summary; capture + steps gained a 1800s timeout; interrupted git clones (dest without + `.git`) are repaired instead of poisoning every re-run; the `export-gnn` + check label now matches the recorded `export_gnn:{tid}` step name. +- [x] **Gate honesty fixes** (`c14e1d9`): `audit_manuscript_math_adjacency` + no longer silently audits unsubstituted text when variable resolution + fails; `check_coverage_table` raises on missing/unparseable METRICS.yaml + instead of degrading benchmark sidecar comparisons. +- [x] **cog-p1-02 — Explicit parser capabilities**: closed. The canonical + registry (`cogant/py/cogant/parsers/registry.py` with per-language + modules and `.pyi` stubs) plus the fallback/unavailable contract tests + (`tests/unit/test_ingest_language_detection_fallbacks.py`, 13/13 green) + satisfy the deliverable and acceptance. cog-p1-01 remains active: the + schema-version migration mechanism is still partial. +- [x] **Template-contract test de-coupled from checkout directory case** + (`9ac186d`): a sidecar clone named `COGANT` no longer fails + `test_project_root_points_to_cogant_staging_root`. +- [x] **REVIEW_LOG_2026-08-02 open items**: the audit-test failure was + already fixed (see the ticked 2026-08-02 item above); METRICS.yaml + regenerated against commit-bound HEAD this pass; the 12 notebook stubs + remain deliberately open. diff --git a/cogant/evaluation/METRICS.yaml b/cogant/evaluation/METRICS.yaml index d281edc2..807a5eca 100644 --- a/cogant/evaluation/METRICS.yaml +++ b/cogant/evaluation/METRICS.yaml @@ -3,8 +3,8 @@ # DO NOT EDIT BY HAND. Run: cd cogant && uv run python ../tools/regenerate_metrics.py schema_version: '1.0' -generated_at: '2026-08-19T15:55:54.237757Z' -generator_git_sha: 36371f163920e1f5e13e4c93d65e4d09f6a8317c +generated_at: '2026-09-08T22:48:12.814800Z' +generator_git_sha: c14e1d9887cff323702329b5b9d2995d015346e5 cogant_version: 0.6.0 package: name: cogant @@ -12,8 +12,8 @@ package: python_min: '3.11' testing: test_count_total: 9659 - test_count_passing: 9606 - test_count_failing: 6 + test_count_passing: 9612 + test_count_failing: 0 test_count_skipped: 47 test_count_xfailed: 0 test_count_xpassed: 0 @@ -517,7 +517,7 @@ literature: source: cogant/docs/evaluation/LITERATURE.md bibliography_entries: 104 benchmark: - suite_runtime_s: 446.5 + suite_runtime_s: 608.32 shipped_fixture_count: 12 benchmark_suite_file: suite_20260818.md benchmark_python_version: 3.12.13 diff --git a/cogant/tests/property/test_correctness_laws.py b/cogant/tests/property/test_correctness_laws.py index 1f22451c..2190c177 100644 --- a/cogant/tests/property/test_correctness_laws.py +++ b/cogant/tests/property/test_correctness_laws.py @@ -120,21 +120,20 @@ def test_law3_role_symmetry(a: dict[str, float], b: dict[str, float]) -> None: # --------------------------------------------------------------------------- -@given( - n_states=st.integers(min_value=1, max_value=10), - n_obs=st.integers(min_value=1, max_value=10), - n_actions=st.integers(min_value=1, max_value=5), -) +@given(n_states=st.integers(min_value=1, max_value=10)) @settings(max_examples=100, deadline=None) -def test_law4_d_vector_normalization(n_states: int, n_obs: int, n_actions: int) -> None: - """Generated D vector must sum to approximately 1.0.""" +def test_law4_d_vector_normalization(n_states: int) -> None: + """Broadcast D vector must be uniform and sum to approximately 1.0.""" model = ReverseGNNModel( model_name="test_model", - hidden_states=[f"s_f{i}" for i in range(n_states)], - observations=[f"o_m{i}" for i in range(n_obs)], - actions=[f"u_c{i}" for i in range(n_actions)], - # D is left empty so render_matrices_module generates the default - # uniform prior which must sum to 1.0. + hidden_states=["s_f0"], + cardinalities={"s_f0": n_states}, + # The forward pipeline's degenerate aggregate prior (all mass on a + # single aggregate state). The renderer is fail-closed and never + # invents matrices, but it must broadcast the degenerate single-entry + # D uniformly over the declared factor cardinality, so the emitted D + # has n_states entries summing to 1.0. + D=[1.0], ) source = render_matrices_module(model) @@ -145,28 +144,39 @@ def test_law4_d_vector_normalization(n_states: int, n_obs: int, n_actions: int) assert len(d_values) == n_states, f"D has {len(d_values)} entries, expected {n_states}" d_sum = sum(d_values) assert math.isclose(d_sum, 1.0, abs_tol=1e-4), f"D vector sum={d_sum}, expected ~1.0" + expected = 1.0 / n_states + for i, value in enumerate(d_values): + assert math.isclose(value, expected, abs_tol=1e-4), ( + f"D[{i}]={value}, expected uniform {expected}" + ) # --------------------------------------------------------------------------- # Law 5: A shape consistency -# If n_obs and n_states > 0 and A is empty -> generated A has shape -# [n_obs][n_states] +# A degenerate single-column aggregate A broadcasts to shape +# [n_obs][n_states] with column-stochastic output # --------------------------------------------------------------------------- @given( - n_states=st.integers(min_value=1, max_value=8), + n_states=st.integers(min_value=2, max_value=8), n_obs=st.integers(min_value=1, max_value=8), ) @settings(max_examples=100, deadline=None) def test_law5_a_shape_consistency(n_states: int, n_obs: int) -> None: - """When A is empty but n_obs and n_states > 0, generated A must be [n_obs][n_states].""" + """A degenerate aggregate A must broadcast to [n_obs][n_states], columns stochastic.""" + # Column-stochastic single-column aggregate A (one aggregate state + # variable). The renderer is fail-closed, but it must broadcast the + # aggregate column across the declared hidden-state cardinality while + # keeping every emitted column a proper distribution. model = ReverseGNNModel( model_name="shape_test", - hidden_states=[f"s_f{i}" for i in range(n_states)], - observations=[f"o_m{i}" for i in range(n_obs)], - actions=["u_c0"], - # A left empty to trigger fallback generation. + hidden_states=["s_f0"], + cardinalities={"s_f0": n_states}, + observations=["o_m0"], + C=[0.0] * n_obs, + A=[[1.0 / n_obs] for _ in range(n_obs)], + D=[1.0], ) source = render_matrices_module(model) @@ -180,9 +190,17 @@ def test_law5_a_shape_consistency(n_states: int, n_obs: int) -> None: assert a_block_match is not None, "A matrix block not found in generated source" row_matches = re.findall(r"\[([^\[\]]+)\]", a_block_match.group(1)) assert len(row_matches) == n_obs, f"A has {len(row_matches)} rows, expected {n_obs}" + columns: list[list[float]] = [[] for _ in range(n_states)] for i, row_str in enumerate(row_matches): cols = [float(x.strip()) for x in row_str.split(",")] assert len(cols) == n_states, f"A row {i} has {len(cols)} cols, expected {n_states}" + for j, value in enumerate(cols): + columns[j].append(value) + for j, column in enumerate(columns): + col_sum = sum(column) + assert math.isclose(col_sum, 1.0, abs_tol=1e-4), ( + f"A column {j} sums to {col_sum}, expected ~1.0" + ) # --------------------------------------------------------------------------- @@ -203,11 +221,25 @@ def test_law6_transition_normalization( action_idx: int, ) -> None: """transition(uniform_dist, k) must return a distribution summing to ~1.0.""" + # The renderer is fail-closed, so supply the identity B the aggregate + # broadcast convention emits for a single-factor model: every action + # column of transition(uniform, k) must still be a proper distribution. model = ReverseGNNModel( model_name="trans_test", - hidden_states=[f"s_f{i}" for i in range(n_states)], - observations=[f"o_m{i}" for i in range(max(1, n_states))], + hidden_states=["s_f0"], + cardinalities={"s_f0": n_states}, + observations=["o_m0"], + C=[0.0], actions=[f"u_c{i}" for i in range(n_actions)], + A=[[1.0]], + D=[1.0], + B=[ + [ + [1.0 if row == col else 0.0 for _ in range(n_actions)] + for col in range(n_states) + ] + for row in range(n_states) + ], ) source = render_matrices_module(model) diff --git a/cogant/tests/property/test_law_matrix_stochasticity.py b/cogant/tests/property/test_law_matrix_stochasticity.py index d00006dd..b0986aea 100644 --- a/cogant/tests/property/test_law_matrix_stochasticity.py +++ b/cogant/tests/property/test_law_matrix_stochasticity.py @@ -58,29 +58,47 @@ def reverse_model(draw) -> ReverseGNNModel: model.actions = [f"u_c{i}" for i in range(n_actions)] model.raw_model_name = "law5_model" - # Randomly decide whether to supply an explicit A/B/C/D or let - # the renderer fall back to shape-consistent defaults. Both - # branches must produce stochastic columns. - if draw(st.booleans()): - # Provide an A where each column is a random positive vector; - # the renderer is documented to leave A untouched on the - # happy path, so we also normalise it up-front to match the - # post-render invariant. - raw_rows = [ - [draw(st.floats(min_value=0.1, max_value=10.0)) for _ in range(n_states)] - for _ in range(n_obs) + # The reverse renderer is fail-closed: it never invents a missing + # matrix, so the strategy always supplies a complete, valid source + # model — column-stochastic A, column-stochastic B per action, zero + # preferences, and a normalized D. Both the source model and the + # emitted module must satisfy the stochasticity invariants. + raw_rows = [ + [draw(st.floats(min_value=0.1, max_value=10.0)) for _ in range(n_states)] + for _ in range(n_obs) + ] + col_sums = [sum(raw_rows[i][j] for i in range(n_obs)) for j in range(n_states)] + model.A = [[raw_rows[i][j] / col_sums[j] for j in range(n_states)] for i in range(n_obs)] + + raw_d = [draw(st.floats(min_value=0.1, max_value=10.0)) for _ in range(n_states)] + d_total = sum(raw_d) + model.D = [v / d_total for v in raw_d] + + raw_b = [ + [ + [draw(st.floats(min_value=0.1, max_value=10.0)) for _ in range(n_actions)] + for _ in range(n_states) ] - col_sums = [sum(raw_rows[i][j] for i in range(n_obs)) for j in range(n_states)] - model.A = [[raw_rows[i][j] / col_sums[j] for j in range(n_states)] for i in range(n_obs)] - # else: leave A empty -> renderer emits uniform 1/n_obs columns. - - if draw(st.booleans()): - # Leave D empty so the renderer synthesises a uniform prior. - pass - else: - raw_d = [draw(st.floats(min_value=0.1, max_value=10.0)) for _ in range(n_states)] - total = sum(raw_d) - model.D = [v / total for v in raw_d] + for _ in range(n_states) + ] + b_col_sums = [ + [ + sum(raw_b[target][source][action] for target in range(n_states)) + for action in range(n_actions) + ] + for source in range(n_states) + ] + model.B = [ + [ + [ + raw_b[target][source][action] / b_col_sums[source][action] + for action in range(n_actions) + ] + for source in range(n_states) + ] + for target in range(n_states) + ] + model.C = [0.0] * n_obs return model diff --git a/tests/test_template_contract.py b/tests/test_template_contract.py index 50fd2088..8120df5b 100644 --- a/tests/test_template_contract.py +++ b/tests/test_template_contract.py @@ -8,7 +8,11 @@ def test_project_root_points_to_cogant_staging_root() -> None: root = project_root() - assert root.name == "cogant" + # The staging root's directory name is "cogant" at the template render + # location (projects/working/cogant) but "COGANT" in a sidecar clone of + # the repo, so only the case-insensitive name plus the layout contract + # are stable across checkouts. + assert root.name.lower() == "cogant" assert (root / "AGENTS.md").is_file() assert (root / "cogant" / "pyproject.toml").is_file() diff --git a/tools/audit_manuscript_math_adjacency.py b/tools/audit_manuscript_math_adjacency.py index 418123b6..588375e3 100644 --- a/tools/audit_manuscript_math_adjacency.py +++ b/tools/audit_manuscript_math_adjacency.py @@ -82,17 +82,16 @@ def find_digit_adjacent_math(text: str) -> list[tuple[int, int, str]]: def _resolve(text: str) -> str: - """Best-effort manuscript-variable resolution so token-adjacent leaks - (``$-${{COUNT}}`` -> ``$-$10``) are caught. Falls back to raw text if the - injector or METRICS.yaml is unavailable.""" - try: - sys.path.insert(0, str(ROOT / "tools")) - import inject_manuscript_vars as inj # type: ignore - - resolved, _ = inj.inject(text, inj.load_metrics(), dry_run=False) - return resolved - except Exception: - return text + """Resolve manuscript variables so token-adjacent leaks + (``$-${{COUNT}}`` -> ``$-$10``) are caught. The audit runs from the + inner package environment (``uv run --directory cogant``), where + ``load_metrics`` is importable; a resolution failure is a gate error, + not a reason to audit unsubstituted text, so it propagates.""" + sys.path.insert(0, str(ROOT / "tools")) + import inject_manuscript_vars as inj # type: ignore + + resolved, _ = inj.inject(text, inj.load_metrics(), dry_run=False) + return resolved def audit(manuscript_dir: Path) -> list[str]: diff --git a/tools/check_coverage_table.py b/tools/check_coverage_table.py index b11ea6f8..3520fe1c 100644 --- a/tools/check_coverage_table.py +++ b/tools/check_coverage_table.py @@ -151,16 +151,20 @@ def _load_json(path: Path) -> Any | None: def _load_metrics_yaml(package_root: Path) -> dict[str, Any]: + """Load METRICS.yaml; a missing or unparseable file is a gate error, + never a silent degradation of the benchmark sidecar comparison.""" metrics_path = package_root / "evaluation" / "METRICS.yaml" if not metrics_path.is_file(): - return {} + raise SystemExit(f"check_coverage_table: METRICS.yaml missing at {metrics_path}") try: import yaml data = yaml.safe_load(metrics_path.read_text(encoding="utf-8")) - except Exception: - return {} - return data if isinstance(data, dict) else {} + except Exception as exc: + raise SystemExit(f"check_coverage_table: cannot parse {metrics_path}: {exc}") from exc + if not isinstance(data, dict): + raise SystemExit(f"check_coverage_table: {metrics_path} is not a mapping") + return data def _compare_metric_row( diff --git a/tools/run_all_runner.py b/tools/run_all_runner.py index 3363ea0d..2bdc5881 100644 --- a/tools/run_all_runner.py +++ b/tools/run_all_runner.py @@ -210,6 +210,7 @@ def run_cmd_capture( out_path: Path, log_fp: Any, step: str = "capture", + timeout: float | None = None, ) -> CommandResult: print(f"+ {' '.join(argv)} > {out_path}", file=log_fp, flush=True) if dry_run: @@ -217,13 +218,21 @@ def run_cmd_capture( return CommandResult(0, 0.0) out_path.parent.mkdir(parents=True, exist_ok=True) t0 = time.monotonic() - proc = subprocess.run( - argv, - cwd=str(cwd), - env=os.environ.copy(), - capture_output=True, - text=True, - ) + try: + proc = subprocess.run( + argv, + cwd=str(cwd), + env=os.environ.copy(), + capture_output=True, + text=True, + timeout=timeout, + ) + except subprocess.TimeoutExpired as exc: + dt = time.monotonic() - t0 + partial = f"{exc.stdout or ''}{exc.stderr or ''}" + out_path.write_text(partial, encoding="utf-8") + report(f" [{step}] TIMEOUT after {timeout}s", log_fp=log_fp) + return CommandResult(124, dt) dt = time.monotonic() - t0 combined = (proc.stdout or "") + (proc.stderr or "") out_path.write_text(combined, encoding="utf-8") @@ -306,6 +315,11 @@ def ensure_git_clone( if dest.exists() and (dest / ".git").is_dir(): report(f" [git_clone] skip (existing clone) {dest}", log_fp=log_fp) return CommandResult(0, 0.0) + if dest.exists(): + # An interrupted clone leaves a partial directory without .git that + # would fail every subsequent clone with "already exists"; remove it. + report(f" [git_clone] removing incomplete clone at {dest}", log_fp=log_fp) + shutil.rmtree(dest) dest.parent.mkdir(parents=True, exist_ok=True) cmd = ["git", "clone"] @@ -357,6 +371,14 @@ class BundleSummaryError(RuntimeError): """Raised when a completed target cannot provide verifiable evidence.""" +class FailFastExit(RuntimeError): + """Raised when --fail-fast aborts the batch; carries the process exit code.""" + + def __init__(self, exit_code: int) -> None: + super().__init__(f"fail-fast exit {exit_code}") + self.exit_code = exit_code + + def _read_bundle(run_dir: Path) -> dict[str, Any]: """Return a parsed bundle or fail closed with an actionable error.""" for candidate in (run_dir / "data" / "bundle.json", run_dir / "bundle.json"): @@ -785,7 +807,7 @@ def check(result: CommandResult | int, label: str) -> int: report(f"warning: {label} exited {code}", log_fp=log_fp) failures.append(label) if options.fail_fast: - return code + raise FailFastExit(code) return 0 if manuscript.get("enabled"): @@ -806,8 +828,7 @@ def check(result: CommandResult | int, label: str) -> int: log_fp=log_fp, step="manuscript_variables", ) - if rc := check(result, "manuscript_variables"): - return rc + check(result, "manuscript_variables") if steps.get("doctor"): result = run_cmd( @@ -817,8 +838,7 @@ def check(result: CommandResult | int, label: str) -> int: log_fp=log_fp, step="doctor", ) - if rc := check(result, "doctor"): - return rc + check(result, "doctor") output_root.mkdir(parents=True, exist_ok=True) @@ -861,7 +881,7 @@ def check(result: CommandResult | int, label: str) -> int: entry["failure_reason"] = "target needs either path or git_url" manifest["targets"].append(entry) if options.fail_fast: - return 2 + raise FailFastExit(2) continue manifest["targets"].append(entry) @@ -885,8 +905,7 @@ def check(result: CommandResult | int, label: str) -> int: entry["commands"].append( command_record("git clone", step=f"git_clone:{tid}", result=result) ) - if rc := check(result, f"git_clone:{tid}"): - return rc + check(result, f"git_clone:{tid}") if not options.dry_run and not target_path.exists(): print(f"error: missing target path {target_path}", file=sys.stderr) @@ -902,7 +921,7 @@ def check(result: CommandResult | int, label: str) -> int: "dashboard": False, } if options.fail_fast: - return 2 + raise FailFastExit(2) continue # With --layout-output the bundle is relocated to /data/bundle.json. @@ -943,8 +962,7 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: entry["commands"].append( command_record(tr, step=f"translate:{tid}", result=result) ) - if rc := check(result, f"translate:{tid}"): - return rc + check(result, f"translate:{tid}") # Re-resolve after translate so downstream gates see the real path. bundle_json = _bundle_path() if not options.dry_run and not bundle_json.is_file(): @@ -964,12 +982,12 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: out_path=run_dir / "scan.json", log_fp=log_fp, step=f"scan:{tid}", + timeout=1800, ) entry["commands"].append( command_record(scan_cmd, step=f"scan:{tid}", result=result) ) - if rc := check(result, f"scan:{tid}"): - return rc + check(result, f"scan:{tid}") if steps.get("graph_stdout"): graph_cmd = ["uv", "run", "cogant", "graph", str(target_path)] @@ -980,12 +998,12 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: out_path=run_dir / "graph.txt", log_fp=log_fp, step=f"graph:{tid}", + timeout=1800, ) entry["commands"].append( command_record(graph_cmd, step=f"graph:{tid}", result=result) ) - if rc := check(result, f"graph:{tid}"): - return rc + check(result, f"graph:{tid}") if steps.get("export_gnn") and (options.dry_run or bundle_json.is_file()): eg_dir = run_dir / "export_gnn" @@ -1011,8 +1029,7 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: entry["commands"].append( command_record(ex, step=f"export_gnn:{tid}", result=result) ) - if rc := check(result, f"export-gnn:{tid}"): - return rc + check(result, f"export_gnn:{tid}") if steps.get("render_site") and (options.dry_run or bundle_json.is_file()): site_dir = run_dir / "site" @@ -1031,8 +1048,7 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: entry["commands"].append( command_record(rv, step=f"render:{tid}", result=result) ) - if rc := check(result, f"render:{tid}"): - return rc + check(result, f"render:{tid}") if steps.get("viz_png") and (options.dry_run or run_dir.is_dir()): vz = ["uv", "run", "cogant", "viz", str(run_dir)] @@ -1042,8 +1058,7 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: entry["commands"].append( command_record(vz, step=f"viz:{tid}", result=result) ) - if rc := check(result, f"viz:{tid}"): - return rc + check(result, f"viz:{tid}") if steps.get("validate_run_dir") and (options.dry_run or run_dir.is_dir()): val = ["uv", "run", "cogant", "validate", str(run_dir)] @@ -1056,12 +1071,12 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: out_path=run_dir / "validate.txt", log_fp=log_fp, step=f"validate:{tid}", + timeout=1800, ) entry["commands"].append( command_record(val, step=f"validate:{tid}", result=result) ) - if rc := check(result, f"validate:{tid}"): - return rc + check(result, f"validate:{tid}") explain_node = t.get("explain") if explain_node and isinstance(explain_node, str) and explain_node.strip(): @@ -1082,12 +1097,12 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: out_path=run_dir / "explain.json", log_fp=log_fp, step=f"explain:{tid}", + timeout=1800, ) entry["commands"].append( command_record(ex, step=f"explain:{tid}", result=result) ) - if rc := check(result, f"explain:{tid}"): - return rc + check(result, f"explain:{tid}") if steps.get("roundtrip") and (options.dry_run or target_path.exists()): rt_dir = run_dir / "roundtrip" @@ -1116,8 +1131,7 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: entry["commands"].append( command_record(rt, step=f"roundtrip:{tid}", result=result) ) - if rc := check(result, f"roundtrip:{tid}"): - return rc + check(result, f"roundtrip:{tid}") batch_api = STAGING_ROOT / "tools" / "batch_api.py" @@ -1143,8 +1157,7 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: entry["commands"].append( command_record(ag, step=f"analyze_graph:{tid}", result=result) ) - if rc := check(result, f"analyze_graph:{tid}"): - return rc + check(result, f"analyze_graph:{tid}") if steps.get("analyze_static") and (options.dry_run or target_path.exists()): asta = [ @@ -1168,8 +1181,7 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: entry["commands"].append( command_record(asta, step=f"analyze_static:{tid}", result=result) ) - if rc := check(result, f"analyze_static:{tid}"): - return rc + check(result, f"analyze_static:{tid}") if steps.get("export_multi") and (options.dry_run or bundle_json.is_file()): exm = [ @@ -1193,8 +1205,7 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: entry["commands"].append( command_record(exm, step=f"export_multi:{tid}", result=result) ) - if rc := check(result, f"export_multi:{tid}"): - return rc + check(result, f"export_multi:{tid}") if steps.get("visualize_diagrams") and (options.dry_run or target_path.exists()): vzd = [ @@ -1218,8 +1229,7 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: entry["commands"].append( command_record(vzd, step=f"visualize:{tid}", result=result) ) - if rc := check(result, f"visualize:{tid}"): - return rc + check(result, f"visualize:{tid}") if steps.get("inspection_artifacts") and (options.dry_run or run_dir.is_dir()): insp = [ @@ -1241,8 +1251,7 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: entry["commands"].append( command_record(insp, step=f"inspection_artifacts:{tid}", result=result) ) - if rc := check(result, f"inspection_artifacts:{tid}"): - return rc + check(result, f"inspection_artifacts:{tid}") command_failures = [ str(command.get("step")) @@ -1368,11 +1377,45 @@ def _bundle_path(_run_dir: Path = run_dir) -> Path: manifest["summary"]["failed_steps"] = failures man_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8") + except FailFastExit as exc: + # --fail-fast abort: persist the partial evidence instead of + # discarding the manifest and summary for every completed target. + report( + f"fail-fast: aborting remaining batch steps (exit {exc.exit_code})", + log_fp=log_fp, + ) + for target in manifest["targets"]: + if target.get("status") != "running": + continue + target["status"] = "failed" + abort_tid = str(target.get("id")) + abort_steps = _failures_for_target(failures, abort_tid) + target["failed_steps"] = sorted(set(abort_steps)) or ["fail_fast"] + target["failure_reason"] = "batch aborted by --fail-fast" + manifest["finished_at"] = datetime.now(UTC).isoformat() + manifest["summary"] = { + "total_wall_time_s": round(time.monotonic() - batch_start, 3), + "target_count": n_targets, + "failed_steps": failures, + "fail_fast": True, + } + if not options.dry_run: + _write_cross_target_summary( + output_root, + manifest, + package_root=package_root, + log_fp=log_fp, + ) + (output_root / "run_manifest.json").write_text( + json.dumps(manifest, indent=2), encoding="utf-8" + ) + return exc.exit_code + finally: if log_fp is not sys.stdout: log_fp.close() # Propagate batch failure through the process exit code so CI and # run.sh callers can detect a failed sweep without parsing summary.json. - # (--fail-fast still early-returns the nonzero code mid-loop.) + # (--fail-fast aborts mid-loop after persisting the partial manifest.) return 1 if failures else 0