Skip to content

ci(policy): read git path listings with -z so quoted names reach the gates - #1474

Merged
Brad-Edwards merged 10 commits into
devfrom
1473-nul-path-listings
Nov 11, 2026
Merged

Brad-Edwards merged 10 commits into
devfrom
1473-nul-path-listings

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Four helpers feed git path listings to the gates: runner._git_lines, common.changed_paths, check_schema_publication._git_paths and the merge-time diff in check_requirement_governance.requirement_changed_paths. On dev the first three lack -z, so git quotes names with non-ASCII bytes, control characters or double quotes, and the gates drop or mangle them (#1473). All four read text mode, which turns a carriage return or CRLF in a name into LF, so even with -z the runner drops it.

Each listing now runs with -z, reads bytes, splits on NUL and decodes every name as strict UTF-8, so names reach the gates exactly as stored.

Encoding policy: none is stated; its bytes-mode -z listings decode strict UTF-8 (check_identity_cutover, check_tooling_artifact_policy, verification_plan), and #1473's Chosen Architecture has a non-UTF-8 name raise UnicodeDecodeError. Not os.fsdecode: a surrogate-escaped name can still raise later, wherever a gate prints or encodes it.

Requirement UIDs

  • None. Repository tooling fix with no requirement scope.

Related Issues

Closes #1473

ADR Impact

  • None.

Changes

  • common.py:git_paths (new) runs git <command> -z <options> and reads bytes. changed_paths, the runner's _git_paths and requirement_changed_paths use it. run_git stays text for rev-parse and merge-base.
  • check_schema_publication.py:_git_paths decodes ls-tree -z bytes the same way (it imports nothing from tools).
  • CI, run 40186476870: the Cambridge mirror now serves only Isabelle2026, so generic-tool-local-inputs got a 404 and skipped checks, integration and proof. 1db12336 selects official-isabelle-release, the lock's other same-byte locator, in the workflow and operations.md.
  • Evidence: dev's source at ecd7b205 matches no capture, failing the evidence tests in every shard. collect_research_evidence.py ran both pipelines on this branch (38 of 42 spec stages passed, 4 preregistered; 20 of 20 formal observations as expected), recorded as capture-c31b568f (spec) and capture-602eb23f (formal).
  • ef9c8de4 merges origin/dev ecd7b205 (no conflicts).
Done-when (verbatim) Change Fails without it Lane
Review: "Capture path listings as bytes, split on NUL, and decode each pathname without newline translation under the repository's chosen filename-encoding policy." git_paths, both _git_paths 6 of 10 on 68dc0474's files test-shard
Review: "Add real temporary-repository cases containing carriage return and CRLF, alongside the existing LF/tab/quote/non-ASCII names; verify all three helpers return exact names and the gates select them." the test module 10 of 10 on dev's files test-shard
Review: "Run the targeted path-listing/caller tests and required CI." - - test-shard, integration
#1473: "A regression test [...] builds a real git repository [...]. It fails on dev." same module as above test-shard
#1473: "Run the existing tests that cover these helpers' callers [...]" - - test-shard, integration
#1473: "Run nox -s verify-fast-feedback [...], nox -s lint and make policy." - - fast-feedback, checks

Test Plan

  • Unit tests pass
  • Integration tests pass if applicable
  • Full completion suite required in CI before merge
  • No coverage regression

Repositories hold names with CR, CRLF, LF, tab, double quote, non-ASCII letter, and leading or trailing space; every listing returns them exactly:

  • staged, base revision, working tree: changed_paths, _changed_paths, the hygiene selection, and the 500-line cap rejecting each 501-line module;
  • whole tree: _tracked_repo_paths, --all-files, ls-tree, and _load_base_manifest loading every record;
  • during a dev merge: requirement_changed_paths, staged and unstaged;
  • a Latin-1 name stops the policy, hygiene and schema publication listings with UnicodeDecodeError.

On 9716654e: 10 passed. One change at a time: text-mode git_paths fails 6, runner.py 4, check_schema_publication.py 1, check_requirement_governance.py 2.

Caller modules (36): 1045 passed, -m integration 47. Evidence checkers and the 6 evidence modules (554, -m integration 25), verify-fast-feedback -- --base-rev origin/dev, lint, make policy and docs-local: passed.

CI on 9716654e: run 40189123506, 32 checks passed; Sonar gate OK.

Ground Control Checks

  • Repository policy command passes

make policy passed.

Traceability

  • IMPLEMENTS: tools/{policy/common,nox_support/runner,check_schema_publication,check_requirement_governance}.py, .github/workflows/canonical-verification.yml
  • TESTS: implementations/python/tests/test_issue_1473_nul_path_listings.py

Checklist

  • FM: not applicable, no semantic change; no executable or runtime adoption is claimed; no package source changes.
  • PR title is a Conventional Commit; ci does not cut a release.

…gates

Without -z, git quotes path names that contain a byte above 0x80 under
the default core.quotePath, a control character, a double quote or a
backslash. The nox runner dropped such names from the hygiene
selection, tools/policy/common.changed_paths handed git's quoted text
to the repository policy and JSON artifact checks, and the schema
publication base listing dropped them.

List with -z and split on NUL in all three, as
check_requirement_governance.py and verification_plan.py already do.
changed_paths now runs one listing for its three comparisons.
Without -z, git quotes a newline in a name as it does any other control
character. The test now covers such a name next to the tab, non-ASCII
and double-quote names; dev's three helpers fail on it alone.
@doublewhy
doublewhy marked this pull request as ready for review October 10, 2026 14:09
doublewhy added a commit that referenced this pull request Oct 12, 2026
…ter entry

Each open entry in the divergence register now names its issue: #1525
to #1557 for the production changes, and #1521 for the smaller
corrections. The entries that open pull requests now address move to
"Addressed by pull requests" with their issues: GH-7 (#1491, #1490),
NS-1 (#1476, #1475), NS-2 (#1480, #1479), MF-2 (#1483, #1482), CL-2
(#1474, #1473), ST-2 (#1472, #1559) and NS-7 and NS-8 (#1488, #1560
and #1561). ST-1 names #1558, GH-3 records that #1415 merged, and
"What remains" names #1522 for the unverified and unexamined
boundaries.
@Brad-Edwards

Copy link
Copy Markdown
Collaborator

Recommendation: do not merge as is.

Reviewed head 68dc047446af864e3775e3deab5423407f08dca8, targeting dev.

  1. The new NUL split still runs all three listings in subprocess text mode (including common.run_git), which performs universal-newline translation. A legitimate POSIX filename containing CR is changed to LF, and CRLF is collapsed to LF, before split('\0'). The runner then drops the non-existent translated path, so the same policy/hygiene bypass remains for a class of control-character names covered by ci(policy): read git path listings with -z so quoted names reach the gates #1473.

Required action:

  • Capture path listings as bytes, split on NUL, and decode each pathname without newline translation under the repository's chosen filename-encoding policy.
  • Add real temporary-repository cases containing carriage return and CRLF, alongside the existing LF/tab/quote/non-ASCII names; verify all three helpers return exact names and the gates select them.
  • Run the targeted path-listing/caller tests and required CI.

Coverage and outstanding work: #1473's currently tested names are handled and -z is the correct Git interface, but text-mode decoding violates the stated verbatim-name invariant. #1538 is separately tracked pre-existing test setup behavior, outside this fix.

Verification limits: Confirmed Python subprocess text-mode normalization with an isolated byte-output probe; no contributor code or repository mutations executed.

…mes survive

Text mode turned a carriage return or CRLF in a name into LF before the
NUL split, so the runner dropped the name and the policy helpers
returned a path that does not exist. tools.policy.common.git_paths now
runs each listing with -z, reads bytes and decodes every name as strict
UTF-8, like the repository's other -z listings. The runner, changed_paths
and the requirement gate's merge-time diff use it; the schema
publication listing reads bytes the same way.

The regression test drives real repositories with CR, CRLF, LF, tab,
quote, non-ASCII and space-edged names through the hygiene selection,
the 500-line cap, the base schema publication manifest and the merge
listing, and checks that a name that is not UTF-8 stops each listing.
…e locator

The Cambridge mirror's dist directory now holds Isabelle2026 only, so
the locked Isabelle2025-2_linux.tar.gz answers 404 there and
canonical / generic-tool-local-inputs fails exact verification, which
skips the checks, integration and proof jobs. The official release
locator serves the same bytes; the fetch still verifies the unchanged
lock size and SHA-256.
Dev's source at ecd7b20, where #1458 merged after #1444, matches no
admitted capture, so the evidence checkers and the tests that load the
current capture fail on every branch built on it. This branch changes no
package source. collect_research_evidence.py ran both pipelines on it:
38 of 42 specification stages passed (4 preregistered non-executions)
and all 20 formal observations matched their expected outcomes.
@doublewhy

Copy link
Copy Markdown
Contributor Author

Updated to 9716654e (merges dev ecd7b205, no conflicts).

  • Bytes, NUL split, no newline translation (8cff64dc): new tools/policy/common.py:git_paths runs git <command> -z, reads bytes and decodes each name as strict UTF-8, the policy of the repo's bytes-mode -z listings and ci(policy): read git path listings with -z so quoted names reach the gates #1473's Chosen Architecture; a non-UTF-8 name raises. changed_paths, the runner and the requirement gate's merge-time diff (the other run_git listing) use it; check_schema_publication._git_paths reads bytes the same way.
  • CR/CRLF cases: test_issue_1473_nul_path_listings.py adds CR, CRLF and space-edged names beside LF/tab/quote/non-ASCII. Every helper returns them exactly; the hygiene selection, 500-line cap, base schema manifest and merge listing select them; a Latin-1 name raises. 6 of 10 fail with 68dc0474's files, 10 of 10 with dev's.
  • Tests and CI: 36 caller modules, 1045 passed (47 with -m integration). Two CI fixes outside the listings: the Isabelle fetch uses official-isabelle-release, since the Cambridge mirror now 404s (1db12336), and combined-source captures bind dev's source (9716654e). Run 40189123506 passed (32 checks, Sonar gate OK).

doublewhy added a commit that referenced this pull request Oct 25, 2026
… module

SEM-204 reached semantics-expansion only through the shared ^SEM- pattern,
whose ownership roots exclude raes_runtime/control_plane_workflows.py, so
requirement governance for the #1462 compensation fix reported an ownership
mismatch (#1510). Add a workflow-compensation-semantics phase listed before
semantics-expansion, with the same reference-implementations prerequisite.
It keeps the shared semantic roots SEM-204 already reached, without the
paths named there for SEM-218 and SEM-233, and adds the compensation module
and the carried #1474 locator doc. The shared phase's roots and the other
#1510 requirements are unchanged.

SEM-204's record gains the IMPLEMENTS and TESTS links governance requires.
The new policy test pins the route, its bounds and SEM-223's unchanged
phase, and runs SEM-204 governance over the delivery files.
doublewhy added a commit that referenced this pull request Oct 25, 2026
…ce runs on this branch

DSL-120 reached sdl-language only through the shared ^DSL- pattern, whose
ownership roots exclude the episode-policy fixtures, the participant-control
guide, the semantic profile and the assurance record that #307 amends, so
forced requirement governance reported ownership mismatches and the branch
carried no UID (#1510). Add a participant-episode-policy-language phase
listed before sdl-language, with sdl-language's prerequisites (none). It
keeps the shared SDL roots DSL-120 already reached, without the paths named
there for other issues and requirements, and adds the files #307 changes and
the carried #1474 locator doc. The shared phase and the other #1510
requirements are unchanged.

docs/governance/requirement-scopes/307.json binds every delivery file to
DSL-120, so the canonical resolver selects DSL-120 for this issue branch and
CI evaluates requirement governance instead of skipping it. The DSL-120
record gains the IMPLEMENTS and TESTS links governance requires. The new
policy test pins the route and its bounds, keeps DSL-121 on the shared phase,
and evaluates the issue scope over its bound files.
@doublewhy

Copy link
Copy Markdown
Contributor Author

This branch conflicts with dev again after the Quality refactor merges, in tools/nox_support/runner.py. The fix is not on dev: the runner still reads git diff --name-only output as text lines (lines 115-118), so #1473 still reproduces. Do you want this as a standalone fix now, or will you fold it into the tools refactor (#1731)? If it is folded in, I'll close this PR and leave #1473 open.

@Brad-Edwards
Brad-Edwards merged commit e207f16 into dev Nov 11, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants