Repository navigation
feat(governance): add Gap register traceability matrix #2591
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
seonghobae
wants to merge
16
commits into
main
Choose a base branch
from
feat/governance-traceability-matrix
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,120
−0
Draft
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
1ec9c66
feat(governance): add Gap register traceability matrix
seonghobae 5d5d337
test(governance): require traceability references
seonghobae d45dbc1
docs(governance): cite traceability standards
seonghobae 3da9a6a
docs(governance): keep citation contract stable
seonghobae 131f292
docs(governance): scope exact-head verification
seonghobae 7574152
test(governance): cover Markdown parser boundaries
seonghobae da846ec
fix(governance): enforce Markdown parser boundaries
seonghobae 7153a8b
docs(governance): record Markdown boundary repair
seonghobae 4b72e59
test(governance): bind live event documentation
seonghobae ccfa9b7
docs(governance): align citation evidence count
seonghobae 20c0817
fix(governance): scan unclosed comments linearly
seonghobae bdad301
docs(governance): record linear parser evidence
seonghobae 2d52d0b
test(governance): reject IDs in Markdown link targets
seonghobae c7efe93
fix(governance): strip Markdown link targets
seonghobae 951534f
test(governance): cover unmatched Markdown delimiters
seonghobae 752f383
docs(governance): align exact-head verification evidence
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,171 @@ | ||
| # Gap traceability matrix | ||
|
|
||
| Status: proposed. Scope: ContextualWisdomLab/.github control plane. | ||
| Register: [`docs/product-technical-gap-baseline.md`](../product-technical-gap-baseline.md). | ||
| Tool: [`scripts/ci/gap_traceability_matrix.py`](../../scripts/ci/gap_traceability_matrix.py). | ||
|
|
||
| ## Problem | ||
|
|
||
| The Gap baseline requires new work to link its Gap ID in the PR description | ||
| and test evidence. Nothing measured whether that happens. On 2026-10-06 | ||
| 12:30 KST, the authenticated `gh pr list` and `gh issue list` exports held 402 | ||
| open PRs and 221 open issues. The tool found these results: | ||
|
|
||
| | Measure | Result | | ||
| | --- | --- | | ||
| | Register IDs (`G-`, `PRD-`, `CONTROL-` rows under a `Gap ID` or `ID` header) | 27, no duplicates | | ||
| | Open work items that mention a known register ID | 13 of 623 | | ||
| | Register IDs that no open work item mentions | 19 of 27 | | ||
| | IDs that open work cites but that the register does not define | 14 | | ||
|
|
||
| The 14 undefined IDs include `G-18` to `G-22`, `PRD-08`, | ||
| `CONTROL-PERSONAL-LITELLM-REVIEW-01` and seven other `CONTROL-*` incident IDs. | ||
| Some of them are defined only in an unmerged PR branch, and some appear in the | ||
| baseline only as prose instead of as a register row. For example, | ||
| `CONTROL-PERSONAL-LITELLM-REVIEW-01` is a row on the `align-pr-issue-runner` | ||
| branch (#2577), not on protected `main`. In both cases a reader of `main` | ||
| cannot resolve the reference. | ||
|
|
||
| These counts describe one snapshot. They do not authorize a merge and they | ||
| do not rank work. | ||
|
|
||
| ## Decision | ||
|
|
||
| Add an offline, standard-library-only CLI that writes a JSON and Markdown | ||
| matrix from the register and the inventory exports: | ||
|
|
||
| ```sh | ||
| gh pr list --state open --limit 1000 \ | ||
| --json number,title,body,isDraft,url > prs.json | ||
| gh issue list --state open --limit 1000 \ | ||
| --json number,title,body,url > issues.json | ||
| python3 scripts/ci/gap_traceability_matrix.py \ | ||
| --prs prs.json --issues issues.json \ | ||
| --output-json matrix.json --output-md matrix.md \ | ||
| --generated-at "$(date '+%Y-%m-%d %H:%M %Z')" | ||
| ``` | ||
|
|
||
| The report shows, for each register ID, the PRs and issues that mention it. | ||
| It also lists IDs without work, work without a known ID, dangling references | ||
| and duplicate register rows. The JSON records the SHA-256 digest of each input | ||
| so a reader can bind the report to its exact inputs. | ||
|
|
||
| `--require-link-event "$GITHUB_EVENT_PATH"` checks the PR in a live event | ||
| payload. `--require-link N` checks PR `N` in an exported inventory. Both exit 1 | ||
| when the PR names no known register ID. This change does not add a workflow | ||
| or a required check. The current backlog would fail such a gate, so turning it | ||
| on needs a separate owner decision. That decision must say which PRs it | ||
| applies to, for example new non-bot PRs only. | ||
|
|
||
| ## Matching rules | ||
|
|
||
| - A register row is a body row of a Markdown table whose header's first cell | ||
| is `Gap ID` or `ID`, whose next row is a valid Markdown delimiter row, | ||
| and whose first cell is exactly one ID. IDs in prose, in other cells and in | ||
| other tables (such as the PR inventory) are mentions. | ||
| - Mentions are taken from the title and body. Fenced code, HTML comments, | ||
| URLs and Markdown link targets are removed first. A fence closes only with | ||
| the same marker, at least the opening length and no trailing non-whitespace. | ||
| Inline code of any backtick-run length is removed unless its whole content | ||
| is one ID, because PR bodies here usually quote IDs as code. | ||
| - An ID must not be attached to a letter, digit, `_`, `-`, `/` or a dotted | ||
| name on its left, or to a letter, digit, `_` or `-` on its right. Korean | ||
| particles such as `G-15를` still match. | ||
| - A range such as `G-17..G-22` yields only its two endpoints. Bodies use | ||
| ranges both for "these gaps" and for "outside these gaps", so expanding | ||
| them would create links that the author did not claim. | ||
|
|
||
| ## Limits | ||
|
|
||
| - A mention is not evidence that the work implements, verifies or closes a | ||
| gap. A body can list an ID without doing the work. Reviewers still judge | ||
| substance. | ||
| - The tool reads one repository's inventory. Cross-repository work such as | ||
| `naruon#974` is not linked. | ||
| - `gh` returns 30 items unless `--limit` is set. Check that the summary | ||
| counts match the expected inventory size. | ||
| - Renamed or reused IDs, such as the earlier `G-15` collision, are reported | ||
| only when two register rows exist at the same time. | ||
|
|
||
| ## Method | ||
|
|
||
| The design used a mixture of agents from different model families. Codex | ||
| and OpenCode each proposed parsing rules, an exit-code contract and edge cases | ||
| independently. A Claude proposer ran twice and timed out both times without | ||
| an answer, so it contributed nothing. The implementation combines the points | ||
| both answers shared: the header-anchored register, removal of code and URLs, | ||
| exit codes 0/1/2/3 and checking the live event payload. | ||
|
|
||
| The external sources support the goals, not repository-specific parser | ||
| mechanics. ISO/IEC/IEEE 29148 defines requirements-engineering processes and | ||
| their information items, which supports maintaining explicit links from work | ||
| to registered requirements | ||
| (International Organization for Standardization, 2018). PROV-O supports | ||
| describing provenance and derivation across systems; the SHA-256 digest is this | ||
| repository's local mechanism for binding a report to the exact input artifacts | ||
| (World Wide Web Consortium, 2013). GitHub defines `GITHUB_EVENT_PATH` as the | ||
| runner file containing the complete webhook payload, so the live-event mode | ||
| reads that authoritative input (GitHub, n.d.). Header anchoring, the meanings | ||
| of exit codes 1/2/3 and Markdown parsing remain local choices validated by the | ||
| tests below. | ||
|
|
||
| A separate read-only Codex pass then reviewed the implementation | ||
| adversarially. It timed out before writing a final answer, so its partial | ||
| reasoning trace was used. Each candidate finding was reproduced before it was | ||
| accepted: | ||
|
|
||
| - `G-04/G-09` lost the second ID. A slash now blocks a match only after a | ||
| path segment, not after another ID. | ||
| - Example tables in fenced code were parsed as register rows. Fenced code, | ||
| including indented, tilde and longer fences, is now skipped. | ||
| - An unclosed fence or HTML comment exposed the rest of the body. Both now | ||
| run to the end of the text. | ||
| - A URL swallowed an ID attached to it after punctuation such as `)`. | ||
| - `--fail-on-duplicates` was ignored by the link-check modes, and an output | ||
| write failure exited 1, the "unlinked" code. Both are fixed. | ||
| - `"isDraft": "false"` was read as a draft. Only JSON `true` counts now. | ||
|
|
||
| A timing probe found quadratic behavior that hostile PR bodies could trigger, | ||
| including an unclosed `<!--` repeated many times and a long run of hyphenated | ||
| letters treated as a URL scheme. A fresh exact-head probe reproduced the | ||
| unclosed-comment case after the earlier regex fix, so comment removal now uses | ||
| one forward `find` scan. 100,000 repetitions of `G-01-` took 13.8 seconds | ||
| before the repairs. At exact head, all six hostile fixtures of 80,000 to | ||
| 200,000 characters complete in 0.001 to 0.029 seconds. Regression tests bound | ||
| these cases. | ||
|
|
||
|
|
||
| ## References | ||
|
|
||
| International Organization for Standardization. (2018). | ||
| *ISO/IEC/IEEE 29148:2018 systems and software engineering—Life cycle | ||
| processes—Requirements engineering.* | ||
| https://www.iso.org/standard/72089.html | ||
|
|
||
| World Wide Web Consortium. (2013, April 30). *PROV-O: The PROV ontology* | ||
| (W3C Recommendation). https://www.w3.org/TR/prov-o/ | ||
|
|
||
| GitHub. (n.d.). *Variables reference*. GitHub Docs. | ||
| https://docs.github.com/en/actions/reference/workflows-and-actions/variables | ||
|
|
||
| ## Verification | ||
|
|
||
| - On predecessor head `1ec9c665b47dd3fe7320e1db7db569dd06ea6e7c`, | ||
| the focused suite passed 55 tests in | ||
| `tests/test_gap_traceability_matrix.py`, under both normal and | ||
| `GITHUB_ACTIONS=true` runs. | ||
| - Implementation head `951534fa0825786550b2f002ca88f86028d1363c` | ||
| adds the repaired Markdown-link target case plus unmatched-fence and | ||
| unmatched-inline-run boundaries. The focused suite passes 63 tests under | ||
| both normal and `GITHUB_ACTIONS=true` runs. Direct probes also pass all | ||
| defect reproductions, adjacent regressions and six hostile-input performance | ||
| fixtures (worst: 0.029 seconds). | ||
| - On the same implementation tree, the warning-fatal full suite reports 5,221 | ||
| passed and 11 skipped, with 40 subtests passed; `git diff --check` passes. | ||
| - The new module has 100% statement and branch coverage, and `interrogate` | ||
| reports 100% docstring coverage for it. | ||
| - One test parses the repository's own baseline and requires its register | ||
| IDs to be unique. | ||
| - `interrogate scripts/ci` reports 97.0% on the clean protected-main tree | ||
| `37b10243c`. This change does not lower that figure. The gap belongs to | ||
| other modules. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.