Skip to content

docs(research): record the fixture provenance audit and the boundary fixture convention - #1461

Closed
doublewhy wants to merge 5 commits into
devfrom
1344-fixture-provenance-audit
Closed

doublewhy wants to merge 5 commits into
devfrom
1344-fixture-provenance-audit

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Part of #1344: the audit record and the convention it asks for. Most divergences registered need production changes, several involving release-admission or semantics choices. The register names the issue that tracks each open divergence: #1525, #1527, #1529, #1531, #1533, #1535, #1537, #1539, #1541, #1543, #1545, #1547, #1549, #1551, #1553, #1555 and #1557.

The PR stands alone: no file in it points at a file that exists only in an open pull request. Clear-cut divergences are fixed in separate pull requests, which the register cites: #1453, #1455 (#1454), #1459 (#1558), #1464 (#1462), #1472 (#1559), #1474 (#1473), #1476 (#1475), #1480 (#1479), #1483 (#1482), #1488 (#1560, #1561) and #1491 (#1490); #1415 (#1414) merged on 2026-10-11.

Requirement UIDs

  • None. Research record and testing convention with no requirement scope.

Related Issues

Closes #1460

ADR Impact

  • None.

Changes

Documentation only: the four new files and the index entry described in the Summary; the reference README also has sentence-case heading and plainer entry wording.

Acceptance item (verbatim, #1460) Change Check that fails without it CI lane Evidence
a written inventory of the test boundaries and, for each, whether its fixture shape was captured or inferred; inventory.md - (docs only) verify CI
the high-blast-radius inferred fixtures validated against the real system, with the observed response recorded ("leave the captured shapes in the repository"); captures.md - (docs only) verify CI
a short convention for how a new boundary fixture establishes its provenance; docs/explain/reference/fixture-provenance.md - (docs only) verify CI
the findings that apply to sibling repositories. index.md (sibling-repository findings) - (docs only) verify CI
Check every cited line against dev at 35122105. all four record files path:line resolver script local 285 references resolve
Run repository fast feedback (hygiene and policy, including the identity-cutover check) and nox -s docs-local. - - fast-feedback passed, below

Test Plan

  • Unit tests pass
  • Integration tests pass if applicable
  • Full completion suite required in CI before merge
  • No coverage regression

CI on head f7a757fd (branch base 35122105), merged with dev 348c4903: run 38516317607 succeeded; Docs run 38516317552 built.

Local checks on f7a757fd: nox -s verify-fast-feedback -- --base-rev origin/dev passed (28 stages PASS, 1 SKIP; it includes gitleaks and policy / identity cutover), make policy passed its 15 stages, and nox -s docs-local passed its 5 stages, including Sphinx. Vale 3.15.2 with the repository config reports no alerts on the five files.

A script resolved each of the 285 path:line references in the four record files against 35122105: each names a file that exists there and stays within its length. GH-1, GH-2, GH-5, LV-1, LV-4, NS-1, NS-2 and CL-1 were captured again while the record was written, and the store, time and runtime-snapshot probes rerun; libvirt source citations were read at its v12.7.0 tag.

Ground Control Checks

  • Repository policy command passes: make policy on f7a757fd.
  • Pre-push review: the captures carry no credentials, tokens or signed URL query strings (gitleaks and private-key stages pass), and every path:line reference resolves at 35122105.

Traceability

Checklist

  • Code follows the project coding standards
  • FM: not applicable, no semantic change (documentation only); no executable or runtime adoption is claimed.
  • No production module, test, published schema, research capture or other evidence-bound file changes.
  • PR title is a Conventional Commit (release-please derives the version and CHANGELOG.md from it)

Documentation

Updated: docs/explain/reference/README.md. Added: docs/research/fixture-provenance/ and docs/explain/reference/fixture-provenance.md. Both sit outside docs/public/.

…fixture convention

Inventory the test boundaries the audit examined that stand in for an
external system, with the fixture's provenance, the real producer's
observed response, a verdict and a blast-radius rating; keep the observed
shapes; register each divergence with its fix status; and write down how a
new boundary fixture establishes where its shape came from.
The convention and the captures no longer point at files that exist only
in #1453: the LV-2 and LV-3 readback and missing-object codes are recorded
in captures.md, and the convention names the existing tests/data directory.
Align provenance labels and verdicts with the record's own definitions,
give capture dates, versions and the expired artifact behind NS-3, add
fixture locations to the register, and use present tense for current
behavior.
GH-6 misread a deliberate check. The fixture's distinct GITHUB_SHA and
GITHUB_WORKFLOW_SHA values let the release-identity test prove that the
two fields map separately, as the issue-1226 preflight record requires.
Rate that row a match and drop GH-6 from the register.

The record covers the boundaries the audit examined. Say so, and name
examples it did not examine: the pinned z3 binding and its clock fakes,
the TechVault host probe stub, and podman.

Also rate NS-6 n/a, since nothing reads its fields; define the qualified
labels and "match in effect"; label the two LilRAE rows by how their
fixtures were made; show the nested REST shape for GH-1 and mark the
field GH-2 derives; name the NS-5 manifest; call the store, time, MCP and
snapshot entries summaries; mark libvirt code 62 as taken from source;
and correct the lease close() range.
@doublewhy
doublewhy force-pushed the 1344-fixture-provenance-audit branch from 62899a8 to 0a528fc Compare October 10, 2026 17:24
…ter entry

Each open entry in the divergence register now names its issue: #1525
to #1557 for the production changes, and #1521 for the smaller
corrections. The entries that open pull requests now address move to
"Addressed by pull requests" with their issues: GH-7 (#1491, #1490),
NS-1 (#1476, #1475), NS-2 (#1480, #1479), MF-2 (#1483, #1482), CL-2
(#1474, #1473), ST-2 (#1472, #1559) and NS-7 and NS-8 (#1488, #1560
and #1561). ST-1 names #1558, GH-3 records that #1415 merged, and
"What remains" names #1522 for the unverified and unexamined
boundaries.
@Brad-Edwards

Copy link
Copy Markdown
Collaborator

Recommendation: do not merge as is.

Reviewed head f7a757fdb62f273408eff8fad97f901f3b5c32d4, targeting dev.

  1. docs(research): record which test-boundary fixtures were captured and which were inferred #1460 explicitly requires high-blast-radius inferred fixtures to be validated against the real system with observed responses retained. inventory.md still labels missing nwfilter (blast H) and store migration/request-commitment drift (blast H) unverified; index.md says migration probes recorded no verdict. Tracking unverified work in Validate the remaining unverified fixture boundaries #1522 does not fulfill that acceptance criterion.
  2. The review snapshot also truncates the linked issue list (linked_issue_list_truncated), so outstanding-work verification is incomplete.

Required action:

Coverage and outstanding work: The inventory, captures, sibling findings and fixture convention are useful and fit the repository research/reference surfaces. High-risk observation coverage is incomplete, and #1344 correctly remains the broader audit umbrella. The existing linked follow-ups are tracked rather than silently omitted.

Verification limits: Complete file patches but incomplete linked-issue inventory. #1461 is OPEN, so conditional audit updates in #1459 are currently not applicable.

@doublewhy

Copy link
Copy Markdown
Contributor Author

Closing: several inventory rows, including the high-blast ones the review asked to validate, describe the libvirt backend that #1621 removed. #1460 stays open.

@doublewhy doublewhy closed this Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants