Repository navigation
ci(gates): verify pipeline with ratchet baselines and entry-surface check - #186
Open
titanwings wants to merge 6 commits into
Open
titanwings wants to merge 6 commits into
titanwings wants to merge 6 commits into
Conversation
…test:audit) Requirement-driven tests written against docs/v2/CONTRACT.md, ACCEPTANCE.md, RENDER.md, STATUS.md, SKILL.md and the v2 plan. They spawn bin/distilly.mjs on the public corpus fixtures and assert literal values. 58 tests; 37 pass and 21 fail on dot-skill-test @ e16cc8a (Node 20.20.2 and 22.14.0, same set). Existing sources and tests are untouched; the only manifest change is the separate test:audit script.
… collect channels PR #185's 已知缺口 says discord/reddit/notion/gmail answer collect/planned-channel and doctor lists them; the CLI answers collect/unknown-channel and doctor's unavailable is empty. Literal contrast recorded as a failing black-box test.
…orpus CI's acceptance job only runs synthetic-interview and synthetic-multisource, so the FAIL verdict STATUS §5 documents is never exercised by a green CI. After retrospect the gate fails with exit 1 and the documented reason; without the flag it only reports.
scripts/run-gates.mjs runs every tests/gates/*.json and is the only thing CI calls. A gate's pass predicate is literal (exit code, a stdout line, a count) or a ratchet against an expected-failure baseline that can only shrink. scripts/check-entry-surface.mjs diffs docs/v2/CONTRACT.md §1 against every level of bin/distilly.mjs --help and against a runtime probe of each collect channel, then diffs error codes documented in markdown (and in a claims file such as a PR body) against the codes the source can emit. Baseline at tests/gates/baseline/entry-surface.txt records the 14 known gaps at e16cc8a. tests/gates/baseline/audit-tests.txt records the 22 tests/audit cases that fail at e16cc8a. Acceptance baselines are empty: 18/18 is required on the three public corpora.
Two jobs. Gates (Node 20 and 22) runs every gate without a browser and feeds the PR body to the entry-surface gate as a claims file. Gates needing Playwright (Node 22) runs the three acceptance corpora. No branch filter and no remote name, so a stacked PR gets the same verdict as trunk.
…t state Row 13 required HEAD to be 0 commits ahead of <remote>/dot-skill-test with a clean working tree, and tests/audit-objective.test.mjs required the audit to pass, so npm test was 394/396 on every unpushed branch and every dirty checkout (measured on this branch before this commit) while CI passed the row unconditionally. Push state is a forge property the program verifies; it is not a property of the code. The row is gone, the test now forbids any row that depends on push state, and the objective-audit gate pins 14/14.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Later fix PRs need a ratchet that keeps today's red tests red until their own change lands.
The objective audit still reads the operator git state. That makes
npm testfail on any branch that leads trunk or has a dirty tree.PR-0 stands up the gate runner, pins the known reds, and stops the audit from reading push state.
Scope
Adds
scripts/run-gates.mjsandscripts/check-entry-surface.mjs.Adds eleven
tests/gates/*.jsonfiles andtests/gates/baseline/*.txt.Adds
tests/audit/**from the requirements audit (60 black-box cases).Adds
.github/workflows/verify.yml.Edits
package.jsonby onescripts.test:auditline.Edits
scripts/audit-objective.mjsso the audit reads the tree, not push state.Edits
tests/audit-objective.test.mjsso no assertion depends on push state.Does not touch
.github/workflows/ci.yml.Blast Radius
CI grows a second workflow. The existing
ci.ymlpath stays.npm teston a dirty tree that leads trunk is# pass 396and# fail 0.No user-facing command path is rewritten except the audit-objective 推送 line.
Ratchet baselines pin 22 known audit failures. A new failure name still fails the gate.
Verification
PR-0 turns no T number from red to green.
The 22 reds are pinned in
tests/gates/baseline/audit-tests.txtas pre-red evidence for later PRs.node scripts/run-gates.mjs --id audit-testson Node 22.14.0 printedBaseline lines deleted. none.
node scripts/run-gates.mjs --without playwrightprinted8/8 gates passedon Node 22.14.0 and on Node 20.20.2.DISTILLY_PLAYWRIGHT_ROOT=/tmp/pw-root node scripts/run-gates.mjs --requires playwrightprinted3/3 gates passedon Node 22.14.0. Each corpus printed验收结果：18/18 通过.Rollback
Revert the six commits on this branch, or close the PR and delete the branch.
ci.ymlis unchanged, so the previous CI path stays.