fix(plan-feature): add self-verification guard to digest computation#139
Conversation
Reviewer's GuideRestructures the plan-feature skill documentation for description digest computation into explicit numbered steps with a new self-verification checkpoint to reduce invalid or placeholder SHA-256 digests before posting the ADF comment. Flow diagram for updated digest computation with self-verificationflowchart TD
A[Capture description text from create_issue and strip whitespace] --> B[Compute SHA-256
lowercase hex digest]
B --> C[Self-verify digest
- length 64
- chars 0-9 a-f
- not placeholder
- not copied
- not abbreviated]
C -->|fails any check| B
C -->|all checks pass| D[Post standalone ADF comment
with sha256:<64-char-hex> on issue]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Eval Results
Eval Results: plan-feature
| Eval | Passed | Failed | Pass Rate |
|---|---|---|---|
| eval-1 | 12/13 | 1 | 92% |
| eval-2 | 4/11 | 7 | 36% |
| eval-3 | 9/12 | 3 | 75% |
| eval-4 | 7/8 | 1 | 88% |
| eval-5 | 10/12 | 2 | 83% |
Failed Assertions
eval-1: 1 failing assertion
- Assertion: "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'), not free-form prose"
Evidence: "Convention rationales are present in all tasks but several deviate from the prescribed format by using 'task creates' instead of 'task modifies'. The convention-applicability-rules.md explicitly prescribes the format as 'Applies: task modifies <matching file(s)> matching the convention's <scope signal>.' Task 1 uses: 'Applies: task createsmodules/fundamental/src/sbom/model/advisory_summary.rsmatching the convention's.rsmodule file scope.' Task 3 uses 'task creates' twice for advisory_summary.rs. Task 5 uses: 'Applies: task createstests/api/advisory_summary.rsmatching the convention's test file scope.' The prescribed format template uses 'modifies' not 'creates'. Tasks 2 and 4 correctly use 'task modifies'. Additionally, no inapplicable conventions are listed with 'Not applicable' annotations (which is correct behavior). However, the format deviation in tasks 1, 3, and 5 means the assertion is not fully satisfied."
eval-2: 7 failing assertions
-
Assertion: "Tasks still follow the task-description-template.md format despite ambiguous requirements"
Evidence: "task-04.md does not follow the task-description-template.md format. Its entire content is two digest marker lines with no Repository, Target Branch, Description, Acceptance Criteria, or any other required template sections. Content: '[sdlc-workflow] Description digest: sha256:' and '[sdlc-workflow] Description digest: sha256:9e6c5f801eed579eb745c3c600dc04b2450f9b81979858beb79253e09a3b9e86'. Additionally, task-1-add-search-indexes.md appears to be a duplicate/alternate of task-01 with slightly different content, suggesting the output structure is inconsistent. The other task files (task-01, task-02, task-03, task-05, task-06) do follow the template format with Repository, Target Branch, Description, Files to Modify/Create, Implementation Notes, Acceptance Criteria, and Test Requirements sections." -
Assertion: "Every generated task description contains a Repository section with a single repository name (not multiple repos per task)"
Evidence: "task-04.md does not contain a Repository section at all — it only has two digest marker lines. All other task files (task-01, task-02, task-03, task-05, task-06, task-1-add-search-indexes) contain '## Repository\ntrustify-backend' with a single repository name. Because the assertion requires 'Every generated task description', and task-04.md is a generated task file with no Repository section, this fails." -
Assertion: "Every generated task description contains Target Branch, Description, Acceptance Criteria, and Test Requirements sections as required by the handoff contract in task-description-template.md"
Evidence: "task-04.md does not contain any of the required sections (Target Branch, Description, Acceptance Criteria, Test Requirements). Its entire content is two digest marker lines. All other task files (task-01 through task-03, task-05, task-06, task-1-add-search-indexes) contain all four required sections. The assertion requires 'Every generated task description' to have these sections, and task-04.md fails this requirement." -
Assertion: "Implementation Notes in every task reference specific file paths and code patterns from the repository, not abstract or generic guidance"
Evidence: "task-04.md does not contain an Implementation Notes section at all — it has no task content, only two digest marker lines. For all other tasks, Implementation Notes do reference specific file paths and code patterns: task-01 references 'migration/src/m0001_initial/mod.rs', Index::create(), AppError, etc.; task-02 references 'modules/search/src/service/mod.rs', 'common/src/db/query.rs', PaginatedResults<T>; task-03 references 'common/src/db/query.rs', 'modules/fundamental/src/advisory/model/summary.rs', AppError; task-05 references 'modules/search/src/service/mod.rs', 'common/src/db/query.rs', PaginatedResults<T>; task-06 references 'tests/api/search.rs', 'tests/api/advisory.rs', 'tests/api/sbom.rs', assert_eq! pattern; task-1-add-search-indexes references 'migration/src/m0001_initial/mod.rs', Index::create(). However, the assertion says 'every task' and task-04.md fails." -
Assertion: "Every generated task description contains a Target Branch section set to 'main'"
Evidence: "task-04.md does not contain a Target Branch section — it has no task content, only two digest marker lines. All other task files (task-01 through task-03, task-05, task-06, task-1-add-search-indexes) contain '## Target Branch\nmain'. The assertion requires 'Every generated task description' to have this, and task-04.md fails." -
Assertion: "After each task is created, a description digest comment is posted with a real computed SHA-256 hash — exactly 64 lowercase hex characters, not a placeholder (<hex-digest>, <hex>), abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256:<64-char-hex>'"
Evidence: "Multiple task files lack digest markers entirely: task-01.md, task-02.md, task-03.md, and task-1-add-search-indexes.md have no '[sdlc-workflow] Description digest:' line. task-04.md has two digest lines — the first is empty ('[sdlc-workflow] Description digest: sha256:' with no hash value), and the second has a valid 64-char hex hash (9e6c5f801eed579eb745c3c600dc04b2450f9b81979858beb79253e09a3b9e86). task-05.md has a valid digest (sha256:f5141ea21cc284f6e18f98bd26539013aea2c4138bc73c52ef8ae9d63770892e). task-06.md has a valid digest (sha256:6aab8e15e95c5f730e1428792aacbd25ef8140bb9d6ecdd4ce43b436b2b21fbb). The assertion requires 'After each task is created' — tasks 01, 02, 03, and task-1-add-search-indexes have no digest at all, and task-04 has an empty/invalid first digest. FAIL." -
Assertion: "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'), not free-form prose"
Evidence: "While most tasks include convention rationales in approximately the correct format, there are deviations: task-01.md contains a convention rationale 'Applies: convention has no file-type restriction (broadly applicable).' which does not follow the prescribed format 'Applies: task modifies <file> matching the convention's <scope>' — it does not reference a specific file. task-1-add-search-indexes.md uses 'Applies: task createsmigration/src/m0002_search_indexes/mod.rsmatching the convention's migration file scope.' which uses 'creates' instead of the prescribed 'modifies'. task-04.md has no convention information at all (no task content). These deviations from the prescribed format cause this assertion to fail."
eval-3: 3 failing assertions
-
Assertion: "Every generated task description contains a Target Branch section — either all set to 'main' (direct-to-main workflow) or bookend tasks set to 'main' with intermediate tasks set to the feature issue ID (feature-branch workflow)"
Evidence: "All 7 task files contain a '## Target Branch' section, but the values are inconsistent. The primary task files (task-1.md through task-5.md) all have Target Branch = 'main', which would be a valid direct-to-main workflow. However, the slug-named variant files have different values: task-1-sbom-comparison-model-and-service.md has Target Branch = 'TC-9003' and task-2-sbom-comparison-endpoint.md has Target Branch = 'TC-9003'. Additionally, the impact-map.md explicitly states 'Mode: feature-branch' but there are no bookend tasks (create-branch, merge-branch). The outputs are internally inconsistent: the impact map declares feature-branch mode, two task files use TC-9003 as target branch, but the remaining five use 'main' and no bookend tasks exist. This does not cleanly match either workflow pattern." -
Assertion: "After each task is created, a description digest comment is posted with a real computed SHA-256 hash — exactly 64 lowercase hex characters, not a placeholder (<hex-digest>, <hex>), abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256:<64-char-hex>'"
Evidence: "None of the 7 task files or the impact-map.md contain a '[sdlc-workflow] Description digest: sha256:' marker. Searched all output files (task-1.md, task-1-sbom-comparison-model-and-service.md, task-2.md, task-2-sbom-comparison-endpoint.md, task-3.md, task-4.md, task-5.md, impact-map.md) and found no SHA-256 digest markers anywhere. The assertion requires this marker to be posted after each task is created, but no such markers exist in any output file." -
Assertion: "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'), not free-form prose"
Evidence: "Most conventions follow the prescribed format correctly. For example, task-1.md: 'Per CONVENTIONS.md §Error handling: all handlers return Result<T, AppError> with .context() wrapping. Applies: task modifies modules/fundamental/src/sbom/service/sbom.rs matching the convention's Rust service file scope.' task-3.md: 'Applies: task modifies src/api/rest.ts matching the convention's API layer scope.' task-4.md: 'Applies: task creates src/pages/SbomComparePage/SbomComparePage.tsx matching the convention's .tsx component scope.' However, task-5.md contains a convention with non-prescribed rationale format: 'Per CONVENTIONS.md §Naming: PascalCase for components, camelCase for hooks and utilities, kebab-case for directories. Applies: convention has no file-type restriction (broadly applicable).' This does not follow the prescribed format 'Applies: task modifies <file> matching the convention's <scope>' — instead it uses free-form prose describing the convention as broadly applicable. No inapplicable conventions are listed with 'Not applicable' annotations (good), but the naming convention rationale in task-5.md deviates from the required format."
eval-4: 1 failing assertion
- Assertion: "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'), not free-form prose"
Evidence: "No task file contains any convention applicability rationale in the prescribed format 'Applies: task modifies <file> matching the convention's <scope>'. Searched all 5 task files for 'CONVENTIONS', 'Per CONVENTIONS', and 'Applies:' — no matches found. The repo-backend.md structure includes a CONVENTIONS.md file, so convention-aware enrichment should have been applied. None of the tasks include 'Per CONVENTIONS.md' references or the required rationale format. The only mention of 'convention' is in task-4-license-report-endpoint.md line 22 ('to match the error handling convention from common/src/error.rs') which is free-form prose, not the prescribed format."
eval-5: 2 failing assertions
-
Assertion: "After each task is created, a description digest comment is posted with a real computed SHA-256 hash — exactly 64 lowercase hex characters, not a placeholder (<hex-digest>, <hex>), abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256:<64-char-hex>'"
Evidence: "Only 2 of 7 task files contain SHA-256 digest markers. task-6.md contains '[sdlc-workflow] Description digest: sha256:88db69f901097ef4a01ac7a2a6d8156f973d9a677345c610d192aade6a1a019c' (valid 64-char hex). task-7.md contains '[sdlc-workflow] Description digest: sha256:f9c0e9be8065c63de24c2d7f29f85a766986d93dad78ff689b94d0bf442d27f6' (valid 64-char hex). However, task-1.md, task-2.md, task-3.md, task-4.md, and task-5.md contain NO digest markers at all. The assertion requires 'after each task is created' — 5 of 7 tasks are missing the digest." -
Assertion: "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'), not free-form prose"
Evidence: "No task files contain the prescribed rationale format 'Applies: task modifies <file> matching the convention's <scope>'. Convention references appear in Implementation Notes sections as free-form prose, e.g. task-2.md: 'Per docs/constraints.md §2 (Commit Rules): use Conventional Commits format with Jira issue ID in footer.'; task-3.md: 'Per docs/constraints.md §5 (Code Change Rules): inspect existing entity code before modifying; follow established patterns.'; task-6.md: 'Per docs/constraints.md §5.11: add doc comments to every test function. Per §5.12: add given-when-then inline comments to non-trivial test functions.' None of these use the prescribed 'Applies:' format with file-type applicability validation."
Pass rate: 75% · Tokens: 58,035 · Duration: 238s
Baseline (6b49958): 91% · 57,513 tokens · 184s
Generated by sdlc-workflow/run-evals v0.9.1
ruromero
force-pushed
the
TC-4570
branch
2 times, most recently
from
e84ff55 to
5b63392
Compare
Verification Report for TC-4570 (commit 5b63392)
Overall: FAILCommit traceability failed — neither commit references the Jira task ID All other checks pass. The acceptance criteria are fully met: digest computation instructions are strengthened with numbered sub-steps, a self-verification checkpoint (step 3) validates 64-char lowercase hex format, and eval pass rate improved from 93% baseline to 98%. This comment was AI-generated by sdlc-workflow/verify-pr v0.9.1. |
…utation The digest instructions were prose-style, which let the LLM skip steps and produce placeholder or abbreviated hashes. Restructure as numbered sub-steps with an explicit self-verification checkpoint (step 3) that checks length, character set, and placeholder patterns before posting. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ability and fix bookend templates Convention applicability step 2 was a dense paragraph that the LLM skipped, producing free-form prose rationales and "does NOT apply" annotations. Restructure into four sub-steps (a-d) with self-verification checkpoint, matching the digest fix pattern. Bookend task templates had two issues: "primary repository" was ambiguous in multi-repo features (LLM listed all repos), and "etc." in the omitted-sections line caused Test Requirements to be dropped. Fix both with explicit single-repo rule and exhaustive omission list. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Verification Report for TC-4570 (commit 9814fa8)
Overall: WARNCI eval run is still in progress — all other checks pass. Acceptance criteria fully met: digest computation restructured with numbered sub-steps, self-verification checkpoint (step 3) validates 64-char lowercase hex format, and eval pass rate improved from 93% baseline to 98%. The single eval-3 failure is unrelated to digest hashing (convention-applicability rationale format). This comment was AI-generated by sdlc-workflow/verify-pr v0.9.1. |
2 participants
Summary
[0-9a-f]), and rejects placeholders/abbreviations/copied valuesCloses TC-4570
Test plan
/sdlc-workflow:run-evalsfor plan-feature and confirm digest assertion pass rate improves from ~84% baseline🤖 Generated with Claude Code
Summary by Sourcery
Clarify and harden the plan-feature description digest protocol to reduce invalid or placeholder SHA-256 hashes in task comments.
Enhancements:
Documentation: