Skip to content

docs(constraints): add eval-aware Test Quality constraints 1.28-1.32#146

Merged
mrizzi merged 1 commit into
mainfrom
TC-4574
May 29, 2026
Merged

docs(constraints): add eval-aware Test Quality constraints 1.28-1.32#146
mrizzi merged 1 commit into
mainfrom
TC-4574

Conversation

@mrizzi
Copy link
Copy Markdown
Owner

@mrizzi mrizzi commented May 29, 2026

Summary

  • Add five new deterministic constraints (1.28–1.32) to §1 Skill Scope Rules in docs/constraints.md, codifying the eval-aware Test Quality behavior from TC-4572 and TC-4573
  • Update Traceability Index entries for verify-pr/SKILL.md (adding Step 4a.1, Step 6a, Step 6d, Step 7 references) and verify-pr/style-conventions.md (adding Check 5)

Implements TC-4574

Test plan

  • Constraint numbering is sequential (1.1–1.32, no gaps or duplicates)
  • Each new constraint's Source references a verified section in the target file
  • Traceability index entries match the new constraints
  • Existing constraints 1.1–1.27 are unchanged

Summary by Sourcery

Document eval-aware Test Quality behavior in the constraints spec and align traceability mappings for verify-pr and style-conventions skills.

New Features:

  • Define new constraints for detecting eval result reviews in PR conversations based on specific author and content markers.
  • Specify constraints for incorporating eval results into the Test Quality verdict, including PASS/WARN/N/A semantics.
  • Introduce constraints for creating and routing eval failure sub-tasks into the existing root-cause investigation pipeline.

Enhancements:

  • Extend the traceability index to reference the new verify-pr SKILL steps and style-conventions check associated with eval handling.

Add five new deterministic constraints to §1 (Skill Scope Rules) codifying
the eval-aware Test Quality behavior introduced by TC-4572 and TC-4573:
- 1.28: eval result review detection criteria
- 1.29: false positive prevention
- 1.30: Test Quality verdict incorporation
- 1.31: autonomous eval failure sub-task creation
- 1.32: eval failure root-cause pipeline integration

Update traceability index for verify-pr/SKILL.md and style-conventions.md.

Implements TC-4574

Assisted-by: Claude Code
@sourcery-ai
Copy link
Copy Markdown
Contributor

sourcery-ai Bot commented May 29, 2026

Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Adds five new eval-aware Test Quality constraints (1.28–1.32) to the Skill Scope Rules in docs/constraints.md and updates the traceability index to map the new constraints to verify-pr SKILL and style-conventions checks.

File-Level Changes

Change Details Files
Add eval-aware Test Quality constraints to the Skill Scope Rules table.
  • Extend the constraints table with entries 1.28–1.32 describing eval detection, false-positive avoidance, Test Quality verdict logic, eval-failure sub-task creation, and routing into root-cause investigation.
  • Ensure numbering continuity by appending new constraints after 1.27 with no renumbering of existing entries.
docs/constraints.md
Wire new constraints into the traceability index for verify-pr and style-conventions documentation.
  • Extend the verify-pr/SKILL.md traceability entry to reference Step 4a.1 for eval detection (constraints 1.28–1.29) and Steps 6a, 6d, and 7 for eval-aware verdicting and sub-task handling (constraints 1.30–1.32).
  • Extend the verify-pr/style-conventions.md traceability entry to reference new Check 5 as the source for constraint 1.30.
docs/constraints.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@mrizzi
Copy link
Copy Markdown
Owner Author

mrizzi commented May 29, 2026

Verification Report for TC-4574 (commit bdfb40f)

Check Result Details
Review Feedback N/A No inline review comments on the PR
Root-Cause Investigation N/A No sub-tasks created
Scope Containment PASS Only docs/constraints.md modified, matching task spec exactly
Diff Size PASS 7 additions / 2 deletions in 1 file, proportionate to task
Commit Traceability PASS Commit bdfb40f contains "Implements TC-4574"
Sensitive Patterns PASS No secrets or credentials in added lines
CI Status PASS All 2 CI checks pass (Plugin Validation, Sourcery review)
Acceptance Criteria PASS 5 of 5 criteria met — all source references verified against actual file content
Test Quality N/A No test files in PR; no eval results
Test Change Classification N/A No test files in PR
Verification Commands N/A No verification commands specified; no eval infrastructure modified

Overall: PASS

All checks pass. Constraints 1.28–1.32 are correctly added with verified source references, sequential numbering, and updated traceability index.


This comment was AI-generated by sdlc-workflow/verify-pr v0.9.1.

@mrizzi mrizzi merged commit 289f20f into main May 29, 2026
2 checks passed
@mrizzi mrizzi deleted the TC-4574 branch May 29, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant