Teach the Dependabot reviewer what this repo actually reports - #132
Merged
Conversation
Adds the prompt a weekly routine runs to triage, validate, and merge Dependabot PRs, written against the live repository rather than against a generic npm project. The rules that matter are the ones a generic version gets wrong here: - `mergeStateStatus: CLEAN` is not evidence. Every substantive job in squad-ci.yml is gated behind `needs: changes` and an `if:`, so a PR can report CLEAN with `test` skipped or cancelled outright. The prompt requires a named SUCCESS from a check that exercised the changed paths, and treats SKIPPED and CANCELLED as the absence of a result rather than a pass. - Dependabot sometimes bumps a workspace member's package.json without touching the root package-lock.json. `npm ci` then refuses to install, and the PR's green CI is green for the old versions. Two currently-open PRs are in this state; both report MERGEABLE/CLEAN. - Three ecosystems are in play, not one. nuget and github-actions PRs have their own expected paths, and action bumps get privilege scrutiny since the upgraded thing runs with the workflow token. - `npm run check` is the gate, not `npx vitest run`, which skips the lint and docs passes a toolchain bump is most likely to break. Also records the merge cap, the report/comment/merge modes, and the fact table the run re-verifies before trusting any of it.
Contributor
🛫 PR Readiness Check
PR Scope: 🔧 Infrastructure
|
| Status | Check | Details |
|---|---|---|
| ❌ | Single commit | 2 commits — consider squashing before review |
| ✅ | Not in draft | Ready for review |
| ✅ | Branch up to date | Up to date with dev |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | No source files changed — changeset not required |
| ✅ | Scope clean | No .squad/ or docs/proposals/ files |
| ✅ | No merge conflicts | No merge conflicts |
| ✅ | Copilot threads resolved | No Copilot review threads |
| ✅ | CI passing | All checks passing |
Files Changed (2 files, +512 −0)
| File | +/− |
|---|---|
.github/prompts/dependabot-review.md |
+503 −0 |
CLAUDE.md |
+9 −0 |
Total: +512 −0
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
Contributor
🟡 Impact Analysis — PR #132Risk tier: 🟡 MEDIUM 📊 Summary
🎯 Risk Factors
📦 Modules Affectedci-workflows (1 file)
root (1 file)
This report is generated automatically for every PR. See #733 for details. |
Section 5's positive-evidence table covered npm, docs and workflows but not src/Squad.Agents.AI or test/Squad.Agents.AI.Tests, which five of the fourteen open Dependabot PRs touch. Without a row those PRs fell through to 'no substantive check identified' and would have been escalated to a human regardless of two green .NET jobs. Requires SUCCESS from both .NET ubuntu-latest and .NET windows-latest, not either: the two have reported differently before. Since dotnet is absent from the runner, that CI evidence is the only evidence a NuGet PR will ever have, so the report must say local validation was unavailable rather than showing the row as a clean pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
.github/prompts/dependabot-review.md— the prompt a weekly Claude routine runs to triage, validate, and merge Dependabot PRs — plus a pointer to it inCLAUDE.md.It is written against this repository rather than a generic npm project, because the generic version is unsafe here. Each rule below was checked against the 14 currently-open Dependabot PRs.
What a generic reviewer gets wrong here
mergeStateStatus: CLEANis not evidence. Every substantive job insquad-ci.ymlis gated behindneeds: changesand anif:, so a PR can reportCLEANwhiletestwas skipped or cancelled. Observed right now:MERGEABLE/CLEANtestrun wasCANCELLED;npm cion its head exits 1MERGEABLE/CLEAN(before this PR)MERGEABLEtestandsdk-exports-validationbothFAILUREThe prompt requires a named
SUCCESSfrom a check that exercised the changed paths, and treatsSKIPPEDandCANCELLEDas the absence of a result rather than a pass.Workspace lockfile drift. Dependabot sometimes bumps a workspace member's
package.jsonwithout touching the rootpackage-lock.json.npm cithen refuses to install and the PR's green CI is green for the old versions. Confirmed by runningnpm cion #112's head:#111 and #26 are in the same state. All three report
MERGEABLE.Three ecosystems, not one. Five open PRs are NuGet and one is github-actions. Each gets its own expected-paths list; action bumps get extra scrutiny, since the upgraded thing runs with the workflow token.
npm run check, notnpx vitest run. The latter skips thelintandcheck:docspasses — exactly what a TypeScript or tooling bump breaks.Companion change to the repo (already applied)
The
default-branchruleset existed but was set toenforcement: disabled, sodevwas unprotected. It is now active, withclaude-mdandchangesas required status checks — the only two jobs here that run unconditionally, so requiring anything deeper would block PRs that legitimately skip it. Repository auto-merge is now enabled so the routine can queue merges behind checks instead of merging directly.Effect: #90 stays
CLEAN; #55 correctly flipped toBLOCKED.Protection here proves a PR is fresh and well-formed — not that the update was tested. The prompt says so explicitly and does not treat the required-checks list as validation.
Validation
npm run check:docspasses with the newCLAUDE.mdreference.npm ciclean, lockfile untouched,npm run buildgreen.EUSAGEfailure above.Of the 14 open PRs, exactly one (#90) reaches
SAFE_TO_MERGE.