Skip to content

Teach the Dependabot reviewer what this repo actually reports - #132

Merged
DeDuva merged 2 commits into
devfrom
chore/dependabot-review-prompt
Sep 1, 2026
Merged

Teach the Dependabot reviewer what this repo actually reports#132
DeDuva merged 2 commits into
devfrom
chore/dependabot-review-prompt

Conversation

@DeDuva

@DeDuva DeDuva commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Adds .github/prompts/dependabot-review.md — the prompt a weekly Claude routine runs to triage, validate, and merge Dependabot PRs — plus a pointer to it in CLAUDE.md.

It is written against this repository rather than a generic npm project, because the generic version is unsafe here. Each rule below was checked against the 14 currently-open Dependabot PRs.

What a generic reviewer gets wrong here

mergeStateStatus: CLEAN is not evidence. Every substantive job in squad-ci.yml is gated behind needs: changes and an if:, so a PR can report CLEAN while test was skipped or cancelled. Observed right now:

PR Reported Actually
#112 MERGEABLE / CLEAN only test run was CANCELLED; npm ci on its head exits 1
#55 MERGEABLE / CLEAN (before this PR) Astro 6→7 major with no substantive check ever run
#111 MERGEABLE test and sdk-exports-validation both FAILURE

The prompt requires a named SUCCESS from a check that exercised the changed paths, and treats SKIPPED and CANCELLED as the absence of a result rather than a pass.

Workspace lockfile drift. Dependabot sometimes bumps a workspace member's package.json without touching the root package-lock.json. npm ci then refuses to install and the PR's green CI is green for the old versions. Confirmed by running npm ci on #112's head:

npm error `npm ci` can only install packages when your package.json and
npm error package-lock.json ... are in sync.
npm error Missing: esbuild@0.28.2 from lock file

#111 and #26 are in the same state. All three report MERGEABLE.

Three ecosystems, not one. Five open PRs are NuGet and one is github-actions. Each gets its own expected-paths list; action bumps get extra scrutiny, since the upgraded thing runs with the workflow token.

npm run check, not npx vitest run. The latter skips the lint and check:docs passes — exactly what a TypeScript or tooling bump breaks.

Companion change to the repo (already applied)

The default-branch ruleset existed but was set to enforcement: disabled, so dev was unprotected. It is now active, with claude-md and changes as required status checks — the only two jobs here that run unconditionally, so requiring anything deeper would block PRs that legitimately skip it. Repository auto-merge is now enabled so the routine can queue merges behind checks instead of merging directly.

Effect: #90 stays CLEAN; #55 correctly flipped to BLOCKED.

Protection here proves a PR is fresh and well-formed — not that the update was tested. The prompt says so explicitly and does not treat the required-checks list as validation.

Validation

Of the 14 open PRs, exactly one (#90) reaches SAFE_TO_MERGE.

Adds the prompt a weekly routine runs to triage, validate, and merge
Dependabot PRs, written against the live repository rather than against a
generic npm project.

The rules that matter are the ones a generic version gets wrong here:

- `mergeStateStatus: CLEAN` is not evidence. Every substantive job in
  squad-ci.yml is gated behind `needs: changes` and an `if:`, so a PR can
  report CLEAN with `test` skipped or cancelled outright. The prompt requires
  a named SUCCESS from a check that exercised the changed paths, and treats
  SKIPPED and CANCELLED as the absence of a result rather than a pass.
- Dependabot sometimes bumps a workspace member's package.json without
  touching the root package-lock.json. `npm ci` then refuses to install, and
  the PR's green CI is green for the old versions. Two currently-open PRs are
  in this state; both report MERGEABLE/CLEAN.
- Three ecosystems are in play, not one. nuget and github-actions PRs have
  their own expected paths, and action bumps get privilege scrutiny since the
  upgraded thing runs with the workflow token.
- `npm run check` is the gate, not `npx vitest run`, which skips the lint and
  docs passes a toolchain bump is most likely to break.

Also records the merge cap, the report/comment/merge modes, and the fact
table the run re-verifies before trusting any of it.
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

🛫 PR Readiness Check

ℹ️ This comment updates on each push. Last checked: commit 9e91c52

PR Scope: 🔧 Infrastructure

⚠️ 2 item(s) to address before review

Status Check Details
Single commit 2 commits — consider squashing before review
Not in draft Ready for review
Branch up to date Up to date with dev
Copilot review No Copilot review yet — it may still be processing
Changeset present No source files changed — changeset not required
Scope clean No .squad/ or docs/proposals/ files
No merge conflicts No merge conflicts
Copilot threads resolved No Copilot review threads
CI passing All checks passing

Files Changed (2 files, +512 −0)

File +/−
.github/prompts/dependabot-review.md +503 −0
CLAUDE.md +9 −0

Total: +512 −0


This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

🟡 Impact Analysis — PR #132

Risk tier: 🟡 MEDIUM

📊 Summary

Metric Count
Files changed 2
Files added 1
Files modified 1
Files deleted 0
Modules touched 2

🎯 Risk Factors

  • 2 files changed (≤5 → LOW)
  • 2 modules touched (2-4 → MEDIUM)

📦 Modules Affected

ci-workflows (1 file)
  • .github/prompts/dependabot-review.md
root (1 file)
  • CLAUDE.md

This report is generated automatically for every PR. See #733 for details.

Section 5's positive-evidence table covered npm, docs and workflows but not
src/Squad.Agents.AI or test/Squad.Agents.AI.Tests, which five of the fourteen
open Dependabot PRs touch. Without a row those PRs fell through to 'no
substantive check identified' and would have been escalated to a human
regardless of two green .NET jobs.

Requires SUCCESS from both .NET ubuntu-latest and .NET windows-latest, not
either: the two have reported differently before. Since dotnet is absent from
the runner, that CI evidence is the only evidence a NuGet PR will ever have, so
the report must say local validation was unavailable rather than showing the
row as a clean pass.
@DeDuva
DeDuva merged commit 046df81 into dev Sep 1, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant