Skip to content

feat(sdk-review): add automated PR review skill (AFSDK-3937)#201

Closed
tiagoek wants to merge 1 commit into
mainfrom
feat/sdk-review-skill
Closed

feat(sdk-review): add automated PR review skill (AFSDK-3937)#201
tiagoek wants to merge 1 commit into
mainfrom
feat/sdk-review-skill

Conversation

@tiagoek

@tiagoek tiagoek commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Description

Adds the SDK Module Review skill — an automated, deterministic PR review that runs on every PR in this repository. Its check-run (sdk-module-review) is intended to become a required status check so PRs cannot be merged until the automated review is green.

The skill runs 20 checks (secrets, license, disclosure, hardcode, telemetry, docs, BDD parity, patterns, versioning + BREAKING family, commits, errors/logging, testing-depth, HTTP hygiene, concurrency, deps supply, deletion hygiene, constants, binding shape, quality-gate parity, PR size) and posts four signals per run: inline comments, aggregated summary comment, GitHub check-run, and a status label.

Zero LLM calls in CI — same input, same output. Every run is reproducible. Cost is only GitHub Actions minutes.

Related Issue

Ref: AFSDK-3937

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Documentation update

How to Test

Automated (in CI):

  1. Once merged, every PR against main triggers .github/workflows/sdk-module-review.yml
  2. The check-run appears in the PR's Checks tab
  3. Findings appear as inline comments, a summary comment, and a status label

Local dry-run (against any open PR):

gh pr diff <PR_NUMBER> > /tmp/pr.diff
DRY_RUN=true LOCAL_DIFF=/tmp/pr.diff REPO_ROOT=$(pwd) \
  bash .claude/scripts/orchestrate.sh <PR_NUMBER> --dry-run

Or via the Claude Code slash command:

/review-new-module <PR_NUMBER> --dry-run

Unit tests:

bats tests/sdk-review/

57 tests pass; shellcheck is clean.

Structure

  • .claude/scripts/ — 20 check scripts + orchestrator + aggregator
  • .claude/scripts/lib/ — 12 shared libs (2 Python AST + 10 bash)
  • .claude/config/rules.yaml (tiers per rule), module-aliases.yaml, baseline.json
  • .github/workflows/sdk-module-review.yml — auto-trigger on every PR
  • .github/workflows/sdk-skill-isolation-check.yml — validates sub-PRs into this feature branch
  • docs/PR-REVIEW.md — user-facing docs for contributors
  • docs/BRANCH-PROTECTION-SETUP.md — admin guide for configuring the required status check
  • tests/sdk-review/ — bats tests + fixture diffs
  • CONTRIBUTING.md — amended: sdk-module-review must pass before maintainer review

Checklist

  • I have read the Contributing Guidelines
  • I have added automated tests (57 bats tests)
  • All tests pass locally
  • I have verified that my code follows the Code Guidelines
  • I have added documentation (docs/PR-REVIEW.md, docs/BRANCH-PROTECTION-SETUP.md, CONTRIBUTING.md amendment)
  • My code does not contain sensitive information (fixture strings are placeholder patterns designed to match SEC regexes for unit-testing — not real credentials)
  • I have followed Conventional Commits

Validation

E2E-tested against 5 real PRs in cloud-sdk-python and cloud-sdk-java:

PR Result
py#196 (external contributor bug fix) 0 BLOCK, 11 FLAG, 0 false positives
py#191 (converter fix) 0/0/0
py#192 (breaking release framed as fix) 4 real BLOCK (DEL-01, COM-01), 0 false positives
py#185 (large feat) 0 BLOCK, 8 FLAG, 0 false positives
jv#9 (new module in Java) Flagship BND-02 caught at DocumentManagementConfig.java:106

Self-test on this PR: 0 BLOCK, 0 FLAG (skill correctly skips its own files via lib/skill-self-skip.sh).

Rollout

Recommended SHADOW → FLAG → BLOCK progression (2 weeks each). See docs/BRANCH-PROTECTION-SETUP.md.

Additional Notes

  • The skill is deterministic — no LLM calls at any point of the CI pipeline. When invoked via Claude Code locally, the LLM can enrich comments narratively but the underlying rule evaluation stays deterministic.
  • BLOCK_LOCKED rules are non-suppressible: SEC-* (secrets), HC-03 (SAP-internal URL leak), DIS-06 (internal artifactory), LIC-01/02 (SPDX headers), BND-02 (BTP token URL concat), BND-05 (binding logs credentials), BREAKING-* (breaking-change declarations).
  • REUSE.toml aggregate at repo root is auto-detected — per-file SPDX rules stay silent for repos using aggregate licensing.
  • Cross-language BDD parity — module aliases seeded in .claude/config/module-aliases.yaml (currently dms ↔ documentmanagement).

Adds the SDK Module Review skill to this repository:

## What this PR adds

**20 deterministic checks** (no LLM calls in CI):
- secrets, license-spdx, disclosure, hardcode, telemetry, docs, bdd,
  patterns, versioning + BREAKING family, commits, errors-logging,
  testing-depth, http-hygiene, concurrency, deps-supply,
  deletion-hygiene, constants, binding-shape, quality-gate-parity,
  pr-size

**Rule tiers:** SHADOW | FLAG | BLOCK | BLOCK_LOCKED, configured in
`.claude/config/rules.yaml`

**BLOCK_LOCKED rules cannot be suppressed:**
- SEC-* (all secrets), HC-03 (SAP-internal URL leak),
  DIS-06 (internal artifactory), LIC-01/02 (SPDX headers),
  BND-02 (BTP token URL concat), BND-05 (binding logs credentials),
  BREAKING-* (all breaking-change declarations)

**REUSE.toml aggregate exemption** — this repo uses REUSE.toml at root,
so per-file SPDX rules are baseline-exempted automatically.

**Cross-language BDD parity** — module-aliases.yaml seeded with
`dms ↔ documentmanagement` (Java sibling).

**4 PR signals posted per run:**
1. Inline comments per finding
2. Aggregated summary comment
3. GitHub check-run (Checks tab)
4. Status label (`sdk-review: ✅ passed` / `❌ blocked` / `⚠️ flagged`)

**Idempotent re-runs:** HTML markers let subsequent runs replace prior
artifacts rather than duplicate them.

## Structure

- `.claude/scripts/`: 20 check-*.sh + orchestrate.sh + aggregate.sh
- `.claude/scripts/lib/`: 12 shared libs (2 Python AST + 10 bash)
- `.claude/config/`: rules.yaml, module-aliases.yaml, baseline.json
- `.github/workflows/sdk-module-review.yml`: auto-trigger on every PR
- `.github/workflows/sdk-skill-isolation-check.yml`: validates sub-PRs
- `docs/PR-REVIEW.md`: user-facing docs
- `docs/BRANCH-PROTECTION-SETUP.md`: admin guide for required-check config
- `tests/sdk-review/`: 57 bats tests + 6 fixtures (placeholder secrets,
  not real credentials)
- `CONTRIBUTING.md`: amended — sdk-module-review is required before review

## Validation

- 57/57 bats tests passing
- shellcheck clean on all bash scripts
- E2E on 5 real PRs (#196, #191, #192, #185 in this repo + Java jv#9):
  zero false positives observed
- Self-test on this PR: 0 BLOCK, 0 FLAG (skill correctly skips its own files)
- Flagship BND-02 finding verified on Java new-module PR (token URL concat)

## Rollout

Recommended SHADOW → FLAG → BLOCK progression, 2 weeks each.
See `docs/BRANCH-PROTECTION-SETUP.md`.

## Note on test fixtures

`tests/sdk-review/fixtures/secrets-*.diff` contain intentional placeholder
strings that match secret regexes (e.g. `AKIATESTFIXTUREXXXXX`,
`eyJTESTFIXTURE...`) so the check-secrets rule can be unit-tested.
These are NOT real credentials.

Ref: https://jira.tools.sap/browse/AFSDK-3937
@tiagoek tiagoek force-pushed the feat/sdk-review-skill branch from 0710032 to 025d15b Compare July 6, 2026 14:39
@tiagoek

tiagoek commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by 6 incremental sub-PRs (see AFSDK-3937/06-INCREMENTAL-DELIVERY.md). Original squash preserved on branch feat/sdk-review-skill.bak-mega-pr for reference.

@tiagoek tiagoek closed this Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant