Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions .github/agents/pyright-pr-review.agent.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: pyright-pr-review
description: Coordinates Pyright PR reviews, discusses remedies, and repairs or publishes changes when explicitly requested.
description: Runs author preflights, coordinates Pyright PR reviews, and repairs or publishes changes when explicitly requested.
tools: ['read', 'search', 'execute', 'edit', 'agent', 'web', 'create_pull_request', 'update_pull_request', 'reply_and_resolve_review_thread']
disable-model-invocation: true
include-custom-instructions: true
Expand All @@ -15,10 +15,19 @@ evidence, not votes.
Read `docs/pr-review-agents.md`, `.github/copilot-instructions.md`, and `.github/agents/pyright-test-policy.md` before
reviewing. Follow the shared review protocol in the guide. A request to review is not permission to fix or publish.

Review is the default. Continue the conversation after a report: explain findings, compare remedies, and help the user
decide whether a nearly correct PR is worth repairing. Discussion does not authorize source edits. Enter the repair phase
below only when the user explicitly requests implementation. Commits and PR updates require their own explicit
authorization, as described in the publishing phase; neither a review nor a repair request implicitly authorizes them.
Full review is the default unless the user asks for an author preflight. Continue the conversation after a report:
explain findings, compare remedies, and help the user decide whether a nearly correct PR is worth repairing. Discussion
does not authorize source edits. Enter the repair phase below only when the user explicitly requests implementation.
Commits and PR updates require their own explicit authorization, as described in the publishing phase; neither a review
nor a repair request implicitly authorizes them.

## Author preflight

When asked for a preflight, check the selected local changes or draft PR head directly using the
[TypeSpec skill's author preflight](../skills/pyright-pr-typespec-review/SKILL.md#author-preflight).
This is a separate path: do not launch the five specialists, require a PR or feedback ledger, or issue a merge verdict.
Use the local diff and available builds first. Return replayable findings, the checks run, and the most important gap.
Do not edit tracked files or publish anything in this mode.

## Establish the review boundary

Expand Down
4 changes: 3 additions & 1 deletion .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ Select `pyright-pr-review` for the coordinated review workflow described in
[PR review agents](../docs/pr-review-agents.md). Its five independent specialists include the ordinary
`pyright-pr-monkey` adversarial reviewer and a dedicated `pyright-pr-safety-reviewer` using the
[PR type-specification review skill](skills/pyright-pr-typespec-review/SKILL.md) for an additional investigation and example-led feedback.
The skill also supports standalone assessments and QA callers with their own execution and result contracts.
For a quick, read-only check of local changes or a draft PR, ask `pyright-pr-review` for an
[author preflight](../docs/pr-review-agents.md#preflight-for-authors). It uses the same skill without launching the
review panel. The skill also supports standalone assessments and QA callers with their own execution and result contracts.
Specialists return evidence and drafts; the coordinator owns the final assessment and explicitly authorized publication.

## Build and Test
Expand Down
15 changes: 13 additions & 2 deletions .github/skills/pyright-pr-typespec-review/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: pyright-pr-typespec-review
description: Investigate whether a Pyright PR changes typing behavior safely by tracing its diff, designing adversarial cases, comparing exact revisions, and checking relevant CPython behavior. Use for requests to find regressions, assess merge safety, or test effects beyond a PR's own examples.
description: Investigate whether Pyright changes typing behavior safely by tracing the diff, designing adversarial cases, comparing exact revisions, and checking relevant CPython behavior. Use for author preflights, regression investigations, merge safety, or testing beyond a PR's own examples.
---

# Pyright PR TypeSpec Review
Expand All @@ -13,8 +13,19 @@ Use this testing method without starting another review coordinator or delegatin

- When a QA runner supplies built revisions, execution tools, test passes, and an output schema, add skill-derived hypotheses and cases to those existing passes. Preserve independent expectations, negative neighbors, unchanged controls, and the final coverage check through execution. Use the runner's prepared builds and replay mechanism rather than creating another review session or invoking a comparison script with incompatible execution limits. Return the runner's declared result format, not a second review report. The runner owns verification, verdicts, and its QA comment.
- When the PR review coordinator assigns a specialist investigation, work in the assigned scope and scratch directory. Return the specialist report, executable evidence, coverage limits, and a concise feedback draft using [Draft PR feedback](#draft-pr-feedback). The coordinator reconciles duplicate findings and owns the final decision and authorized publication; the specialist never posts independently.
- When an author asks for an early check on local changes or a draft, use [Author preflight](#author-preflight).
- For a standalone assessment, follow the complete workflow below. Drafting feedback is not authorization to publish it.

## Author preflight

This is a focused, read-only check, not a full PR review. It does not need a GitHub PR or a second agent.

1. **Identify what is under test.** Record the target branch and its merge base with the candidate. For local edits, record `HEAD` and the selected staged and unstaged diff, plus any selected untracked inputs; do not describe that working tree as a commit. Keep unrelated edits out of the comparison. If you cannot isolate the selected changes, report the limitation rather than attributing mixed results to the patch. Never reset, stash, or overwrite the author's files.
2. **Check the riskiest changed behavior first.** Identify the [semantic contract of every new or changed shortcut](references/discovery-coverage.md#challenge-the-claimed-implication). Probe the riskiest contracts first and mark any remaining ones untested. For a typing change, start with the motivating case, a nearby valid case that must still work, and a nearby invalid case that must still produce a specific diagnostic. Derive their expectations independently. Compare the same probes on the base and candidate with matching settings. Assert exact inferred types and diagnostic rules and locations. If a changed helper serves multiple paths, include the most relevant alternate caller or syntax; an operator and a direct method call may exercise different paths. Check the downstream use if a type or reachability fact could be lost.
3. **Match the evidence to the claim.** For each claimed fix, run a repro that reaches the changed path and shows the old behavior on the base and the intended behavior on the candidate. A case that meets the same expectation on both is a control, not proof of the fix. For shared-state changes, observe a later operation or independent consumer after the first operation. For protocol changes, check the user-visible lifecycle through completion, not just the first response. When rechecking feedback, replay the original repro on the exact new candidate before reporting it fixed. Mark claims without that evidence as unverified.
4. **Run the smallest relevant integration checks.** For TypeScript changes, run the covering build/type check and focused tests. If the branch needs an update or resolves a conflict, check the merge result in an isolated checkout and report it separately from base-to-candidate behavior. For workflow changes, run the existing YAML-reading guards; if generated files change, also regenerate them with the pinned compiler in an isolated checkout and compare the results. Passing a YAML guard does not show the generated change will survive regeneration.
5. **Give the author a replay, not an approval.** For each finding, provide the base and candidate identity, a minimal repro, expected vs. actual behavior, the command to replay it, and the smallest useful test to add. Separate new regressions from baseline failures and missing evidence. Stop after the focused checks and call out the highest-risk untested path. If you cannot run a comparison, describe the result as unverified, not passing.

## Establish the comparison

- Read the current PR description, complete diff, added tests, review feedback, QA comments, CI and relevant primer/benchmark results. Associate each result with its tested commit. Distinguish a green QA feedback comment from an unrelated check mark.
Expand All @@ -32,7 +43,7 @@ Trace changed functions through callers, downstream consumers and early exits. B
- New flags or cached metadata: initialization, every mutation path, cloning, specialization and synthesized types.
- The point where an inferred type is combined with later arguments, expected return types, union members or narrowing state.

For each changed decision that accepts, rejects, widens, or skips analysis, distinguish what its predicate actually establishes from the conclusion the caller draws. Look for a counterexample where the predicate holds but that conclusion does not. Inspect the helper's contract and returned information rather than inferring semantics from its name or a truthy result.
Identify and challenge the semantic contract of every new or changed shortcut, including early exits, fast paths, cache reuse, and fallbacks that skip work. For each, distinguish what enables it, what that condition actually guarantees, and what the skipped path would have produced or done. Look for a counterexample where the shortcut still triggers but fails to preserve a required result, diagnostic, or state change. Inspect the helper's contract and returned information rather than inferring semantics from its name or a truthy result.

Use [references/discovery-coverage.md](references/discovery-coverage.md) to turn that inventory into concrete witnesses, relevant input combinations, and consumer tests, and later to audit missing hypotheses and classify individual findings. Choose feature-specific dimensions from [references/probe-design.md](references/probe-design.md); do not generate an arbitrary quota of cases or test every feature for every PR.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,14 @@ For a consequential changed branch, write down:
- The stronger conclusion used to accept, reject, transform, or skip something.
- A concrete witness that could separate those two claims, and a control where the implication is valid.

For a shortcut, also name the work it replaces and the behavior that must stay equivalent. Keep its guard true while varying a condition it does not check, such as binding identity, specialization, or evaluation state. Verify what downstream consumers observe, not just whether the fast path returns a value. For example, an identifier spelled `issubclass` does not prove it binds to the builtin; a shadowed callable can leave a branch reachable that builtin-specific narrowing would incorrectly skip. Check the diagnostic inside that branch.

For example, failure of containment in both directions does not prove that two sets are disjoint. Construct a shared member instead of treating failed containment checks as an overlap oracle. Similarly, finding an applicable operation does not by itself establish its result type, runtime completion, or permission to discard later alternatives. Read and exercise the actual contracts; these are reasoning prompts, not presumed bugs in every use of such helpers.

Trace where the new result goes next. A positional check followed by an aggregate check, or a precise result reduced to a boolean and then recomputed, can lose the fact that justified the earlier decision. Probe the boundary between those stages as well as the stages separately. Reuse of an existing helper is not sufficient evidence that it establishes the new caller's required property.

For shared state, make the sequence part of the repro. For example, run a starred tuple match before an independent match that relies on the same cached tuple shape. Compare the second match with and without the first so that a passing standalone match cannot hide the state leak. For aliases, distinguish mutating the current object from mutating an old alias after rebinding.

## Coverage belongs to a hypothesis and a combination

Derive dimensions from changed branches, callers, representations, and consumers. For each hypothesis, select the combinations that can exercise different semantics. Small relevant products can be generated; larger spaces need justified representatives and targeted interactions. Do not require an exhaustive product of unrelated language features.
Expand Down
17 changes: 15 additions & 2 deletions docs/pr-review-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ copilot --agent pyright-pr-review --prompt "Analyze microsoft/pyright#11571, inc
```

The coordinator is explicitly selected rather than automatically invoked for ordinary tasks. The five specialist
profiles are subagent-only. Models inherit the user's settings. This requires a host that supports custom-agent
profiles are subagent-only. Models inherit the user's settings. Full review requires a host that supports custom-agent
delegation; if it cannot load or invoke the specialists, the coordinator reports that limitation instead of pretending
the panel ran. Shell/GitHub access and test execution still require the host's usual permissions.

Expand Down Expand Up @@ -44,9 +44,22 @@ skill's example-led style. Already-covered issues are omitted from that suppleme
strengthen the established finding without changing its style. If no additional findings remain, there is no supplemental
comment. Full findings and evidence remain in the reports regardless of their feedback destination.

## Preflight for authors

Before opening a PR or asking for full review, select `pyright-pr-review` and ask:

> Preflight my local analyzer changes against upstream main. Run the motivating case and nearby valid and invalid
> controls. Show me how to replay any failures. Do not edit or publish anything.

The [author preflight](../.github/skills/pyright-pr-typespec-review/SKILL.md#author-preflight) also accepts a draft PR.
It runs directly without the five specialists and reports tested revisions, results, replay commands, and coverage
gaps. It distinguishes a repro that demonstrates the fix from a control that passes on both revisions. For shared state
or protocol changes, it checks what a later consumer or the end of the interaction observes. It does not give a merge
verdict or replace full review.

## Conversation and optional repair

For an ongoing conversation, select the coordinator with `/agent` in an interactive session. The `--prompt` example above
For an ongoing conversation, select the coordinator with `/agent` in an interactive session. The `--prompt` example in Usage
is a one-shot review, not an ongoing conversation. In interactive use, continue with questions or a scoped repair request:

| Request | Coordinator behavior |
Expand Down
Loading