diff --git a/.github/agents/pyright-pr-review.agent.md b/.github/agents/pyright-pr-review.agent.md index f54eb6a21022..5c8777b427e4 100644 --- a/.github/agents/pyright-pr-review.agent.md +++ b/.github/agents/pyright-pr-review.agent.md @@ -1,6 +1,6 @@ --- name: pyright-pr-review -description: Coordinates Pyright PR reviews, discusses remedies, and repairs or publishes changes when explicitly requested. +description: Runs author preflights, coordinates Pyright PR reviews, and repairs or publishes changes when explicitly requested. tools: ['read', 'search', 'execute', 'edit', 'agent', 'web', 'create_pull_request', 'update_pull_request', 'reply_and_resolve_review_thread'] disable-model-invocation: true include-custom-instructions: true @@ -15,10 +15,19 @@ evidence, not votes. Read `docs/pr-review-agents.md`, `.github/copilot-instructions.md`, and `.github/agents/pyright-test-policy.md` before reviewing. Follow the shared review protocol in the guide. A request to review is not permission to fix or publish. -Review is the default. Continue the conversation after a report: explain findings, compare remedies, and help the user -decide whether a nearly correct PR is worth repairing. Discussion does not authorize source edits. Enter the repair phase -below only when the user explicitly requests implementation. Commits and PR updates require their own explicit -authorization, as described in the publishing phase; neither a review nor a repair request implicitly authorizes them. +Full review is the default unless the user asks for an author preflight. Continue the conversation after a report: +explain findings, compare remedies, and help the user decide whether a nearly correct PR is worth repairing. Discussion +does not authorize source edits. Enter the repair phase below only when the user explicitly requests implementation. +Commits and PR updates require their own explicit authorization, as described in the publishing phase; neither a review +nor a repair request implicitly authorizes them. + +## Author preflight + +When asked for a preflight, check the selected local changes or draft PR head directly using the +[TypeSpec skill's author preflight](../skills/pyright-pr-typespec-review/SKILL.md#author-preflight). +This is a separate path: do not launch the five specialists, require a PR or feedback ledger, or issue a merge verdict. +Use the local diff and available builds first. Return replayable findings, the checks run, and the most important gap. +Do not edit tracked files or publish anything in this mode. ## Establish the review boundary diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 0c92a0e9b016..39e9b33bb167 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -21,7 +21,9 @@ Select `pyright-pr-review` for the coordinated review workflow described in [PR review agents](../docs/pr-review-agents.md). Its five independent specialists include the ordinary `pyright-pr-monkey` adversarial reviewer and a dedicated `pyright-pr-safety-reviewer` using the [PR type-specification review skill](skills/pyright-pr-typespec-review/SKILL.md) for an additional investigation and example-led feedback. -The skill also supports standalone assessments and QA callers with their own execution and result contracts. +For a quick, read-only check of local changes or a draft PR, ask `pyright-pr-review` for an +[author preflight](../docs/pr-review-agents.md#preflight-for-authors). It uses the same skill without launching the +review panel. The skill also supports standalone assessments and QA callers with their own execution and result contracts. Specialists return evidence and drafts; the coordinator owns the final assessment and explicitly authorized publication. ## Build and Test diff --git a/.github/skills/pyright-pr-typespec-review/SKILL.md b/.github/skills/pyright-pr-typespec-review/SKILL.md index 6986e00c9214..79e2c342556a 100644 --- a/.github/skills/pyright-pr-typespec-review/SKILL.md +++ b/.github/skills/pyright-pr-typespec-review/SKILL.md @@ -1,6 +1,6 @@ --- name: pyright-pr-typespec-review -description: Investigate whether a Pyright PR changes typing behavior safely by tracing its diff, designing adversarial cases, comparing exact revisions, and checking relevant CPython behavior. Use for requests to find regressions, assess merge safety, or test effects beyond a PR's own examples. +description: Investigate whether Pyright changes typing behavior safely by tracing the diff, designing adversarial cases, comparing exact revisions, and checking relevant CPython behavior. Use for author preflights, regression investigations, merge safety, or testing beyond a PR's own examples. --- # Pyright PR TypeSpec Review @@ -13,8 +13,19 @@ Use this testing method without starting another review coordinator or delegatin - When a QA runner supplies built revisions, execution tools, test passes, and an output schema, add skill-derived hypotheses and cases to those existing passes. Preserve independent expectations, negative neighbors, unchanged controls, and the final coverage check through execution. Use the runner's prepared builds and replay mechanism rather than creating another review session or invoking a comparison script with incompatible execution limits. Return the runner's declared result format, not a second review report. The runner owns verification, verdicts, and its QA comment. - When the PR review coordinator assigns a specialist investigation, work in the assigned scope and scratch directory. Return the specialist report, executable evidence, coverage limits, and a concise feedback draft using [Draft PR feedback](#draft-pr-feedback). The coordinator reconciles duplicate findings and owns the final decision and authorized publication; the specialist never posts independently. +- When an author asks for an early check on local changes or a draft, use [Author preflight](#author-preflight). - For a standalone assessment, follow the complete workflow below. Drafting feedback is not authorization to publish it. +## Author preflight + +This is a focused, read-only check, not a full PR review. It does not need a GitHub PR or a second agent. + +1. **Identify what is under test.** Record the target branch and its merge base with the candidate. For local edits, record `HEAD` and the selected staged and unstaged diff, plus any selected untracked inputs; do not describe that working tree as a commit. Keep unrelated edits out of the comparison. If you cannot isolate the selected changes, report the limitation rather than attributing mixed results to the patch. Never reset, stash, or overwrite the author's files. +2. **Check the riskiest changed behavior first.** Identify the [semantic contract of every new or changed shortcut](references/discovery-coverage.md#challenge-the-claimed-implication). Probe the riskiest contracts first and mark any remaining ones untested. For a typing change, start with the motivating case, a nearby valid case that must still work, and a nearby invalid case that must still produce a specific diagnostic. Derive their expectations independently. Compare the same probes on the base and candidate with matching settings. Assert exact inferred types and diagnostic rules and locations. If a changed helper serves multiple paths, include the most relevant alternate caller or syntax; an operator and a direct method call may exercise different paths. Check the downstream use if a type or reachability fact could be lost. +3. **Match the evidence to the claim.** For each claimed fix, run a repro that reaches the changed path and shows the old behavior on the base and the intended behavior on the candidate. A case that meets the same expectation on both is a control, not proof of the fix. For shared-state changes, observe a later operation or independent consumer after the first operation. For protocol changes, check the user-visible lifecycle through completion, not just the first response. When rechecking feedback, replay the original repro on the exact new candidate before reporting it fixed. Mark claims without that evidence as unverified. +4. **Run the smallest relevant integration checks.** For TypeScript changes, run the covering build/type check and focused tests. If the branch needs an update or resolves a conflict, check the merge result in an isolated checkout and report it separately from base-to-candidate behavior. For workflow changes, run the existing YAML-reading guards; if generated files change, also regenerate them with the pinned compiler in an isolated checkout and compare the results. Passing a YAML guard does not show the generated change will survive regeneration. +5. **Give the author a replay, not an approval.** For each finding, provide the base and candidate identity, a minimal repro, expected vs. actual behavior, the command to replay it, and the smallest useful test to add. Separate new regressions from baseline failures and missing evidence. Stop after the focused checks and call out the highest-risk untested path. If you cannot run a comparison, describe the result as unverified, not passing. + ## Establish the comparison - Read the current PR description, complete diff, added tests, review feedback, QA comments, CI and relevant primer/benchmark results. Associate each result with its tested commit. Distinguish a green QA feedback comment from an unrelated check mark. @@ -32,7 +43,7 @@ Trace changed functions through callers, downstream consumers and early exits. B - New flags or cached metadata: initialization, every mutation path, cloning, specialization and synthesized types. - The point where an inferred type is combined with later arguments, expected return types, union members or narrowing state. -For each changed decision that accepts, rejects, widens, or skips analysis, distinguish what its predicate actually establishes from the conclusion the caller draws. Look for a counterexample where the predicate holds but that conclusion does not. Inspect the helper's contract and returned information rather than inferring semantics from its name or a truthy result. +Identify and challenge the semantic contract of every new or changed shortcut, including early exits, fast paths, cache reuse, and fallbacks that skip work. For each, distinguish what enables it, what that condition actually guarantees, and what the skipped path would have produced or done. Look for a counterexample where the shortcut still triggers but fails to preserve a required result, diagnostic, or state change. Inspect the helper's contract and returned information rather than inferring semantics from its name or a truthy result. Use [references/discovery-coverage.md](references/discovery-coverage.md) to turn that inventory into concrete witnesses, relevant input combinations, and consumer tests, and later to audit missing hypotheses and classify individual findings. Choose feature-specific dimensions from [references/probe-design.md](references/probe-design.md); do not generate an arbitrary quota of cases or test every feature for every PR. diff --git a/.github/skills/pyright-pr-typespec-review/references/discovery-coverage.md b/.github/skills/pyright-pr-typespec-review/references/discovery-coverage.md index 3c1ea4bfdb80..790b8d799f7b 100644 --- a/.github/skills/pyright-pr-typespec-review/references/discovery-coverage.md +++ b/.github/skills/pyright-pr-typespec-review/references/discovery-coverage.md @@ -34,10 +34,14 @@ For a consequential changed branch, write down: - The stronger conclusion used to accept, reject, transform, or skip something. - A concrete witness that could separate those two claims, and a control where the implication is valid. +For a shortcut, also name the work it replaces and the behavior that must stay equivalent. Keep its guard true while varying a condition it does not check, such as binding identity, specialization, or evaluation state. Verify what downstream consumers observe, not just whether the fast path returns a value. For example, an identifier spelled `issubclass` does not prove it binds to the builtin; a shadowed callable can leave a branch reachable that builtin-specific narrowing would incorrectly skip. Check the diagnostic inside that branch. + For example, failure of containment in both directions does not prove that two sets are disjoint. Construct a shared member instead of treating failed containment checks as an overlap oracle. Similarly, finding an applicable operation does not by itself establish its result type, runtime completion, or permission to discard later alternatives. Read and exercise the actual contracts; these are reasoning prompts, not presumed bugs in every use of such helpers. Trace where the new result goes next. A positional check followed by an aggregate check, or a precise result reduced to a boolean and then recomputed, can lose the fact that justified the earlier decision. Probe the boundary between those stages as well as the stages separately. Reuse of an existing helper is not sufficient evidence that it establishes the new caller's required property. +For shared state, make the sequence part of the repro. For example, run a starred tuple match before an independent match that relies on the same cached tuple shape. Compare the second match with and without the first so that a passing standalone match cannot hide the state leak. For aliases, distinguish mutating the current object from mutating an old alias after rebinding. + ## Coverage belongs to a hypothesis and a combination Derive dimensions from changed branches, callers, representations, and consumers. For each hypothesis, select the combinations that can exercise different semantics. Small relevant products can be generated; larger spaces need justified representatives and targeted interactions. Do not require an exhaustive product of unrelated language features. diff --git a/docs/pr-review-agents.md b/docs/pr-review-agents.md index d02b2fa7c09b..837f643e61bd 100644 --- a/docs/pr-review-agents.md +++ b/docs/pr-review-agents.md @@ -14,7 +14,7 @@ copilot --agent pyright-pr-review --prompt "Analyze microsoft/pyright#11571, inc ``` The coordinator is explicitly selected rather than automatically invoked for ordinary tasks. The five specialist -profiles are subagent-only. Models inherit the user's settings. This requires a host that supports custom-agent +profiles are subagent-only. Models inherit the user's settings. Full review requires a host that supports custom-agent delegation; if it cannot load or invoke the specialists, the coordinator reports that limitation instead of pretending the panel ran. Shell/GitHub access and test execution still require the host's usual permissions. @@ -44,9 +44,22 @@ skill's example-led style. Already-covered issues are omitted from that suppleme strengthen the established finding without changing its style. If no additional findings remain, there is no supplemental comment. Full findings and evidence remain in the reports regardless of their feedback destination. +## Preflight for authors + +Before opening a PR or asking for full review, select `pyright-pr-review` and ask: + +> Preflight my local analyzer changes against upstream main. Run the motivating case and nearby valid and invalid +> controls. Show me how to replay any failures. Do not edit or publish anything. + +The [author preflight](../.github/skills/pyright-pr-typespec-review/SKILL.md#author-preflight) also accepts a draft PR. +It runs directly without the five specialists and reports tested revisions, results, replay commands, and coverage +gaps. It distinguishes a repro that demonstrates the fix from a control that passes on both revisions. For shared state +or protocol changes, it checks what a later consumer or the end of the interaction observes. It does not give a merge +verdict or replace full review. + ## Conversation and optional repair -For an ongoing conversation, select the coordinator with `/agent` in an interactive session. The `--prompt` example above +For an ongoing conversation, select the coordinator with `/agent` in an interactive session. The `--prompt` example in Usage is a one-shot review, not an ongoing conversation. In interactive use, continue with questions or a scoped repair request: | Request | Coordinator behavior |