Skip to content

Report deferred primer and benchmark checks on PR heads - #11847

Merged
Bill Schnurr (bschnurr) merged 2 commits into
mainfrom
fix/pr-check-gating
Oct 9, 2026
Merged

Bill Schnurr (bschnurr) merged 2 commits into
mainfrom
fix/pr-check-gating

Conversation

@bschnurr

Copy link
Copy Markdown
Member

Summary

  • Publish mypy_primer and Type checker benchmark checks on the PR head instead of leaving their native workflow_run checks attached to main.
  • Queue both checks while Validation runs; execute only after all Validation jobs pass and report the follow-up result with a workflow link.
  • Run mypy_primer on every validated PR, including documentation and workflow-only changes; retain the analyzer-change filter for automatic benchmarks.
  • Keep check-write permissions confined to trusted default-branch reporting jobs, serialize check initialization, and isolate rerun results by check ID.
  • Leave branch protection unchanged. These default-branch workflows take effect after merge.

Review and verification

Reviewed the complete workflow change set and made check start/completion timestamps explicit. All 72 existing benchmark tests pass, including executable YAML guards for lifecycle, filters, reporting outcomes, and permissions. Changed-file ESLint and Prettier checks pass.

Addresses the missing PR-visible checks observed on #11839.

Queue PR-visible follow-up checks while Validation runs, report execution results from trusted jobs, and run mypy_primer for every validated PR. Keep the benchmark analyzer filter and branch protection unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rchiodo

Rich Chiodo (rchiodo) commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR.

@rchiodo

Copy link
Copy Markdown
Collaborator

Result: ⚠️ partially-verified

Verification details

Verification: Isolated verification observed failures whose relationship to this PR could not be determined: Dependency and test discovery; this review is not fully verified.

Summary: All 31 targeted benchmark-comparison tests passed, including the PR&#x27;s executable workflow guards and new lifecycle test. Changed-file ESLint and Prettier checks passed, and six additional failed-Validation reporting scenarios passed. Initial discovery failed because Git metadata and installed YAML dependencies were unavailable; the offline dependency bootstrap succeeded. Local verification is strong, but hosted GitHub Actions execution and the claimed full 72-test benchmark suite were not verified.

Test runs: 5 passed, 1 failed

  • ❌ Failed | relationship unknown | Dependency and test discovery | pwd; git status --short; git log -1 --oneline; printf '\nSandbox profile: %s\n' "$AUTOMATION_SANDBOX_PROFILE"; printf '\nDependency and runner discovery\n'; node --version; npm --version; python --version; python -m pytest --version; test ! -d node_modules || echo root-node_modules-present; git ls-files 'lock' '*package.json' 'pytest.ini' 'pyproject.toml' 'requirements' 'build/benchmark/' '.github/workflows/'; printf '\nRoot manifest\n'; cat package.json; printf '\nBenchmark test discovery\n'; grep -nE '^class |^ def test_|main|import |node_modules' build/benchmark/test_compare_benchmarks.py; printf '\nYAML dependency probe\n'; node -e "console.log(require.resolve('yaml'))"
  • ✅ Passed | Benchmark comparison and executable workflow guards | python -m pytest test_compare_benchmarks.py -q
  • ✅ Passed | Changed JavaScript ESLint | ESLINT_USE_FLAT_CONFIG=false node ./node_modules/eslint/bin/eslint.js build/reportPullRequestCheck.js build/resolveValidatedPullRequest.js
  • ✅ Passed | Changed-file Prettier | node ./node_modules/prettier/bin-prettier.js --check .github/workflows/mypy_primer_pr.yaml .github/workflows/pr_follow_up_checks.yml .github/workflows/typecheck_benchmark_trigger.yml build/benchmark/README.md build/reportPullRequestCheck.js build/resolveValidatedPullRequest.js
  • ✅ Passed | Offline root dependency bootstrap | pnpm install --offline --frozen-lockfile --filter pyright-root
  • ✅ Passed | Valid failed-Validation queue completion and stale-head rejection | node <<'NODE'
    const assert = require('node:assert/strict');
    const fs = require('node:fs');
    const workflow = require('yaml').parse(fs.readFileSync('.github/workflows/pr_follow_up_checks.yml', 'utf8'));
    const execute = new (Object.getPrototypeOf(async function () {}).constructor)('github', 'context', 'core', 'require', workflow.jobs.report.steps[1].with.script);
    const repo = { owner: 'microsoft', repo: 'pyright' };
    const sha = 'a'.repeat(40);
    const pr = { number: 7, state: 'open', head: { sha, repo: { id: 17 } } };
    const core = { notice() {}, setFailed(message) { throw new Error(message); } };
    (async () => {
    let passed = 0;
    for (const name of workflow.jobs.report.strategy.matrix.check) {
    for (const conclusion of ['failure', 'cancelled', 'timed_out']) {
    const checks = [];
    const github = {
    rest: {
    pulls: { list: 'pulls', listFiles: 'files', get: async () => ({ data: pr }) },
    checks: {
    listForRef: 'checks',
    create: async p => { const check = { ...p, id: checks.length + 1 }; checks.push(check); return { data: check }; },
    update: async p => { const check = checks.find(c => c.id === p.check_run_id); assert.ok(check); Object.assign(check, p); return { data: check }; },
    },
    },
    paginate: async method => method === 'pulls' ? [pr] : method === 'files' ? [] : checks,
    };
    const context = { repo, serverUrl: 'https://github.com&#x27;, runId: 123, eventName: 'pull_request_target', payload: { pull_request: pr } };
    process.env.CHECK_NAME = name;
    await execute(github, context, core, require);
    assert.equal(checks[0].status, 'queued');
    context.eventName = 'workflow_run';
    context.payload = { workflow_run: { name: 'Validation', path: '.github/workflows/validation.yml', event: 'pull_request', conclusion, repository: { full_name: 'microsoft/pyright' }, head_repository: { id: 17, owner: { login: 'external' } }, head_branch: 'feature', head_sha: sha } };
    await execute(github, context, core, require);
    assert.equal(checks.length, 1);
    assert.equal(checks[0].head_sha, sha);
    assert.equal(checks[0].status, 'completed');
    assert.equal(checks[0].conclusion, 'skipped');
    assert.ok(Number.isFinite(Date.parse(checks[0].completed_at)));
    assert.equal(checks[0].output.summary, 'Not run because Validation did not pass.');
    context.payload.workflow_run.head_sha = 'b'.repeat(40);
    await execute(github, context, core, require);
    assert.equal(checks.length, 1);
    passed++;
    }
    }
    console.log(${passed} valid failed-Validation reporting scenarios passed; stale heads ignored);
    })().catch(error => { console.error(error); process.exitCode = 1; });
    NODE
❌ Dependency and test discovery diagnostic output
fatal: not a git repository (or any parent up to mount point /)
Sandbox profile: typescript
v24.19.0
11.17.0
Python 3.13.16
pytest 9.1.1
Error: Cannot find module 'yaml'
Require stack:
- /workspace/[eval]
[container exit=1]

@rchiodo Rich Chiodo (rchiodo) added the review-auto:changes-requested Automated review: posted blocking findings to address. label Oct 8, 2026
@rchiodo

Copy link
Copy Markdown
Collaborator

Existing review feedback remains actionable.

@bschnurr

Copy link
Copy Markdown
Member Author

Copilot resolve the merge conflicts in this pull request

# Conflicts:
#	.github/workflows/typecheck_benchmark_trigger.yml

Co-authored-by: bschnurr <1946977+bschnurr@users.noreply.github.com>

Copilot AI commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Copilot resolve the merge conflicts in this pull request

Merged main and resolved the benchmark workflow conflict, preserving PR check reporting and incorporating main’s candidate-ref validation. The merge commit is 819a1b01.

@rchiodo

Copy link
Copy Markdown
Collaborator

Result: ⚠️ partially-verified

Verification details

Verification: Isolated verification observed failures that were not classified as caused by this PR: Batched dependency and test discovery.

Summary: All 32 tests in the changed comparator/workflow suite passed, including the new check-lifecycle test and expanded path-filter coverage. Offline dependency installation, changed-file ESLint, and Prettier checks passed. An additional mocked workflow test passed 12 failed-Validation scenarios covering both checks and stale-head rejection. Discovery Git probes failed because the container snapshot lacked Git metadata; live GitHub Actions execution was not verified.

Test runs: 5 passed, 1 failed

  • ❌ Failed | unrelated to this PR | Batched dependency and test discovery | printf 'Sandbox profile: %s\n' "$AUTOMATION_SANDBOX_PROFILE"; git status --short; git log -1 --oneline; git diff HEAD^ HEAD --stat; python --version; node --version; pnpm --version; python -c "import importlib.util; print('pytest available:', importlib.util.find_spec('pytest') is not None)"; node -e "for (const name of ['yaml', 'eslint', 'prettier']) { try { console.log(name + ': ' + require.resolve(name)); } catch { console.log(name + ': unavailable'); } }"; if test -d node_modules; then printf 'node_modules present\n'; else printf 'node_modules absent\n'; fi; git grep -n -E 'unittest|pytest|test_compare_benchmarks|test_typecheck_benchmark' -- .github/workflows build/benchmark/README.md; git grep -n '^ def test_' -- build/benchmark/test_compare_benchmarks.py
  • ✅ Passed | Comparator and workflow guards | python -m unittest discover -s build/benchmark -p test_compare_benchmarks.py -v
  • ✅ Passed | Changed JavaScript ESLint | ESLINT_USE_FLAT_CONFIG=false node ./node_modules/eslint/bin/eslint.js build/reportPullRequestCheck.js build/resolveValidatedPullRequest.js
  • ✅ Passed | Changed workflow and JavaScript formatting | node ./node_modules/prettier/bin-prettier.js --check .github/workflows/mypy_primer_pr.yaml .github/workflows/pr_follow_up_checks.yml .github/workflows/typecheck_benchmark_trigger.yml build/reportPullRequestCheck.js build/resolveValidatedPullRequest.js
  • ✅ Passed | Offline dependency bootstrap | pnpm install --offline --frozen-lockfile
  • ✅ Passed | Failed-Validation queue workflow integration | node <<'NODE'
    const assert = require('node:assert/strict');
    const fs = require('node:fs');
    const YAML = require('yaml');
    const workflow = YAML.parse(fs.readFileSync('.github/workflows/pr_follow_up_checks.yml', 'utf8'));
    const execute = new (Object.getPrototypeOf(async function () {}).constructor)('github', 'context', 'core', 'require', workflow.jobs.report.steps[1].with.script);
    (async () => {
    let scenarios = 0;
    for (const name of workflow.jobs.report.strategy.matrix.check) {
    for (const conclusion of ['failure', 'cancelled', 'timed_out']) {
    for (const queued of [false, true]) {
    const head = 'a'.repeat(40);
    const checks = queued ? [{ id: 1, name, head_sha: head, external_id: pyright-follow-up:${name}, status: 'queued' }] : [];
    const pr = { number: 7, state: 'open', head: { sha: head, repo: { id: 17 } } };
    const context = {
    repo: { owner: 'microsoft', repo: 'pyright' }, serverUrl: 'https://github.com&#x27;, runId: 123,
    eventName: 'workflow_run', payload: { workflow_run: {
    name: 'Validation', path: '.github/workflows/validation.yml', event: 'pull_request', conclusion,
    repository: { full_name: 'microsoft/pyright' }, head_repository: { id: 17, owner: { login: 'external' } },
    head_branch: 'feature', head_sha: head,
    } },
    };
    const github = {
    rest: {
    pulls: { list: 'pulls', listFiles: 'files', get: async () => ({ data: pr }) },
    checks: {
    listForRef: 'checks',
    create: async (p) => { const data = { ...p, id: 1 }; checks.push(data); return { data }; },
    update: async (p) => { assert.equal(p.check_run_id, 1); Object.assign(checks[0], p); return { data: checks[0] }; },
    },
    },
    paginate: async (method) => method === 'pulls' ? [pr] : method === 'files' ? [] : checks,
    };
    const core = { notice: () => {}, setFailed: (message) => { throw new Error(message); } };
    process.env.CHECK_NAME = name;
    await execute(github, context, core, require);
    assert.equal(checks.length, 1);
    assert.equal(checks[0].status, 'completed');
    assert.equal(checks[0].conclusion, 'skipped');
    assert.equal(checks[0].summary, undefined);
    assert.equal(checks[0].output.summary, 'Not run because Validation did not pass.');
    assert.equal(checks[0].head_sha, head);
    assert.ok(Number.isFinite(Date.parse(checks[0].completed_at)));
    assert.equal(checks[0].details_url, 'https://github.com/microsoft/pyright/actions/runs/123&#x27;);
    const snapshot = JSON.stringify(checks);
    pr.head.sha = 'b'.repeat(40);
    await execute(github, context, core, require);
    assert.equal(JSON.stringify(checks), snapshot);
    scenarios++;
    }
    }
    }
    console.log(PASS: ${scenarios} failed-Validation scenarios; both check names, queued/missing checks, failure/cancelled/timed_out, timestamps, links, and stale-head rejection.);
    })().catch((error) => { console.error(error); process.exitCode = 1; });
    NODE
❌ Batched dependency and test discovery diagnostic output
Sandbox profile: typescript
fatal: not a git repository (or any parent up to mount point /)
Python 3.13.16
v24.19.0
11.21.0
pytest available: True
yaml: unavailable
eslint: unavailable
prettier: unavailable
node_modules absent
[container exit=128]

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@rchiodo Rich Chiodo (rchiodo) added review-auto:approved Automated review: no blocking findings (approval posted). and removed review-auto:changes-requested Automated review: posted blocking findings to address. labels Oct 9, 2026
@bschnurr
Bill Schnurr (bschnurr) merged commit 6692d09 into main Oct 9, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants