Skip to content

Audit repository configs and enforce benchmark evidence compatibility - #11943

Open
Bill Schnurr (bschnurr) wants to merge 6 commits into
microsoft:mainfrom
bschnurr:benchmark/config-fairness
Open

Bill Schnurr (bschnurr) wants to merge 6 commits into
microsoft:mainfrom
bschnurr:benchmark/config-fairness

Conversation

@bschnurr

@bschnurr Bill Schnurr (bschnurr) commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Purpose

Make corpus-specific configuration and observable analysis differences visible, and refuse unsupported comparisons. This PR does not establish apples-to-apples analysis or explain Pyright's additional runtime.

Changes

  • Inventory native checker configs and TOML/INI settings immediately after cloning, before dependency installation. Preserve the inventory on preparation failure. JSON, console output, and weekly HTML show config paths/settings and explicit benchmark-config handling without publishing values. Repository configs remain intact.
  • Keep explicit checker configs, Python/interpreter selection, ignore/exclusion isolation, and ty user-config isolation. Preserve native .py/.pyi discovery without choosing either file kind for the checker.
  • Audit native source counts outside timing. Missing/zero/changed counts reject measurements; differing counts make a package Not comparable. Equal counts are explicitly not proof of equal file identities, dependency work, or analysis.
  • Introduce configuration/evidence version 8. Successful records require audited counts, corpus scope and preparation evidence. Record resolved target dependencies, ambient Python package versions, and hashes of inherited import-path overrides. Reject incompatible source commits/roots/exclusions, environments, dependency versions and package sets in the weekly report. Fix missing-candidate-metric crashes.
  • Apply configuration/audit/import-environment compatibility checks to release history, retain native counts, and reject changed successful native scopes. Older baselines/history must be regenerated rather than mixed with version 8.
  • Separate individual totals from totals over the shared successful package intersection; remove comparative total-time bars. Shared scope is still not equivalent analysis. Threading speedups additionally require matching Pyright versions and Node options.
  • Strengthen the untimed capability suite to check diagnostic categories and rule IDs, not just source lines. Notes/source excerpts and unrelated same-line lint errors cannot satisfy the assertion control. Add installed-package precision probes with and without py.typed, using the same dependency-path mechanism as corpus runs. JSON retains raw and categorized evidence; HTML shows the ten per-case results without using them as a runtime divisor or quality score.
  • Install the same pinned checker requirements in every weekly job, rather than separate unpinned checker environments. Snapshot comparison catches differing transitive resolutions; this is not a full corpus dependency lock or identical hardware.
  • Keep default ty and ty-infer separate, Pyrefly all, Pyright basic, mypy with untyped-body checking, and explicit Zuban default mode. Document their non-equivalent inference/import behavior and parent-only RSS/memory-cap limitations.

Validation

  • Windows: complete 116-test suite passed with every real checker profile and executable YAML-reading workflow guards; the final 12 report tests also passed after adding the threading-version/heap guard.
  • Linux/WSL Python 3.12.3: 72 tests passed, including native configuration isolation, ten category-based capability probes, compatibility validation, history and shard/report checks. This is not the hosted Python 3.14.6 environment.
  • ESLint, workflow Prettier, and whitespace checks passed. Existing CRLF files retain their formatting; the whitespace check uses core.whitespace=cr-at-eol.
  • Fresh Windows and Linux Click runs verified 17 native files for all seven profiles and rendered version-8 reports. Actual cross-platform data was rejected when deliberately mixed.
  • Fresh Windows NumPy run verified 356 files for Pyright/Pyrefly/ty/ty-infer, 204 for mypy, 206 for Zuban. Scope differences remain non-comparable and are excluded from totals; no forced file selection masks them.
  • A real internal-feed preparation failure retained NumPy's four config inventory records. For the successful local NumPy retry, Hypothesis was constrained to cached internal-feed version 6.170.8 after newer metadata returned 404; resolved versions are recorded. No switch to public package feeds was made.
  • Installed-package probes demonstrate a real analysis difference: mypy loses precision on the untyped installed dependency while the other tested profiles retain the declared return type. Native project-file counts alone miss that difference. Pyrefly all's two existing mismatches are additional annotation requirements, not proof of weaker inference.

Remaining limits

The full nine-project hosted weekly run and published page have not been refreshed. Local single-run validations are correctness checks, not clean speed rankings. Independent hosted hardware, corpus dependency resolution, source-level suppressions, inference semantics, and unequal process-tree resource budgets still prevent a general equal-work claim. Do not divide runtime by rules, diagnostic counts, or capability matches. Regenerate baselines and release history for evidence version 8 before comparing old and new results.

Use explicit checker configurations and a shared implementation-first source manifest, reject results without verified native file counts, and report inference and memory-comparison limitations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35cc6021-56a0-45b2-9b66-3817d947cd99
@heejaechang

Heejae Chang (heejaechang) commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

🔒 Automated review in progress — Heejae Chang (@heejaechang) is auto-reviewing this PR.

Use Pyrefly's all preset, prove an optional diagnostic is enabled with a real-binary regression test, and invalidate older configuration baselines. Document that this is not equivalent to Pyright basic analysis.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35cc6021-56a0-45b2-9b66-3817d947cd99
Verify exact inferred types and intended negative diagnostics on paired examples using the same isolated checker configurations. Preserve native counts and diagnostic evidence, distinguish behavior mismatches from evaluation failures, and reject incompatible suite identities or outcomes during comparisons and shard merging. Do not normalize runtime by probe or rule counts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35cc6021-56a0-45b2-9b66-3817d947cd99
@bschnurr

Copy link
Copy Markdown
Member Author

/benchmark

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@heejaechang Heejae Chang (heejaechang) added the review-auto:approved Automated review: no blocking findings (approval posted). label Oct 28, 2026

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@github-actions

This comment has been minimized.

@bschnurr

Copy link
Copy Markdown
Member Author

/benchmark

@github-actions

This comment has been minimized.

Recursively inventory native checker configs and TOML/INI settings, including Pyrefly sub-config overrides, without publishing values. Surface parse and traversal failures and include discovered settings in JSON, console output, and weekly HTML.

Pass configured roots and benchmark exclusions to each checker instead of selecting implementations or stubs. Record native counts, mark differing scopes non-comparable, and exclude those timings from report totals. Keep explicit configuration isolation and validate it against hostile root/nested settings using real native discovery.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35cc6021-56a0-45b2-9b66-3817d947cd99
@bschnurr Bill Schnurr (bschnurr) changed the title Isolate benchmark configs and enforce source-file parity Audit repository configs and isolate type-checker benchmarks Oct 28, 2026
for field in ("check_paths", "exclude_directories"):
if old.get("files_selected") != new.get("files_selected"):
failures.append(f"{package}/{checker}: native source-file count changed")
scope_changed = True

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue · Please address or respond

When the baseline contains a checker that is absent from the candidate, new is None here and new.get(...) raises before the existing missing-result handling can report the comparison failure. Handle a missing candidate metric before comparing files_selected, and add a regression case for differing checker sets.

@heejaechang

Copy link
Copy Markdown
Collaborator

GitHub cannot anchor PR review comments to unchanged lines in the diff. Falling back to a general PR comment for build/benchmark/typecheck_benchmark.py:L1389.

Warning · Non-blocking recommendation

The load_history compatibility profile still accepts an unversioned, zero-audit result alongside a version-4, one-audit result for the same package commit because it omits the new metadata. Preserve and compare the configuration version, audit policy, and package manifest identity so workload changes are not presented as release-performance changes. [verified]

@heejaechang

Copy link
Copy Markdown
Collaborator

Issue · Please address or respond

The repository configuration audit records tool.ty.rules.all in JSON but prints only tool.ty in the console. This contradicts the documented console inventory of setting names. Include detected setting names (without values) in the console output alongside the path, affected checker, and explicit benchmark-config policy.

@heejaechang Heejae Chang (heejaechang) added review-auto:changes-requested Automated review: posted blocking findings to address. and removed review-auto:approved Automated review: no blocking findings (approval posted). labels Oct 28, 2026
@github-actions

This comment has been minimized.

@bschnurr

Copy link
Copy Markdown
Member Author

/benchmark

@github-actions

This comment has been minimized.

@heejaechang Heejae Chang (heejaechang) added review-auto:approved Automated review: no blocking findings (approval posted). and removed review-auto:changes-requested Automated review: posted blocking findings to address. labels Nov 2, 2026
Require environment, corpus, dependency and native-count evidence; protect history compatibility and handle missing metrics. Separate individual totals from shared successful coverage without implying equivalent analysis. Retain configuration inventories on preparation failures.

Validate diagnostic categories and assertion controls, add installed typed/untyped dependency probes, and pin a common Python checker environment for weekly jobs. Record resolved package versions and import-path override fingerprints. Preserve native discovery and explicitly withhold apples-to-apples speed claims.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35cc6021-56a0-45b2-9b66-3817d947cd99
@bschnurr Bill Schnurr (bschnurr) changed the title Audit repository configs and isolate type-checker benchmarks Audit repository configs and enforce benchmark evidence compatibility Nov 2, 2026
@github-actions

This comment has been minimized.

@bschnurr

Copy link
Copy Markdown
Member Author

Local review: f907f97e integrated with ff00b031.

I ran the benchmark tooling suite including its YAML-reading workflow checks: 110 passed, six skipped, and test_workflows_use_current_pnpm_setup failed. The same test fails with the same action-pin mismatch on main, so this is not evidence that the PR regressed pin compatibility. This run validates the tooling tests, not benchmark performance.

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After this merges, /benchmark produces v8 profiles, but the PR comment workflow still renders them against the checked-in docs/typecheck-benchmark/history/history.json.

With two valid synthetic v8 results and that history from GitHub's synthetic main+PR merge, the exact renderer CLI exits 0 and makes charts on base, but exits 1 with uses a different benchmark profile and makes none on this head. A matching v8 history succeeds. That renderer failure prevents the benchmark PR comment from being posted.

The description already mentions regenerating history, but the release-history job publishes artifacts and pages without committing the regenerated file back to main. Could we include compatible regenerated history, or skip charts that use incompatible history and still post the report? A focused test with v8 results and the checked-in history would help.

There's another issue in compare_benchmarks.py: it checks the file count before checking whether a metric failed. In a mocked run through the actual driver, a successful native audit followed by a timed-out benchmark reports native source-file count changed and drops the package row. Base instead reports that the candidate failed or is missing and includes a Failed row. A genuine count mismatch also loses the row. Could we check failure first and keep a useful row in both cases, with focused tests?

Classification: C (test coverage). No type precision changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants