Repository navigation
Audit repository configs and enforce benchmark evidence compatibility - #11943
Bill Schnurr (bschnurr) wants to merge 6 commits into
Conversation
Use explicit checker configurations and a shared implementation-first source manifest, reject results without verified native file counts, and report inference and memory-comparison limitations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35cc6021-56a0-45b2-9b66-3817d947cd99
|
🔒 Automated review in progress — Heejae Chang (@heejaechang) is auto-reviewing this PR. |
Use Pyrefly's all preset, prove an optional diagnostic is enabled with a real-binary regression test, and invalidate older configuration baselines. Document that this is not equivalent to Pyright basic analysis. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35cc6021-56a0-45b2-9b66-3817d947cd99
Verify exact inferred types and intended negative diagnostics on paired examples using the same isolated checker configurations. Preserve native counts and diagnostic evidence, distinguish behavior mismatches from evaluation failures, and reject incompatible suite identities or outcomes during comparisons and shard merging. Do not normalize runtime by probe or rule counts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35cc6021-56a0-45b2-9b66-3817d947cd99
|
/benchmark |
Heejae Chang (heejaechang)
left a comment
There was a problem hiding this comment.
Approved via Review Center.
Rich Chiodo (rchiodo)
left a comment
There was a problem hiding this comment.
Approved via Review Center.
This comment has been minimized.
This comment has been minimized.
|
/benchmark |
This comment has been minimized.
This comment has been minimized.
Recursively inventory native checker configs and TOML/INI settings, including Pyrefly sub-config overrides, without publishing values. Surface parse and traversal failures and include discovered settings in JSON, console output, and weekly HTML. Pass configured roots and benchmark exclusions to each checker instead of selecting implementations or stubs. Record native counts, mark differing scopes non-comparable, and exclude those timings from report totals. Keep explicit configuration isolation and validate it against hostile root/nested settings using real native discovery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35cc6021-56a0-45b2-9b66-3817d947cd99
| for field in ("check_paths", "exclude_directories"): | ||
| if old.get("files_selected") != new.get("files_selected"): | ||
| failures.append(f"{package}/{checker}: native source-file count changed") | ||
| scope_changed = True |
There was a problem hiding this comment.
Issue · Please address or respond
When the baseline contains a checker that is absent from the candidate, new is None here and new.get(...) raises before the existing missing-result handling can report the comparison failure. Handle a missing candidate metric before comparing files_selected, and add a regression case for differing checker sets.
|
GitHub cannot anchor PR review comments to unchanged lines in the diff. Falling back to a general PR comment for build/benchmark/typecheck_benchmark.py:L1389.
The |
The repository configuration audit records |
This comment has been minimized.
This comment has been minimized.
|
/benchmark |
This comment has been minimized.
This comment has been minimized.
Require environment, corpus, dependency and native-count evidence; protect history compatibility and handle missing metrics. Separate individual totals from shared successful coverage without implying equivalent analysis. Retain configuration inventories on preparation failures. Validate diagnostic categories and assertion controls, add installed typed/untyped dependency probes, and pin a common Python checker environment for weekly jobs. Record resolved package versions and import-path override fingerprints. Preserve native discovery and explicitly withhold apples-to-apples speed claims. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35cc6021-56a0-45b2-9b66-3817d947cd99
This comment has been minimized.
This comment has been minimized.
|
Local review: I ran the benchmark tooling suite including its YAML-reading workflow checks: 110 passed, six skipped, and |
Rich Chiodo (rchiodo)
left a comment
There was a problem hiding this comment.
After this merges, /benchmark produces v8 profiles, but the PR comment workflow still renders them against the checked-in docs/typecheck-benchmark/history/history.json.
With two valid synthetic v8 results and that history from GitHub's synthetic main+PR merge, the exact renderer CLI exits 0 and makes charts on base, but exits 1 with uses a different benchmark profile and makes none on this head. A matching v8 history succeeds. That renderer failure prevents the benchmark PR comment from being posted.
The description already mentions regenerating history, but the release-history job publishes artifacts and pages without committing the regenerated file back to main. Could we include compatible regenerated history, or skip charts that use incompatible history and still post the report? A focused test with v8 results and the checked-in history would help.
There's another issue in compare_benchmarks.py: it checks the file count before checking whether a metric failed. In a mocked run through the actual driver, a successful native audit followed by a timed-out benchmark reports native source-file count changed and drops the package row. Base instead reports that the candidate failed or is missing and includes a Failed row. A genuine count mismatch also loses the row. Could we check failure first and keep a useful row in both cases, with focused tests?
Classification: C (test coverage). No type precision changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
Purpose
Make corpus-specific configuration and observable analysis differences visible, and refuse unsupported comparisons. This PR does not establish apples-to-apples analysis or explain Pyright's additional runtime.
Changes
.py/.pyidiscovery without choosing either file kind for the checker.py.typed, using the same dependency-path mechanism as corpus runs. JSON retains raw and categorized evidence; HTML shows the ten per-case results without using them as a runtime divisor or quality score.ty-inferseparate, Pyreflyall, Pyright basic, mypy with untyped-body checking, and explicit Zuban default mode. Document their non-equivalent inference/import behavior and parent-only RSS/memory-cap limitations.Validation
core.whitespace=cr-at-eol.Remaining limits
The full nine-project hosted weekly run and published page have not been refreshed. Local single-run validations are correctness checks, not clean speed rankings. Independent hosted hardware, corpus dependency resolution, source-level suppressions, inference semantics, and unequal process-tree resource budgets still prevent a general equal-work claim. Do not divide runtime by rules, diagnostic counts, or capability matches. Regenerate baselines and release history for evidence version 8 before comparing old and new results.