Skip to content

Use square-root performance scoring - #118

Merged
irvineoy merged 3 commits into
mainfrom
codex/power-score
Oct 20, 2026
Merged

irvineoy merged 3 commits into
mainfrom
codex/power-score

Conversation

@irvineoy

@irvineoy irvineoy commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Replace the additive compilation/correctness/speedup score with max(0, 1 - 1.05 / speedup_ratio) ** 0.5. Correct, comparable candidates earn a positive score only above 1.05x speedup; failed checks and invalid comparisons score zero. The existing score field and mean-per-case speedup aggregation remain in use.

Document SCORE_POWER and SCORE_SPEEDUP_THRESHOLD, curve examples, and historical score comparability. Text reports and the dashboard display up to four decimal places. For an illustrative 1.25x speedup, the old score of 245 becomes 0.4; these are formula examples, not new GPU measurements.

The branch is rebased onto current main and adds two follow-up commits:

  • Tests added on main after this branch was cut (tests/test_task_run_v2.py, tests/test_candidate_verification.py) still asserted the 120/220/320 scale; they now assert the bounded score. The review-gate fixture uses a synthetic 2x candidate, so its score assertion still distinguishes accepted from rejected candidates instead of reading 0 for both.
  • Score cases that repeated an already covered branch are removed: equivalent invalid speedups and constants, a default-curve point duplicated by the power test, method-consistency inputs that have dedicated tests, and the all-zero run of the report-format integration test.
  • The per-measurement average score in text reports uses four decimals, matching the other score fields.
  • src/utils/utils.py and src/utils/report_generation.py are removed. They had no callers, imports, or documentation references, and they inferred compilation and correctness from score thresholds (20/120/220) that the bounded score no longer encodes. Current reports count those from pass_compilation and pass_correctness.

Validation (CPU only):

  • Linux x86_64, Docker python:3.12-bookworm with Python 3.12.15 and CPU PyTorch 2.14.1, using the CI dependency set and commands on this branch (80dca1e) and on main (c0da09d) from the same checkout bundle:
    • make check-perf-helpers: pass on both.
    • python -m pytest -q tests --durations=25: this branch 14844 passed, 143 skipped, 0 failed; main 14819 passed, 143 skipped, 0 failed.
    • python -m compileall -q src tasks tests: pass on both.
  • macOS arm64, Python 3.12: the focused scoring, evaluator, held-out, task lifecycle, candidate verification, reporting, and visualization tests passed (156). The full suite showed no failure on this branch that does not also occur on main (Linux-only process containment and GNU tool assumptions); three timing-sensitive process tests failed only while both suites ran concurrently and passed when rerun alone.
  • No GPU evaluation was run. This change re-scores evaluator outputs and does not change measurement, so no before/after task evaluation is included; scores from earlier runs are not directly comparable with new ones.

@irvineoy
irvineoy marked this pull request as ready for review October 13, 2026 03:34
Tests added on main after this branch was cut still asserted the additive
120/220/320 scale. Update them to the bounded square-root score. The review
gate fixture now uses a synthetic 2x candidate, so its score assertion still
distinguishes accepted from rejected candidates instead of reading 0 for both.

Remove score cases that repeat an already covered branch: equal-class invalid
speedups and constants, a default-curve point duplicated by the power test,
method-consistency inputs covered by dedicated tests, and the all-zero
report-format run. Format the per-measurement average score with four
decimals, matching the other score fields.
src/utils/utils.py and src/utils/report_generation.py have no callers,
imports, or documentation references. They inferred compilation and
correctness from score thresholds of 20, 120, and 220, which the bounded
score no longer encodes: failed checks and unimproved correct candidates
now both score 0. Current reports derive those counts from the
pass_compilation and pass_correctness fields instead.
@irvineoy
irvineoy merged commit 8bb7092 into main Oct 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant