Repository navigation
Use a linear default score curve (SCORE_POWER 0.5 -> 1.0) - #137
Open
sharareh-y wants to merge 1 commit into
Open
sharareh-y wants to merge 1 commit into
sharareh-y wants to merge 1 commit into
Conversation
The square-root curve has an unbounded slope at the 1.05x threshold, so the score of a kernel just above the cutoff swings with timing noise and can dominate a run's sensitivity. The linear curve keeps a finite slope there (t/x^2), scores the fraction of runtime saved beyond the threshold, and leaves more room to separate large speedups. The 1.05x threshold is unchanged. Update the README and API reference scoring sections, the release notes, and the expected scores in tests (2x: 0.475, 1.1x: 1/22, 1.5x: 0.3); pin the default constants in a test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR changes
SCORE_POWERinsrc/score.pyfrom0.5(square root) to1.0(linear).SCORE_SPEEDUP_THRESHOLDstays at1.05. The formula, thezero-score rules (compile/correctness failure, invalid or non-comparable
speedup, speedup at or below the threshold) and the aggregation rules do not
change.
Why
The figure plots score against task speedup (log scale, 1x to 1000x) at
t = 1.05forp = 0.25, 0.5, 1, 2. All curves are zero up to 1.05x. Thep < 1curves rise almost vertically just after the threshold and then flattenearly: under
p = 0.5a 2x kernel already gets 0.689. Thep = 1curve leavesthe threshold with a finite slope and keeps more room between 10x and 100x.
1. The square root has an infinite slope at the threshold
With
t = SCORE_SPEEDUP_THRESHOLDand speedupx > t:For
p < 1the factor(1 - t/x)^(p-1)diverges asx -> t+, so the slope isinfinite at the threshold. For
p = 1the slope ist / x^2, which is finite(at most
1 / t ~ 0.95per unit of speedup at the threshold). So underp = 0.5, a kernel just above 1.05x gets a score that swings strongly withordinary timing noise; under
p = 1it does not.Monte Carlo with lognormal noise on the measured speedup (50,000 draws per
point,
t = 1.05), peak score standard deviation:On a real 32-task hip2hip run (all correct), one task sat 0.2% above the
threshold (1.0517x). Under
p = 0.5that single task carried 59% of therun's sensitivity to small speedup changes (its share of d(mean score)/d ln x).
Under
p = 1it carries 10%.2. The square root saturates early
Under
p = 0.5, going from 10x to 100x adds 0.049. Underp = 1it adds0.095, about twice the headroom, so large wins remain distinguishable.
3. The linear score has a plain meaning
With
p = 1,score = 1 - candidate_time / (baseline_time / 1.05): thefraction of runtime saved beyond the threshold. Over a set of tasks that all
score above zero, the mean score is
1 - 1.05 / H, whereHis theharmonic-mean speedup of those tasks.
Trade-off
Modest wins earn less. A 1.5x kernel goes from 0.548 to 0.300, and a 1.1x
kernel from 0.213 to 0.045. If the project wants to reward breadth of small
wins strongly,
p = 0.5does that, but at the cost of the near-thresholdnoise described above.
Changes
src/score.py:SCORE_POWER = 1.0; docstring describes the linear default.README.md("Reward and Scoring Signals"):p = 1(linear), recomputedexample table (1.1x 0.0455, 1.2x 0.1250, 2x 0.4750), and a description of the
linear score and its mean.
docs/reference/api-reference.md(Scoring): default curve, constants table(
SCORE_POWERdefault1.0), slope note, and the 1.25x worked example(default 0.16; 0.4 with
p = 0.5).tests/test_score.py: new test pinning the defaults (power 1.0, threshold1.05); default-curve test now checks linear values at 1.0x, 1.05x, 1.1x, 1.2x,
1.25x and 2x; configurable-power test covers 0.5, 1.0 and 2.0; the
task_result_scoring1.5x case expects 0.3.tests/test_task_run_v2.py,tests/test_candidate_verification.py,tests/test_held_out_v2.py: synthetic 2x fixtures expect 0.475 (was 0.6892).tests/test_postprocessing_outcomes_v2.py: the 1.1x reporting fixtureexpects
1/22(0.0455 in four-decimal reports; was 0.2132).tests/test_score.py(threshold test): with the threshold set to 1.2, a 1.25xspeedup expects 0.04 (was
sqrt(0.04) = 0.2).docs/reference/release-notes.md: "Unreleased" entry for the curve change.Testing
CPU-only tests, run on an MI355X node (no GPU validation is claimed):
Result: 139 passed.