Skip to content

Use a linear default score curve (SCORE_POWER 0.5 -> 1.0) - #137

Open
sharareh-y wants to merge 1 commit into
mainfrom
linear-score-power
Open

sharareh-y wants to merge 1 commit into
mainfrom
linear-score-power

Conversation

@sharareh-y

Copy link
Copy Markdown
Collaborator

Summary

This PR changes SCORE_POWER in src/score.py from 0.5 (square root) to
1.0 (linear). SCORE_SPEEDUP_THRESHOLD stays at 1.05. The formula, the
zero-score rules (compile/correctness failure, invalid or non-comparable
speedup, speedup at or below the threshold) and the aggregation rules do not
change.

score = max(0, 1 - 1.05 / speedup) ** p        p: 0.5 -> 1.0
Speedup Score, p = 0.5 (before) Score, p = 1 (after)
<= 1.05x 0 0
1.1x 0.2132 0.0455
1.2x 0.3536 0.1250
1.5x 0.5477 0.3000
2x 0.6892 0.4750

Why

fig1_power_sensitivity

The figure plots score against task speedup (log scale, 1x to 1000x) at
t = 1.05 for p = 0.25, 0.5, 1, 2. All curves are zero up to 1.05x. The
p < 1 curves rise almost vertically just after the threshold and then flatten
early: under p = 0.5 a 2x kernel already gets 0.689. The p = 1 curve leaves
the threshold with a finite slope and keeps more room between 10x and 100x.

1. The square root has an infinite slope at the threshold

With t = SCORE_SPEEDUP_THRESHOLD and speedup x > t:

d score / d x = p * (1 - t/x)^(p-1) * t / x^2

For p < 1 the factor (1 - t/x)^(p-1) diverges as x -> t+, so the slope is
infinite at the threshold. For p = 1 the slope is t / x^2, which is finite
(at most 1 / t ~ 0.95 per unit of speedup at the threshold). So under
p = 0.5, a kernel just above 1.05x gets a score that swings strongly with
ordinary timing noise; under p = 1 it does not.

Monte Carlo with lognormal noise on the measured speedup (50,000 draws per
point, t = 1.05), peak score standard deviation:

Noise on speedup p = 0.5 p = 1
2% 0.074 (at 1.06x) 0.019 (at 1.10x)
5% 0.116 (at 1.08x) 0.044 (at 1.16x)

On a real 32-task hip2hip run (all correct), one task sat 0.2% above the
threshold (1.0517x). Under p = 0.5 that single task carried 59% of the
run's sensitivity to small speedup changes (its share of d(mean score)/d ln x).
Under p = 1 it carries 10%.

2. The square root saturates early

p 2x 10x 100x
0.5 (before) 0.689 0.946 0.995
1 (after) 0.475 0.895 0.990

Under p = 0.5, going from 10x to 100x adds 0.049. Under p = 1 it adds
0.095, about twice the headroom, so large wins remain distinguishable.

3. The linear score has a plain meaning

With p = 1, score = 1 - candidate_time / (baseline_time / 1.05): the
fraction of runtime saved beyond the threshold. Over a set of tasks that all
score above zero, the mean score is 1 - 1.05 / H, where H is the
harmonic-mean speedup of those tasks.

Trade-off

Modest wins earn less. A 1.5x kernel goes from 0.548 to 0.300, and a 1.1x
kernel from 0.213 to 0.045. If the project wants to reward breadth of small
wins strongly, p = 0.5 does that, but at the cost of the near-threshold
noise described above.

Changes

  • src/score.py: SCORE_POWER = 1.0; docstring describes the linear default.
  • README.md ("Reward and Scoring Signals"): p = 1 (linear), recomputed
    example table (1.1x 0.0455, 1.2x 0.1250, 2x 0.4750), and a description of the
    linear score and its mean.
  • docs/reference/api-reference.md (Scoring): default curve, constants table
    (SCORE_POWER default 1.0), slope note, and the 1.25x worked example
    (default 0.16; 0.4 with p = 0.5).
  • tests/test_score.py: new test pinning the defaults (power 1.0, threshold
    1.05); default-curve test now checks linear values at 1.0x, 1.05x, 1.1x, 1.2x,
    1.25x and 2x; configurable-power test covers 0.5, 1.0 and 2.0; the
    task_result_scoring 1.5x case expects 0.3.
  • tests/test_task_run_v2.py, tests/test_candidate_verification.py,
    tests/test_held_out_v2.py: synthetic 2x fixtures expect 0.475 (was 0.6892).
  • tests/test_postprocessing_outcomes_v2.py: the 1.1x reporting fixture
    expects 1/22 (0.0455 in four-decimal reports; was 0.2132).
  • tests/test_score.py (threshold test): with the threshold set to 1.2, a 1.25x
    speedup expects 0.04 (was sqrt(0.04) = 0.2).
  • docs/reference/release-notes.md: "Unreleased" entry for the curve change.

Testing

CPU-only tests, run on an MI355X node (no GPU validation is claimed):

python3 -m pytest -q tests/test_score.py tests/test_task_run_v2.py tests/test_candidate_verification.py tests/test_held_out_v2.py tests/test_postprocessing_outcomes_v2.py tests/test_held_out.py tests/test_task_boundary_failures_v2.py

Result: 139 passed.

The square-root curve has an unbounded slope at the 1.05x threshold, so
the score of a kernel just above the cutoff swings with timing noise and
can dominate a run's sensitivity. The linear curve keeps a finite slope
there (t/x^2), scores the fraction of runtime saved beyond the threshold,
and leaves more room to separate large speedups. The 1.05x threshold is
unchanged.

Update the README and API reference scoring sections, the release notes,
and the expected scores in tests (2x: 0.475, 1.1x: 1/22, 1.5x: 0.3); pin
the default constants in a test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant