fix(tier3): use one overall-score formula - #124
Conversation
Use the five-dimension mean consistently for Harbor rewards, pass@k, lift, reports, comparisons, and best-agent selection. Persist the score policy while preserving legacy artifact semantics. Signed-off-by: Saibernard Yogendran <bernie97@seas.upenn.edu>
Signed-off-by: Saibernard Yogendran <bernie97@seas.upenn.edu>
rng1995
left a comment
There was a problem hiding this comment.
The unified formula is correct for newly recorded policy-aware results, but current-head rendering and comparison reinterpret historical default-v2 artifacts under the new dimension policy. That can change stored scores and verdicts. The focused scoring, reporting, and history suites passed (419 tests, 4 skipped apart from one unchanged host-Python subprocess issue); Ruff and diff checks are clean. Requesting changes for the two historical-compatibility paths noted inline.
Signed-off-by: Saibernard Yogendran <bernie97@seas.upenn.edu>
|
@rng1995 Thanks for catching this. I addressed both historical-compatibility issues in 1237c59. Missing-policy default-v2 artifacts now preserve their recorded scoring behavior in reports and comparisons, including overall score, baseline, Skill Lift, pass@k, and best-agent ordering. Explicit current-policy artifacts still use the five-dimension formula. I added focused regression coverage, and the full suite passes with 5,495 tests passed and 18 skipped. |
Summary
Tier 3 already defines its overall score as the equal-weight mean of Security, Correctness, Discoverability, Effectiveness, and Efficiency. Some scoring paths instead averaged the six evaluator outputs directly. That gave
goal_accuracyandbehavior_checkseparate votes even though they are the two halves of Effectiveness.This PR makes Harbor rewards, pass@k, Skill Lift, reports, comparisons, and best-agent selection use the published five-dimension formula.
The difference is visible when Effectiveness pulls away from the other dimensions:
The old calculation reported
0.6667 - 0.5833 = +0.0833. The documented formula gives0.6000 - 0.6500 = -0.0500, which matches the dimension table. Scores do not always change, but when they do, pass@k results can cross the threshold, best-agent selection can change, and Skill Lift can reverse sign.New artifacts identify the formula as
skill-evaluator-dimension-mean-v1. Historical and partial artifacts keep their recorded scoring behavior._SCHEMA_VERSIONremains"2.0"becausescore_policyis additive and does not break existing consumers.The documentation now explains the score ranges, which evaluators use an LLM judge, which use deterministic trace checks, how to find judge provenance, how Skill Lift is calculated, and why the Tier 1 Quality Score shown out of 100 is separate from Tier 3.
Relates to #61. This PR documents the current judge provenance, but it does not claim that the LLM judges have been validated. The repository does not publish inter-rater agreement, calibration, or a comparison against human-labeled ground truth, so that part of the issue remains open.
Two existing assertions change with the documented contract.
test_judge_failure_artifacts.pyexpects0.6because three of five dimensions score1.0, andtest_tier3_schema.pynow checks the named dimensions and versioned policy rather than raw metric names.Verification
make lintmake testmake buildThe score-policy regression tests and lint checks pass. The full suite matches the base commit, and the package and documentation checks pass.
Release Impact
CHANGELOG.md