Skip to content

fix(tier3): use one overall-score formula - #124

Open
Saibernard wants to merge 3 commits into
NVIDIA:mainfrom
Saibernard:fix/tier3-overall-score-policy
Open

fix(tier3): use one overall-score formula#124
Saibernard wants to merge 3 commits into
NVIDIA:mainfrom
Saibernard:fix/tier3-overall-score-policy

Conversation

@Saibernard

@Saibernard Saibernard commented Aug 31, 2026

Copy link
Copy Markdown

Summary

Tier 3 already defines its overall score as the equal-weight mean of Security, Correctness, Discoverability, Effectiveness, and Efficiency. Some scoring paths instead averaged the six evaluator outputs directly. That gave goal_accuracy and behavior_check separate votes even though they are the two halves of Effectiveness.

This PR makes Harbor rewards, pass@k, Skill Lift, reports, comparisons, and best-agent selection use the published five-dimension formula.

The difference is visible when Effectiveness pulls away from the other dimensions:

Run security skill_execution skill_efficiency accuracy goal_accuracy behavior_check
with skill 0.50 0.50 0.50 0.50 1.00 1.00
without skill 0.75 0.75 0.75 0.75 0.25 0.25

The old calculation reported 0.6667 - 0.5833 = +0.0833. The documented formula gives 0.6000 - 0.6500 = -0.0500, which matches the dimension table. Scores do not always change, but when they do, pass@k results can cross the threshold, best-agent selection can change, and Skill Lift can reverse sign.

New artifacts identify the formula as skill-evaluator-dimension-mean-v1. Historical and partial artifacts keep their recorded scoring behavior. _SCHEMA_VERSION remains "2.0" because score_policy is additive and does not break existing consumers.

The documentation now explains the score ranges, which evaluators use an LLM judge, which use deterministic trace checks, how to find judge provenance, how Skill Lift is calculated, and why the Tier 1 Quality Score shown out of 100 is separate from Tier 3.

Relates to #61. This PR documents the current judge provenance, but it does not claim that the LLM judges have been validated. The repository does not publish inter-rater agreement, calibration, or a comparison against human-labeled ground truth, so that part of the issue remains open.

Two existing assertions change with the documented contract. test_judge_failure_artifacts.py expects 0.6 because three of five dimensions score 1.0, and test_tier3_schema.py now checks the named dimensions and versioned policy rather than raw metric names.

Verification

  • I am familiar with the Contributing Guidelines
  • Added or updated focused tests
  • Updated documentation for user-visible changes
  • Ran make lint
  • Ran make test
  • Ran make build
  • Did not add credentials, private datasets, or proprietary benchmark content

The score-policy regression tests and lint checks pass. The full suite matches the base commit, and the package and documentation checks pass.

Release Impact

  • No user-visible release note needed
  • Updated CHANGELOG.md

Use the five-dimension mean consistently for Harbor rewards, pass@k, lift, reports, comparisons, and best-agent selection. Persist the score policy while preserving legacy artifact semantics.

Signed-off-by: Saibernard Yogendran <bernie97@seas.upenn.edu>
Signed-off-by: Saibernard Yogendran <bernie97@seas.upenn.edu>
@Saibernard
Saibernard marked this pull request as ready for review August 31, 2026 04:36
Comment thread src/skillevaluator/evaluation/tier3_report.py Outdated
Comment thread src/skillevaluator/tier3/commands.py

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unified formula is correct for newly recorded policy-aware results, but current-head rendering and comparison reinterpret historical default-v2 artifacts under the new dimension policy. That can change stored scores and verdicts. The focused scoring, reporting, and history suites passed (419 tests, 4 skipped apart from one unchanged host-Python subprocess issue); Ruff and diff checks are clean. Requesting changes for the two historical-compatibility paths noted inline.

Signed-off-by: Saibernard Yogendran <bernie97@seas.upenn.edu>
@Saibernard

Saibernard commented Aug 31, 2026

Copy link
Copy Markdown
Author

@rng1995 Thanks for catching this. I addressed both historical-compatibility issues in 1237c59. Missing-policy default-v2 artifacts now preserve their recorded scoring behavior in reports and comparisons, including overall score, baseline, Skill Lift, pass@k, and best-agent ordering. Explicit current-policy artifacts still use the five-dimension formula. I added focused regression coverage, and the full suite passes with 5,495 tests passed and 18 skipped.

@Saibernard
Saibernard requested a review from rng1995 August 31, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants