eval-invariance-engine

If an evaluation score changes substantially under semantics-preserving transformations, part of the measurement is sensitive to the evaluation representation rather than only the intended capability. This library quantifies that instability and ships a drop-in adapter, native metrics, and a CLI for Inspect AI.

It generalizes the option-order robustness method from mmlu-robustness-audit into a reusable, framework-agnostic tool.

Why

Leaderboards often present a headline score as if it were a stable property of a model. But the same model, on the same semantic questions, can move when answer labels rotate or other meaning-preserving presentation details change. If a score is not invariant to a declared non-semantic perturbation, part of the measurement depends on the evaluation representation. This engine makes that fragility measurable and rerunnable.

This package does not claim that instability proves contamination, memorization, or any specific causal mechanism. It measures the narrower quantity directly supported by the experiment: sensitivity of the evaluation result to declared semantics-preserving perturbations.

Install

pip install eval-invariance-engine          # core, zero dependencies
pip install "eval-invariance-engine[inspect]"  # + Inspect AI integration

CLI

invariance-check demo                          # built-in fragility demo
invariance-check report results.json           # score a results file, print the report
invariance-check report results.json --fail-on-fragile   # exit 1 if fragile — use it as a CI gate

results.json is either a mapping {"shift0": [true, false, ...], ...} or a list of {"condition": "shift0", "correct": true} rows. --fail-on-fragile turns the check into a regression gate: wire it into CI and a benchmark that starts drifting fails the build.

Quickstart (framework-agnostic)

from eval_invariance_engine import MCQItem, all_cyclic_variants, build_report

item = MCQItem("Capital of France?", ["Paris", "Rome", "Bonn", "Madrid"], answer_index=0)
results = {"shift0": [...], "shift1": [...], "shift2": [...], "shift3": [...]}  # bool per item
print(build_report(results).summary())
# [FRAGILE] n=... | drift=0.180 (95% CI 0.090-0.270) | flip_rate=0.31 | shift0=..., ...

One line: audit any Inspect multiple-choice task

from inspect_ai import eval
from eval_invariance_engine.inspect_adapter import invariance_task

# my_task is any Inspect Task whose samples have `choices` + `target`
eval(invariance_task(my_task), model="openai/gpt-4o-mini")

invariance_task() expands every item into all cyclic option orderings, scores them with the invariance scorer, and attaches the drift + flip-rate metrics — so a single call turns an existing benchmark into a fragility audit. Non-MCQ samples pass through untouched (skipped).

Using Inspect already? I am looking for outside tasks to test this wrapper against. See issue #1.

Inspect AI integration — native metrics

invariance_scorer() attaches two custom @metrics — invariance_drift (max accuracy spread across conditions) and invariance_flip_rate (fraction of items whose correctness flips) — so the fragility numbers appear inside the eval log alongside accuracy, not just in a post-hoc script.

from inspect_ai import Task, eval
from inspect_ai.solver import generate
from inspect_ai.dataset import MemoryDataset
from eval_invariance_engine import MCQItem
from eval_invariance_engine.inspect_adapter import cyclic_variant_samples, invariance_scorer

items = [MCQItem("2+2=?", ["4", "5", "6", "7"], 0)]
samples = [s for i, it in enumerate(items) for s in cyclic_variant_samples(it, group_id=str(i))]

task = Task(dataset=MemoryDataset(samples), solver=generate(), scorer=invariance_scorer())
logs = eval(task, model="openai/gpt-4o-mini")
# logs[0] now carries `invariance_drift` and `invariance_flip_rate` in its metrics.

Method

• Perturbations are deterministic and semantics-preserving: cyclic option reordering (the answer set is identical, only labels rotate) and whitespace restyling. The correct real-world answer never changes.
• Report: per-condition accuracy, max_drift (best - worst condition), flip_rate (fraction of items whose correctness flipped across conditions — the sharpest non-semantic-sensitivity signal), a bootstrap 95% CI on the drift, and a fragile / invariant verdict.
• Deterministic given a seed; the core is pure standard library.

Status

v0.3.0 — core, Inspect adapter, native @metrics, the invariance-check CLI, and the invariance_task() one-call wrapper, all tested (21 tests). Roadmap: whitespace/seed conditions surfaced as first-class Inspect conditions, plus outside usage and reproduction evidence.

License

MIT. AI-assisted authorship disclosed in AI_USAGE.md; all logic is deterministic and unit-tested.