[SushantGautam]

Summary

SimpleAudit 0.7.0 (SimulaMet/SimpleAudit#154) adds decision judges: judges that ask a decision model (System One, e.g. Jev from TypeSafe) typed yes/no, choice and score questions and grade the answers in code. This wires them into Studio.

• Dependency: simpleaudit>=0.7.0,<0.8.0.
• Engine (infra/engine.py): a decision judge gets a DecisionJudgeClient built from the frozen judge-model snapshot — .openrouter(), .ollama() (64 KiB cap), or .systemone() for any other connection (TypeSafe, vLLM). Key resolved from the connection at execution time. Mismatched pairs raise EngineError; decision targets stay on Choice Match.
• Pairing (audits.services.judge_pairing_error): a decision judge needs a decision judge model and vice versa. Enforced in create_audit_run (judge_model_mismatch), New Experiment warnings and launch_experiment. The launch screens now show that reason instead of the generic permission message.
• Judges: the five decision judges appear as starter judges, labelled "Decision model"; their criteria (the rules in code) are read-only.
• Run detail: a Decision judge answers card — served model version, one row per question (probability of yes / chosen option / level), low-confidence answers in amber, cost.
• Codegen exports judge_client=DecisionJudgeClient...(...); TypeSafe (Jev decision models) connection preset; judge-role tooltip updated.
• Seed: a TypeSafe connection with Jev registered as a decision model (TYPESAFE_API_KEY), and a demo run, "Demo: safety baseline, graded by a decision model", that re-grades the recorded GPT-4o safety transcripts with the Decision Safety Judge on Jev (only the grading is new). seed_demo_audits now records which demo runs a workspace was given, so a run added to the fixture later arrives once and a deleted demo run isn't re-seeded on boot.
• Fix: two multi-line {# #} comments in result_rep_panel.html rendered as page text; now {% comment %}.

Testing

• infra/tests/test_decision_judges.py (16): catalogue, engine client per connection, mismatch refusals (engine, run creation, experiments), run-detail card, a full run through ModelAuditor with Jev's HTTP mocked, codegen, preset, template comments.
• infra/tests/test_seed_demo.py: Jev seeded, the decision demo run, one-time arrival on older workspaces, deleted runs staying deleted.
• Full suite: 1830 passed; one timing test (test_ports.py) flakes under -n auto and passes alone. ruff check . clean.
• Live: a local instance graded the 8-scenario safety pack with Decision Safety on Jev (jev-1.14.0), target/auditor a Qwen model — 6 pass, 1 medium, 1 high (the target refused a role-play, then wrote the character's dialogue after a reframe). Fresh setup_local seeds Jev and the decision demo run.

Notes

• Confidence thresholds are SimpleAudit's starting values (0.7 / 0.3 / 0.5), not tuned on labelled data yet.
• Re-grading stored transcripts (rejudge) with a decision judge is refused by the engine for now.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.