[JacobStephens2]

Summary

0.27.1 adds blind pairwise grading of process Trials by a Grader panel, people, or both. The Bar fixes whether their calls sit beside the Verdict or count toward it (#1944, #1953).

• thirdshift experiment <candidate> gains panel seats, voting and family rules, --panel off|beside|counted, --people off|beside|counted, seeded reader samples and overlap, and --panel-spending-limit <usd>. The Console accepts the same settings (Grade each pair of process Trials with a Grader panel, beside the Verdict #1948, Estimate and cap the Grader panel's spend (#1884) #1951, Count the panel or people in the Verdict, with seeded samples for several readers #1953).
• thirdshift experiments <id> --pair <pair> and the Console’s blind pair page show the Ticket, checklist, diffs and evidence. Readers seal merge calls, then submit a preference and optional rubric before revealing Arms, panel calls and readers’ calls (Add the blind pairwise review page to the Console #1946). Identifying diff lines and error paths are removed from the blind page (Spec: blind pairwise grading of process Trials by a Grader panel and people #1944).
• Counted people’s calls require every assigned reader to call an Arm ready; missing calls leave the Verdict at waiting_on_people. A counted panel can stand in outside the sample, subject to family and agreement requirements. Low people–panel agreement makes the Verdict inconclusive; people-only sampling uses process pairs as statistical units (Count the panel or people in the Verdict, with seeded samples for several readers #1953, Spec: blind pairwise grading of process Trials by a Grader panel and people #1944).
• Panels judge each pair in both presentation orders and default to voting by provider family. Failed grading pauses the Experiment; resumes request only missing calls (Grade each pair of process Trials with a Grader panel, beside the Verdict #1948). Panel estimates and spending are reported separately from the Arms. The spending limit applies per command run and starts afresh on resume (Estimate and cap the Grader panel's spend (#1884) #1951).
• Tasks opting into [process_grading.new_tests] get offline base-tests and cross-tests for each pair, with evidence beside the Scores and a Console table (Run base-tests and cross-tests for each pair of Trials #1943).
• OpenCode Arms and Graders can use Fireworks and Baseten, with per-provider keys under [opencode.keys] and opencode.api_key as the fallback (Allow Fireworks and Baseten through OpenCode, with a key per provider #1942).
• The model Grader is now chosen from a family neither Arm uses. Experiment JSON moves to format 10 (Spec: blind pairwise grading of process Trials by a Grader panel and people #1944).

Also: site and docs updates (#1942, #1943, #1944, #1946, #1948, #1951, #1953), and the 0.27.0 release pull request (#1957).

Version diff

diff --git a/Cargo.lock b/Cargo.lock
index 0c8249db..29825a74 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -2432,3 +2432,3 @@ dependencies = [
 name = "thirdshift"
-version = "0.27.0"
+version = "0.27.1"
 dependencies = [
diff --git a/Cargo.toml b/Cargo.toml
index cb2bb090..a283d20c 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -2,3 +2,3 @@
 name = "thirdshift"
-version = "0.27.0"
+version = "0.27.1"
 edition = "2024"