@@ -0,0 +1,31 @@
+---
+"@hyperdx/hdx-eval": minor
+---
+
+feat(evals): support an independent inference model/provider for LLM-as-judge
+grading. The grader can now differ from the run model (e.g. run with Anthropic,
+grade with OpenAI) to reduce same-model bias. `--judge-model` accepts a
+`provider:model` spec (`anthropic`, `openai`; a bare model name defaults to
+anthropic), configurable per repo via `eval.config.json` `grading.judgeModel`.
+The judge now uses the Vercel AI SDK, so grader credentials come from
+`AI_API_KEY`/`OPENAI_API_KEY`/`ANTHROPIC_API_KEY` (plus `AI_BASE_URL` /
+`AI_REQUEST_HEADERS`), mirroring the API package. The provider-specific key
+takes precedence over `AI_API_KEY` so a runner key and a differing grader key
+don't collide.
+
+The judge prompt now defines explicit 0-5 scoring anchors so scores stay
+calibrated across judge models, and the judge is more robust to intermittent
+structured-output malformations (recovers nested/stringified `scores` payloads
+and retries once on schema failure). Reasoning judge models (OpenAI gpt-5.x /
+o-series) get a larger output-token budget so hidden reasoning tokens don't
+truncate the JSON.
+
+fix(evals): re-grading a batch with a different judge model now actually
+re-runs that judge instead of silently returning the previously cached judge's
+scores. The `needsJudge` check and per-run reuse guard were keyed on the mere
+presence of a cached judge, not its identity, so `grade <batch> --judge-model
+openai:gpt-5.6-sol` over an Opus-graded batch would skip the LLM call and hand
+back the stale Opus scores. Both now key on the judge spec: a cached grade from
+a different judge model is treated as stale. `--rerun-judge` still forces a
+refresh with the same judge. This is required for grader-bias comparisons
+(grading one batch with two judges).
@@ -337,14 +337,44 @@ dev runs-instrument <batch>                 # enrich with ClickHouse query_log t
 ### Grading & Reporting
 
 ```bash
-dev grade <batch>                           # programmatic + LLM-as-judge
-dev grade <batch> --judge-model claude-sonnet-4-6
-dev grade <batch> --no-judge                # programmatic checks only
-dev grade <batch> --rerun-judge             # re-run judge even if grades exist
-dev report <batch>                          # render _summary.md + _summary.json
-dev report <batch> --baseline hyperdx-main  # override baseline for deltas
+dev grade <batch>                            # programmatic + LLM-as-judge
+dev grade <batch> --judge-model openai:gpt-4o        # grade with a different provider
+dev grade <batch> --no-judge                 # programmatic checks only
+dev grade <batch> --rerun-judge              # force re-grade with the current judge
+dev report <batch>                           # render _summary.md + _summary.json
+dev report <batch> --baseline hyperdx-main   # override baseline for deltas
 ```
 
+#### Independent grader (different provider/model)
+
+The LLM-as-judge can run on a **different provider/model than the run model**
+(Anthropic run graded by OpenAI, and vice-versa), which reduces same-model bias.
+Pass `--judge-model` as a `provider:model` spec — supported providers
+`anthropic` and `openai`; a bare model name defaults to `anthropic`:
+
+```bash
+dev grade <batch> --judge-model openai:gpt-4o
+dev grade <batch> --judge-model claude-opus-4-7   # == anthropic:claude-opus-4-7
+```
+
+Precedence: `--judge-model` flag → `eval.config.json` `grading.judgeModel` →
+built-in default (`anthropic:claude-opus-4-7`):
+
+```json
+{ "grading": { "judgeModel": "openai:gpt-4o" } }
+```
+
+Grades are cached per run; re-grading a batch with a **different** judge
+re-runs that judge automatically (a cached grade from another judge model is
+treated as stale). Use `--rerun-judge` to force a refresh with the same judge.
+
+**Grader credentials** come from the environment, mirroring `packages/api`:
+`OPENAI_API_KEY` / `ANTHROPIC_API_KEY` per provider, `AI_API_KEY` as a fallback,
+`AI_BASE_URL` for a custom endpoint (Azure AI, LiteLLM proxy), and
+`AI_REQUEST_HEADERS` (OpenAI) as a JSON object. The **provider-specific key
+wins** over `AI_API_KEY`, so the runner's Anthropic key can live in `AI_API_KEY`
+while an OpenAI grader uses `OPENAI_API_KEY` without the two colliding.
+
 ### Viewer
 
 ```bash
@@ -366,10 +396,16 @@ Combined score = `0.4 * programmatic + 0.6 * judge - toolErrorPenalty`
   answer, defined per-scenario in `ground-truth.json`. Includes both positive
   checks (must find the right thing) and negative checks (must not blame a
   distractor).
-- **LLM-as-judge (60%)** — multi-criteria rubric scored 0-5 by Claude Opus
-  (configurable). Criteria are scenario-specific. Answers are **blinded** —
-  MCP tool names and brand terms are redacted so the judge can't tell which
-  MCP produced the answer.
+- **LLM-as-judge (60%)** — scenario-specific, multi-criteria rubric scored 0-5
+  by an LLM judge (configurable — see [Independent grader](#independent-grader-different-providermodel)).
+  The judge prompt defines **explicit 0-5 scoring anchors** (0 = wrong/absent …
+  5 = fully correct) to keep scores calibrated across judge models, and answers
+  are **blinded** — MCP tool names and brand terms are redacted so the judge
+  can't tell which MCP produced the answer.
+
+  > Judge scores are only comparable **within the same judge model** (anchors
+  > reduce but don't erase per-model scale differences). Hold the judge constant
+  > across runs you compare directly.
 - **Tool error penalty** — up to 20% deducted for high tool-call error rates.
 
 ### Baseline + Challengers
 
@@ -13,10 +13,13 @@
     "dist/*"
   ],
   "dependencies": {
-    "@anthropic-ai/sdk": "^0.92.0",
+    "@ai-sdk/anthropic": "^3.0.58",
+    "@ai-sdk/openai": "^3.0.47",
     "@clickhouse/client": "1.23.0-head.fae5998.1",
+    "ai": "^6.0.116",
     "commander": "^12.1.0",
-    "ms": "^2.1.3"
+    "ms": "^2.1.3",
+    "zod": "3.25"
   },
   "devDependencies": {
     "@dotenvx/dotenvx": "^1.51.1",
 
@@ -0,0 +1,255 @@
+/**
+ * Verifies that gradeBatch records the fully-qualified `provider:model` judge
+ * spec on the GradeRecord and threads the resolved spec into judgeTrajectory.
+ * The judge module is mocked so no provider credentials or network are needed.
+ */
+
+import { mkdirSync, readFileSync, rmSync, writeFileSync } from 'fs';
+import { join } from 'path';
+
+jest.mock('@/grading/judge', () => ({
+  judgeTrajectory: jest.fn(),
+}));
+
+import { gradeBatch } from '@/grading/grade';
+import { judgeTrajectory } from '@/grading/judge';
+import type { JudgeResult } from '@/grading/types';
+import type { RunRecord } from '@/harness/types';
+
+const mockJudge = judgeTrajectory as jest.MockedFunction<
+  typeof judgeTrajectory
+>;
+
+const ANSWER =
+  'Root cause: payment-service ConnectionTimeoutError on db.payment.connect ' +
+  'reaching db-payment.internal — DB connection timeout — cascading into ' +
+  'checkout-api 5xx errors. Ruled out concurrent SMTP and CDN bursts ' +
+  '(separate trace trees, no checkout parent).';
+
+function buildRun(): RunRecord {
+  return {
+    schemaVersion: 1,
+    runId: 'error-root-cause-test-0',
+    scenario: 'error-root-cause',
+    mcp: 'hyperdx',
+    model: 'claude-sonnet-4-6',
+    plugin: 'none',
+    runIndex: 0,
+    seed: 42,
+    startedAt: '2026-05-10T00:00:00.000Z',
+    endedAt: '2026-05-10T00:01:00.000Z',
+    durationMs: 60_000,
+    agentPrompt: 'p',
+    systemPromptAppend: 's',
+    termination: 'final_answer',
+    exitCode: 0,
+    tools: [],
+    toolCalls: [
+      {
+        name: 'mcp__hyperdx__hyperdx_query',
+        input: null,
+        output: null,
+        isError: false,
+        startedAt: '',
+        endedAt: null,
+        durationMs: null,
+      },
+    ],
+    messages: [],
+    finalAnswer: ANSWER,
+    tokens: { input: 0, output: 0, cacheCreation: 0, cacheRead: 0 },
+    totalCostUsd: 0,
+    stderr: '',
+  };
+}
+
+function writeBatch(batchDir: string): void {
+  const dir = join(batchDir, 'error-root-cause', 'hyperdx');
+  mkdirSync(dir, { recursive: true });
+  writeFileSync(join(dir, '0.json'), JSON.stringify(buildRun(), null, 2));
+}
+
+function fakeJudgeResult(model: string): JudgeResult {
+  return {
+    model,
+    scores: {},
+    weightedScore: 0.5,
+    rawResponse: '{}',
+    durationMs: 1,
+    tokens: { input: 1, output: 1, cacheCreation: 0, cacheRead: 0 },
+  };
+}
+
+describe('gradeBatch judge model spec', () => {
+  const tmpRoot = join('/tmp', `hdx-eval-judge-model-test-${Date.now()}`);
+  const savedEnv: Record<string, string | undefined> = {};
+
+  beforeAll(() => {
+    for (const k of ['AI_API_KEY', 'ANTHROPIC_API_KEY', 'OPENAI_API_KEY']) {
+      savedEnv[k] = process.env[k];
+    }
+  });
+
+  afterAll(() => {
+    for (const [k, v] of Object.entries(savedEnv)) {
+      if (v === undefined) delete process.env[k];
+      else process.env[k] = v;
+    }
+    rmSync(tmpRoot, { recursive: true, force: true });
+  });
+
+  beforeEach(() => {
+    mockJudge.mockReset();
+  });
+
+  it('threads the resolved spec to the judge and records it', async () => {
+    process.env.OPENAI_API_KEY = 'sk-oai-test';
+    delete process.env.AI_API_KEY;
+    const batchDir = join(tmpRoot, 'openai');
+    writeBatch(batchDir);
+
+    // The judge echoes back the spec it was called with.
+    mockJudge.mockImplementation(async opts =>
+      fakeJudgeResult(opts.judgeModel as string),
+    );
+
+    const result = await gradeBatch(batchDir, { judgeModel: 'openai:gpt-4o' });
+
+    expect(mockJudge).toHaveBeenCalledTimes(1);
+    expect(mockJudge.mock.calls[0][0].judgeModel).toBe('openai:gpt-4o');
+    expect(result.graded[0].judgeModel).toBe('openai:gpt-4o');
+    expect(result.graded[0].judge?.model).toBe('openai:gpt-4o');
+  });
+
+  it('normalizes a bare model name to the anthropic provider', async () => {
+    process.env.ANTHROPIC_API_KEY = 'sk-ant-test';
+    delete process.env.AI_API_KEY;
+    delete process.env.OPENAI_API_KEY;
+    const batchDir = join(tmpRoot, 'anthropic-bare');
+    writeBatch(batchDir);
+
+    mockJudge.mockImplementation(async opts =>
+      fakeJudgeResult(opts.judgeModel as string),
+    );
+
+    const result = await gradeBatch(batchDir, {
+      judgeModel: 'claude-opus-4-7',
+    });
+
+    expect(mockJudge.mock.calls[0][0].judgeModel).toBe(
+      'anthropic:claude-opus-4-7',
+    );
+    expect(result.graded[0].judgeModel).toBe('anthropic:claude-opus-4-7');
+  });
+
+  it('throws a provider-aware error when the grader key is missing', async () => {
+    delete process.env.AI_API_KEY;
+    delete process.env.ANTHROPIC_API_KEY;
+    delete process.env.OPENAI_API_KEY;
+    const batchDir = join(tmpRoot, 'nokey');
+    writeBatch(batchDir);
+
+    await expect(
+      gradeBatch(batchDir, { judgeModel: 'openai:gpt-4o' }),
+    ).rejects.toThrow(/No API key set for the "openai" judge/);
+    expect(mockJudge).not.toHaveBeenCalled();
+  });
+
+  it('records "skipped" and never calls the judge with --no-judge', async () => {
+    delete process.env.AI_API_KEY;
+    delete process.env.ANTHROPIC_API_KEY;
+    delete process.env.OPENAI_API_KEY;
+    const batchDir = join(tmpRoot, 'skipped');
+    writeBatch(batchDir);
+
+    const result = await gradeBatch(batchDir, {
+      judgeModel: 'openai:gpt-4o',
+      skipJudge: true,
+    });
+
+    expect(mockJudge).not.toHaveBeenCalled();
+    expect(result.graded[0].judge).toBeNull();
+    expect(result.graded[0].judgeModel).toBe('skipped');
+  });
+
+  it('re-runs the judge when re-grading a batch with a DIFFERENT judge model', async () => {
+    // Regression guard for the silent-skip bug: a cached grade from judge A
+    // must NOT suppress judge B on a second pass. Without keying needsJudge /
+    // the per-run guard on judge identity, pass B returns pass A's stale
+    // scores relabeled — which would quietly corrupt an Opus-vs-GPT grader
+    // comparison.
+    process.env.ANTHROPIC_API_KEY = 'sk-ant-test';
+    process.env.OPENAI_API_KEY = 'sk-oai-test';
+    delete process.env.AI_API_KEY;
+    const batchDir = join(tmpRoot, 'judge-swap');
+    writeBatch(batchDir);
+
+    mockJudge.mockImplementation(async opts =>
+      fakeJudgeResult(opts.judgeModel as string),
+    );
+
+    // Pass A — Anthropic judge.
+    await gradeBatch(batchDir, { judgeModel: 'anthropic:claude-opus-4-7' });
+    expect(mockJudge).toHaveBeenCalledTimes(1);
+
+    // Pass B — different judge, NO --rerun-judge. Must re-run, not skip.
+    mockJudge.mockClear();
+    const passB = await gradeBatch(batchDir, {
+      judgeModel: 'openai:gpt-5.6-sol',
+    });
+
+    expect(mockJudge).toHaveBeenCalledTimes(1);
+    expect(mockJudge.mock.calls[0][0].judgeModel).toBe('openai:gpt-5.6-sol');
+    expect(passB.graded[0].judgeModel).toBe('openai:gpt-5.6-sol');
+    expect(passB.graded[0].judge?.model).toBe('openai:gpt-5.6-sol');
+
+    // Sidecar now reflects judge B, not the stale judge A.
+    const sidecar = JSON.parse(
+      readFileSync(
+        join(batchDir, 'error-root-cause', 'hyperdx', '0.grade.json'),
+        'utf8',
+      ),
+    ) as { judgeModel: string };
+    expect(sidecar.judgeModel).toBe('openai:gpt-5.6-sol');
+  });
+
+  it('does NOT re-run the judge when re-grading with the SAME judge model', async () => {
+    // The other half of the identity check: an unchanged judge should still be
+    // cached (no needless re-spend). Only --rerun-judge forces a refresh.
+    process.env.OPENAI_API_KEY = 'sk-oai-test';
+    delete process.env.AI_API_KEY;
+    const batchDir = join(tmpRoot, 'judge-same');
+    writeBatch(batchDir);
+
+    mockJudge.mockImplementation(async opts =>
+      fakeJudgeResult(opts.judgeModel as string),
+    );
+
+    await gradeBatch(batchDir, { judgeModel: 'openai:gpt-5.6-sol' });
+    expect(mockJudge).toHaveBeenCalledTimes(1);
+
+    mockJudge.mockClear();
+    await gradeBatch(batchDir, { judgeModel: 'openai:gpt-5.6-sol' });
+    expect(mockJudge).not.toHaveBeenCalled();
+  });
+
+  it('reuses the sidecar-persisted judgeModel spec when re-reading', async () => {
+    process.env.OPENAI_API_KEY = 'sk-oai-test';
+    delete process.env.AI_API_KEY;
+    const batchDir = join(tmpRoot, 'persist');
+    writeBatch(batchDir);
+
+    mockJudge.mockImplementation(async opts =>
+      fakeJudgeResult(opts.judgeModel as string),
+    );
+
+    await gradeBatch(batchDir, { judgeModel: 'openai:gpt-4o' });
+    const sidecar = JSON.parse(
+      readFileSync(
+        join(batchDir, 'error-root-cause', 'hyperdx', '0.grade.json'),
+        'utf8',
+      ),
+    ) as { judgeModel: string };
+    expect(sidecar.judgeModel).toBe('openai:gpt-4o');
+  });
+});