Skip to content

Commit 0480c3e

Browse files
jongioCopilot
andauthored
feat(report): add Ship / No-ship verdict table and collapsible PR comment sections (#35)
* feat(report): add Ship / No-ship verdict table and collapsible PR comment sections - Add `classifyGroupVerdict` and `renderVerdictSummaryTable` to surface per-eval `✅ Ship`, `🛑 No ship`, and `⚠️ Re-run` verdicts at the top of comparison PR comments. - Order actionable `🛑 No ship` and `⚠️ Re-run` `<details>` sections above `---`, and place passing `<details>`, unchanged eval coverage, and full `Baseline vs. Candidate` / `Quality & token summary` telemetry below `---`. - Align the aggregate `Outcome` row in `renderComparisonSection` with the headline verdict and preserve comment byte-budget notices. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(report): bound unchanged/telemetry rows to comment budget and classify zero-trial baselines - Bound unchanged coverage rows and telemetry Eval/Trials/Result rows against the 60 KB PR comment limit. - Sum stimuliTotal separately in unchanged coverage summaries and report skipped scenario counts accurately. - Let failing candidate groups with zero-trial baselines reach the No-ship classification. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(report): prevent unverified baseline failures from reporting as fixed in Outcome Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(report): show dash for missing candidate model and keep unchanged failure delta Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(report): require usable baseline for Ship verdict and distinguish score regressions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * refactor(report): remove duplicate per-group details blocks in PR comment Keep only the top-level Ship / No-ship verdict table above the fold and a single collapsible Telemetry, token deltas & run evidence section. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(report): address review findings on still-failing verdicts, incomplete runs, and byte limits Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent 1925d17 commit 0480c3e

5 files changed

Lines changed: 262 additions & 27 deletions

File tree

‎README.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -254,11 +254,13 @@ Artifacts uploaded by older Action versions require explicit IDs/digests.
254254

255255
### Baseline vs. Candidate PR comparison
256256

257-
To include a **Baseline vs. Candidate comparison** table (Outcome, Passed/Failed,
258-
Score delta, Turns delta, Duration delta, and Token `in/out` + `%` delta) in the sticky PR
259-
comment and job summary, run the baseline arm with a distinct `report-id`
260-
(such as `evals-baseline`), run the candidate arm with `report-id: evals`, and
261-
pass `baseline-report-id: evals-baseline` to the `report` step:
257+
To include a top-level **Ship / No-ship / Re-run verdict summary table** (`✅ Ship`,
258+
`🛑 No ship`, `⚠️ Re-run`) and a collapsible **Telemetry, token deltas & run
259+
evidence** `<details>` section (`Baseline vs. Candidate comparison`, `Quality &
260+
token summary`, and evidence links) in the sticky PR comment, run the baseline
261+
arm with a distinct `report-id` (such as `evals-baseline`), run the candidate
262+
arm with `report-id: evals`, and pass `baseline-report-id: evals-baseline` to
263+
the `report` step:
262264

263265
```yaml
264266
report:

‎dist/index.cjs‎

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

‎src/reporter.ts‎

Lines changed: 33 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
import { parseDocument } from 'yaml';
22
import { commentMarker, orderedBody, selectComment, type Comment, type CommentIdentity, type ReportOrder } from './comments.js';
33
import { object } from './policy.js';
4-
import { escapeMarkdown, hasQualityDetails, renderComparisonSection, renderQualitySection, type Report } from './reporting.js';
4+
import {
5+
escapeMarkdown, hasQualityDetails, renderComparisonSection, renderQualitySection, renderVerdictSummaryTable,
6+
type Report,
7+
} from './reporting.js';
58
import { ReportingError, type GitHubClient } from './github.js';
69

710
export const commentConcurrencyGroup = 'evals-${{ github.workflow_ref }}';
@@ -171,7 +174,8 @@ export function renderComment(
171174
const status = evaluation?.outcome ?? lint?.outcome ?? report.outcome;
172175
const rows = evaluation?.groups ?? [];
173176
const suite = evaluation?.suite;
174-
const base = `# Evals: ${status}\n\n` +
177+
const maxEvalRows = baseline ? 25 : 100;
178+
const detailsBody =
175179
(suite ? `**Suite:** ${escapeMarkdown(suite)}\n\n` : evaluation ? '**Selection:** Native eval paths or tags\n\n' : '') +
176180
(lint ? `**Lint:** ${lint.outcome}${lint.outcome === 'validated' ? ' (no model execution)' : ` - ${escapeMarkdown(lint.code)}`}` +
177181
((companion?.artifactId ?? artifactId) !== undefined ? ` - [Evidence](${run}/artifacts/${companion?.artifactId ?? artifactId})` : '') + '\n\n' : '') +
@@ -181,21 +185,36 @@ export function renderComment(
181185
(['error', 'failed', 'cancelled'].includes(evaluation.outcome) ? `Diagnostic: ${escapeMarkdown(evaluation.code)}.\n\n` : '') +
182186
(evaluation.outcome === 'not-run' ? '**Not an evaluation pass.**\n\n' : '') +
183187
(rows.length ? '| Eval | Trials | Result |\n| --- | ---: | --- |\n' +
184-
rows.slice(0, 100).map((group) =>
188+
rows.slice(0, maxEvalRows).map((group) =>
185189
`| ${escapeMarkdown(group.name ?? group.identity.slice(0, 12))} | ${group.trials} | ${group.trials ? group.passed ? 'passed' : 'failed' : 'not-run'} |`,
186190
).join('\n') + '\n\n' +
187-
(rows.length > 100 ? `${rows.length - 100} further groups are in the structured artifact.\n\n` : '') : '') : '');
188-
if (!evaluation) return base;
189-
let output = base;
191+
(rows.length > maxEvalRows ? `${rows.length - maxEvalRows} further groups are in the structured artifact.\n\n` : '') : '') : '');
192+
if (!evaluation) return `# Evals: ${status}\n\n${detailsBody}`;
193+
if (!baseline) {
194+
let output = `# Evals: ${status}\n\n${detailsBody}`;
195+
if (rows.length && hasQualityDetails(rows)) {
196+
const maxQualityBytes = Math.min(16_000, Math.max(0, 57_500 - Buffer.byteLength(output, 'utf8')));
197+
const quality = renderQualitySection('### Quality & token summary', rows, 25, maxQualityBytes);
198+
if (quality) output = `${output}${quality}\n\n`;
199+
}
200+
return output;
201+
}
202+
203+
const { overallVerdict, markdown: verdictTable } = renderVerdictSummaryTable(baseline.report, evaluation, 25);
204+
const header = `# Evals: ${overallVerdict} (${status})\n\n` +
205+
(verdictTable ? `${verdictTable}\n\n` : '') +
206+
(['error', 'failed', 'cancelled'].includes(evaluation.outcome) ? `Diagnostic: ${escapeMarkdown(evaluation.code)}.\n\n` : '') +
207+
(evaluation.outcome === 'not-run' ? '**Not an evaluation pass.**\n\n' : '');
208+
209+
let telemetry = detailsBody;
190210
if (rows.length && hasQualityDetails(rows)) {
191-
const maxQualityBytes = Math.min(16_000, Math.max(0, 57_500 - Buffer.byteLength(output, 'utf8')));
211+
const maxQualityBytes = Math.min(16_000, Math.max(0, 56_000 - Buffer.byteLength(header + telemetry, 'utf8')));
192212
const quality = renderQualitySection('### Quality & token summary', rows, 25, maxQualityBytes);
193-
if (quality) output = `${output}${quality}\n\n`;
213+
if (quality) telemetry = `${telemetry}${quality}\n\n`;
194214
}
195-
if (baseline) {
196-
const maxComparisonBytes = Math.min(16_000, Math.max(0, 57_500 - Buffer.byteLength(output, 'utf8')));
197-
const comparison = renderComparisonSection('### Baseline vs. Candidate comparison', baseline.report, evaluation, 25, maxComparisonBytes);
198-
if (comparison) output = `${output}${comparison}\n\n`;
199-
}
200-
return output;
215+
const maxComparisonBytes = Math.min(16_000, Math.max(0, 56_000 - Buffer.byteLength(header + telemetry, 'utf8')));
216+
const comparison = renderComparisonSection('### Baseline vs. Candidate comparison', baseline.report, evaluation, 25, maxComparisonBytes, overallVerdict);
217+
if (comparison) telemetry = `${telemetry}${comparison}\n\n`;
218+
219+
return `${header}<details>\n<summary>📊 <strong>Telemetry, token deltas &amp; run evidence</strong></summary>\n\n${telemetry.trim()}\n\n</details>\n\n`;
201220
}

‎src/reporting.ts‎

Lines changed: 169 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -208,15 +208,177 @@ function formatGroupCell(group?: ReportGroup): string {
208208
return group.overallScore !== undefined ? `${verdict} (${group.overallScore.toFixed(2)})` : verdict;
209209
}
210210

211+
export interface GroupVerdictEntry {
212+
key: string;
213+
name: string;
214+
baseGroup: ReportGroup | undefined;
215+
candGroup: ReportGroup | undefined;
216+
verdict: '✅ Ship' | '🛑 No ship' | '⚠️ Re-run';
217+
transition: string;
218+
changed: boolean;
219+
why: string;
220+
}
221+
222+
export function classifyGroupVerdict(baseline: ReportGroup | undefined, candidate: ReportGroup | undefined): {
223+
verdict: '✅ Ship' | '🛑 No ship' | '⚠️ Re-run';
224+
transition: string;
225+
changed: boolean;
226+
why: string;
227+
} {
228+
const transition = classifyGroupTransition(baseline, candidate);
229+
if (!candidate || candidate.trials === 0) {
230+
return {
231+
verdict: '⚠️ Re-run',
232+
transition,
233+
changed: true,
234+
why: !candidate ? 'Eval removed from candidate run' : 'No trials executed (not-run)',
235+
};
236+
}
237+
if (baseline && baseline.trials === 0 && candidate.passed) {
238+
return {
239+
verdict: '⚠️ Re-run',
240+
transition,
241+
changed: true,
242+
why: 'Baseline had no trials executed (not-run)',
243+
};
244+
}
245+
const ship = candidate.passed && transition !== 'regressed';
246+
const verdict = ship ? '✅ Ship' : '🛑 No ship';
247+
const changed = transition !== 'unchanged';
248+
const parts: string[] = [];
249+
if (baseline && baseline.trials === 0 && !candidate.passed) {
250+
parts.push('Failed in candidate (baseline not-run)');
251+
} else if (!candidate.passed) {
252+
if (transition === 'regressed' && baseline?.passed) parts.push('Regressed to failing');
253+
else if (transition === 'regressed') parts.push('Still failing (score regressed)');
254+
else if (transition === 'improved-score') parts.push('Still failing (score improved)');
255+
else if (transition === 'added') parts.push('New eval failing');
256+
else parts.push('Still failing');
257+
} else if (transition === 'fixed') {
258+
parts.push('Fixed failing eval');
259+
} else if (transition === 'regressed') {
260+
parts.push('Score regressed');
261+
} else if (transition === 'improved-score') {
262+
parts.push('Improved score');
263+
} else if (transition === 'reduced-tokens') {
264+
parts.push('Reduced token usage at parity');
265+
} else if (transition === 'increased-tokens') {
266+
parts.push('Increased token usage at parity');
267+
} else if (transition === 'added') {
268+
parts.push('New eval passing');
269+
} else {
270+
parts.push(`Passing (${candidate.stimuliRun}/${candidate.stimuliTotal} scenarios)`);
271+
}
272+
if (baseline?.overallScore !== undefined && candidate.overallScore !== undefined) {
273+
parts.push(`score ${baseline.overallScore.toFixed(2)} → ${candidate.overallScore.toFixed(2)} (${formatScoreDelta(baseline.overallScore, candidate.overallScore)})`);
274+
} else if (candidate.overallScore !== undefined) {
275+
parts.push(`score ${candidate.overallScore.toFixed(2)}`);
276+
}
277+
const sameTrials = baseline !== undefined && baseline.trials === candidate.trials;
278+
if (sameTrials && baseline?.tokens?.output !== undefined && candidate.tokens?.output !== undefined && baseline.tokens.output > 0) {
279+
const outDelta = candidate.tokens.output - baseline.tokens.output;
280+
if (outDelta !== 0) {
281+
const sign = outDelta > 0 ? '+' : '';
282+
const pct = ((outDelta / baseline.tokens.output) * 100).toFixed(1);
283+
parts.push(`${sign}${pct}% output tokens (${sign}${outDelta})`);
284+
}
285+
}
286+
if (sameTrials && baseline?.durationMs !== undefined && candidate.durationMs !== undefined && baseline.durationMs > 0) {
287+
const durDelta = candidate.durationMs - baseline.durationMs;
288+
if (durDelta !== 0) {
289+
const sign = durDelta > 0 ? '+' : '';
290+
const pct = ((durDelta / baseline.durationMs) * 100).toFixed(1);
291+
parts.push(`${sign}${pct}% duration`);
292+
}
293+
}
294+
return { verdict, transition, changed, why: parts.join(' · ') };
295+
}
296+
297+
export function renderVerdictSummaryTable(
298+
baseline: Report | undefined, candidate: Report, maxGroups = 25, maxBytes = 16_000,
299+
): { overallVerdict: '✅ Ship' | '🛑 No ship' | '⚠️ Re-run'; entries: GroupVerdictEntry[]; markdown: string } {
300+
const baseGroups = baseline?.groups ?? [];
301+
const candGroups = candidate.groups ?? [];
302+
const baseMap = new Map(baseGroups.map((group) => [group.identity, group]));
303+
const candMap = new Map(candGroups.map((group) => [group.identity, group]));
304+
const keys = [...new Set([...baseMap.keys(), ...candMap.keys()])];
305+
const baseIncomplete = baseline !== undefined && ['error', 'not-run', 'cancelled'].includes(baseline.outcome);
306+
const candIncomplete = ['error', 'not-run', 'cancelled'].includes(candidate.outcome);
307+
const severityOrder: Record<'🛑 No ship' | '⚠️ Re-run' | '✅ Ship', number> = {
308+
'🛑 No ship': 0,
309+
'⚠️ Re-run': 1,
310+
'✅ Ship': 2,
311+
};
312+
const entries: GroupVerdictEntry[] = keys
313+
.map((key) => {
314+
const baseGroup = baseMap.get(key);
315+
const candGroup = candMap.get(key);
316+
const name = escapeMarkdown((candGroup ?? baseGroup)!.name ?? (candGroup ?? baseGroup)!.identity.slice(0, 12));
317+
let { verdict, transition, changed, why } = classifyGroupVerdict(baseGroup, candGroup);
318+
if (verdict !== '🛑 No ship' && (baseIncomplete || candIncomplete)) {
319+
verdict = '⚠️ Re-run';
320+
changed = true;
321+
if (baseIncomplete && !baseGroup) why = 'Baseline did not complete';
322+
else if (candIncomplete) why = 'Candidate run did not complete';
323+
}
324+
return { key, name, baseGroup, candGroup, verdict, transition, changed, why };
325+
})
326+
.sort((a, b) => severityOrder[a.verdict] - severityOrder[b.verdict]);
327+
328+
if (entries.length === 0) {
329+
const overallVerdict = candidate.outcome === 'failed'
330+
? '🛑 No ship'
331+
: candidate.outcome === 'passed' && !baseIncomplete
332+
? '✅ Ship'
333+
: '⚠️ Re-run';
334+
return { overallVerdict, entries, markdown: '' };
335+
}
336+
let hasNoShip = candidate.outcome === 'failed';
337+
let hasRerun = candIncomplete || baseIncomplete;
338+
for (const entry of entries) {
339+
if (entry.verdict === '🛑 No ship') hasNoShip = true;
340+
if (entry.verdict === '⚠️ Re-run') hasRerun = true;
341+
}
342+
const overallVerdict = hasNoShip ? '🛑 No ship' : hasRerun ? '⚠️ Re-run' : '✅ Ship';
343+
if (maxBytes <= 0) return { overallVerdict, entries, markdown: '' };
344+
const header = '| Skill | Verdict | Why |\n| --- | --- | --- |\n';
345+
const footer = '\n\nFurther groups are available in the structured artifact.';
346+
const footerBytes = Buffer.byteLength(footer, 'utf8');
347+
let usedBytes = Buffer.byteLength(header, 'utf8');
348+
const rows: string[] = [];
349+
const limit = Math.min(entries.length, maxGroups);
350+
for (let index = 0; index < limit; index++) {
351+
const entry = entries[index]!;
352+
const boldVerdict = entry.verdict.replace(/^(✅|🛑|⚠️)\s+(.*)$/u, '$1 **$2**');
353+
const row = `| **${entry.name}** | ${boldVerdict} | ${entry.why} |`;
354+
const rowBytes = Buffer.byteLength((rows.length ? '\n' : '') + row, 'utf8');
355+
const needsFooter = index + 1 < entries.length;
356+
if (usedBytes + rowBytes + (needsFooter ? footerBytes : 0) > maxBytes) break;
357+
rows.push(row);
358+
usedBytes += rowBytes;
359+
}
360+
const markdown = rows.length === 0
361+
? ''
362+
: header + rows.join('\n') + (rows.length < entries.length ? footer : '');
363+
return { overallVerdict, entries, markdown };
364+
}
365+
211366
export function renderComparisonSection(
212-
heading: string, baseline: Report, candidate: Report, maxGroups = 25, maxBytes = 16_000,
367+
heading: string, baseline: Report, candidate: Report, maxGroups = 25, maxBytes = 16_000, overallVerdict?: string,
213368
): string {
214369
if (maxBytes <= 0) return '';
215-
const outcomeTransition = baseline.outcome === 'failed' && candidate.outcome === 'passed'
216-
? 'fixed'
217-
: baseline.outcome === 'passed' && candidate.outcome === 'failed'
218-
? 'regressed'
219-
: baseline.outcome === candidate.outcome ? 'unchanged' : `${baseline.outcome} -> ${candidate.outcome}`;
370+
const hasGroupRegression = (baseline.groups ?? []).some((baseGroup) => {
371+
const candGroup = (candidate.groups ?? []).find((g) => g.identity === baseGroup.identity);
372+
return classifyGroupTransition(baseGroup, candGroup) === 'regressed';
373+
});
374+
const outcomeTransition = (baseline.outcome === 'passed' && candidate.outcome === 'failed') || hasGroupRegression
375+
? 'regressed'
376+
: overallVerdict === '⚠️ Re-run'
377+
? '⚠️ Re-run'
378+
: baseline.outcome === 'failed' && candidate.outcome === 'passed' && overallVerdict !== '🛑 No ship'
379+
? 'fixed'
380+
: baseline.outcome === candidate.outcome ? 'unchanged' : `${baseline.outcome} -> ${candidate.outcome}`;
381+
const candidateOutcomeCell = overallVerdict ? `${candidate.outcome} (${overallVerdict})` : candidate.outcome;
220382
const hasBothGroups = baseline.groups !== undefined && candidate.groups !== undefined;
221383
const baseGroups = baseline.groups ?? [];
222384
const candGroups = candidate.groups ?? [];
@@ -234,7 +396,7 @@ export function renderComparisonSection(
234396
const summaryTable =
235397
`${heading}\n\n` +
236398
'| Metric | Baseline | Candidate | Delta |\n| --- | --- | --- | --- |\n' +
237-
`| Outcome | ${baseline.outcome} | ${candidate.outcome} | ${outcomeTransition} |\n` +
399+
`| Outcome | ${baseline.outcome} | ${candidateOutcomeCell} | ${outcomeTransition} |\n` +
238400
`| Passed | ${baseline.passed ?? '-'} | ${candidate.passed ?? '-'} | ${formatSignedCount(baseline.passed, candidate.passed)} |\n` +
239401
`| Failed | ${baseline.failed ?? '-'} | ${candidate.failed ?? '-'} | ${formatSignedCount(baseline.failed, candidate.failed)} |\n` +
240402
`| Turns | ${baseline.turns ?? '-'} | ${candidate.turns ?? '-'} | ${workloadsMatch ? formatSignedCount(baseline.turns, candidate.turns) : '-'} |\n` +

‎tests/reporter.test.ts‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -256,10 +256,62 @@ describe('one sticky PR report for direct Action calls', () => {
256256
report: baselineReport, artifactId: 125,
257257
})).toBe('created');
258258
const body = String(test.writes()[0]?.[2]?.body);
259+
expect(body).toContain('# Evals: ✅ Ship (passed)');
260+
expect(body).toContain('| Skill | Verdict | Why |');
261+
expect(body).toContain('| **pr-description** | ✅ **Ship** | Fixed failing eval · score 0.94 → 1.00 (+0.06) · -31.5% output tokens (-456) · -11.4% duration |');
262+
expect(body).toContain('<summary>📊 <strong>Telemetry, token deltas &amp; run evidence</strong></summary>');
259263
expect(body).toContain('**Baseline:** failed - 2 trials - [Evidence](https://github.com/o/r/actions/runs/1000/artifacts/125)');
260264
expect(body).toContain('### Baseline vs. Candidate comparison');
265+
expect(body).toContain('| Outcome | failed | passed (✅ Ship) | fixed |');
261266
expect(body).toContain('| pr-description | failed (0.94) | passed (1.00) | +0.06 | -2 | -4401ms (-11.4%) | -191 tokens (-0.3%; out: -456 / -31.5%) | fixed |');
262267

268+
// Mixed verdicts (No ship, Re-run, still-failing token deltas) and 100-group comment byte budget
269+
const mixedBase: Report = {
270+
...report, reportId: 'smoke-baseline', suite: 'smoke', outcome: 'failed',
271+
executed: 2, passed: 1, failed: 1, turns: 4, durationMs: 20_000,
272+
tokens: { input: 40_000, output: 1_000, total: 41_000 },
273+
groups: [
274+
{ identity: '1'.repeat(64), name: 'still-failing', passed: false, trials: 1, stimuliRun: 1, stimuliTotal: 1, skipped: 0, overallScore: 0.5, threshold: 1, turns: 2, durationMs: 10_000, tokens: { input: 20_000, output: 600, total: 20_600 } },
275+
{ identity: '2'.repeat(64), name: 'skipped-cand', passed: true, trials: 1, stimuliRun: 1, stimuliTotal: 1, skipped: 0, overallScore: 1, threshold: 1, turns: 2, durationMs: 10_000, tokens: { input: 20_000, output: 400, total: 20_400 } },
276+
],
277+
};
278+
const mixedCand: Report = {
279+
...report, suite: 'smoke', outcome: 'failed',
280+
executed: 1, passed: 0, failed: 1, turns: 2, durationMs: 9_000,
281+
tokens: { input: 20_000, output: 500, total: 20_500 },
282+
groups: [
283+
{ identity: '1'.repeat(64), name: 'still-failing', passed: false, trials: 1, stimuliRun: 1, stimuliTotal: 1, skipped: 0, overallScore: 0.5, threshold: 1, turns: 2, durationMs: 9_000, tokens: { input: 20_000, output: 500, total: 20_500 } },
284+
{ identity: '2'.repeat(64), name: 'skipped-cand', passed: true, trials: 0, stimuliRun: 0, stimuliTotal: 1, skipped: 1 },
285+
],
286+
};
287+
const mixedTest = fixture();
288+
expect(await postReport(mixedTest.client, context, mixedCand, 123, undefined, { report: mixedBase, artifactId: 125 })).toBe('created');
289+
const mixedBody = String(mixedTest.writes()[0]?.[2]?.body);
290+
expect(mixedBody).toContain('# Evals: 🛑 No ship (failed)');
291+
expect(mixedBody).toContain('| **still-failing** | 🛑 **No ship** | Still failing · score 0.50 → 0.50 (0.00) · -16.7% output tokens (-100) · -10.0% duration |');
292+
expect(mixedBody).toContain('| **skipped-cand** | ⚠️ **Re-run** | No trials executed (not-run) |');
293+
294+
const heavyGroups = Array.from({ length: 100 }, (_, index) => ({
295+
identity: index.toString(16).padStart(64, '0'),
296+
name: `${'&'.repeat(90)}-${index}`,
297+
model: 'claude-haiku-5.5',
298+
passed: true,
299+
trials: 1,
300+
stimuliRun: 1,
301+
stimuliTotal: 1,
302+
skipped: 0,
303+
overallScore: 1,
304+
threshold: 1,
305+
turns: 2,
306+
durationMs: 50,
307+
tokens: { input: 20, output: 30, total: 50 },
308+
}));
309+
const heavyBase: Report = { ...report, reportId: 'smoke-baseline', suite: 'smoke', outcome: 'passed', executed: 100, passed: 100, failed: 0, turns: 200, durationMs: 5000, tokens: { input: 2000, output: 3000, total: 5000 }, groups: heavyGroups };
310+
const heavyCand: Report = { ...report, suite: 'smoke', outcome: 'passed', executed: 100, passed: 100, failed: 0, turns: 200, durationMs: 5000, tokens: { input: 2000, output: 3000, total: 5000 }, groups: heavyGroups };
311+
const heavyTest = fixture();
312+
expect(await postReport(heavyTest.client, context, heavyCand, 123, undefined, { report: heavyBase, artifactId: 125 })).toBe('created');
313+
expect(Buffer.byteLength(String(heavyTest.writes()[0]?.[2]?.body), 'utf8')).toBeLessThanOrEqual(60_000);
314+
263315
for (const badBaseline of [
264316
{ report: { ...baselineReport, reportId: 'smoke' }, artifactId: 125 },
265317
{ report: { ...baselineReport, command: 'lint' }, artifactId: 125 },

0 commit comments

Comments
 (0)