Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 45 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,43 @@ only the lint ID/digest pair to override lint while discovering run evidence.
Overrides still undergo digest and repository/revision/run/attempt checks.
Artifacts uploaded by older Action versions require explicit IDs/digests.

### Baseline vs. Candidate PR comparison

To include a **Baseline vs. Candidate comparison** table (Outcome, Passed/Failed,
Score delta, Turns delta, Duration delta, and Token `in/out` + `%` delta) in the sticky PR
comment and job summary, run the baseline arm with a distinct `report-id`
(such as `evals-baseline`), run the candidate arm with `report-id: evals`, and
pass `baseline-report-id: evals-baseline` to the `report` step:
Comment thread
jongio marked this conversation as resolved.
Resolved

```yaml
report:
needs: [lint, run-baseline, run]
if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
actions: read
contents: read
pull-requests: write
concurrency:
group: evals-${{ github.workflow_ref }}
cancel-in-progress: false
steps:
- uses: github/evals-action@v0
with:
command: report
report-id: evals
baseline-report-id: evals-baseline
```

`run-baseline` runs in the same workflow run, attempt, and PR checkout revision
(restoring only the baseline skill or target files before invoking `command: run`
with `report-id: evals-baseline`), and aggregate turns, duration, and token deltas
are shown when both arms share the same selection and per-eval trial counts.
You can also pass `baseline-artifact-id` and `baseline-artifact-digest`
explicitly instead of discovering the baseline artifact by `baseline-report-id`
(matching `baseline-report-id`, or `<report-id>-baseline` when omitted).

## Commands

| `command` | Behavior |
Expand Down Expand Up @@ -289,6 +326,8 @@ explicit lint rejects execution overrides.
| `max-retries` | Nonnegative run override. All mode defaults to 0; explicit run preserves native default. |
| `source-artifact-id`, `source-artifact-digest` | Optional paired overrides for report: exact primary artifact ID and its 64-character lowercase SHA-256 digest. Omit for automatic discovery. Providing both bypasses all discovery. |
| `lint-artifact-id`, `lint-artifact-digest` | Optional paired lint override for report. Uses the shared report ID or legacy primary ID prefix plus `-lint`. Without a primary override, run evidence is still discovered. |
| `baseline-report-id` | Optional baseline report identity for report mode (e.g. `evals-baseline`). Discovers the matching current-run/attempt run artifact unless `baseline-artifact-id` is set. |
| `baseline-artifact-id`, `baseline-artifact-digest` | Optional paired baseline run override for report mode. Must be distinct from primary and lint artifacts. |
| `upload-transcripts` | Default false; opt in to raw native JSONL and stdout/stderr, potentially sensitive. |
| `artifact-retention-days` | 1-90, default 7; subject to repository retention policy. |

Expand Down Expand Up @@ -318,11 +357,13 @@ A passing check doesn't guarantee the AI will get it right every time.
| `lint-artifact-id`, `lint-artifact-digest` | Companion lint evidence produced by all after successful lint. |
| `native-diagnostics-path` | Local raw logs/output directory under runner temporary storage. |

Structured artifacts contain status, revision, suite, bounded native eval names,
engine/dependency identities, counts, verdicts and hashed group identities.
Structured artifacts contain status, revision, suite, bounded native eval and
model names, engine/dependency identities, counts, verdicts, optional
score/turn/duration/token summaries, and hashed group identities.
They exclude full source paths, prompts, model output and raw exception text.
Names are Markdown-escaped in comments; do not store secrets in suite or eval
names. Comments show at most 100 groups and link to the complete artifact.
Names are Markdown-escaped in comments; do not store secrets in suite, eval, or
model names. Comments show at most 100 groups (and up to 25 groups in quality
and comparison tables) and link to the complete artifact.
Raw JSONL above 64 MiB is refused rather than partially interpreted.

Optional transcripts may contain prompts, secrets and proprietary code.
Expand Down
6 changes: 6 additions & 0 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,12 @@ inputs:
description: Optional lint artifact override, paired with lint-artifact-digest. Otherwise discovered alongside the run. Supports the shared report ID or legacy prefix plus -lint; report command only.
lint-artifact-digest:
description: SHA-256 digest for the optional companion lint artifact; report command only.
baseline-report-id:
description: Optional baseline report identity for report mode. Discovers the matching current-run/attempt run artifact unless baseline-artifact-id is set.
baseline-artifact-id:
description: Optional baseline run artifact ID override for report mode, paired with baseline-artifact-digest.
baseline-artifact-digest:
description: SHA-256 digest for the optional baseline run artifact; report command only.
upload-transcripts:
description: Opt in to native JSONL and stdout/stderr logs, which may contain sensitive content.
default: 'false'
Expand Down
2 changes: 1 addition & 1 deletion dist/index.cjs

Large diffs are not rendered by default.

39 changes: 34 additions & 5 deletions src/action.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ export async function runAction(): Promise<void> {
let inputs: Inputs | undefined;
let pullNumber: number | undefined;
let companion: Evidence | undefined;
let baseline: Evidence | undefined;
const nativeFiles: string[] = [];
let nativeDirectory: string | undefined;
try {
Expand Down Expand Up @@ -102,9 +103,30 @@ export async function runAction(): Promise<void> {
artifactId: discoveredLint.id,
};
}
await commentReport(imported, reportToken || githubToken, pullNumber, primary.id, companion);
if (imported.command === 'run') {
if (inputs.baselineArtifactId !== undefined && inputs.baselineArtifactDigest) {
const expectedBaselineId = inputs.baselineReportId
? [inputs.baselineReportId, reportPrefix(inputs.baselineReportId), `${reportPrefix(inputs.baselineReportId)}-run`]
: [`${reportPrefix(inputs.reportId)}-baseline`, `${reportPrefix(inputs.reportId)}-baseline-run`];
baseline = {
report: await readReportArtifact(inputs.baselineArtifactId, inputs.baselineArtifactDigest, expectedBaselineId, 'run'),
artifactId: inputs.baselineArtifactId,
};
} else if (inputs.baselineReportId) {
const discoveredBaseline = await discoverReportArtifacts(githubClient(githubToken || reportToken, requiredEnv('RUNNER_TEMP')), {
repository: requiredEnv('GITHUB_REPOSITORY'), runId: positiveEnv('GITHUB_RUN_ID'),
attempt: positiveEnv('GITHUB_RUN_ATTEMPT'), reportId: inputs.baselineReportId,
}, false);
if (!discoveredBaseline.run) throw new NativeError('report-artifacts-not-found', 'evidence');
baseline = {
report: await readReportArtifact(discoveredBaseline.run.id, discoveredBaseline.run.digest, discoveredBaseline.run.reportId, 'run'),
artifactId: discoveredBaseline.run.id,
};
}
}
await commentReport(imported, reportToken || githubToken, pullNumber, primary.id, companion, baseline);
publishOutputs(imported);
await core.summary.addRaw(renderSummary(imported)).write();
await core.summary.addRaw(renderSummary(imported, baseline?.report)).write();
return;
}

Expand Down Expand Up @@ -169,6 +191,7 @@ export async function runAction(): Promise<void> {
report.code = 'vally-execution-failed';
report.category = 'execution';
delete report.executed;
delete report.turns;
delete report.durationMs;
delete report.tokens;
if (executionInputs.command === 'run') {
Expand Down Expand Up @@ -226,15 +249,21 @@ export async function runAction(): Promise<void> {

function reportPrefix(id: string): string { return id.replace(/-(?:lint|run)$/, ''); }

async function commentReport(report: Report, token: string, pullNumber: number, artifactId?: number, companion?: Evidence): Promise<void> {
async function commentReport(
report: Report, token: string, pullNumber: number, artifactId?: number, companion?: Evidence, baseline?: Evidence,
): Promise<void> {
try {
const outcome = await postReport(githubClient(token, requiredEnv('RUNNER_TEMP')), {
const client = githubClient(token, requiredEnv('RUNNER_TEMP'));
const context = {
repository: report.repository, workflowRef: requiredEnv('GITHUB_WORKFLOW_REF'),
workflowSha: requiredEnv('GITHUB_WORKFLOW_SHA'), job: requiredEnv('GITHUB_JOB'),
reportId: reportPrefix(report.reportId), selection: report.selection,
pullNumber, runId: report.workflowRunId,
order: { runNumber: positiveEnv('GITHUB_RUN_NUMBER'), attempt: report.workflowRunAttempt },
}, report, artifactId, companion);
};
const outcome = baseline
? await postReport(client, context, report, artifactId, companion, baseline)
: await postReport(client, context, report, artifactId, companion);
if (outcome === 'stale') core.warning('A newer run or attempt already owns this report comment.');
} catch (error) {
throw new NativeError(error instanceof ReportingError ? error.code : 'comment-reporting-failed', 'execution');
Expand Down
27 changes: 26 additions & 1 deletion src/inputs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ export interface Inputs {
sourceArtifactDigest?: string;
lintArtifactId?: number;
lintArtifactDigest?: string;
baselineReportId?: string;
baselineArtifactId?: number;
baselineArtifactDigest?: string;
uploadTranscripts: boolean;
retentionDays: number;
credentialEnv: string[];
Expand Down Expand Up @@ -68,25 +71,45 @@ export function parseInputs(read: InputReader): Inputs {
const sourceArtifactDigest = text('source-artifact-digest');
const lintArtifactId = integer('lint-artifact-id', 1);
const lintArtifactDigest = text('lint-artifact-digest');
const baselineReportId = text('baseline-report-id');
const baselineArtifactId = integer('baseline-artifact-id', 1);
const baselineArtifactDigest = text('baseline-artifact-digest');
if (baselineReportId && !/^[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}$/.test(baselineReportId)) {
throw new Error('baseline-report-id must be 1-64 letters, digits, underscores or hyphens.');
}
if (baselineReportId && baselineReportId.replace(/-(?:lint|run)$/, '') === reportId.replace(/-(?:lint|run)$/, '')) {
throw new Error('baseline-report-id must be distinct from report-id.');
}
if (sourceArtifactDigest && !/^[a-f0-9]{64}$/.test(sourceArtifactDigest)) {
throw new Error('Input source-artifact-digest must be a SHA-256 artifact digest.');
}
if (lintArtifactDigest && !/^[a-f0-9]{64}$/.test(lintArtifactDigest)) {
throw new Error('Input lint-artifact-digest must be a SHA-256 artifact digest.');
}
if (baselineArtifactDigest && !/^[a-f0-9]{64}$/.test(baselineArtifactDigest)) {
throw new Error('Input baseline-artifact-digest must be a SHA-256 artifact digest.');
}
if ((lintArtifactId !== undefined) !== Boolean(lintArtifactDigest)) {
throw new Error('Inputs lint-artifact-id and lint-artifact-digest must be provided together.');
}
if ((sourceArtifactId !== undefined) !== Boolean(sourceArtifactDigest)) {
throw new Error('Inputs source-artifact-id and source-artifact-digest must be provided together.');
}
if ((baselineArtifactId !== undefined) !== Boolean(baselineArtifactDigest)) {
throw new Error('Inputs baseline-artifact-id and baseline-artifact-digest must be provided together.');
}
if (command !== 'report' && (sourceArtifactId !== undefined || sourceArtifactDigest ||
lintArtifactId !== undefined || lintArtifactDigest)) {
lintArtifactId !== undefined || lintArtifactDigest ||
baselineReportId || baselineArtifactId !== undefined || baselineArtifactDigest)) {
throw new Error('Artifact identity inputs are only valid for report.');
}
if (lintArtifactId !== undefined && lintArtifactId === sourceArtifactId) {
throw new Error('Lint and primary report artifacts must be distinct.');
}
if (baselineArtifactId !== undefined &&
(baselineArtifactId === sourceArtifactId || baselineArtifactId === lintArtifactId)) {
throw new Error('Baseline report artifact must be distinct from primary and lint artifacts.');
}
const retentionDays = integer('artifact-retention-days', 1) ?? 7;
if (retentionDays > 90) throw new Error('artifact-retention-days must not exceed 90.');
const overrides: Inputs['overrides'] = {};
Expand All @@ -103,6 +126,8 @@ export function parseInputs(read: InputReader): Inputs {
reportId, ...(sourceArtifactId !== undefined ? { sourceArtifactId } : {}),
...(sourceArtifactDigest ? { sourceArtifactDigest } : {}),
...(lintArtifactId !== undefined ? { lintArtifactId, lintArtifactDigest } : {}),
...(baselineReportId ? { baselineReportId } : {}),
...(baselineArtifactId !== undefined ? { baselineArtifactId, baselineArtifactDigest } : {}),
uploadTranscripts: boolean('upload-transcripts'), retentionDays,
credentialEnv: lines('credential-env'),
};
Expand Down
19 changes: 19 additions & 0 deletions src/native.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ export interface NativeResult {
executed?: number;
passed?: number;
failed?: number;
turns?: number;
durationMs?: number;
tokens?: ReportTokens;
nativeRunId?: string;
Expand Down Expand Up @@ -143,6 +144,8 @@ interface BucketAccumulator {
stimuli: Set<string>;
skipped: number;
name?: string;
turns: number;
turnTrials: number;
durationMs: number;
durationTrials: number;
totalTokens: number;
Expand All @@ -155,6 +158,7 @@ interface BucketAccumulator {
function emptyBucket(): BucketAccumulator {
return {
trials: 0, stimuli: new Set<string>(), skipped: 0,
turns: 0, turnTrials: 0,
durationMs: 0, durationTrials: 0,
totalTokens: 0, tokenTrials: 0,
inputTokens: 0, outputTokens: 0, ioTokenTrials: 0,
Expand Down Expand Up @@ -217,6 +221,11 @@ export function parseNativeEvidence(text: string, runId: string, exitCode: numbe
if (trajectory.metrics !== undefined && trajectory.metrics !== null &&
typeof trajectory.metrics === 'object' && !Array.isArray(trajectory.metrics)) {
const metrics = trajectory.metrics as Record<string, unknown>;
const turnCount = safeNonNegativeInteger(metrics.turnCount);
if (turnCount !== undefined) {
bucket.turns += turnCount;
bucket.turnTrials++;
}
if (metrics.tokenUsage !== undefined && metrics.tokenUsage !== null &&
typeof metrics.tokenUsage === 'object' && !Array.isArray(metrics.tokenUsage)) {
const usage = metrics.tokenUsage as Record<string, unknown>;
Expand Down Expand Up @@ -279,6 +288,10 @@ export function parseNativeEvidence(text: string, runId: string, exitCode: numbe
if (group.error !== undefined) executionErrors = true;
const name = bucket.name ?? basename(group.evalFilePath);
if (!name || name.length > 256 || /[\r\n]/.test(name)) throw new NativeError('invalid-native-eval-name', 'evidence');
const model = typeof group.model === 'string' && group.model.length > 0 && group.model.length <= 128 && !/[\r\n]/.test(group.model)
? group.model
: undefined;
const groupTurns = bucket.trials > 0 && bucket.turnTrials === bucket.trials ? bucket.turns : undefined;
const groupDurationMs = bucket.trials > 0 && bucket.durationTrials === bucket.trials ? bucket.durationMs : undefined;
const groupTokens: ReportTokens | undefined = bucket.trials > 0 && bucket.tokenTrials === bucket.trials
? {
Expand All @@ -291,12 +304,14 @@ export function parseNativeEvidence(text: string, runId: string, exitCode: numbe
groups.push({
identity: key,
name,
...(model ? { model } : {}),
passed: group.passed,
trials: bucket.trials,
stimuliRun,
stimuliTotal,
skipped,
...(bucket.trials > 0 && group.scoringApplied ? { overallScore: group.overallScore as number, threshold: group.threshold as number } : {}),
...(groupTurns !== undefined ? { turns: groupTurns } : {}),
...(groupDurationMs !== undefined ? { durationMs: groupDurationMs } : {}),
...(groupTokens !== undefined ? { tokens: groupTokens } : {}),
});
Expand All @@ -306,6 +321,9 @@ export function parseNativeEvidence(text: string, runId: string, exitCode: numbe
}
const executed = groups.reduce((total, group) => total + group.trials, 0);
const executedGroups = groups.filter((group) => group.trials > 0);
const turns = executedGroups.length > 0 && executedGroups.every((group) => group.turns !== undefined)
? executedGroups.reduce((total, group) => total + group.turns!, 0)
: undefined;
const durationMs = executedGroups.length > 0 && executedGroups.every((group) => group.durationMs !== undefined)
? executedGroups.reduce((total, group) => total + group.durationMs!, 0)
: undefined;
Expand All @@ -325,6 +343,7 @@ export function parseNativeEvidence(text: string, runId: string, exitCode: numbe
executed,
nativeRunId: runId,
groups,
...(turns !== undefined ? { turns } : {}),
...(durationMs !== undefined ? { durationMs } : {}),
...(tokens !== undefined ? { tokens } : {}),
};
Expand Down
Loading
Loading