Skip to content

feat(harness): run a single epoch of a run #132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
abhinav-pola wants to merge 1 commit into from
Closed

feat(harness): run a single epoch of a run #132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions src/harness/run.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,19 @@ describe("runBenchmark", () => {
expect(result.usage.generationTimeMs).toBe(600);
expect(result.sampleScores[0]?.generationIds).toEqual(["fake-Q1 target B"]);
});
it("runs only the requested epoch", async () => {
const model = fakeModel(() => "Answer: B");
const solver = generate(model.service, {
temperature: 0.5,
reasoningEffort: "high",
});
const result = await runPromise(
runBenchmark({ epochs: 3, epoch: 2, maxConcurrency: 4 }).pipe(
provide(makeLayers(model, solver))
)
);
expect(result.sampleScores.map((score) => score.epoch)).toEqual([2, 2]);
});
it("captures per-sample message trajectories", async () => {
const model = fakeModel(() => "Answer: B");
const solver = chain(
Expand Down
9 changes: 6 additions & 3 deletions src/harness/run.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ import { Solver } from "./solver";

export interface RunConfig {
readonly epochs: number;
readonly epoch?: number;
readonly maxConcurrency: number;
readonly range?: {
readonly start?: number;
Expand Down Expand Up @@ -83,7 +84,7 @@ type EvalOutcome = {

function sampleEpochStream(
dataset: DatasetService,
epochs: number,
epochs: readonly number[],
range:
| {
readonly start?: number;
Expand All @@ -96,7 +97,7 @@ function sampleEpochStream(
streamZipWithIndex,
streamFlatMap(([sample, i]) =>
streamFromIterable(
Array.from({ length: epochs }, (_, epoch) => ({
epochs.map((epoch) => ({
sample,
epoch,
sampleIndex: baseIndex + i,
Expand Down Expand Up @@ -337,7 +338,9 @@ export function runBenchmark(
effectFlatMap((dataset) => {
const sampleEpochs = sampleEpochStream(
dataset,
config.epochs,
config.epoch === undefined
? Array.from({ length: config.epochs }, (_, epoch) => epoch)
: [config.epoch],
Comment on lines +341 to +343

@devin-ai-integration devin-ai-integration Bot Oct 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

馃煛 Out-of-range epoch produces invalid scores

When epoch is negative or at least epochs, runBenchmark still evaluates that nonexistent epoch. Its scores and request IDs carry the invalid index.

Learn more

epochs defines the number of valid zero-based epoch indices. The selected epoch bypasses that count, so an invalid index still flows through scoring, logging, request session IDs, and cache salts. The public input accepts numbers without checking their range.

Example: runBenchmark({ epochs: 3, epoch: 3, maxConcurrency: 1 }) evaluates every sample as epoch 3 instead of rejecting the request; the valid indices are 0, 1, and 2.

Recommended fix: Validate the selected index as a safe integer in [0, config.epochs) before streaming. Surface an explicit input error to runBenchmarkById callers and add tests for negative, fractional, and out-of-range indices.

Devin Review

Was this helpful? React with 馃憤 or 馃憥 to provide feedback.

config.range
);
const initialAcc: FoldAccumulator = {
Expand Down
2 changes: 2 additions & 0 deletions src/runner/run-by-id.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ export interface RunBenchmarkInput {
readonly baseUrl?: string;
readonly benchmarkConfig: BenchmarkRunConfig;
readonly epochs: number;
readonly epoch?: number;
readonly maxConcurrency: number;
readonly range?: {
readonly start?: number;
Expand Down Expand Up @@ -96,6 +97,7 @@ export function runBenchmarkById(
const model = modelFromConfig(input.benchmarkConfig);
const runConfig: RunConfig = definedValues({
epochs: input.epochs,
epoch: input.epoch,

@devin-ai-integration devin-ai-integration Bot Oct 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

馃敶 Single-epoch result files overwrite each other

When separate epoch runs reuse a session ID, makeLocalResultStore writes them to the same filename. The later write discards the earlier epoch's scores.

Learn more

The local result store names files by benchmark, model, and session ID, not epoch. The new single-epoch mode permits distinct calls with the same session ID and different epoch indices. Both calls write valid results but target one file, so the last write replaces the first. This affects callers that use resultStore for multiple epoch runs under the same benchmark, model, and session ID; callers without a result store are unaffected.

Example: Run epoch 0 and then epoch 1 with epochs: 2, sessionId: "trial", and the same local store. Both runs return scores, but the resulting parquet file contains only epoch 1.

Recommended fix: Pass the selected epoch through the result-store write options and include it in the filename when present. Keep the existing filename for runs without a selected epoch, and test two writes sharing a session ID.

Devin Review

Was this helpful? React with 馃憤 or 馃憥 to provide feedback.

maxConcurrency: input.maxConcurrency,
range: input.range,
degradeSolverErrors: benchmark.degradeSolverErrors,
Expand Down
Loading