Skip to content

feat(eval): add trial count support#162

Merged
Abhijeet Prasad (AbhiPrasad) merged 1 commit into
mainfrom
abhi-feat-eval-trial-count
Jun 15, 2026
Merged

feat(eval): add trial count support#162
Abhijeet Prasad (AbhiPrasad) merged 1 commit into
mainfrom
abhi-feat-eval-trial-count

Conversation

@AbhiPrasad

Copy link
Copy Markdown
Member

Run eval cases multiple times with a global TrialCount option and allow each case to override the default. Expose the zero-based TrialIndex through task hooks so tasks can distinguish repeated executions.

Public API additions:

// eval.Opts
TrialCount int

// eval.Case
TrialCount int

// eval.TaskHooks
TrialIndex int

Example usage:

result, err := evaluator.Run(ctx, eval.Opts[string, string]{
    Experiment: "trial-count-demo",
    Dataset: eval.NewDataset([]eval.Case[string, string]{
        {Input: "hello", Expected: "HELLO"},
        {Input: "world", Expected: "WORLD", TrialCount: 3}, // per-case override
    }),
    Task: eval.T(task),
    Scorers: []eval.Scorer[string, string]{scorer},
    TrialCount: 2, // default trials per case
})
task := func(ctx context.Context, input string, hooks *eval.TaskHooks) (eval.TaskOutput[string], error) {
    log.Printf("running trial %d", hooks.TrialIndex)
    return eval.TaskOutput[string]{Value: strings.ToUpper(input)}, nil
}

Run eval cases multiple times with a global TrialCount option and allow each
case to override the default. Expose the zero-based TrialIndex through task
hooks so tasks can distinguish repeated executions.

Public API additions:

```go
// eval.Opts
TrialCount int

// eval.Case
TrialCount int

// eval.TaskHooks
TrialIndex int
```

Example usage:

```go
result, err := evaluator.Run(ctx, eval.Opts[string, string]{
    Experiment: "trial-count-demo",
    Dataset: eval.NewDataset([]eval.Case[string, string]{
        {Input: "hello", Expected: "HELLO"},
        {Input: "world", Expected: "WORLD", TrialCount: 3}, // per-case override
    }),
    Task: eval.T(task),
    Scorers: []eval.Scorer[string, string]{scorer},
    TrialCount: 2, // default trials per case
})
```

```go
task := func(ctx context.Context, input string, hooks *eval.TaskHooks) (eval.TaskOutput[string], error) {
    log.Printf("running trial %d", hooks.TrialIndex)
    return eval.TaskOutput[string]{Value: strings.ToUpper(input)}, nil
}
```
@AbhiPrasad Abhijeet Prasad (AbhiPrasad) merged commit f567198 into main Jun 15, 2026
6 checks passed
@AbhiPrasad Abhijeet Prasad (AbhiPrasad) deleted the abhi-feat-eval-trial-count branch June 15, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants