Skip to content

[DO NOT MERGE] exp #651 E2: digested trajectories - #661

Draft
eranra wants to merge 1 commit into
feat/optimizer-profilefrom
exp/651-E2
Draft

eranra wants to merge 1 commit into
feat/optimizer-profilefrom
exp/651-E2

Conversation

@eranra

@eranra eranra commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

Experiment E2 of #651 (closes the run for #654 when filled in). Do not merge. This branch only adds run configuration on top of #657:

  • runs-on: [self-hosted, ibm-vpc, skillberry-1], because RITS is reachable only there;
  • in full_verified/overrides.env: SB_EMPTY_SEED=1 and SB_REUSE_LATEST_BASELINE=1, the same overrides as baseline 37204406089;
  • CAPEVOLVE_OPTIMIZER_PROFILE=digest_trajectories.

Dispatch:

gh workflow run benchmarks.yml --ref exp/651-E2 -f benchmark=spreadsheetbench -f tier=full_verified -f iterations=5 \
  -f agent_model=ibm-rits/google/gemma-4-31B-it -f optimizer_model=ibm-rits/google/gemma-4-31B-it \
  -f gate_k_se=0.2 -f algorithm=hill-climb-all

The measures and the decision rule are pre-registered in #651 and #654. The result will be posted on #654.

🤖 Generated with Claude Code

…profile=digest_trajectories)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Eran Raichstein <eranra@il.ibm.com>
@skillberry-bot skillberry-bot added paper-experiment Publication-grade held-out experiments for the NAACL/ACL paper algorithm Optimization algorithms: GEPA / SkillOpt / hill-climb benchmark-full Run all FULL benchmarks (whole/representative; expensive) on this PR wontfix This will not be worked on labels Oct 5, 2026
@skillberry-bot

Copy link
Copy Markdown
Contributor

🏷️ Automatic Labeling

I've analyzed this pull request and added the following labels:

  • paper-experiment - algorithm - benchmark-full - paper-experiment - algorithm - benchmark-full - wontfix

These labels were selected based on the PR title, description, and changed files. If you believe any labels are incorrect or missing, feel free to adjust them manually.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

algorithm Optimization algorithms: GEPA / SkillOpt / hill-climb benchmark-full Run all FULL benchmarks (whole/representative; expensive) on this PR paper-experiment Publication-grade held-out experiments for the NAACL/ACL paper wontfix This will not be worked on

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants