Repository navigation
feat(h4): add the three-arm results and the T1-vs-G2 run handoff - #696
Merged
Merged
Conversation
bcarmeli
force-pushed
the
results/h4-arms
branch
from
October 8, 2026 15:47
78397ca to
65cb5e8
Compare
Placed per reports/README.md's scheme: reports/<exp>/ mirrors artifacts/<exp>/, and cross-cutting analyses sit at reports/ root with an experiment prefix beside v4-skillset-variant-comparison.md and v4-t4-merge-decisions.md. reports/v4/h4/all-arms-per-trial.md the three-arm table reports/v4/h4/handoff-t1-vs-g2.md how to run the next arm reports/v4-cap-evolve-question-answers.md v4/h4/ groups the held-out suite — 30 scenarios authored after the original 34 — rather than splitting per task, because its material is cross-arm by nature. The question answers are about the v4 arms themselves, not h4, so they stay at the root with the other cross-cutting documents. README.md's naming section records both choices so the next writer does not have to re-derive them. all-arms-per-trial.md lists every clean trial for the three T1 arms plus the reference column. Per-trial rather than per-task-mean deliberately: of the differences first attributed to the Oct-06 suite drop, one was real, two were measurement artifacts, and the rest was sampling noise — only the vectors made that separable. That matches this directory's existing caution that a number without its n is not a result. What the arms establish: icinga-045 went [0.47 x5] -> [1.00 x5], matching the reference, because the Oct-06 drop seeds the icinga service on every task where Oct-04 did not. That also explains an earlier unexplained icinga result: the agent had been searching a database that did not contain the host it was asked about. The aggregate is nevertheless flat (0.6735 vs 0.6799). That one gain is cancelled by two platform catalog tasks the Oct-06 drop broke or exposed, where lookup_catalog_item returns "Operation schema unavailable". The gap to the reference is concentrated, not systemic: T1 matches or beats it on 12 of 30 tasks. The shortfall sits in a few stable, diagnosable tasks — icinga-048 at a deterministic 0.500 against 1.000, where the agent fails to discriminate two comments by their service_name field. Stable tasks make the better optimization targets; a bimodal task's gains vanish into variance. No new prompt bundles. The h4 arms run the EXISTING v4 artifacts, verified byte-identical across all 8 files each: T1 is artifacts/v4/seed/ and G2 is artifacts/v4/v4_g_e1/best/ (cand_0004). $PARSEC_H4_BUNDLE on main points straight at those. An h4-specific copy was staged here first and dropped — duplicating a bundle would let the arms silently drift from their v4 provenance, and "the same bundle as v4" is the entire basis for comparing the two experiments. v4-cap-evolve-question-answers.md answers the questions raised ahead of an external post. The finding that cuts across three of their four publication blockers: 12 trials failed for infrastructure reasons, were scored 0.0 anyway, and reached the published v4 numbers. icinga-013's cand_0002 is published at 0.0000 with all five trials dead — no valid measurement exists — and platform-004's cand_0001 is published at 0.4000 when both its surviving trials scored a perfect 1.000, so a candidate was discarded on a gateway outage. platform-031's published +0.6480 gain is measured against a baseline containing dead trials. handoff-t1-vs-g2.md leads with three silent failure modes because those cost whole task sets, documents that both branches are needed at run time and why (anything importing cap-evolve cannot live on this orphan branch), and covers what parallel execution requires: the five harness services are shared mutable state, so each worker needs its own stack, and they all share one LLM gateway whose request rate was already observed failing. Signed-off-by: boazc <boazc@il.ibm.com>
bcarmeli
force-pushed
the
results/h4-arms
branch
from
October 8, 2026 15:59
65cb5e8 to
9fbc04f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The h4 held-out suite: the three-arm T1 results, the answers to the cap-evolve team's publication questions, and the handoff for running T1 vs G2.
Placement follows
reports/README.mdThat file is the naming authority here:
reports/<exp>/mirrorsartifacts/<exp>/, and cross-cutting analyses sit at the root with an experiment prefix.reports/v4/h4/all-arms-per-trial.mdreports/v4/h4/handoff-t1-vs-g2.mdreports/v4-cap-evolve-question-answers.mdv4-skillset-variant-comparison.mdandv4-t4-merge-decisions.mdv4/h4/is a deliberate exception to "one file per task": h4 is 30 scenarios (bench-v4-035…-064) authored after the original 34, and its material is cross-arm by nature.README.md's naming section now records that, and where per-task h4 reports would go if written.No new prompt bundles
An earlier revision added
artifacts/h4/{t1,g2}— 16 files, 5,426 lines, all byte-identical to artifacts already on this branch. Dropped.artifacts/v4/seed/cand_0004)artifacts/v4/v4_g_e1/best/$PARSEC_H4_BUNDLE(set on main, PR #695) points straight at those. Duplicating would let the h4 arms silently drift from their v4 provenance — and "the same bundle as v4" is the entire basis for comparing the two experiments.Why the code isn't here
parsec-historyis an orphan branch with no cap-evolve packages — nocore/, nocapevolve_harbor, noskills/phases/baseline/scripts/run.py. The split is by whether something imports cap-evolve:scripts/parsec/, PRs #694 / #695)All 13 existing
scripts/on this branch import zero cap-evolve symbols, which confirms the line.What the three arms establish
icinga-045:[0.47 ×5]→[1.00 ×5], matching the reference. The Oct-06 drop seeds the icinga service on every task where Oct-04 did not — which also explains an earlier icinga result nobody could account for: the agent had been searching a database that didn't contain the host it was asked about.But the aggregate is flat — 0.6735 vs 0.6799. That gain is cancelled by two platform catalog tasks the Oct-06 drop broke or exposed (
lookup_catalog_item→Operation schema unavailable).The gap to the reference is concentrated, not systemic. T1 matches or beats it on 12 of 30 tasks. The shortfall sits in a few stable, diagnosable tasks —
icinga-048holds a deterministic 0.500 against 1.000, where the agent fails to discriminate two comments by theirservice_namefield. Those stable tasks are the good optimization targets; bimodal tasks' gains vanish into variance.The table is per-trial, not per-task-mean — consistent with this directory's existing caution that a number without its
nis not a result. Of the differences first attributed to the Oct-06 drop, one was real, two were measurement artifacts, and the rest was sampling noise. Only the vectors made that separable.reports/v4-cap-evolve-question-answers.mdOne finding cuts across three of their four publication blockers: 12 trials failed for infrastructure reasons, were scored 0.0, and reached the published v4 numbers.
icinga-013'scand_0002is published at 0.0000 with all five trials dead — no valid measurement exists, so it must not be described as a failed candidateplatform-004'scand_0001is published at 0.4000 when both surviving trials scored a perfect 1.000 — a candidate was discarded on a gateway outageplatform-031's published +0.6480 gain is measured against a baseline containing dead trialsTheir sharpest question — whether the "after" score is held out, given the two source documents contradict each other — is deliberately marked as needing the team rather than inferred.
reports/v4/h4/handoff-t1-vs-g2.mdLeads with the three silent failure modes because those cost whole task sets here. Also documents what parallel execution requires: the five harness services are shared mutable state, so each parallel worker needs its own stack, and they all share one LLM gateway whose request rate was already observed failing at ~50% hangs.