Skip to content

feat(h4): add the three-arm results and the T1-vs-G2 run handoff - #696

Merged
bcarmeli merged 1 commit into
parsec-historyfrom
results/h4-arms
Oct 8, 2026
Merged

bcarmeli merged 1 commit into
parsec-historyfrom
results/h4-arms

Conversation

@bcarmeli

@bcarmeli bcarmeli commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

The h4 held-out suite: the three-arm T1 results, the answers to the cap-evolve team's publication questions, and the handoff for running T1 vs G2.

Placement follows reports/README.md

That file is the naming authority here: reports/<exp>/ mirrors artifacts/<exp>/, and cross-cutting analyses sit at the root with an experiment prefix.

file why there
reports/v4/h4/all-arms-per-trial.md h4 suite material, grouped by suite
reports/v4/h4/handoff-t1-vs-g2.md same
reports/v4-cap-evolve-question-answers.md about the v4 arms, not h4 — root, beside v4-skillset-variant-comparison.md and v4-t4-merge-decisions.md

v4/h4/ is a deliberate exception to "one file per task": h4 is 30 scenarios (bench-v4-035 … -064) authored after the original 34, and its material is cross-arm by nature. README.md's naming section now records that, and where per-task h4 reports would go if written.

No new prompt bundles

An earlier revision added artifacts/h4/{t1,g2} — 16 files, 5,426 lines, all byte-identical to artifacts already on this branch. Dropped.

arm bundle it actually uses verified
T1 (original instructions) artifacts/v4/seed/ 8/8 identical
G2 (cand_0004) artifacts/v4/v4_g_e1/best/ 8/8 identical

$PARSEC_H4_BUNDLE (set on main, PR #695) points straight at those. Duplicating would let the h4 arms silently drift from their v4 provenance — and "the same bundle as v4" is the entire basis for comparing the two experiments.

Why the code isn't here

parsec-history is an orphan branch with no cap-evolve packages — no core/, no capevolve_harbor, no skills/phases/baseline/scripts/run.py. The split is by whether something imports cap-evolve:

what branch
adapter, arm driver, detectors main (scripts/parsec/, PRs #694 / #695)
bundles, results, report builders here

All 13 existing scripts/ on this branch import zero cap-evolve symbols, which confirms the line.

What the three arms establish

icinga-045: [0.47 ×5] → [1.00 ×5], matching the reference. The Oct-06 drop seeds the icinga service on every task where Oct-04 did not — which also explains an earlier icinga result nobody could account for: the agent had been searching a database that didn't contain the host it was asked about.

But the aggregate is flat — 0.6735 vs 0.6799. That gain is cancelled by two platform catalog tasks the Oct-06 drop broke or exposed (lookup_catalog_item → Operation schema unavailable).

The gap to the reference is concentrated, not systemic. T1 matches or beats it on 12 of 30 tasks. The shortfall sits in a few stable, diagnosable tasks — icinga-048 holds a deterministic 0.500 against 1.000, where the agent fails to discriminate two comments by their service_name field. Those stable tasks are the good optimization targets; bimodal tasks' gains vanish into variance.

The table is per-trial, not per-task-mean — consistent with this directory's existing caution that a number without its n is not a result. Of the differences first attributed to the Oct-06 drop, one was real, two were measurement artifacts, and the rest was sampling noise. Only the vectors made that separable.

reports/v4-cap-evolve-question-answers.md

One finding cuts across three of their four publication blockers: 12 trials failed for infrastructure reasons, were scored 0.0, and reached the published v4 numbers.

  • icinga-013's cand_0002 is published at 0.0000 with all five trials dead — no valid measurement exists, so it must not be described as a failed candidate
  • platform-004's cand_0001 is published at 0.4000 when both surviving trials scored a perfect 1.000 — a candidate was discarded on a gateway outage
  • platform-031's published +0.6480 gain is measured against a baseline containing dead trials

Their sharpest question — whether the "after" score is held out, given the two source documents contradict each other — is deliberately marked as needing the team rather than inferred.

reports/v4/h4/handoff-t1-vs-g2.md

Leads with the three silent failure modes because those cost whole task sets here. Also documents what parallel execution requires: the five harness services are shared mutable state, so each parallel worker needs its own stack, and they all share one LLM gateway whose request rate was already observed failing at ~50% hangs.

@bcarmeli bcarmeli changed the title feat(h4): add the T1/G2 prompt bundles, the three-arm results, and the run handoff feat(h4): add the three-arm results and the T1-vs-G2 run handoff Oct 8, 2026
Placed per reports/README.md's scheme: reports/<exp>/ mirrors artifacts/<exp>/,
and cross-cutting analyses sit at reports/ root with an experiment prefix
beside v4-skillset-variant-comparison.md and v4-t4-merge-decisions.md.

  reports/v4/h4/all-arms-per-trial.md     the three-arm table
  reports/v4/h4/handoff-t1-vs-g2.md       how to run the next arm
  reports/v4-cap-evolve-question-answers.md

v4/h4/ groups the held-out suite — 30 scenarios authored after the original 34
— rather than splitting per task, because its material is cross-arm by nature.
The question answers are about the v4 arms themselves, not h4, so they stay at
the root with the other cross-cutting documents. README.md's naming section
records both choices so the next writer does not have to re-derive them.

all-arms-per-trial.md lists every clean trial for the three T1 arms plus the
reference column. Per-trial rather than per-task-mean deliberately: of the
differences first attributed to the Oct-06 suite drop, one was real, two were
measurement artifacts, and the rest was sampling noise — only the vectors made
that separable. That matches this directory's existing caution that a number
without its n is not a result.

What the arms establish:

  icinga-045 went [0.47 x5] -> [1.00 x5], matching the reference, because the
  Oct-06 drop seeds the icinga service on every task where Oct-04 did not. That
  also explains an earlier unexplained icinga result: the agent had been
  searching a database that did not contain the host it was asked about.

  The aggregate is nevertheless flat (0.6735 vs 0.6799). That one gain is
  cancelled by two platform catalog tasks the Oct-06 drop broke or exposed,
  where lookup_catalog_item returns "Operation schema unavailable".

  The gap to the reference is concentrated, not systemic: T1 matches or beats
  it on 12 of 30 tasks. The shortfall sits in a few stable, diagnosable tasks —
  icinga-048 at a deterministic 0.500 against 1.000, where the agent fails to
  discriminate two comments by their service_name field. Stable tasks make the
  better optimization targets; a bimodal task's gains vanish into variance.

No new prompt bundles. The h4 arms run the EXISTING v4 artifacts, verified
byte-identical across all 8 files each: T1 is artifacts/v4/seed/ and G2 is
artifacts/v4/v4_g_e1/best/ (cand_0004). $PARSEC_H4_BUNDLE on main points
straight at those. An h4-specific copy was staged here first and dropped —
duplicating a bundle would let the arms silently drift from their v4
provenance, and "the same bundle as v4" is the entire basis for comparing the
two experiments.

v4-cap-evolve-question-answers.md answers the questions raised ahead of an
external post. The finding that cuts across three of their four publication
blockers: 12 trials failed for infrastructure reasons, were scored 0.0 anyway,
and reached the published v4 numbers. icinga-013's cand_0002 is published at
0.0000 with all five trials dead — no valid measurement exists — and
platform-004's cand_0001 is published at 0.4000 when both its surviving trials
scored a perfect 1.000, so a candidate was discarded on a gateway outage.
platform-031's published +0.6480 gain is measured against a baseline containing
dead trials.

handoff-t1-vs-g2.md leads with three silent failure modes because those cost
whole task sets, documents that both branches are needed at run time and why
(anything importing cap-evolve cannot live on this orphan branch), and covers
what parallel execution requires: the five harness services are shared mutable
state, so each worker needs its own stack, and they all share one LLM gateway
whose request rate was already observed failing.

Signed-off-by: boazc <boazc@il.ibm.com>
@bcarmeli
bcarmeli merged commit 1d11dae into parsec-history Oct 8, 2026
2 checks passed
@bcarmeli
bcarmeli deleted the results/h4-arms branch October 14, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant