Repository navigation
feat(kernelforge): export-runs reads AgentKernelArena campaign logs - #1951
amd-ama10002-2 wants to merge 4 commits into
Conversation
Code metrics gate: PASSEDJudged: the 6 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.
105 findings in files this change does not touchOutside this change (informational): 105
...and 80 more; the job log lists every row.
Thresholds and sources: |
amd-ama10002-2
added
the
run-forge-e2e
There was a problem hiding this comment.
🟡 Changes recommended
Campaign exports can overwrite original source timings and omit valid implementer and planner transcripts.
4 open issues
What changed in this PR
Extends kernelforge export-runs to collect AgentKernelArena campaigns for Hyperloom’s Experience KB under the launcher’s run identity.
Changes:
- Discovers campaign logs before plain forge-loop artifacts.
- Enriches iterations with archives, agent transcripts, identity, and provenance.
- Adds campaign tests and updates exporter documentation.
| File | Description |
|---|---|
src/kernelforge/tests/test_run_export_arena.py |
Adds campaign export coverage. |
src/kernelforge/run_export/document.py |
Integrates campaign discovery. |
src/kernelforge/run_export/command.py |
Updates CLI help. |
src/kernelforge/run_export/arena.py |
Implements campaign parsing and enrichment. |
docs/reference/experience-kb-collect.md |
Documents campaign planner transcripts. |
docs/kernelforge/reference/cli.md |
Documents campaign export support. |
Review effort: Balanced
|
/review |
|
Automated review started for head 7c09f21. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/38980864196 |
|
Automated review: no verdict applied -- the review job ended with result 'failure'. Comment |
7c09f21 to
f9e237b
Compare
ab6ac35 to
2e6b440
Compare
b87fc25 to
13c3c72
Compare
PR #1951 -- feat(kernelforge): export-runs reads AgentKernelArena campaign logsWhat it does: Blocking issues: none SKIPPED: V4 -- # PARTIAL: #1032 changes 1075 files, only its first 100 were compared, overlap there unchecked Checked: src/kernelforge/run_export/ (arena, archive, document, forge_loop, session_log, units, files, command), orchestrator/agent.py, orchestrator/summary_line.py, loop/runner.py (_lane_prompt, _implementer_history), hyperloom_kb mapping kernelforge-v1 and declaration kernelforge-run-v1, hyperloom_kb/collect/engine.py; tree-wide sweep for every non-test writer and reader of the claimed set and of read_transcript/is_transcript | Ran: fetch.sh; pytest test_run_export_arena.py, test_run_export.py, test_collect_kernelforge.py at head (83 passed); the new transcript-claim test at 13c3c72 (2 failed) and at head (passed); normalize on a campaign beside one and two log-less campaigns at head; ruff check and format, lint-imports (7 kept), code-metrics gate against the base (passed), pr_hygiene size (306 lines); ruff C901 (10, 20) and PLR0913 (5) at head and base; CI at head: 31 check runs success, 2 skipped (main-only), docs build success | Base: 78558a9 | Head: 8c0e9c5 LGTM |
An AgentKernelArena campaign keeps its launcher's log under logs/, next to the run's KernelForge artifacts directory (<workspace>/<run>/<task>-forge-<id>/engine/forge_experiments) and the Claude Code transcripts of its agents. export-runs read that artifacts directory as a plain forge-loop run, under the config's campaign_id and the engine/ path instead of the launcher's run name. A campaign log that names a KernelForge artifacts directory is now one run, every unit under the launcher's run name. An iteration's fields also come from KernelForge's archive of it, from the implementer's messages in the transcript of the session that produced its candidate, linked by its path under the log root, and from the reasoning log that session kept beside its plan. The transcripts of the round's planners are raw logs of the run. The artifacts directory a campaign log claims is not read again as a plain forge-loop run. Observable effect: `kernelforge export-runs` on AgentKernelArena campaign logs writes each iteration under the launcher's run name with its implementer's transcript and the round's planner transcripts as raw logs; before, the same iterations came out as a plain forge-loop run under the campaign_id, with no planner transcripts. Co-authored-by: Cursor <cursoragent@cursor.com>
…implementer, planners and source time Four ways the AgentKernelArena campaign reader dropped or replaced evidence KernelForge had written for an iteration. An implementer runs in its kernel's directory, so the implementer of a kernel at the workspace root shares the planners' working directory, and the reader filed it as a planner: its iteration lost the transcript's reasoning, PLAN headline and reasoning_file. A session in the workspace itself is now an implementer when its prompt names a reasoning log or ends with the line KernelForge ends every implementer's prompt with. An iteration first takes the sessions told to keep the reasoning log beside its plan, the link a plain forge-loop run already uses, and only then the plan marker and time window that runs made before reasoning logs need. A lane's transcript was matched by its plan's first heading, and a lane plan is free Markdown, so a plan without a heading matched nothing. Such a plan is now matched whole; the lane's prompt holds it verbatim. For a forge-rewrite campaign, source_ms took the archive's baseline over the original kernel's time that forge-rewrite logged as its source baseline; for a run state that records no pristine baseline, the archive's is the time of the port the forge-loop started from. The logged source baseline now wins, and the archive's is the fallback. A fan-out round plans its lanes together and measures them in later iterations, but planner transcripts were taken from the measuring iteration's time window, so every lane after the first lost its round's planners. They now come from the round whose orchestration/iter_NNN holds the candidate's plan, shared by every candidate of that round. Observable effect: `kernelforge export-runs` on an AgentKernelArena campaign gives the implementer of a kernel at the workspace root, and the lane that ran a plan without a heading, their transcript's reasoning, PLAN headline and rationale.reasoning_file again; such a lane from a run that kept no reasoning log had no reasoning at all, so collection skipped it. A forge-rewrite campaign's iterations report the original kernel's logged time as source_ms instead of the port's. Every lane of a fan-out round lists that round's planner transcripts in its raw logs, in place of whichever planners ran while it was measured. Co-authored-by: Cursor <cursoragent@cursor.com>
…ut of the plain runs beside it The AgentKernelArena campaign reader claimed its log and its KernelForge artifacts directory but not the Claude Code transcripts of its agents, so the plain forge-loop reader read them again. That reader gives a transcript whose prompt names a reasoning log to the plain run whose workspace path ends most like the log's, and an arena workspace always ends in engine: another campaign whose log is missing, or a plain run made in a directory named engine, took the campaign implementer's transcript, and two such runs made the transcript fit both equally well. The campaign now claims the transcripts it read, as the document promises for every file a layout reads. Observable effect: `kernelforge export-runs` on a log root that holds an AgentKernelArena campaign log beside another KernelForge run read as a plain forge-loop run (another campaign without its log, or a plain run in a directory named engine) no longer gives that run's Experience the campaign implementer's transcript, PLAN headline, reasoning and rationale.reasoning_file, and no longer stops with "fits the runs in ... equally well" when two such runs are there; each transcript stays with the campaign that ran it. Co-authored-by: Cursor <cursoragent@cursor.com>
… launches as one campaign AgentKernelArena writes one launcher log for every task of a run, and a resumed run appends to the same log: its runner launches each task under the same logger, the forge adapter logs "Forge artifacts: <dir>" at every launch, and each engine's output comes through as [FORGE] lines. The campaign reader took the first artifacts line and read the whole log as that launch's, while the session log reader keeps the last source baseline and PORT lines it sees, so the first launch's PORT unit took a later launch's decision, and every one of its units took a later launch's source baseline as source_ms, which also wins over the archive's value. A campaign log that names more than one artifacts directory is now one unit saying so, and each of those directories is read as a plain forge-loop run, as it is without the campaign layer. Observable effect: `kernelforge export-runs` on the log of an AgentKernelArena run of several tasks, or of a resumed run that launched KernelForge again, writes one entry naming the artifacts directories, which collection skips with that reason, and each launch's iterations as a plain forge-loop run under its campaign_id; before, the first launch's PORT step and iterations came out under the launcher's run name with another launch's source_ms and PORT decision. Co-authored-by: Cursor <cursoragent@cursor.com>
b312a44 to
18d8a9f
Compare
|
Superseded by #2037: the four stacked PRs were combined into one PR at the reviewer's request. |
Motivation
We are building knowledge-base support for KernelForge runs (kernel writing): turning KernelForge's raw logs into structured Experiences in Hyperloom's Experience KB. We are also changing the logs KernelForge writes so they carry what the KB extracts: planner plans open with a
Summary:line, and implementers keep a<plan>.reasoning.mdwork log that opens the same way. This PR teacheskernelforge export-runsthe AgentKernelArena campaign log, after which AgentKernelArena logs can be collected.Stack: merge in this order, one at a time. Each PR targets the branch of the PR above it in this list.
export-runsand thekernelforge-v1mapping, for plain forge-loop runsexport-runsreads AgentKernelArena driver.log sessionsexport-runsreads AgentKernelArena campaign logs (this PR)What changes
kernelforge export-runsreads an AgentKernelArena campaign log (the launcher's log underlogs/that names one KernelForge artifacts directory): each forge-loop iteration is one entry under the launcher's run name, after the PORT step when the campaign ran forge-rewrite. Before, the artifacts directory was read as a plain forge-loop run under the config'scampaign_id.source_ms. Read as a plain run, the same iterations reported the pristine baselinerun_state.jsonrecords or, without one, the time of the port the forge-loop started from. A campaign without forge-rewrite keeps the archive's value.Docker runtime: … image=,config_ok=orforge_engine=<branch> <commit>line; AgentKernelArena's launcher script prints the first two to its console, and none of them reaches the log file AgentKernelArena writes underlogs/. Identity records the operator from the evaluator's--model-class.Known limitations
<run>/<task>-forge-<id>); without it the campaign is one entry saying why.RESUME RUN:, as every parallel worker's is, is one entry saying the run was not logged, and its artifacts directory is read as a plain forge-loop run.test_run_export_arena.py.