Repository navigation
feat(kernelforge): collect plain forge-loop runs as Experiences with export-runs and kernelforge-v1 - #1944
feat(kernelforge): collect plain forge-loop runs as Experiences with export-runs and kernelforge-v1#1944amd-ama10002-2 wants to merge 6 commits into
Conversation
Code metrics gate: PASSEDJudged: the 15 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.
105 findings in files this change does not touchOutside this change (informational): 105
...and 80 more; the job log lists every row.
Thresholds and sources: |
ca8cde8 to
05be3a7
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Evidence association, baseline accuracy, historical transcript support, and read-failure isolation need correction before collection is reliable.
5 open issues
Baseline JSON exports use the wrong source timing · New Basename matching mixes evidence between runs · New Historical transcripts without reasoning logs are incorrectly discarded · New Unreadable candidate files abort export instead of reporting errors · New Unreadable transcripts abort the entire export · New
What changed in this PR
Adds collection of plain KernelForge runs into Hyperloom’s Experience knowledge base, building on the summaries and reasoning logs introduced in #1949.
Changes:
- Adds
kernelforge export-runsto normalize archived candidates and supporting logs. - Packages the
kernelforge-run-v1declaration andkernelforge-v1mapping. - Adds exporter and collection tests, plus usage documentation.
| File | Description |
|---|---|
| src/kernelforge/tests/test_run_export.py | Tests export behavior and collection integration. |
| src/kernelforge/run_export/units.py | Defines shared run and unit structures. |
| src/kernelforge/run_export/forge_loop.py | Converts plain forge-loop archives into units. |
| src/kernelforge/run_export/files.py | Inventories files and provides bounded readers. |
| src/kernelforge/run_export/document.py | Combines runs into an export document. |
| src/kernelforge/run_export/command.py | Implements the export CLI command. |
| src/kernelforge/run_export/archive.py | Extracts artifacts, summaries, and transcript evidence. |
| src/kernelforge/run_export/__init__.py | Introduces the exporter package. |
| src/kernelforge/cli/__init__.py | Registers export-runs. |
| src/hyperloom_kb/tests/test_collect_kernelforge.py | Tests mapping projection and skip conditions. |
| src/hyperloom_kb/mappings/kernelforge-v1.yaml | Maps exported units into Experiences. |
| src/hyperloom_kb/declarations/kernelforge-run-v1.yaml | Defines the KernelForge Experience schema. |
| docs/reference/experience-kb-collect.md | Documents KernelForge collection. |
| docs/kernelforge/reference/cli.md | Documents exporter arguments and behavior. |
| docs/kernelforge/how-to/autonomous-loop.md | Links summaries to Experience export. |
Review effort: Balanced
|
/review |
|
Automated review started for head 05be3a7. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/38980828098 |
|
Automated review: no verdict applied -- the review job ended with result 'failure'. Comment |
05be3a7 to
924313c
Compare
|
This is the result of running the agent review-pr skill locally (the /review skill I think) PR #1944 -- feat(kernelforge): collect plain forge-loop runs as Experiences with export-runs and kernelforge-v1What it does: adds
The docs, the declaration and the mapping state that a plain run made before #1949 links no transcript. Blocking issues: none SKIPPED: V4 -- # PARTIAL: #1032 changes 1075 files, only its first 100 were compared; overlap there unchecked Checked: src/kernelforge/run_export/{files,archive,forge_loop,units,document,command}.py, src/kernelforge/cli/init.py, the declaration and mapping, docs/kernelforge/reference/cli.md, docs/reference/experience-kb-collect.md, and the description against the head; tree-wide sweep for every non-test writer of optimization_plan_path, merge_attempt_staged, kernel_path, campaign_id, pristine_baseline_wall_ms, MERGE_PLAN_PREFIX and reasoning_log_section, and for the mkdtemp callers that create 0700 directories (src/kernelforge, excluding tests and run_export); tree-wide importers of read_json, inventory and _files_in (src, scripts) | Ran: fetch.sh's queries through a copy adapted to gh 2.4.0; ruff C901 (max 20) and PLR0913 (max 5) on the 8 changed modules, both clean; through normalize and the export-runs CLI, a plain run made before #1949 (recorded rationale, no transcript), an unreadable handoff, an unlistable transcript directory, and an unreadable events.jsonl (each exits 1 naming the path), plus a missing events.jsonl (exports); test_run_export.py, test_collect_kernelforge.py, test_collect.py; each fix commit's new cases against its parent, where they fail; the declaration's schema_ref at this head and at #1950's head; an independent second reader at 85cdf56 and again at this head; check runs at the head, all green | Base: d32992a | Head: e5ea4c9 LGTM |
|
This is the result of running the agent review-pr skill locally (the /review skill I think) PR #1944 -- feat(kernelforge): collect plain forge-loop runs as Experiences with export-runs and kernelforge-v1What it does: adds Blocking issues: none SKIPPED: V4 -- # PARTIAL: #1032 changes 1075 files, only its first 100 were compared; overlap there unchecked Checked: src/kernelforge/run_export/{files,archive,forge_loop,units,document,command}.py, src/kernelforge/cli/init.py, the declaration and mapping, both reference pages, and the description against the head (its embedded declaration is identical to the file and its schema_ref matches the one recomputed with load_mapping); tree-wide sweeps for every non-test writer of the run values the export reads, every copy of the transcript claim, and every caller of read_json, inventory and _files_in | Ran: fetch.sh's queries through a copy adapted to gh 2.4.0; ruff C901 (max 20) and PLR0913 (max 5) on the 8 changed modules, both clean; through normalize, the export-runs CLI and a collect dry run: a plain run made before #1949, an unreadable handoff, an unlistable transcript directory, and a pre-#1949 lane with its transcript present; each new test case against its parent commit (candidate record, transcript directory and best bundle fail on 7a1171a, events log on 2aa48b7; all pass here); pytest test_run_export.py, test_collect_kernelforge.py and test_collect.py (87 passed); CI at e5ea4c9: 33 checks succeeded, including all 12 test shards and both coverage jobs (code-metrics and pr-hygiene succeeded on the rerun that cancelled their first runs) | Base: d32992a | Head: e5ea4c9 LGTM |
…-runs and kernelforge-v1
Experiences built from KernelForge logs carried tool narration as their
reasoning ("I'll start by reading the required optimization plan...") and
three long sentences cut from a 25 KB plan as their plan, and the
normalizer that produced them lived outside the repository.
`kernelforge export-runs LOG_ROOT... -o runs.json` writes every run found
under the given directories as one document for
`hyperloom-kb-collect --mapping kernelforge-v1`. It reads a plain
forge-loop run's forge_experiments/ directory, which yields one unit per
archived candidate: run_id from run_state.json's campaign_id, times from
events.jsonl, identity from campaign_config.json, the Implementer's
backend and model and kept candidates' KB records from forge-loop.log, and
transcripts linked by the reasoning log their first prompt names.
rationale.plan is the plan's Summary line and rationale.reasoning the
Summary line of the implementer's reasoning log, each cut to three
sentences; logs without them, as every run made before KernelForge asked
for them, keep the earlier extraction. The packaged kernelforge-run-v1
declaration has an optional rationale.reasoning_log file field holding
the whole log, and the packaged kernelforge-v1 mapping (producer version
10) projects the document onto it. Both are complete as they land: the
AgentKernelArena layouts export-runs reads next add no field to either.
Observable effect: a new `kernelforge export-runs` command, a packaged
kernelforge-run-v1 declaration and a packaged kernelforge-v1 mapping for
hyperloom-kb-collect. Experiences collected through them from plain
forge-loop runs state the plan and the implementer's change in the
agents' own one-to-three-sentence summaries and link the reasoning log as
a file.
Co-authored-by: Cursor <cursoragent@cursor.com>
… each run's own evidence Three problems in how export-runs turns a plain forge-loop run into units. source_ms came from each candidate's meta.json, whose baseline_wall_ms is the time of the kernel the search started from. A run given --baseline-json scores every candidate against that file's times instead, which the run state keeps as pristine_baseline_wall_ms, so its units paired speedups against one kernel with the time of another. source_ms now takes the run state's pristine baseline and falls back to meta.json only when the run state records none; the declaration's source_ms description says so. A forge-loop log went to every run whose workspace held the log's kernel file under a directory of the same name, and a transcript to every run whose workspace name and iteration matched the reasoning log it names, so two runs called ws under one log root shared each other's model, KB record, PLAN headline, reasoning and raw logs. Each log and transcript now goes to the one run whose workspace path shares the most trailing components with the path it names, which still finds a run made in a container, and the export stops, naming the file and both runs, when two runs fit equally well. An unreadable file already stopped the export, with a traceback. It now stops with a message naming the file, and suggests fixing ownership when a container left it root-owned. An unreadable .log is no longer skipped in silence, which dropped its run's provenance. Observable effect: source_ms of a run given --baseline-json is the original kernel's time; logs and transcripts are no longer shared between runs whose workspaces have one name; an export that cannot read a file, or cannot tell which run a log or transcript belongs to, exits 1 with "Error: ..." naming it and writes nothing. Co-authored-by: Cursor <cursoragent@cursor.com>
os.walk without onerror, and Path.rglob, return nothing for a directory the user cannot list. A run copied out of a container, where Claude Code keeps each project's transcripts in a directory only root can list and KernelForge creates its best bundles with mkdtemp, therefore lost those files while the export still exited 0. inventory now raises the listing error, which export-runs reports with its fix-ownership hint, and _files_in lists through inventory instead of a second walker. The declaration and both reference pages also said that a plain run made before #1949 falls back to its Implementer's transcript. A plain session's transcript is linked only by the reasoning log its first prompt names, so such a run gives the rationale KernelForge recorded and its lanes give no reasoning; the text now says so. The reasoning_file description changes, and with it the declaration's schema_ref. Co-authored-by: Cursor <cursoragent@cursor.com>
read_json returned None for a JSON file it could not open, so an unreadable meta.json became a "not a readable JSON object" entry while every other file the export cannot read stops it with the fix-ownership hint. A missing or malformed record is still one entry saying why; a record that is there but cannot be read now raises like any other file. The export-runs reference says which is which.
…soning The mapping's no-reasoning skip reason said the logs hold no Claude Code transcript of the agent and asked for logs that record its reasoning. A lane of a plain run made before #1949 can have its transcript in the logs, but nothing links it, so every such lane got a wrong diagnosis and a remedy that cannot help, contradicting the collection reference. The reason now says no transcript is linked, and that such a lane has no reasoning, like a merge. Co-authored-by: Cursor <cursoragent@cursor.com>
An events.jsonl the export could not open still became one entry for the whole run, while every other file it cannot read stops it, so every candidate of that run was skipped behind a PermissionError that the collect report showed only as a skip reason. The export now stops with the path and the fix-ownership hint; a missing events log still leaves the times to meta.json. The export-runs reference now says that a run is a forge_experiments directory holding campaign_config.json and run_state.json, and which records become entries: a run's records that are not the JSON KernelForge writes, and a candidate's missing or malformed meta.json.
e5ea4c9 to
225058e
Compare
|
Superseded by #2037: the four stacked PRs were combined into one PR at the reviewer's request. |


Motivation
We are building knowledge-base support for KernelForge runs (kernel writing): turning KernelForge's raw logs into structured Experiences in Hyperloom's Experience KB. We are also changing the logs KernelForge writes so they carry what the KB extracts: planner plans open with a
Summary:line, and implementers keep a<plan>.reasoning.mdwork log that opens the same way (#1949). This PR adds the exporter and the packaged declaration and mapping, starting with plainforge-loopruns. There is also an example experience that was generated from using this declaration on a local hyperloom run.Stack: merge in this order, one at a time. Each PR targets the branch of the PR above it in this list.
export-runsand thekernelforge-v1mapping, for plain forge-loop runs (this PR)export-runsreads AgentKernelArena driver.log sessionsexport-runsreads AgentKernelArena campaign logsDon't collect AgentKernelArena logs into a KB until #1951 lands: until then a campaign's artifacts are read as a plain forge-loop run, under the wrong
run_id.What changes
kernelforge export-runs: writes every plainforge-looprun found under the given directories (aforge_experiments/directory holdingcampaign_config.jsonandrun_state.json) as one document, one entry per archived candidate. A run whosecampaign_config.jsonorrun_state.jsonis not the JSON KernelForge writes, or a candidate whosemeta.jsonis missing or malformed, is one entry saying why; a directory it cannot list or a file it cannot read stops the export withError: cannot read <path>: <reason>and writes nothing.baseline.source_msis the pristine baseline the run'srun_state.jsonrecords, so a run given--baseline-jsonreports the original kernel's time, not that of the kernel its search started from.kernelforge-run-v1declaration andkernelforge-v1mapping forhyperloom-kb-collect, both with every field they will have: the arena layouts in feat(kernelforge): export-runs reads AgentKernelArena driver.log sessions #1950 and feat(kernelforge): export-runs reads AgentKernelArena campaign logs #1951 add none, and leave the declaration's schema_ref unchanged.rationale.planis its plan'sSummary:line andrationale.reasoningthe reasoning log's, each at most three sentences, and the whole log is linked asrationale.reasoning_log. Plain runs made before feat(kernelforge): plans open with a Summary line; implementers keep <plan>.reasoning.md #1949 fall back to the plan's first sentences and to the rationale KernelForge recorded: their sessions' first prompts name no reasoning log, so no transcript is linked to them, and their lanes have no reasoning.Usage
Declaration:
kernelforge-run-v1Example Experience
Collected from iteration 1 of a plain forge-loop run with Claude-Opus-5.5 as the Implementer; its
reflection.lessonis KernelForge's machine-written fallback because the run predates #1909.Known limitations
hyperloom-kb-collectkeys raw logs by SHA-256, so files with identical bytes are recorded under one name: above,candidates/iter_001/change.diffappears as a secondforge.patch.kernelforge_commitin provenance and noidentity.operator; their logs hold neither.test_run_export.pyandhyperloom_kb/tests/test_collect_kernelforge.py.