Skip to content

feat(kernelforge): collect plain forge-loop runs as Experiences with export-runs and kernelforge-v1 - #1944

Closed
amd-ama10002-2 wants to merge 6 commits into
feat/kernelforge-summary-linesfrom
feat/kernelforge-run-experiences
Closed

amd-ama10002-2 wants to merge 6 commits into
feat/kernelforge-summary-linesfrom
feat/kernelforge-run-experiences

Conversation

@amd-ama10002-2

@amd-ama10002-2 amd-ama10002-2 commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

We are building knowledge-base support for KernelForge runs (kernel writing): turning KernelForge's raw logs into structured Experiences in Hyperloom's Experience KB. We are also changing the logs KernelForge writes so they carry what the KB extracts: planner plans open with a Summary: line, and implementers keep a <plan>.reasoning.md work log that opens the same way (#1949). This PR adds the exporter and the packaged declaration and mapping, starting with plain forge-loop runs. There is also an example experience that was generated from using this declaration on a local hyperloom run.

Stack: merge in this order, one at a time. Each PR targets the branch of the PR above it in this list.

  1. feat(kernelforge): plans open with a Summary line; implementers keep <plan>.reasoning.md #1949 — Summary line in plans; implementer reasoning logs
  2. feat(kernelforge): collect plain forge-loop runs as Experiences with export-runs and kernelforge-v1 #1944 — export-runs and the kernelforge-v1 mapping, for plain forge-loop runs (this PR)
  3. feat(kernelforge): export-runs reads AgentKernelArena driver.log sessions #1950 — export-runs reads AgentKernelArena driver.log sessions
  4. feat(kernelforge): export-runs reads AgentKernelArena campaign logs #1951 — export-runs reads AgentKernelArena campaign logs

Don't collect AgentKernelArena logs into a KB until #1951 lands: until then a campaign's artifacts are read as a plain forge-loop run, under the wrong run_id.

What changes

  • New kernelforge export-runs: writes every plain forge-loop run found under the given directories (a forge_experiments/ directory holding campaign_config.json and run_state.json) as one document, one entry per archived candidate. A run whose campaign_config.json or run_state.json is not the JSON KernelForge writes, or a candidate whose meta.json is missing or malformed, is one entry saying why; a directory it cannot list or a file it cannot read stops the export with Error: cannot read <path>: <reason> and writes nothing.
  • Each forge-loop log and Claude Code transcript belongs to the one run whose workspace path ends most like the path it names, which still finds a run made in a container; a log or transcript that fits two runs equally well stops the export, naming both runs.
  • baseline.source_ms is the pristine baseline the run's run_state.json records, so a run given --baseline-json reports the original kernel's time, not that of the kernel its search started from.
  • New packaged kernelforge-run-v1 declaration and kernelforge-v1 mapping for hyperloom-kb-collect, both with every field they will have: the arena layouts in feat(kernelforge): export-runs reads AgentKernelArena driver.log sessions #1950 and feat(kernelforge): export-runs reads AgentKernelArena campaign logs #1951 add none, and leave the declaration's schema_ref unchanged.
  • An Experience's rationale.plan is its plan's Summary: line and rationale.reasoning the reasoning log's, each at most three sentences, and the whole log is linked as rationale.reasoning_log. Plain runs made before feat(kernelforge): plans open with a Summary line; implementers keep <plan>.reasoning.md #1949 fall back to the plan's first sentences and to the rationale KernelForge recorded: their sessions' first prompts name no reasoning log, so no transcript is linked to them, and their lanes have no reasoning.
  • Refactor-only parts: none.

Usage

kernelforge export-runs <LOG_ROOT>... -o runs.json
hyperloom-kb-collect --mapping kernelforge-v1 --document runs.json [--dry-run]
Declaration: kernelforge-run-v1
# KernelForge runs, one Experience per run: the forge-rewrite PORT step when the session ran forge-rewrite, and each
# forge-loop iteration, which measures one candidate kernel and keeps or reverts it.
# Hyperloom gives every declaration the rationale fields reasoning, alternatives, and preconditions, so they are not
# listed here. rationale.reasoning holds the Summary line the implementer's reasoning log (rationale.reasoning_log)
# opens with, at most three sentences stating the problem it solved and its approach. When the session kept no such
# log, or its log has no Summary line, as in every run made before KernelForge asked for the log, it holds the first
# three sentences of the implementer's own messages from the Claude Code transcript linked to the session
# (rationale.reasoning_file) or, with none linked, of the rationale KernelForge recorded: the Rationale line it logs,
# which it cuts at 200 characters (kernelforge/loop/runner.py:5417 at 1c14e79e1), or for a plain forge-loop run the
# candidate's meta.json. Reasoning is required: a run whose logs hold none becomes no Experience, and the review asks
# for logs that record it.
# The objective and every identity field always form the repeat group, with the schema: Hyperloom groups Experiences
# that way by itself (hyperloom_kb/query_view.py repeat_group_key) and accepts group: true only on baseline and change
# fields, so none is declared here.
schema_version: 2

objectives:
  - id: kernel_mean_case_speedup@v1
    description: >-
      How many times faster a candidate kernel runs than the original kernel the session started from, averaged with
      equal weight over the task's benchmark cases: for each case, the original kernel's time divided by the
      candidate's time, then the mean of those ratios. A forge-loop iteration takes it as the mean of three
      independent benchmark runs of per-case medians, and keeps the candidate only when it clears a bar KernelForge
      sets from the best result so far and the spread of those runs; the PORT step measures the ported kernel once.
    unit: x
    direction: higher_is_better

identity:
  - {name: operator, description: "Operator the kernel implements: the operator the task's production baseline calls, or the PyTorch module class the task benchmarks, e.g. gemm_a8w8_blockscale_bpreshuffle, SimpleMatmulModule."}
  - {name: kernel, description: "Kernel the session optimizes: the task's logical operator, which KernelForge files its KB records under, e.g. gemm_a16w16_nt_n1024_k4096, hip2hip/gpumode_large/SimpleMatmulModule."}
  - {name: framework, description: "Framework that owns the source operator, e.g. aiter."}
  - {name: backend, description: "Kernel language of the candidate, e.g. flydsl, hip."}
  - {name: backend_version, description: "Release of the candidate's kernel language in the session's runtime, e.g. 0.3.2.", kind: version}
  - {name: gpu, description: "GPU model the candidate was measured on, which also fixes its architecture target, e.g. mi355x (gfx950), mi300x (gfx942)."}
  - {name: dtype, description: "Data type of the operator's first input in the benchmark cases, e.g. bfloat16, float8_e4m3fn."}
  - {name: model, description: "Model family the task bundle files the task under, e.g. deepseek-v4-flash."}

baseline:
  - name: current_best_mean_case_speedup
    description: >-
      The score of the best kernel KernelForge had kept when the iteration began, from the iteration's header line
      or, for a plain forge-loop run, the candidate's meta.json: the score the candidate had to beat, by a margin
      KernelForge sets from the spread of its benchmark runs. KernelForge raises it only when it keeps a candidate.
      Until the first KEEP, it is KernelForge's own measurement of the kernel the forge-loop started from: the
      original kernel, or forge-rewrite's port of it or the KB kernel it accepted instead. A score is how many times
      faster a kernel runs than the original kernel, averaged over the task's benchmark cases; 1.0 is as fast as the
      original.
    kind: number
    role: measurement
  - name: source_ms
    description: >-
      How long the original kernel took before any change, in milliseconds: its average time per benchmark case. It
      comes from forge-rewrite's source baseline line or, for a forge-loop run, from the pristine baseline the run's
      run_state.json records, which for a run given --baseline-json is that file's time rather than the time of the
      kernel the search started from. A run state that records none leaves the baseline wall time KernelForge
      archived for the iteration. Every score compares a kernel's time on each case with the original kernel's time
      on that case.
    kind: number

rationale:
  - name: plan
    description: >-
      What the plan the candidate was built from sets out to do, in at most three sentences: the Summary line the
      plan opens with, the planner's statement of the problem it addresses and of its goal. For a lane, the plan is
      the lane's own; for a single session, the round's optimization plan, read from its original draft when
      KernelForge could not revise the draft and put the critic's review in front of it
      (kernelforge/orchestrator/orchestration.py:1711 at 1c14e79e1); for a merge, the stacked plans as KernelForge
      records them. A plan without a Summary line, as every plan written before KernelForge asked for one is, gives
      its first three sentences instead: lines a planner wrote before the plan's first heading are skipped, as are
      headings, code blocks, and tables.
    kind: text
  - name: plan_file
    description: >-
      The whole plan the candidate was built from, as KernelForge archived it in the planning round's
      orchestration/iter_NNN/: optimization_plan.md for a single session or a round's first lane, lane_NNN.md for
      the round's other lanes. The Experience links to the file by its name, size, and SHA-256.
    kind: file
  - name: reasoning_file
    description: >-
      Path of the Claude Code transcript of the session that made the candidate, relative to the directory the logs
      were collected from: every message the agent wrote, and the record of the iteration it gave KernelForge
      afterwards. The transcript is also one of the Experience's raw_logs, named by the path's last part. A path, not
      a file link: Hyperloom's credential screen for linked files flags the transcript's token-usage counters, such
      as input_tokens. Absent when the export links no transcript in the logs to that session, as for every plain
      forge-loop run made before KernelForge named the reasoning log in the session's first prompt.
    kind: text
  - name: reasoning_log
    description: >-
      The whole reasoning log the implementer kept while it worked, beside the plan it executed:
      orchestration/iter_NNN/<plan>.reasoning.md, such as optimization_plan.reasoning.md or lane_002.reasoning.md.
      It opens with the Summary line rationale.reasoning holds, followed by the implementer's record of its work: what
      it examined and the evidence, what it changed, what each change measured, and which directions it kept or
      dropped on those measurements. The Experience links to the file by
      its name, size, and SHA-256; the log is also one of its raw_logs. Optional: absent when no session kept one,
      as for the PORT step, a merge, and every run made before KernelForge asked for the log.
    kind: file
    required: false

change:
  - name: approach
    description: >-
      The optimization the candidate tried, in one line: the implementer's own PLAN headline, which KernelForge asks
      for as a short headline "naming the optimization NOW IN THE FILE" (kernelforge/orchestrator/agent.py:414 at
      1c14e79e1); for a merge, KernelForge's line naming the two stacked candidates; for the PORT step, the port's op
      line or the KB kernel forge-rewrite accepted instead.
    kind: text
    role: summary
    search: 4
  - name: source_code
    description: >-
      The kernel file that was measured, as KernelForge archived it in candidates/iter_NNN/. The Experience links to
      the file by its name, size, and SHA-256; the knowledge base stores the file once, beside the run's raw logs.
    kind: file

outcome:
  - name: decision
    description: >-
      KernelForge's decision label for an iteration, as its forge-loop runner assigns them, or the PORT step's
      result.
    role: decision
    values:
      - KEEP
      - REVERT_PERF
      - REVERT_VALIDATION
      - REVERT_VALIDATION_TIMEOUT
      - REVERT_VALIDATION_ERROR
      - BUILD_FAILED
      - CRASH
      - REVERT_INTEGRITY
      - REVERT_CONTENDED
      - NO_CHANGES
      - API_ERROR
      - AGENT_ERROR
      - ORCHESTRATION_ERROR
      - PORT OK
      - PORT FAILED
  - name: mean_case_speedup
    description: >-
      The candidate's score: how many times faster it ran than the original kernel, averaged over the task's
      benchmark cases. For a forge-loop iteration it is the mean of three independent benchmark runs; for the PORT
      step, its one interim measurement.
    kind: number
    role: measurement

reflection:
  - name: lesson
    description: >-
      The first three sentences of KernelForge's lesson for the iteration, which it writes after every iteration that
      ran an implementer session (kernelforge/loop/runner.py:3796 at 1c14e79e1): the agent's own record of the
      iteration when KernelForge could get one, otherwise a record KernelForge writes from what the loop observed,
      opening "(no agent summary)". Optional: lanes get no lesson.
    kind: text
    required: false
  - name: lesson_file
    description: >-
      KernelForge's whole lesson for the iteration, lessons/iter_NNN.md. The Experience links to the file by its
      name, size, and SHA-256. Optional, as lesson is.
    kind: file
    required: false
Example Experience

Collected from iteration 1 of a plain forge-loop run with Claude-Opus-5.5 as the Implementer; its reflection.lesson is KernelForge's machine-written fallback because the run predates #1909.

id: exp-ce94db15e7c45728ba82a87e4aedfebe
run_id: bb03b671005f4284ad238d7d9fc1e3da
seq: 1
created_at: "2026-10-16T08:17:52Z"
completed_at: "2026-10-16T08:36:18Z"
objective: kernel_mean_case_speedup@v1
schema_ref: schema:sha256:772e1e3afa2f43ed7162aac5504e2757b193319222c4de59c192f321785e8c85
schema_version: 2
kind: experience
status: complete
parent_id: ""
supersedes: ""
identity:
  kernel: _softmax_kernel
  backend: triton
  gpu: mi300x
baseline:
  current_best_mean_case_speedup: 1.0
  source_ms: 0.0126
rationale:
  reasoning: >-
    The softmax kernel spent most of its VALU work on a precise fp32 divide per element and the range fix-up in exp.
    It now uses exp2 with LOG2E folded into an fma and one reciprocal per row, has mask-free constexpr paths, and
    handles wide rows (N>512) as two rows per 4-warp program.
  reasoning_file: .claude/projects/<run-root>-ws/6579b9c8-37c3-48d0-8c87-1ff52a7a5458.jsonl
  reasoning_log: {name: optimization_plan.reasoning.md, bytes: 2160, sha256: 66b21780d9039055dd081dac73e596ca83cda3a4dcf2f4b6fa7445f61d8330e6}
  plan: >-
    Static ISA analysis indicates that, inside the kernel, the cost is mostly VALU work: a precise IEEE fp32 divide
    on every element (about 43% of instructions at N=1024, about 32% at N=256) and the `tl.exp` range fix-up.
    Run-to-run noise of about ±9–12% on M1024_N256 then blurs any gain. This plan ships one combined candidate (exp2
    with LOG2E folded into an fma, one reciprocal per row, a mask-free constexpr path, and a `num_warps` chosen per
    row width and measured on M4096_N1024) so the gain on the stable large case is big enough to clear the keep
    threshold, targeting about 1.05–1.12× mean.
  plan_file: {name: optimization_plan.md, bytes: 8385, sha256: 5f8fc2314d0fca1b662aebbcc1c1d8e7c329f87daea4f6e369facdb46dbc9667}
change:
  approach: exp2 with one reciprocal per row, mask-free paths, two rows per 4-warp program for wide rows
  source_code: {name: softmax_kernel.py, bytes: 2065, sha256: 07b23e7fed780f63440e1becc1b8734b6cfca7ce24b04747241151c894a2aee6}
outcome:
  decision: KEEP
  mean_case_speedup: 1.1295409194258121
reflection:
  lesson: >-
    (no agent summary) session hit 1 gate rejection(s): The kernel is CORRECT but NOT faster than the current best,
    so it is not good en No summarizer session was available for this iteration, so this record is machine-written
    from what the loop observed. It has no account of directions the agent tried and abandoned.
  lesson_file: {name: iter_001.md, bytes: 2538, sha256: e474930b88b7b0bc8cfaaaf764907dd17f189980c849cd5665c4f48c376e7208}
provenance:
  producer: kernelforge
  producer_version: "10"
  source_ref: ws/iteration-1
  model: ""
  prompt_version: ""
  snapshot_version: ""
  extra:
    candidate_origin: session
    commit: db4b0ffdaaeaeded0def6ab59bcff9fd0e5c0c0c
    kb_record: kernel:forge-loop:softmax:unknown:unknown:triton:mi300x/rewrite-softmax-92fe883d6dc2-d7c8a1a9914c
    llm_backend: claude
    llm_model: Claude-Opus-5.5
notes: {}
rendered_refs: []
raw_logs:  # 22 files, 674,027 bytes; each comment is the file it came from
  - {name: forge-loop.log, bytes: 15017}                               # forge-loop.log
  - {name: campaign_config.json, bytes: 1039}                          # ws/forge_experiments/campaign_config.json
  - {name: run_state.json, bytes: 2713}                                # ws/forge_experiments/run_state.json
  - {name: forge.patch, bytes: 2868}                                   # ws/forge_experiments/candidates/iter_001/change.diff
  - {name: meta.json, bytes: 6468}                                     # ws/forge_experiments/candidates/iter_001/meta.json
  - {name: softmax_kernel.py, bytes: 2065}                             # ws/forge_experiments/candidates/iter_001/softmax_kernel.py
  - {name: validation.txt, bytes: 76}                                  # ws/forge_experiments/candidates/iter_001/validation.txt
  - {name: context.json, bytes: 11978}                                 # ws/forge_experiments/orchestration/iter_001/context.json
  - {name: dispatch.json, bytes: 7210}                                 # ws/forge_experiments/orchestration/iter_001/dispatch.json
  - {name: lane_plans.json, bytes: 75}                                 # ws/forge_experiments/orchestration/iter_001/lane_plans.json
  - {name: optimization_plan.md, bytes: 8385}                          # ws/forge_experiments/orchestration/iter_001/optimization_plan.md
  - {name: optimization_plan.reasoning.md, bytes: 2160}                # ws/forge_experiments/orchestration/iter_001/optimization_plan.reasoning.md
  - {name: specialists.json, bytes: 17904}                             # ws/forge_experiments/orchestration/iter_001/specialists.json
  - {name: structured_output.json, bytes: 8656}                        # ws/forge_experiments/orchestration/iter_001/structured_output.json
  - {name: benchmark.json, bytes: 4876}                                # ws/forge_experiments/best/iter_001/benchmark.json
  - {name: softmax_kernel.py, bytes: 2065}                             # ws/forge_experiments/best/iter_001/files/softmax_kernel.py
  - {name: forge.patch, bytes: 2868}                                   # ws/forge_experiments/best/iter_001/forge.patch
  - {name: publication.json, bytes: 1032}                              # ws/forge_experiments/best/iter_001/publication.json
  - {name: validation.txt, bytes: 76}                                  # ws/forge_experiments/best/iter_001/validation.txt
  - {name: iter_001.json, bytes: 1012}                                 # ws/forge_experiments/handoffs/iter_001.json
  - {name: iter_001.md, bytes: 2538}                                   # ws/forge_experiments/lessons/iter_001.md
  - {name: 6579b9c8-37c3-48d0-8c87-1ff52a7a5458.jsonl, bytes: 572946}  # .claude/projects/<run-root>-ws/6579b9c8-37c3-48d0-8c87-1ff52a7a5458.jsonl

Known limitations

  • hyperloom-kb-collect keys raw logs by SHA-256, so files with identical bytes are recorded under one name: above, candidates/iter_001/change.diff appears as a second forge.patch.
  • Plain runs link only the Implementer's transcript, not the planners'.
  • Plain runs record no kernelforge_commit in provenance and no identity.operator; their logs hold neither.
  • KernelForge writes naive local times, which collection reads as UTC: right in a UTC container, off by the host's offset elsewhere.

  • Tests: new test_run_export.py and hyperloom_kb/tests/test_collect_kernelforge.py.
  • Size/complexity triggers crossed: none
  • Observable effect: see What changes.
  • Breaking changes: no
  • PR addresses single concern: yes

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Code metrics gate: PASSED

Judged: the 15 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension Limit New Worse Baseline out of date Baseline entries vs base
Cyclomatic complexity fails > 20 0 0 0 68 +0
Cognitive complexity fails > 30 0 0 0 274 +0
Function length (lines) fails > 80 0 0 0 412 +0
Nested block depth fails > 5 0 0 0 25 +0
Parameter count fails > 5 0 0 0 95 +0
Module length (lines) warns > 800, fails > 1200 0 0 0 51 +0
Duplicated lines per file fails > 0 0 0 0 97 +0
Dead code (vulture) fails > 0 0 0 0 2 +0
Check Findings Effect
Comments 0
English only 0
Production code imports test code 0
Repeated literals 0
A1 Private name of another package 0
A8 Flag argument 0
A9 Tuple return 0
A10 Raw vocabulary comparison 0
A11 Hidden global write 0
A12 Pure forwarder 0
A12 Single-implementation abstraction 0
105 findings in files this change does not touch

Outside this change (informational): 105

Unit Dimension Baseline Now
src/hyperloom/orchestrator/enablement/recipe/projections.py project_launch_evidence Cognitive complexity absent 36
src/hyperloom/orchestrator/state/shared_state.py resolve_graded_comparison Cognitive complexity absent 34
src/hyperloom_kb/collect/mapping.py _experience Cognitive complexity absent 31
src/hyperloom_kb/schema.py FieldDeclaration.__post_init__ Cognitive complexity absent 32
src/kernelforge/cli/forge_loop_command.py _prepare_measurement_driver Cognitive complexity absent 36
src/kernelforge/cli/forge_loop_command.py _read_kb_warm_start Cognitive complexity absent 40
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Cognitive complexity absent 44
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Cognitive complexity absent 38
src/hyperloom/inference_optimizer/cli/__init__.py _reexport_session_environment Function length (lines) absent 111
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_agent_lineup Function length (lines) absent 89
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_run_topology Function length (lines) absent 92
src/hyperloom/inference_optimizer/cli/__init__.py _resume_session Function length (lines) absent 170
src/hyperloom/inference_optimizer/cli/__init__.py _seed_fresh_session Function length (lines) absent 170
src/hyperloom/inference_optimizer/multi_node/cli.py _restart_multinode Function length (lines) absent 107
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _build_parser Function length (lines) absent 84
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _spawn_rank_actors Function length (lines) absent 84
src/hyperloom_kb/sync.py GlobalSync.pull Function length (lines) absent 84
src/kernelforge/cli/forge_loop_command.py _RunReporter.build_result Function length (lines) absent 91
src/kernelforge/cli/forge_loop_command.py _prepare_measurement_driver Function length (lines) absent 136
src/kernelforge/cli/forge_loop_command.py _read_kb_warm_start Function length (lines) absent 149
src/kernelforge/loop/runner.py IterationLoop._close_iteration Function length (lines) absent 117
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Function length (lines) absent 184
src/kernelforge/loop/runner.py IterationLoop._implementer_history Function length (lines) absent 112
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Function length (lines) absent 102
src/kernelforge/loop/runner.py IterationLoop._source_attempt Function length (lines) absent 169

...and 80 more; the job log lists every row.

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.

Comment thread src/kernelforge/tests/test_run_export_arena.py Fixed
Comment thread src/kernelforge/tests/test_run_export_arena.py Fixed
@amd-ama10002-2 amd-ama10002-2 changed the title feat(kernelforge): turn KernelForge runs into Experiences that carry the agents' own Summary lines feat(kernelforge): collect plain forge-loop runs as Experiences with export-runs and kernelforge-v1 Oct 16, 2026
@amd-ama10002-2
amd-ama10002-2 changed the base branch from main to feat/kernelforge-summary-lines October 16, 2026 20:53
@amd-ama10002-2
amd-ama10002-2 force-pushed the feat/kernelforge-run-experiences branch from ca8cde8 to 05be3a7 Compare October 16, 2026 20:53
@amd-ama10002-2 amd-ama10002-2 added the run-forge-e2e Run the single-GPU Forge e2e smoke on this PR (again on every new commit while set) label Oct 16, 2026
@amd-ama10002-2
amd-ama10002-2 requested a balanced review from Copilot October 16, 2026 23:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Evidence association, baseline accuracy, historical transcript support, and read-failure isolation need correction before collection is reliable.

5 open issues
What changed in this PR

Adds collection of plain KernelForge runs into Hyperloom’s Experience knowledge base, building on the summaries and reasoning logs introduced in #1949.

Changes:

  • Adds kernelforge export-runs to normalize archived candidates and supporting logs.
  • Packages the kernelforge-run-v1 declaration and kernelforge-v1 mapping.
  • Adds exporter and collection tests, plus usage documentation.
File Description
src/​kernelforge/​tests/​test_run_export.py Tests export behavior and collection integration.
src/​kernelforge/​run_export/​units.py Defines shared run and unit structures.
src/​kernelforge/​run_export/​forge_loop.py Converts plain forge-loop archives into units.
src/​kernelforge/​run_export/​files.py Inventories files and provides bounded readers.
src/​kernelforge/​run_export/​document.py Combines runs into an export document.
src/​kernelforge/​run_export/​command.py Implements the export CLI command.
src/​kernelforge/​run_export/​archive.py Extracts artifacts, summaries, and transcript evidence.
src/​kernelforge/​run_export/​__init__.py Introduces the exporter package.
src/​kernelforge/​cli/​__init__.py Registers export-runs.
src/​hyperloom_kb/​tests/​test_collect_kernelforge.py Tests mapping projection and skip conditions.
src/​hyperloom_kb/​mappings/​kernelforge-v1.yaml Maps exported units into Experiences.
src/​hyperloom_kb/​declarations/​kernelforge-run-v1.yaml Defines the KernelForge Experience schema.
docs/​reference/​experience-kb-collect.md Documents KernelForge collection.
docs/​kernelforge/​reference/​cli.md Documents exporter arguments and behavior.
docs/​kernelforge/​how-to/​autonomous-loop.md Links summaries to Experience export.

Review effort: Balanced

Comment thread src/kernelforge/run_export/archive.py Outdated
Comment thread src/kernelforge/run_export/forge_loop.py Outdated
Comment thread src/kernelforge/run_export/archive.py
Comment thread src/kernelforge/run_export/forge_loop.py
Comment thread src/kernelforge/run_export/forge_loop.py Outdated
@amd-ama10002-2

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

Automated review started for head 05be3a7. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/38980828098

@github-actions

Copy link
Copy Markdown

Automated review: no verdict applied -- the review job ended with result 'failure'. Comment /review to retry. Run: https://github.com/AMD-AGI/Hyperloom/actions/runs/38980828098

@amd-ama10002-2
amd-ama10002-2 force-pushed the feat/kernelforge-run-experiences branch from 05be3a7 to 924313c Compare October 16, 2026 23:34
@amd-ama10002-2

amd-ama10002-2 commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator Author

This is the result of running the agent review-pr skill locally (the /review skill I think)

PR #1944 -- feat(kernelforge): collect plain forge-loop runs as Experiences with export-runs and kernelforge-v1

What it does: adds kernelforge export-runs, which reads what a plain forge-loop run leaves behind into one JSON document: archived candidates, plans, reasoning logs, loop logs and Claude Code transcripts. It also adds the packaged kernelforge-run-v1 declaration and kernelforge-v1 mapping, with which hyperloom-kb-collect turns that document into Experiences. Later commits make three changes:

  • baseline.source_ms comes from the run's pristine baseline;
  • each loop log and transcript is tied to the run whose workspace path it fits best, and a tie stops the export;
  • the export stops and names the path when it cannot list a directory or read a file under the log roots. Records the export can read but finds missing or malformed still become entries carrying their reason: a run's records, or a candidate's meta.json.

The docs, the declaration and the mapping state that a plain run made before #1949 links no transcript.

Blocking issues: none

SKIPPED: V4 -- # PARTIAL: #1032 changes 1075 files, only its first 100 were compared; overlap there unchecked
SKIPPED: V4 -- # PARTIAL: #1154 changes 1232 files, only its first 100 were compared; overlap there unchecked
SKIPPED: V4 -- # PARTIAL: #1276 changes 1311 files, only its first 100 were compared; overlap there unchecked
SKIPPED: V4 -- # PARTIAL: #1675 changes 161 files, only its first 100 were compared; overlap there unchecked
SKIPPED: V4 -- # PARTIAL: #1869 changes 334 files, only its first 100 were compared; overlap there unchecked
SKIPPED: V4 -- # PARTIAL: #1922 changes 130 files, only its first 100 were compared; overlap there unchecked

Checked: src/kernelforge/run_export/{files,archive,forge_loop,units,document,command}.py, src/kernelforge/cli/init.py, the declaration and mapping, docs/kernelforge/reference/cli.md, docs/reference/experience-kb-collect.md, and the description against the head; tree-wide sweep for every non-test writer of optimization_plan_path, merge_attempt_staged, kernel_path, campaign_id, pristine_baseline_wall_ms, MERGE_PLAN_PREFIX and reasoning_log_section, and for the mkdtemp callers that create 0700 directories (src/kernelforge, excluding tests and run_export); tree-wide importers of read_json, inventory and _files_in (src, scripts) | Ran: fetch.sh's queries through a copy adapted to gh 2.4.0; ruff C901 (max 20) and PLR0913 (max 5) on the 8 changed modules, both clean; through normalize and the export-runs CLI, a plain run made before #1949 (recorded rationale, no transcript), an unreadable handoff, an unlistable transcript directory, and an unreadable events.jsonl (each exits 1 naming the path), plus a missing events.jsonl (exports); test_run_export.py, test_collect_kernelforge.py, test_collect.py; each fix commit's new cases against its parent, where they fail; the declaration's schema_ref at this head and at #1950's head; an independent second reader at 85cdf56 and again at this head; check runs at the head, all green | Base: d32992a | Head: e5ea4c9

LGTM

@amd-ama10002-2

amd-ama10002-2 commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator Author

This is the result of running the agent review-pr skill locally (the /review skill I think)

PR #1944 -- feat(kernelforge): collect plain forge-loop runs as Experiences with export-runs and kernelforge-v1

What it does: adds kernelforge export-runs, which reads what a plain forge-loop run leaves behind (archived candidates, plans, reasoning logs, events and loop logs, Claude Code transcripts) into one JSON document, and the packaged kernelforge-run-v1 declaration and kernelforge-v1 mapping with which hyperloom-kb-collect turns that document into Experiences. The export takes baseline.source_ms from the run's pristine baseline. It ties each log and transcript to the run whose workspace path it fits best, and stops naming the path when that is a tie, or when it cannot list a directory or read a file under the log roots. A plain run made before #1949 links no transcript, which the docs, the declaration and the collect report's skip reason all say.

Blocking issues: none

SKIPPED: V4 -- # PARTIAL: #1032 changes 1075 files, only its first 100 were compared; overlap there unchecked
SKIPPED: V4 -- # PARTIAL: #1154 changes 1232 files, only its first 100 were compared; overlap there unchecked
SKIPPED: V4 -- # PARTIAL: #1276 changes 1311 files, only its first 100 were compared; overlap there unchecked
SKIPPED: V4 -- # PARTIAL: #1675 changes 161 files, only its first 100 were compared; overlap there unchecked
SKIPPED: V4 -- # PARTIAL: #1869 changes 334 files, only its first 100 were compared; overlap there unchecked
SKIPPED: V4 -- # PARTIAL: #1922 changes 130 files, only its first 100 were compared; overlap there unchecked

Checked: src/kernelforge/run_export/{files,archive,forge_loop,units,document,command}.py, src/kernelforge/cli/init.py, the declaration and mapping, both reference pages, and the description against the head (its embedded declaration is identical to the file and its schema_ref matches the one recomputed with load_mapping); tree-wide sweeps for every non-test writer of the run values the export reads, every copy of the transcript claim, and every caller of read_json, inventory and _files_in | Ran: fetch.sh's queries through a copy adapted to gh 2.4.0; ruff C901 (max 20) and PLR0913 (max 5) on the 8 changed modules, both clean; through normalize, the export-runs CLI and a collect dry run: a plain run made before #1949, an unreadable handoff, an unlistable transcript directory, and a pre-#1949 lane with its transcript present; each new test case against its parent commit (candidate record, transcript directory and best bundle fail on 7a1171a, events log on 2aa48b7; all pass here); pytest test_run_export.py, test_collect_kernelforge.py and test_collect.py (87 passed); CI at e5ea4c9: 33 checks succeeded, including all 12 test shards and both coverage jobs (code-metrics and pr-hygiene succeeded on the rerun that cancelled their first runs) | Base: d32992a | Head: e5ea4c9

LGTM

@amd-ama10002-2
amd-ama10002-2 marked this pull request as ready for review October 17, 2026 02:07
amd-ama10002-2 and others added 3 commits October 19, 2026 17:28
…-runs and kernelforge-v1

Experiences built from KernelForge logs carried tool narration as their
reasoning ("I'll start by reading the required optimization plan...") and
three long sentences cut from a 25 KB plan as their plan, and the
normalizer that produced them lived outside the repository.

`kernelforge export-runs LOG_ROOT... -o runs.json` writes every run found
under the given directories as one document for
`hyperloom-kb-collect --mapping kernelforge-v1`. It reads a plain
forge-loop run's forge_experiments/ directory, which yields one unit per
archived candidate: run_id from run_state.json's campaign_id, times from
events.jsonl, identity from campaign_config.json, the Implementer's
backend and model and kept candidates' KB records from forge-loop.log, and
transcripts linked by the reasoning log their first prompt names.

rationale.plan is the plan's Summary line and rationale.reasoning the
Summary line of the implementer's reasoning log, each cut to three
sentences; logs without them, as every run made before KernelForge asked
for them, keep the earlier extraction. The packaged kernelforge-run-v1
declaration has an optional rationale.reasoning_log file field holding
the whole log, and the packaged kernelforge-v1 mapping (producer version
10) projects the document onto it. Both are complete as they land: the
AgentKernelArena layouts export-runs reads next add no field to either.

Observable effect: a new `kernelforge export-runs` command, a packaged
kernelforge-run-v1 declaration and a packaged kernelforge-v1 mapping for
hyperloom-kb-collect. Experiences collected through them from plain
forge-loop runs state the plan and the implementer's change in the
agents' own one-to-three-sentence summaries and link the reasoning log as
a file.

Co-authored-by: Cursor <cursoragent@cursor.com>
… each run's own evidence

Three problems in how export-runs turns a plain forge-loop run into units.

source_ms came from each candidate's meta.json, whose baseline_wall_ms is
the time of the kernel the search started from. A run given
--baseline-json scores every candidate against that file's times instead,
which the run state keeps as pristine_baseline_wall_ms, so its units
paired speedups against one kernel with the time of another. source_ms
now takes the run state's pristine baseline and falls back to meta.json
only when the run state records none; the declaration's source_ms
description says so.

A forge-loop log went to every run whose workspace held the log's kernel
file under a directory of the same name, and a transcript to every run
whose workspace name and iteration matched the reasoning log it names, so
two runs called ws under one log root shared each other's model, KB
record, PLAN headline, reasoning and raw logs. Each log and transcript
now goes to the one run whose workspace path shares the most trailing
components with the path it names, which still finds a run made in a
container, and the export stops, naming the file and both runs, when two
runs fit equally well.

An unreadable file already stopped the export, with a traceback. It now
stops with a message naming the file, and suggests fixing ownership when
a container left it root-owned. An unreadable .log is no longer skipped
in silence, which dropped its run's provenance.

Observable effect: source_ms of a run given --baseline-json is the
original kernel's time; logs and transcripts are no longer shared between
runs whose workspaces have one name; an export that cannot read a file,
or cannot tell which run a log or transcript belongs to, exits 1 with
"Error: ..." naming it and writes nothing.

Co-authored-by: Cursor <cursoragent@cursor.com>
os.walk without onerror, and Path.rglob, return nothing for a directory the
user cannot list. A run copied out of a container, where Claude Code keeps
each project's transcripts in a directory only root can list and KernelForge
creates its best bundles with mkdtemp, therefore lost those files while the
export still exited 0. inventory now raises the listing error, which
export-runs reports with its fix-ownership hint, and _files_in lists through
inventory instead of a second walker.

The declaration and both reference pages also said that a plain run made
before #1949 falls back to its Implementer's transcript. A plain session's
transcript is linked only by the reasoning log its first prompt names, so such
a run gives the rationale KernelForge recorded and its lanes give no
reasoning; the text now says so. The reasoning_file description changes, and
with it the declaration's schema_ref.

Co-authored-by: Cursor <cursoragent@cursor.com>
amd-ama10002-2 and others added 3 commits October 19, 2026 17:28
read_json returned None for a JSON file it could not open, so an unreadable
meta.json became a "not a readable JSON object" entry while every other file
the export cannot read stops it with the fix-ownership hint. A missing or
malformed record is still one entry saying why; a record that is there but
cannot be read now raises like any other file. The export-runs reference says
which is which.
…soning

The mapping's no-reasoning skip reason said the logs hold no Claude Code
transcript of the agent and asked for logs that record its reasoning. A lane
of a plain run made before #1949 can have its transcript in the logs, but
nothing links it, so every such lane got a wrong diagnosis and a remedy that
cannot help, contradicting the collection reference. The reason now says no
transcript is linked, and that such a lane has no reasoning, like a merge.

Co-authored-by: Cursor <cursoragent@cursor.com>
An events.jsonl the export could not open still became one entry for the
whole run, while every other file it cannot read stops it, so every
candidate of that run was skipped behind a PermissionError that the collect
report showed only as a skip reason. The export now stops with the path and
the fix-ownership hint; a missing events log still leaves the times to
meta.json.

The export-runs reference now says that a run is a forge_experiments
directory holding campaign_config.json and run_state.json, and which records
become entries: a run's records that are not the JSON KernelForge writes, and
a candidate's missing or malformed meta.json.
@amd-ama10002-2

Copy link
Copy Markdown
Collaborator Author

Superseded by #2037: the four stacked PRs were combined into one PR at the reviewer's request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-forge-e2e Run the single-GPU Forge e2e smoke on this PR (again on every new commit while set)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants