Skip to content

refactor(knowledge): build the recipe journal and KB writes from named parts - #2011

Merged
xiaofei-zheng merged 33 commits into
mainfrom
refactor/chenyi/cc20-knowledge
Oct 22, 2026
Merged

xiaofei-zheng merged 33 commits into
mainfrom
refactor/chenyi/cc20-knowledge

Conversation

@chennyiiis

@chennyiiis chennyiiis commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor
  • Description: what and why

src/hyperloom/orchestrator/knowledge/recipe_journal.py carried twelve entries in scripts/code_metrics_baseline.txt: eleven cognitive-complexity and function-length exemptions across seven functions, plus the module-line count. The weight was in boolean operands and ternaries — long chains of str(x or "") defaulting and override if the caller set it else the live value — not in branching, so this names those idioms once instead of restructuring control flow. One entry remains, the module-line count, lowered to match.

What the module no longer does twice:

  • the framework / version / precision identity was derived in both workload_canonical_id and kb_amend_recipe; _framework_identity derives it once;
  • the amend repeated "the override if the caller set it, else the live value" for eleven fields; PRESERVED_RECIPE_FIELDS states each field with the empty value put_recipe falls back to, and amended_recipe_fields reads it;
  • _extra_server_args reimplemented to_str_list's isinstance ladder;
  • four guarded float(x or 0.0) coercions were written out inline. recorded_float in hyperloom.common.coerce is that form named once, and it is not to_float: the two disagree on inf, nan and True, where to_float returns the default. recorded_float has three call sites, all throughput comparisons (recipe_journal.py:1079, :1209, :1210). TestRecordedFloat pins the three inputs they disagree on so a later swap is not silent.

Two groups that never needed the collaborator move to sibling modules, because the module was already at its baselined length:

  • _recipe_attrs.py (242 lines) — the recipe-shaped projections of SharedState: the workload tags, the attempt provenance, and the recipe attrs themselves;
  • _config_timeline.py (169 lines) — the configuration arm's attempt rows on the FRAMEWORK_AGENT event.

Both depend only downward (hyperloom.common, hyperloom.inference_optimizer, the knowledge package's own workload_tags) and neither imports recipe_journal, so the split adds no cycle.

recipe_journal.py goes 1671 → 1269 lines; the three production modules total 1680 against main's 1671, and with coerce.py's new primitive the production net is +22 lines.

Upstream work this branch merges in and carries through the split unchanged: #1985's benched=True on the configuration arm's attempt row, now set in _config_timeline._attempt_fields; and #1986's propagation of recipe-store read and write failures, including the narrow LocalRecipeStoreError / OSError / ValueError handler on the per-variant KB write. The three broad handlers #1986 deleted are absent here.

  • Linked issue(s): close/fix refs

None.

  • Tests: added/updated? commands run?

workload_kb_tags, attempt_provenance and recipe_attrs_from_state were private helpers on the collaborator and are now module functions in _recipe_attrs, re-exported through recipe_journal.__all__. test_coordinator_kb_writes.py, test_coordinator_sync_helpers_coverage_unit.py and test_recipe_sedimentation.py call them there, with every assertion carried across unchanged. test_remote_read_is_wired_into_cli_t0_and_close_write_remains pins that the close path still publishes through HyperloomRemoteKB.from_env().write; it now inspects _publish_remote_recipe, where that call lives. TestRecordedFloat in test_coerce.py is new. #1986's own tests run unchanged against this tree. Full CI.

  • Size/complexity triggers crossed: split, or why the shape is right

This PR is the split. Seven functions were over the complexity or length trigger; the code-metrics gate reports 0 new and 0 worse across the ten touched files. Three record types are added: AttemptProvenance replaces a bare three-element tuple returned and unpacked across modules, and _FrameworkIdentity and _VariantMeasurement each replace a group of locals threaded into one call.

  • If this simplifies or refactors: which mechanism was removed, which contract was preserved, and the tests that show it

Removed: the duplicate identity derivation, the eleven-way field ternary, the inline extra_server_args ladder and the four inline float coercions, and the five single-use units an earlier revision of this branch had added. Preserved: the put_recipe keyword set, the JournalEntry field set and the 24 record_attempt keys are identical to main's, which the tests above exercise.

  • Observable effect: what an operator will now see

None. Every journal row, KB lesson, KB pitfall, recipe amend and framework-event row carries the same fields with the same values, and no log line's wording, level or presence changes. The fields _config_timeline now passes to record_attempt without a caller-side str(x or "") are coerced identically by that recorder's own _ATTEMPT_FIELDS / _ATTEMPT_BLOCKS projections.

  • Breaking changes: no

  • PR addresses single concern: yes

  • Root cause is upstream: no

Test and others added 3 commits October 19, 2026 17:02
…d parts

recipe_journal.py carried its whole job in a handful of very large
functions: the session's close path, the Recipe KB amend, the
per-variant fact write and the configuration arm's timeline rows were
each one long body threading dozens of defaulting expressions together.

Two coherent groups move to sibling modules:

- `_recipe_attrs.py` owns the recipe-shaped view of SharedState: the
  workload tags, the KEEP'd kernel optimizations joined to their
  end-to-end verdict, the attempt provenance and the recipe attrs
  themselves. These are pure projections of settled state and never
  needed the collaborator.
- `_config_timeline.py` owns the configuration arm's run and attempt
  rows on the FRAMEWORK_AGENT event.

What stays is split along the jobs it already did: the close path reads
as its guards, the remote publish and the local amend, and the amend
builds its `put_recipe` arguments from one table of the fields a recipe
carries instead of repeating "the override if the caller set it, else
the live value" for each of eleven fields.

Three duplicated mechanisms go with it. The framework, version and
precision identity was derived twice; the store's authority row was read
through three near-identical `try`/`except` blocks; and the integrate
ledger's float coercion was written out three times, now `to_float`.

`_collect_workload_tags`, `_build_kernel_optimizations_from_state`,
`_collect_attempt_provenance` and `_build_recipe_attrs_from_state` were
private helpers on the collaborator and are now module functions in
`_recipe_attrs`; their tests call them there, with every assertion
carried across unchanged. `test_remote_read_is_wired_into_cli_t0_and_
close_write_remains` now inspects `_publish_remote_recipe`, which is
where the close path's `HyperloomRemoteKB.from_env().write` lives.

No observable change: internal structure only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…knowledge

# Conflicts:
#	scripts/code_metrics_baseline.txt
#	src/hyperloom/inference_optimizer/tests/test_coordinator_sync_helpers_coverage_unit.py
#	src/hyperloom/orchestrator/knowledge/recipe_journal.py
@chennyiiis
chennyiiis requested a review from a team as a code owner October 19, 2026 10:32
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Code metrics gate: PASSED

Judged: the 10 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension Limit New Worse Baseline out of date Baseline entries vs base
Cyclomatic complexity fails > 20 0 0 0 11 +0
Cognitive complexity fails > 30 0 0 0 69 -6
Function length (lines) fails > 80 0 0 0 199 -5
Nested block depth fails > 5 0 0 0 8 +0
Parameter count fails > 5 0 0 0 83 +0
Module length (lines) warns > 800, fails > 1200 0 0 0 47 +0
Duplicated lines per file fails > 0 0 0 0 95 +0
Dead code (vulture) fails > 0 0 0 0 2 +0
Check Findings Effect
Comments 0
English only 0
Production code imports test code 0
Repeated literals 0
A1 Private name of another package 0
A8 Flag argument 0
A9 Tuple return 0
A10 Raw vocabulary comparison 0
A11 Hidden global write 0
A12 Pure forwarder 0
A12 Single-implementation abstraction 0
59 findings in files this change does not touch

Outside this change (informational): 59

Unit Dimension Baseline Now
src/hyperloom_kb/collect/mapping.py _experience Cognitive complexity absent 31
src/kernelforge/cli/forge_loop_command.py forge_loop Cognitive complexity absent 31
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Cognitive complexity absent 44
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Cognitive complexity absent 38
src/hyperloom/orchestrator/phases/kernel.py KernelPhase._record_geak_delegation_timeline Parameter count absent 7
src/hyperloom/inference_optimizer/cli/__init__.py _reexport_session_environment Function length (lines) absent 107
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_agent_lineup Function length (lines) absent 89
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_run_topology Function length (lines) absent 92
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _build_parser Function length (lines) absent 84
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _spawn_rank_actors Function length (lines) absent 84
src/hyperloom_kb/sync.py GlobalSync.pull Function length (lines) absent 84
src/kernelforge/loop/runner.py IterationLoop._close_iteration Function length (lines) absent 117
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Function length (lines) absent 185
src/kernelforge/loop/runner.py IterationLoop._implementer_history Function length (lines) absent 112
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Function length (lines) absent 102
src/kernelforge/loop/runner.py IterationLoop._source_attempt Function length (lines) absent 175
src/hyperloom_kb/http_service.py Module length (lines) absent 1525
src/kernelforge/cli/forge_loop_command.py Module length (lines) absent 2003
src/hyperloom/orchestrator/actions/executors/_kv_metrics.py port_from_server_log Nested block depth absent 6
src/hyperloom/orchestrator/specialists/dispatch.py SpecialistDispatchCollaborator.warm_specialist_params Cognitive complexity 93 94
src/hyperloom/agents/quantization/driver/retry.py quantize_via_prompt Function length (lines) 102 103
src/hyperloom/inference_optimizer/breakdown/exporter.py build Function length (lines) 132 136
src/hyperloom/inference_optimizer/cli/__init__.py _run_optimize Function length (lines) 273 279
src/hyperloom/inference_optimizer/cli/preflight.py _preflight Function length (lines) 203 208
src/hyperloom/inference_optimizer/multi_node/scripts/kernel_patch_multinode.py _do_apply Function length (lines) 98 99

...and 34 more; the job log lists every row.

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.

@chennyiiis chennyiiis added the size-exception Waives the PR diff budget; added by a maintainer with a reason in the PR description label Oct 19, 2026
…w in its own module

The read-modify-write an amend performs is one job: the caller's overrides
win, each field it left alone keeps the value the live row carries, and the
extras another session wrote survive untouched. That rule, its table of the
fields a recipe carries and the row-supersession it shares with the close
path now live in `_recipe_amend.py` rather than among the journal's own
helpers.

No observable change: internal structure only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@chaojhou chaojhou left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #2011 -- refactor(knowledge): build the recipe journal and KB writes from named parts

What it does: PR #2011 splits the over-complex functions in knowledge/recipe_journal.py (finalize_recipe_and_journal, kb_amend_recipe, _record_config_attempts, the recipe-attrs/workload-tag/kernel-opt/provenance projections, the explore-grid loop in fact_write_hook, _record_fact_per_variant) into named helpers and three new sibling modules: _recipe_attrs.py, _config_timeline.py and _recipe_amend.py. It also folds the three authority-row reads into one _authoritative_recipe_row helper. I ran a differential harness over 600 random states and found the extraction behaviour-equivalent apart from log text, error-dict key order and to_float edge cases. The PR still cannot merge: it conflicts semantically with main, because #1986 removed the swallow handlers that this PR consolidates and #1985 added benched=True to a function this PR moved.

Blocking issues: 2

  1. [conflict-main-1986-1985] CONFLICTING with main: taking the PR side reverts #1986's no-swallow fix and silently drops #1985's benched=True [verified]
    Problem: The PR's mergeable state is CONFLICTING (3 files), and the conflict is semantic. Main's 84bcdfb (#1986) deleted the broad except handlers around get_authoritative_recipe and put_recipe in read_local_recipe_row, kb_amend_recipe and the finalize local path. This PR collects those same handlers into a new swallowing helper, _authoritative_recipe_row (except Exception -> {} with an INFO log), and keeps the put_recipe try/except. Main's 00422ae (#1985) added benched=True to the config-arm record_attempt call, which this PR moved to _config_timeline.py. src/hyperloom/orchestrator/knowledge/recipe_journal.py:1229
    Impact: Recipe store read and put failures are swallowed again, which undoes #1986 and fails its new propagation tests. LocalRecipeStoreError is no longer imported while _record_fact_impl's merged handler still names it, so the first store failure raises NameError. Config-arm attempts lose benched=True, which the #1985 variant funnel counts on (framework_event.py iterates adopted/attribution_eligible/benched). Nothing flags this because _config_timeline.py does not conflict.
    Action: Rebase onto main. Remove the try/except from _authoritative_recipe_row (or inline the plain read), drop the put_recipe try/except in kb_amend_recipe, keep the LocalRecipeStoreError import, add benched=True in _config_timeline._attempt_fields, regenerate code_metrics_baseline.txt from main's numbers, then update the description, which still claims three try/except reads were unified.

  2. [desc-unstated-log-and-coercion] Description claims a single log-line difference; other operator-visible changes are unstated [verified]
    Problem: The description says the only behavioural difference is a new INFO log in the KEEP path's read_local_recipe_row. The differential harness found more than that. The INFO messages operators may grep for, 'kb_amend_recipe: authority get_recipe failed (...); proceeding with empty live' and 'recipe read failed (...); finalize appends the current session only', are both replaced by 'recipe authority read failed for cid=... (...); proceeding without it'. The remote-finalize error outcome dict changes key order. The integrate-ledger float coercion now goes through to_float (_recipe_attrs.py:178/191/208, _recipe_amend.py:83), which maps bool and non-finite values to 0.0 where base kept them. The description also says two modules are added and the file goes from 1785 to 1485 lines; at head it adds three modules (_recipe_amend.py too) and goes from 1800 to 1376 lines. It also names a remaining _record_fact_impl baseline entry that does not exist at head. src/hyperloom/orchestrator/knowledge/recipe_journal.py:1238
    Impact: An operator grepping the old log strings stops matching. The persisted micro_speedup, e2e_gain_pct and e2e_tput values differ for non-finite inputs. The description calls the change 'same fields with the same values' and so misstates it.
    Action: Restate the Observable effect section: list the changed log texts and the to_float normalisation, and fix the module, line and baseline counts. After the rebase onto main, most of these log lines disappear (see the conflict finding), so rewrite the description after resolving it.

Checked: RecipeJournalCollaborator.fact_write_hook (+ _record_grid_facts, _grid_attempt_row, _result_payload) (yes); _record_config_run / _record_config_attempts -> _config_timeline.record_config_run / record_config_attempts (yes); RecipeJournalCollaborator._record_fact_per_variant (+ _variant_verdict/_variant_measurement/_variant_failure/_variant_ch (yes); _collect_workload_tags -> _recipe_attrs.workload_kb_tags (yes); _build_kernel_optimizations_from_state -> _recipe_attrs.kept_kernel_optimizations (no); _collect_attempt_provenance -> _recipe_attrs.attempt_provenance (yes); _build_recipe_attrs_from_state -> _recipe_attrs.recipe_attrs_from_state (yes); finalize_recipe_and_journal (+ _unvalidated_stack_outcome, _finalize_journal, _publish_remote_recipe, _final_recipe_over (unsure); kb_amend_recipe (+ _amends_local_recipe, _framework_identity, _recipe_amend.amended_recipe_fields/amended_recipe_extras/ (yes); read_local_recipe_row (no); workload_canonical_id (yes); improved_best_config (from finalize local path) (no) | Conflict with main semantic | Base: ef93838 | Head: 20d0823

…knowledge

# Conflicts:
#	scripts/code_metrics_baseline.txt
#	src/hyperloom/inference_optimizer/tests/test_coordinator_kb_writes.py
#	src/hyperloom/orchestrator/knowledge/recipe_journal.py
@ZhengGong-amd

Copy link
Copy Markdown
Collaborator

Requesting changes: resolving the conflict with main as written silently reverts two merged fixes, the description does not match the diff, and production code grows by 23%.

Production code: +1363/−944 (net +419), functions 31 → 83, 7 new classes. Most of the growth wraps redundant normalization in named helpers. Bring this to net-zero or negative by deleting those mechanisms, not by relocating them.

Blocking

  1. Semantic conflict with main. Rebase and fix all three:
  2. The description does not match the diff. It omits _recipe_amend.py, gives the line counts as 1785→1485 (actual 1800→1376), lists the wrong remaining baseline entry, and omits the merged log text. It also omits to_float now turning True/nan/inf into 0.0 in the persisted micro_speedup/e2e_gain_pct/e2e_tput (_recipe_attrs.py:178/191/208, _recipe_amend.py:83). Rewrite the Observable effect to cover these.

Simplify instead of relocate

  • Delete the caller-side str(x or "") normalization, the six _attempt_* builders and _ConfigAttemptContext (_config_timeline.py:38-187). framework_event.record_attempt already normalizes every field. One ~50-line record_config_attempts is enough. Restore its dropped "timeline only, outcome recorded verbatim" docstring invariant (:245).
  • Delete _IntegrateIndex, _E2eVerdict, AttemptProvenance.unrecorded and the single-use literal constants in _recipe_attrs.py. Keep only workload_kb_tags and recipe_attrs_from_state.
  • Do not mint new names for rules refactor(knowledge): match warm-start candidates in a module of their own #1959 also renames. _RECIPE_OWNED_KEYS, _runtime_version_tags, _authoritative_recipe_row and _framework_identity each belong to one owner (recipe_kb/schema.py, workload_tags.py, recipe_kb).

Test and others added 3 commits October 20, 2026 10:46
…r the merge

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two wrappers whose whole body was a dict literal, one whose body was two
recorder calls, a NamedTuple built in one place and unpacked in the next, and a
classmethod constructor with one caller. Each is inlined at its only call site;
no function crosses the complexity ceiling as a result.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chennyiiis

Copy link
Copy Markdown
Contributor Author

Thanks — all three points addressed at 6658097f.

1. The merge silently reverted two merged fixes. Correct, and the resolution was wrong in both directions. origin/main is merged in and verified by behaviour, not by ancestry:

2. The description does not match the diff. Rewritten against the current head. It had said two new modules when there are three, carried line and complexity numbers from an earlier push, and claimed "no observable change" while two things had changed. Both are now stated and both are reverted:

  • the reworded authority-read log line no longer exists — fix: stop swallowing failures in 43 non-boundary broad handlers #1986 deleted the handler it lived in;
  • inf had begun rendering as 0.0 in micro_speedup, e2e_gain_pct, e2e_tput and the live-row throughput comparison, because the extracted helpers reached for to_float, which rejects non-finite values. That is not an honest rendering: 0.0 is the same value a never-measured field carries, so it hides a broken benchmark behind a plausible number, and it would silently unfreeze a recipe row whose stored best_throughput is inf. hyperloom.common.coerce.recorded_float restores the merge base's behaviour and TestRecordedFloat pins it.

3. Production code grows by 23%. Fair, and now 21% after deleting five single-use units rather than moving them: two wrappers whose whole body was a dict literal, one whose body was two recorder calls, a NamedTuple built in one place and unpacked in the next, and a classmethod constructor with one caller. Split by what the lines actually are:

before after delta
executable lines 1370 1476 +7.7%
docstring lines 202 341 +139
comment lines 85 76 −9
blank lines 129 270 +141

The executable growth is the parameter lists and returns of the units the ceiling required; the rest is each unit's docstring and its two PEP 8 blank lines, plus three module headers. Three duplicated mechanisms are gone outright — the twice-derived framework/version/precision identity, the thrice-written float coercion, and the three near-identical authority-row handlers (that last one jointly with #1986). Nothing further can be inlined without putting a function back over the ceiling: _record_fact_per_variant sits at 12 and the three remaining single-use helpers it calls would take it to 21.

The three new modules import only downward and none imports recipe_journal, so the split adds no cycle; import-linter and the code-metrics gate both pass at head.

…ee on

to_float is not a drop-in replacement for a guarded float(x or 0.0): it returns
its default for inf, nan and bool. Pinning all three keeps the next swap from
being silent, and says so in recorded_float's own docstring.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chennyiiis

Copy link
Copy Markdown
Contributor Author

Follow-up on the inf → 0.0 point, with the mechanism named.

It was not a rendering decision — it was a swap. The integrate ledger's four figures each carried their own try: float(x or 0.0) except (TypeError, ValueError): 0.0, and consolidating them onto the existing hyperloom.common.coerce.to_float looked like the obvious de-duplication. to_float is not a drop-in replacement for that inline form. The two disagree on exactly three inputs:

input float(x or 0.0) guarded to_float(x, 0.0)
inf inf 0.0
nan nan 0.0
True 1.0 0.0

So 0.0 was never a rendering of an infinite value; it was to_float's default standing in for one. It is also the same value a never-measured field carries, so after it is written nobody reading the recipe row can tell "infinite" from "zero", and it would have silently unfrozen a row whose stored best_throughput is inf — a row no session can beat today.

Fixed by de-duplicating onto recorded_float instead: the same four lines, named once, agreeing with the inline form on every input. to_float is untouched everywhere it was already in use, and every remaining to_float call in recipe_journal.py is present at the merge base with the same argument. TestRecordedFloat::test_keeps_what_to_float_rejects pins all three disagreeing inputs and recorded_float's docstring states the non-equivalence, so the next reader does not make the same swap.

Same conclusion was reached independently on #1958 and #1995.

@chennyiiis

chennyiiis commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

PR #2011 -- Step 7 independent reader, run

A second reader, given the diff, base.txt and the earlier card with every finding false until
defended, re-adjudicated this PR at head b005658a2. No finding survives. Blocking issues:
none.

The card's two findings, re-tested by behaviour rather than ancestry.
conflict-main-1986-1985: head carries 8 except Exception handlers, the same count in the same
roles as the merge base; no _authoritative_recipe_row helper exists; the three authority reads
(recipe_journal.py:1136, :1240, :1343) and the put_recipe at :1358 are bare.
LocalRecipeStoreError is imported at :27 and caught at :601, so no NameError.
benched=True is at _config_timeline.py:173 inside _attempt_fields, splatted into
record_attempt at :219, and the 24-key kwarg set is identical to main's.
desc-unstated-log-and-coercion: every log.<level>("...") literal was extracted from the base
file and from all four head files and the sorted sets diffed -- zero differences. The float
consolidation goes through this PR's new recorded_float (coerce.py:26-38), which is
try: float(value or default) except (TypeError, ValueError): default, the merge base's inline
form character for character, not to_float. The description's inventory now matches head
exactly.

A statement that ended up on the other side of the check protecting it. The untouched regions
were byte-diffed (identical but for one helper call in _record_fact_impl) and the four genuinely
reordered paths hand-traced. _final_recipe_overrides (:1128-1152) still reads the authority
row after building attrs and before improved_best_config, matching base order -- that is the
baseline_accuracy-before-re-baseline shape and it is not here. _variant_failure (:772)
receives the derived outcome, not outcome_raw, which is the exact swap a split invites.
_record_grid_facts (:437-458) guards only record_attempt_row, leaving
_record_fact_per_variant outside the guard as base did, with _grid_attempt_row's None
condition the precise negation of base's if. The one real scope narrowing -- the success
log.info and _record_remote_recipe_audit leaving the try in _publish_remote_recipe
(:1064-1126) -- was chased and killed: that helper carries its own except Exception at
:874-875, so no reachable input changes the outcome.

A shared-helper swap that changes the domain. coerce.py was read in full, every coerce call
site the diff introduces enumerated, and each compared to the base expression it replaces
(base:954, :969, :976, :1640, all of which already carried
try/except (TypeError, ValueError)). _config_timeline.py imports no coerce helper at all; no
new to_int or to_str_list appears anywhere; and the non-coerce coercions that moved into
_recipe_amend.amended_recipe_fields stayed unguarded exactly as base had them, so a bad type
still raises rather than becoming a plausible 0.0.

Checked: knowledge/recipe_journal.py, _recipe_attrs.py, _config_timeline.py, _recipe_amend.py, hyperloom/common/coerce.py, tests/test_coerce.py | Ran: log-literal set diff base vs head across all four files; except Exception census; byte-diff of untouched regions; hand-trace of the four reordered paths; coerce call-site census against the base expressions | Base: adbc1879e4601d6ee6f1b09f65f78bc4bf82aa7b | Head: b005658a2 | CI: 40 checks, 0 failures

@ZhengGong-amd

Copy link
Copy Markdown
Collaborator

Production code is still too large. The net growth is +393 lines (it was +419), and it needs to keep shrinking before this can merge.

I classified every changed line in the touched production files with an AST pass (merge-base vs head 2403ced90):

net lines
executable logic +0
docstrings +147
def/class signatures +78
NamedTuple/dataclass fields +17
blank lines between new defs +122
imports / __all__ +26

Functions go from 39 to 86, record types from 2 to 8, and total parameters from 64 to 158. Of the 59 new definitions, 38 are referenced exactly once in src/.

The logic barely changed (+0). The growth is the scaffolding around single-use units: their signatures, docstrings, fields and spacing (about +390 lines). That is the same control flow in more places, not a simpler one (AGENTS.md Simplify by removing a mechanism). The parameter-count and tuple-return limits in [tool.hyperloom.code_metrics] are review triggers, not a reason to add a type or a helper. When a limit and the clearer code disagree, keep the clearer code and say so (Size is a design signal).

Cut next:

  • _ConfigAttemptContext and the _attempt_fields/_attempt_failure/_attempt_artifacts/_attempt_accuracy builders. framework_event.record_attempt already normalizes every field.
  • _IntegrateIndex and the ~9 single-use constants in _recipe_attrs.py.
  • The names that duplicate refactor(knowledge): match warm-start candidates in a module of their own #1959's: give each rule one owner.
  • Inline the remaining helpers that have one caller and only thread arguments through, and drop the docstrings that restate a private helper's name (Comment below the local average).

Target: production net ≤ 0, or name each remaining new type in the description with what it removes.

…knowledge

# Conflicts:
#	src/hyperloom/orchestrator/knowledge/recipe_journal.py
Test and others added 6 commits October 20, 2026 18:53
…knowledge

# Conflicts:
#	scripts/code_metrics_baseline.txt
#	src/hyperloom/orchestrator/knowledge/recipe_journal.py
…ary line

The restructuring added helpers, and each arrived with a multi-paragraph block.
Reduce every added helper to its summary line, keeping only the sentences that
state an invariant or a constraint from outside the file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e made stale

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Test and others added 14 commits October 21, 2026 15:01
fact_write_hook, _record_fact_impl and _record_config_run were inside every
repo threshold before this branch, so they read as they did: the grid loop and
the provenance dict are back in their callers and the configuration arm's run
row is back beside them. Four more single-use units -- a dict-literal wrapper,
a two-call wrapper and two predicates -- go the same way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…knowledge

# Conflicts:
#	scripts/code_metrics_baseline.txt
…scaffolding

The split answered the complexity ceiling by relocation: `_config_timeline.py`,
`_recipe_amend.py` and `_recipe_attrs.py` held 38 helpers, 29 of them with one
caller, each carrying a docstring restating its name and two blank lines.

All three are inlined back into `recipe_journal.py`, and what only existed to
carry arguments across the module boundary is deleted with them:

- `_ConfigAttemptContext`, `_attempt_accuracy` and `_attempt_artifacts`;
- the caller-side `str(x or "")` on every field `record_attempt` already
  projects through `event_fields.text`, and the empty-gate skip
  `record_attempt_gate` already makes;
- `_IntegrateIndex`, nine single-use constants, `_model_identity_tags`,
  `_last_stack_launch_args`, `_stack_action_names`, `_stack_entry_source`,
  `amend_provenance` and `improved_best_config`;
- every docstring that restated a private helper's name; the invariants the
  longer ones carried are kept as comments.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he length gate

Its docstring restated each parameter name; only the no_promote rule said
something the signature could not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…knowledge

#2055 deletes state nothing reads, and this branch had relocated one of the
readers: _build_kernel_optimizations_from_state became the module-level
kept_kernel_optimizations with _index_integrate_attempts,
_kernel_optimization_row and _last_rebenched_throughput beside it. A textual
merge would have resurrected the whole join over the deleted
kernel_opt_task_attempts ledger. Main's deletion wins and is re-applied to the
branch's shape: those four units, the kernel_optimizations attrs key and the
extras merge that carried it into the KB row are gone, and the config arm's
attempt now copies the attribution explore stored instead of re-deriving it
(_attempt_failure no longer takes the outcome).

The two conflicted tests take main's side: it replaced the recipe-attrs
kernel_optimizations assertions with one that pins an earlier session's rows
being carried forward, and deleted the e2e_decision tests of the retired join.
The branch's move from _build_recipe_attrs_from_state to
recipe_attrs_from_state is carried across in the surviving cases.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…knowledge

Only scripts/code_metrics_baseline.txt conflicted. Resolved with the
ratchet rule: whichever side records the lower value wins, an entry the
merge base had and a side dropped is retired, an entry absent from the
base is kept.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…knowledge

Brings the branch up to origin/main; no path conflicted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…knowledge

Brings the branch up to origin/main; no path conflicted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…documented twice

The module carried a second copy of several things it already had. The
framework / version / precision identity was derived at two sites with the
same fallback chain around each; `_extra_server_args` re-implemented
`common.coerce.to_str_list`'s whole None / str / list / scalar ladder; four
figures were parsed through their own `try: float(x or 0.0)` blocks, and the
`gain_pct` / `delta_pct` fallback was an open-coded `first_float`.

Deleted with them: the `isinstance` guards whose producers are all typed
(`lever_kind_for_task`'s two dict arguments, the variant metrics checked twice,
and the tuple shape of a cache this class is the sole writer of), and eleven
docstring `Args:` / `Returns:` blocks that restated the parameter names and
the body back to the reader. What those docstrings said that the code cannot
is kept.

The module goes from 1671 lines to 1654 and its baseline entry comes down with
it; production lines across both changed files are -4 against main. No
observable field, value or log line changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…es their own modules

recipe_journal.py carried the FRAMEWORK_AGENT configuration-arm attempt rows
and the session's recipe attributes alongside the journal itself. Each is a
complete job with one entry point, so each becomes its own module and the
journal keeps only what it owns.

_NON_ATTEMPT_OUTCOMES travels with the attempt rows that read it, so the
dependency points one way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…knowledge

# Conflicts:
#	scripts/code_metrics_baseline.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@xiaofei-zheng xiaofei-zheng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What this PR does

Nine functions in src/hyperloom/orchestrator/knowledge/recipe_journal.py sat above the repo's complexity ceiling and the module was at its baselined length, so this names the parts each of them was already built from and moves two coherent groups sideways: _recipe_attrs.py (the recipe-shaped projections of SharedState — workload tags, attempt provenance, recipe attrs) and _config_timeline.py (the configuration arm's attempt rows on the FRAMEWORK_AGENT event). The close path splits into _publish_remote_recipe / _write_local_recipe / _final_recipe_overrides, the amend builds its put_recipe arguments from one field table, and recorded_float goes to hyperloom.common.coerce as the one home for the guarded float(x or 0.0) form that keeps inf, nan and bool as recorded.

Blocking issues (1)

1. The description still describes an earlier revision of this branch.

body.txt and the diff at feda1ad2 disagree on most of the structural facts:

  • "Three coherent groups move to sibling modules" lists _recipe_amend.py (141 lines). That file does not exist; rows_not_superseded, amended_recipe_fields and amended_recipe_extras stayed in recipe_journal.py:244-290. The later commit refactor(recipe-journal): delete the three sibling modules and their scaffolding undid that direction and the description was never updated.
  • _recipe_attrs.py is given as 408 lines and _config_timeline.py as 252; they are 242 and 169.
  • The Tests section says _build_kernel_optimizations_from_state is now a module function in _recipe_attrs. No such symbol exists anywhere in the tree, and _recipe_attrs carries no kernel-optimization projection.
  • "The three integrate-ledger figures and the live-row throughput comparison use it" — recorded_float's only call sites are recipe_journal.py:1079, :1209 and :1210, all throughput comparisons. micro_speedup, e2e_gain_pct and e2e_tput are untouched by this diff, so the try: float(x or 0.0) rationale the section is built on describes code that is not here.
  • Metrics: recipe_journal.py is given as 1786 → 1362 and the baseline entries as 13 → 1. This PR's own scripts/code_metrics_baseline.txt hunk says -module-lines ... 1671 / +... 1269, and the entry count goes 12 → 1.
  • Line accounting: "the four production modules total 2163 lines against main's 1786, which is +21%". Three production modules total 1680 against 1671 — the split is close to line-neutral, not +21%. The executable / docstring / blank breakdown is derived from the same stale figures.

A reviewer sizing the risk of this refactor is told about a module split and a line-growth profile that did not happen, and is pointed at coercion call sites the diff never touches. Please rewrite the description against the current head.

What was checked

recipe_journal.py compared field by field against the merge base e34bb7af: the close path, kb_amend_recipe (24-key reserved set, 21 put_recipe kwargs, the 11 per-field defaults), _record_fact_per_task / _record_fact_per_variant, extract_kept_best_config, workload_canonical_id, read_local_recipe_row; _recipe_attrs.py and _config_timeline.py against the helpers they were moved from; common/coerce.py (recorded_float, first_float, to_str_list, to_mapping); breakdown/recorder/framework_event.py (_ATTEMPT_FIELDS, _ATTEMPT_BLOCKS, record_attempt, record_attempt_gate); orchestrator/lever.py; session/optimization_journal.py.

The no-observable-change claim holds on every path read. In particular: the record_attempt fields that are now passed raw instead of str(x or "")-coerced are re-coerced identically by _ATTEMPT_FIELDS / _ATTEMPT_BLOCKS; the gate row the base skipped on an empty gate name is dropped by record_attempt_gate itself; _extra_server_args's switch to to_str_list strips a string payload that every call site already strips; amended_recipe_fields' type(default)(live_value) reproduces the base's per-field dict() / list() / str() / float() exactly; first_float(gain_pct, delta_pct) is the base's two-step fallback; narrowing the remote publish's try is safe because _record_remote_recipe_audit catches Exception itself. Dangling-reference check: the three deleted private helpers have no surviving caller outside the four migrated tests.

No local test run — all 12 test shards plus ruff, pylint, mypy, bandit, CodeQL, import-linter and code-metrics are green at this head.

@xiaofei-zheng
xiaofei-zheng dismissed their stale review October 22, 2026 11:23

Resolved: the description has been rewritten against the current head.

@xiaofei-zheng xiaofei-zheng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues.

The one finding from my earlier review — the description describing an earlier revision of this branch — is resolved: the body now matches the diff at feda1ad2 (two sibling modules at their real line counts, 12 to 1 baseline entries, recipe_journal.py 1671 to 1269, production net +22, and recorded_float's three actual call sites). The earlier reviews' blocking items are resolved at this head too: _authoritative_recipe_row and the broad handlers it revived are gone, LocalRecipeStoreError is imported at recipe_journal.py:32 and caught at :559, #1985's benched=True is set in _config_timeline._attempt_fields, the baseline is regenerated, and _ConfigAttemptContext, _IntegrateIndex, _E2eVerdict and AttemptProvenance.unrecorded no longer exist.

Checked: recipe_journal.py compared field by field against the merge base e34bb7af — the close path (_unvalidated_stack_outcome / _publish_remote_recipe / _write_local_recipe / _final_recipe_overrides), kb_amend_recipe (the 24-key reserved set, the 21 put_recipe kwargs, the 11 per-field defaults), _record_fact_per_task / _record_fact_per_variant, extract_kept_best_config, workload_canonical_id, read_local_recipe_row; _recipe_attrs.py and _config_timeline.py against the helpers they were moved from; common/coerce.py (recorded_float, first_float, to_str_list, to_mapping); breakdown/recorder/framework_event.py (_ATTEMPT_FIELDS, _ATTEMPT_BLOCKS, record_attempt, record_attempt_gate); orchestrator/lever.py (lever_kind_for_task); session/optimization_journal.py (OUTCOME_KEEP / OUTCOME_REVERT).

The no-observable-change claim holds on every path read. Specifically: the record_attempt fields now passed raw instead of str(x or "")-coerced are re-coerced identically by the recorder's own projections; the gate row the base skipped on an empty gate name is dropped by record_attempt_gate itself; _extra_server_args's switch to to_str_list strips a string every call site already strips; amended_recipe_fields' type(default)(live_value) reproduces the base's per-field dict() / list() / str() / float(); first_float(gain_pct, delta_pct) is the base's two-step fallback; narrowing the remote publish's try is safe because _record_remote_recipe_audit carries its own except Exception. Dangling-reference check: the three deleted private helpers have no surviving caller outside the four migrated tests.

Ran: no local suite — all 12 test shards plus ruff, pylint, mypy, bandit, CodeQL, gitleaks, import-linter, REUSE and code-metrics are green at this head.

LGTM

@xiaofei-zheng
xiaofei-zheng merged commit 04d3768 into main Oct 22, 2026
42 checks passed
@xiaofei-zheng
xiaofei-zheng deleted the refactor/chenyi/cc20-knowledge branch October 22, 2026 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size-exception Waives the PR diff budget; added by a maintainer with a reason in the PR description

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants