Skip to content

[RSI] Refuse proposals whose AITER_CONFIG file is not on disk when they are admitted; keep run_grid's check as backstop - #1982

Merged
xiaofei-zheng merged 6 commits into
mainfrom
pulse/auto-fix/draft-113-141
Oct 20, 2026
Merged

xiaofei-zheng merged 6 commits into
mainfrom
pulse/auto-fix/draft-113-141

Conversation

@luochen-amd

@luochen-amd luochen-amd commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Submitted automatically by Hyperloom RSI (Pulse auto-fix) for problem #195 config-arm-env-references-unauthored-artifact. Review it like any other change before merging.

Problem

Config-arm variants set an AITER_CONFIG_* env to a tuned-GEMM CSV no stage had written, and the server died in aiter's config loader (FileNotFoundError), then timed out on /health. In the Qwen3-14B-FP8 session of 10-14 both such variants came from candidate_discovery_specialist proposals that were two-step ("have a patch specialist author the CSV, then point the env at it"): only the env was structured, the authoring step lived in prose nothing reads, so the env-only proposal was benched as a plain config arm.

Change

At admission (the specialist patch-safety gate, where a proposal's references are already grounded):

  • patch_safety.drop_env_ungrounded_proposals removes every proposal_set entry whose extra_envs name an AITER_CONFIG_* file (any :-joined element) that is not on disk, and says why, pointing at the gemm_tuning action for tuned-GEMM CSVs.
  • The drops go to proposals_env_ungrounded on the specialist_done payload, and dispatch.record_specialist_result records them as proposal_env_file_missing in last_action_failures -- the same path patches_ungrounded takes to the SEED prompt -- so the orchestrator learns at proposal time, not after a doomed boot. Every consumer of the payload (the orchestrator's round, FrameworkPhase's discovery candidates and config arms) reads the filtered proposal_set.

Backstop: run_grid's pre-boot fast-fail keeps the same check (unsupported_capability_reason now also returns the missing-file reason), for variants that reach it another way.

Shared check: common/env_safety.ungrounded_env_file_reason(envs) -- existence-only, every :-joined element, OSError from Path.is_file() treated as missing. Used by both sides.

Layout, no behaviour change:

  • runner._finalize's post-vetting bookkeeping (numeric scan, forbidden-field strip, writing the vetting results back) moves into patch_safety.admit_payload, which also runs the new drop. runner.py was 1525 lines on main against a 1520 baseline; it is now 1505.
  • _grid_runner.py stays at its 2089 baseline.
  • scripts/code_metrics_baseline.txt is lowered for the entries this shrinks: runner.py module lines and _finalize; run_grid and record_specialist_result function lines.

Testing

  • New tests:
    • admission drop: written vs missing file, flag-only proposals kept, non-list proposal_set;
    • admit_payload bookkeeping;
    • dispatch records proposal_env_file_missing;
    • run_grid fast-fail drops a missing-file variant on any framework.
  • The existing helper tests now call the shared function.
  • On Linux (python 3.11, .[test]): test_specialist_patch_safety_unit.py, test_grid_runner_helpers_coverage_unit.py and test_coordinator_async_batch2_unit.py pass (283), and the -k "patch_safety or specialist or grid_runner or env_safety or proposal or dispatch" selection over the specialists, executors and inference_optimizer tests passes (1703).
  • ruff check and ruff format --check (0.16.2), scripts/code_metrics.py --base-ref main (exit 0) and lint-imports (7 kept, 0 broken) pass.

Verification

Not verified by the auto-fix pipeline (no verification for AMD-AGI/Hyperloom yet).

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Code metrics gate: PASSED

Judged: the 9 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension Limit New Worse Baseline out of date Baseline entries vs base
Cyclomatic complexity fails > 20 0 0 0 64 +0
Cognitive complexity fails > 30 0 0 0 259 +0
Function length (lines) fails > 80 0 0 0 401 +0
Nested block depth fails > 5 0 0 0 24 +0
Parameter count fails > 5 0 0 0 95 +0
Module length (lines) warns > 800, fails > 1200 0 0 0 50 +0
Duplicated lines per file fails > 0 0 0 0 97 +0
Dead code (vulture) fails > 0 0 0 0 2 +0
Check Findings Effect
Comments 0
English only 0
Production code imports test code 0
Repeated literals 0
A1 Private name of another package 0
A8 Flag argument 0
A9 Tuple return 0
A10 Raw vocabulary comparison 0
A11 Hidden global write 0
A12 Pure forwarder 0
A12 Single-implementation abstraction 0

Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 2

Module Lines Limit
src/hyperloom/orchestrator/specialists/dispatch.py 1056 warns > 800
src/hyperloom/orchestrator/specialists/patch_safety.py 1175 warns > 800

Already over at the base (backlog, not this change's): 2

Unit Dimension Base Now
src/hyperloom/orchestrator/specialists/patch_safety.py:320 resolve_patch_apply_root Function length (lines) 112 112
src/hyperloom/orchestrator/specialists/runner.py:559 SpecialistRunner._prepare Function length (lines) 199 199
97 findings in files this change does not touch

Outside this change (informational): 97

Unit Dimension Baseline Now
src/hyperloom/orchestrator/state/shared_state.py resolve_graded_comparison Cognitive complexity absent 34
src/hyperloom_kb/collect/mapping.py _experience Cognitive complexity absent 31
src/hyperloom_kb/schema.py FieldDeclaration.__post_init__ Cognitive complexity absent 32
src/kernelforge/cli/forge_loop_command.py _prepare_measurement_driver Cognitive complexity absent 36
src/kernelforge/cli/forge_loop_command.py _read_kb_warm_start Cognitive complexity absent 40
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Cognitive complexity absent 44
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Cognitive complexity absent 38
src/hyperloom/inference_optimizer/cli/__init__.py _reexport_session_environment Function length (lines) absent 111
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_agent_lineup Function length (lines) absent 89
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_run_topology Function length (lines) absent 92
src/hyperloom/inference_optimizer/cli/__init__.py _resume_session Function length (lines) absent 170
src/hyperloom/inference_optimizer/cli/__init__.py _seed_fresh_session Function length (lines) absent 170
src/hyperloom/inference_optimizer/multi_node/cli.py _restart_multinode Function length (lines) absent 107
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _build_parser Function length (lines) absent 84
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _spawn_rank_actors Function length (lines) absent 84
src/hyperloom_kb/sync.py GlobalSync.pull Function length (lines) absent 84
src/kernelforge/cli/forge_loop_command.py _RunReporter.build_result Function length (lines) absent 91
src/kernelforge/cli/forge_loop_command.py _prepare_measurement_driver Function length (lines) absent 136
src/kernelforge/cli/forge_loop_command.py _read_kb_warm_start Function length (lines) absent 149
src/kernelforge/loop/runner.py IterationLoop._close_iteration Function length (lines) absent 117
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Function length (lines) absent 184
src/kernelforge/loop/runner.py IterationLoop._implementer_history Function length (lines) absent 112
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Function length (lines) absent 102
src/kernelforge/loop/runner.py IterationLoop._source_attempt Function length (lines) absent 173
src/hyperloom_kb/http_service.py Module length (lines) absent 1525

...and 72 more; the job log lists every row.

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.

@luochen-amd luochen-amd changed the title Abort grid variants whose AITER_CONFIG file path is missing [RSI] Abort grid variants whose AITER_CONFIG file path is missing Oct 19, 2026
@luochen-amd
luochen-amd marked this pull request as ready for review October 19, 2026 07:41
@luochen-amd
luochen-amd requested a review from a team as a code owner October 19, 2026 07:41

@xiaofei-zheng xiaofei-zheng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1982 -- [RSI] Abort grid variants whose AITER_CONFIG file path is missing

What it does: a grid variant whose AITER_CONFIG_* env names a CSV that does not exist used to boot a server that died in aiter's config loader with FileNotFoundError and then timed out on /health. The PR adds ungrounded_env_file_reason() in _grid_runner.py and calls it at run_grid's pre-boot capability fast-fail, so such a variant is dropped as capability_unsupported with the key and missing path(s) before any server starts.

Blocking issues: 1

  1. [V1] The follow-up commit rewrote both files with CRLF line endings, which is why the PR is CONFLICTING [verified]
    Problem: eb52f689 converted src/hyperloom/orchestrator/actions/executors/_grid_runner.py and src/hyperloom/inference_optimizer/tests/test_grid_runner_helpers_coverage_unit.py from LF to CRLF. The diff is +2826/-2723 lines; ignoring CR it is +104/-1. 72af614c and the merge base are LF, and .editorconfig sets end_of_line = lf.
    Impact: GitHub reports mergeable: CONFLICTING -- the CR-stripped diff applies cleanly on current main (3fe44e0), so the conflict comes entirely from the line-ending rewrite. It also conflicts with every in-flight PR touching _grid_runner.py (#1964, #1926, #1908, #1896, #1867, #1764, #1667, #1316) and replaces blame on 2.3k lines. At head only CodeQL, Gitleaks and ReadTheDocs reported; tests, lint and code-metrics have not run on this commit.
    Action: convert both files back to LF (e.g. sed -i 's/\r$//' <files>, or pre-commit run mixed-line-ending --files <files>), rebase on main, and let the full CI matrix run on the new head.

Checked: ungrounded_env_file_reason, _is_file, run_grid fast-fail and multi-node restart order, capability_unsupported readers (_attach_grid_launch_evidence, state/failure_evidence.py), _aiter_jit.csvs_aiter_will_load, aiter jit/core.py:get_config_file and every AITER_CONFIG_* name aiter reads (all file paths, so the prefix match is safe), the two FileNotFoundErrors in session Qwen3-14B-FP8/20261014T024938Z-d210b947 | Ran: the 6 new ungrounded_env_file_reason tests at head (6 passed); ruff C901 on base and head (run_grid 61 -> 61); git apply --check of the CR-stripped diff on main | SKIPPED: V4 for #1276 #1907 #1032 #1869 #1154 (only their first 100 files compared) | Base: a56028d | Head: eb52f68

@chaojhou chaojhou left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1982 -- [RSI] Abort grid variants whose AITER_CONFIG file path is missing

What it does: Config-arm variants could set an AITER_CONFIG_* env to a tuned-GEMM CSV no stage had written, and the server then died at config load. The PR adds ungrounded_env_file_reason to the grid runner, which drops a variant before boot when any :-separated element of a non-empty AITER_CONFIG_* value is not an existing file, and records it through the existing capability fast-fail path.

Blocking issues: 1

  1. [V5] CONFLICTING with main, caused only by an LF->CRLF rewrite of both files; CI never ran lint or tests at this head [verified]
    Problem: GitHub reports mergeable: CONFLICTING, and git merge-tree against origin/main 3fe44e0 gives content conflicts in both files. The bot re-saved every line of both files with CRLF (_grid_runner.py 2289/2289, the test file 537/537; 0 at the merge base and on main), so with main having changed both files since a56028d every line differs. With the head copies normalised to LF, git merge-file merges both cleanly: the real edits do not overlap. Because of the conflict, only Analyze, Gitleaks and ReadTheDocs ran at this head; Lint, Tests with Coverage and code-metrics did not, so the description's ruff format --check and "45 passed" claims are unchecked by CI. The CRLF rewrite also breaks .editorconfig (end_of_line = lf) and the pre-commit mixed-line-ending --fix=lf hook. src/hyperloom/orchestrator/actions/executors/_grid_runner.py:1
    Impact: The PR cannot merge; and the reviewable change (a ~42-line helper, a 1-line call-site edit and 6 tests) is buried in a +2826/-2723 whole-file diff that also wipes git blame for the grid runner.
    Action: Rebase onto current main keeping both files LF (e.g. git add --renormalize or dos2unix on the two files) so the diff shows only the helper, the run_grid call site and the tests, then let the full CI matrix run at the new head.

Checked: _grid_runner.py (ungrounded_env_file_reason, _is_file, run_grid fast-fail), _grid_base.py (GridVariant), env_safety.py (filter_untrusted_env_mapping), _aiter_jit.py (csvs_aiter_will_load), phases/kernel.py AITER_CONFIG candidate checks, multi-node sandbox path; same files on origin/main | Ran: git merge-tree against origin/main; CR byte counts per blob at base, head and main; git merge-file with the head copies normalised to LF (0 conflicts) | Base: a56028d | Head: eb52f68

@xiaofei-zheng

Copy link
Copy Markdown
Collaborator

Root cause is upstream of the grid runner; this PR only adds the last-line guard.

In session Qwen3-14B-FP8/20261014T024938Z-d210b947, both failing variants came from candidate_discovery_specialist proposals (rounds 52d55174, e9cbde17) that were explicitly two-step: "have a patch specialist author the CSV, then point AITER_CONFIG_GEMM_A8W8_BLOCKSCALE_BPRESHUFFLE at it". Only step two was structured (extra_envs); step one lived in reason / residual_questions text that nothing reads. No patch naming either path was ever submitted, so "the patch route was rejected first" in the description does not match the session. Explore then benched the env-only proposal as a plain config arm. The canonical route for this work, the GEMM tuning action, ran later (09:38, KEEP) and wrote its CSVs under the session's runs/gemm_tuning/.

Please fix it where proposals are admitted:

  • specialists/runner.py: the patch-safety gate (vet_patches / strip_forbidden_proposal_fields, ~L1207) is where a proposal's references are grounded. Ground AITER_CONFIG_* file values there too, and surface drops the same way patches_ungrounded reaches the SEED prompt via record_action_failure (specialists/dispatch.py:895), so the orchestrator learns at proposal time rather than after a doomed boot.
  • A proposal whose premise is "missing tuned GEMM rows" should route to the GEMM tuning action instead of becoming a config arm pointing at a file that does not exist yet.

Keep ungrounded_env_file_reason in run_grid as the backstop.

@luochen-amd luochen-amd changed the title [RSI] Abort grid variants whose AITER_CONFIG file path is missing [RSI] Refuse proposals whose AITER_CONFIG file is not on disk when they are admitted; keep run_grid's check as backstop Oct 20, 2026
@luochen-amd

Copy link
Copy Markdown
Contributor Author

Thanks @xiaofei-zheng and @chaojhou. Pushed:

  • Line endings: both files are back to LF. That rewrite came from our follow-up commit, not from the fix.
  • Rebased: merged with main, so the diff is only the change.
  • Root cause: fixed where proposals are admitted, as suggested. The patch-safety gate drops a proposal whose AITER_CONFIG_* file is not on disk, and the drop reaches last_action_failures as proposal_env_file_missing, the way patches_ungrounded does. The reason points at the gemm_tuning action, which writes tuned-GEMM CSVs.
  • Backstop: run_grid keeps its check.

On routing a "missing tuned GEMM rows" premise straight to gemm_tuning: this PR does it through the denial the orchestrator reads, rather than by rewriting the proposal into a different action. Happy to add explicit routing if you would rather have it here.

The description is updated with the details, and the test, lint, code-metrics and import-linter runs.

@xiaofei-zheng xiaofei-zheng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1982 -- [RSI] Refuse proposals whose AITER_CONFIG file is not on disk when they are admitted; keep run_grid's check as backstop

What it does: specialist proposals that point an AITER_CONFIG_* env at a CSV no stage has written are now dropped at the specialist patch-safety gate (patch_safety.admit_payload -> drop_env_ungrounded_proposals), and the drop reaches the SEED prompt as proposal_env_file_missing through the same record_action_failure path as patches_ungrounded. run_grid keeps the same check (shared common/env_safety.ungrounded_env_file_reason) as a pre-boot backstop for variants that reach it another way.

Blocking issues: none

The earlier CRLF rewrite is reverted; the PR is mergeable and the full CI matrix is green at this head.

Checked: SpecialistRunner._finalize is the only producer of proposal_set that reaches record_specialist_result, and admission runs before specialist_done.json is written; _ADMISSION_DROPS in dispatch.record_specialist_result; unsupported_capability_reason (only caller run_grid) now checks missing files before the vllm-only probe; every AITER_CONFIG_* name aiter reads is a file env, so the prefix match is safe | Ran: the new admission/grid/dispatch tests at head (17 passed); replayed specialist_rounds 0/5/8 from session Qwen3-14B-FP8/20261014T024938Z-d210b947 through drop_env_ungrounded_proposals -- both failing proposals (aiter-tuned-gemm-k5120-shapes, aiter-tuned-gemm-fill-keyschema-fix) are dropped, the other proposals kept; ruff C901 at head (_finalize 25 -> 22, run_grid 61 unchanged, new functions under 20) | Base: adbc187 | Head: 5459375

LGTM

@xiaofei-zheng
xiaofei-zheng merged commit 0311f1d into main Oct 20, 2026
41 of 42 checks passed
@xiaofei-zheng
xiaofei-zheng deleted the pulse/auto-fix/draft-113-141 branch October 20, 2026 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants