Repository navigation
[RSI] Refuse proposals whose AITER_CONFIG file is not on disk when they are admitted; keep run_grid's check as backstop - #1982
Conversation
Code metrics gate: PASSEDJudged: the 9 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.
Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 2
Already over at the base (backlog, not this change's): 2
97 findings in files this change does not touchOutside this change (informational): 97
...and 72 more; the job log lists every row.
Thresholds and sources: |
xiaofei-zheng
left a comment
There was a problem hiding this comment.
PR #1982 -- [RSI] Abort grid variants whose AITER_CONFIG file path is missing
What it does: a grid variant whose AITER_CONFIG_* env names a CSV that does not exist used to boot a server that died in aiter's config loader with FileNotFoundError and then timed out on /health. The PR adds ungrounded_env_file_reason() in _grid_runner.py and calls it at run_grid's pre-boot capability fast-fail, so such a variant is dropped as capability_unsupported with the key and missing path(s) before any server starts.
Blocking issues: 1
- [V1] The follow-up commit rewrote both files with CRLF line endings, which is why the PR is CONFLICTING [verified]
Problem:eb52f689convertedsrc/hyperloom/orchestrator/actions/executors/_grid_runner.pyandsrc/hyperloom/inference_optimizer/tests/test_grid_runner_helpers_coverage_unit.pyfrom LF to CRLF. The diff is +2826/-2723 lines; ignoring CR it is +104/-1.72af614cand the merge base are LF, and.editorconfigsetsend_of_line = lf.
Impact: GitHub reportsmergeable: CONFLICTING-- the CR-stripped diff applies cleanly on current main (3fe44e0), so the conflict comes entirely from the line-ending rewrite. It also conflicts with every in-flight PR touching_grid_runner.py(#1964, #1926, #1908, #1896, #1867, #1764, #1667, #1316) and replaces blame on 2.3k lines. At head only CodeQL, Gitleaks and ReadTheDocs reported; tests, lint and code-metrics have not run on this commit.
Action: convert both files back to LF (e.g.sed -i 's/\r$//' <files>, orpre-commit run mixed-line-ending --files <files>), rebase on main, and let the full CI matrix run on the new head.
Checked: ungrounded_env_file_reason, _is_file, run_grid fast-fail and multi-node restart order, capability_unsupported readers (_attach_grid_launch_evidence, state/failure_evidence.py), _aiter_jit.csvs_aiter_will_load, aiter jit/core.py:get_config_file and every AITER_CONFIG_* name aiter reads (all file paths, so the prefix match is safe), the two FileNotFoundErrors in session Qwen3-14B-FP8/20261014T024938Z-d210b947 | Ran: the 6 new ungrounded_env_file_reason tests at head (6 passed); ruff C901 on base and head (run_grid 61 -> 61); git apply --check of the CR-stripped diff on main | SKIPPED: V4 for #1276 #1907 #1032 #1869 #1154 (only their first 100 files compared) | Base: a56028d | Head: eb52f68
chaojhou
left a comment
There was a problem hiding this comment.
PR #1982 -- [RSI] Abort grid variants whose AITER_CONFIG file path is missing
What it does: Config-arm variants could set an AITER_CONFIG_* env to a tuned-GEMM CSV no stage had written, and the server then died at config load. The PR adds ungrounded_env_file_reason to the grid runner, which drops a variant before boot when any :-separated element of a non-empty AITER_CONFIG_* value is not an existing file, and records it through the existing capability fast-fail path.
Blocking issues: 1
- [V5] CONFLICTING with main, caused only by an LF->CRLF rewrite of both files; CI never ran lint or tests at this head [verified]
Problem: GitHub reportsmergeable: CONFLICTING, andgit merge-treeagainst origin/main 3fe44e0 gives content conflicts in both files. The bot re-saved every line of both files with CRLF (_grid_runner.py2289/2289, the test file 537/537; 0 at the merge base and on main), so with main having changed both files since a56028d every line differs. With the head copies normalised to LF,git merge-filemerges both cleanly: the real edits do not overlap. Because of the conflict, only Analyze, Gitleaks and ReadTheDocs ran at this head; Lint, Tests with Coverage and code-metrics did not, so the description'sruff format --checkand "45 passed" claims are unchecked by CI. The CRLF rewrite also breaks.editorconfig(end_of_line = lf) and the pre-commitmixed-line-ending --fix=lfhook. src/hyperloom/orchestrator/actions/executors/_grid_runner.py:1
Impact: The PR cannot merge; and the reviewable change (a ~42-line helper, a 1-line call-site edit and 6 tests) is buried in a +2826/-2723 whole-file diff that also wipesgit blamefor the grid runner.
Action: Rebase onto current main keeping both files LF (e.g.git add --renormalizeordos2unixon the two files) so the diff shows only the helper, therun_gridcall site and the tests, then let the full CI matrix run at the new head.
Checked: _grid_runner.py (ungrounded_env_file_reason, _is_file, run_grid fast-fail), _grid_base.py (GridVariant), env_safety.py (filter_untrusted_env_mapping), _aiter_jit.py (csvs_aiter_will_load), phases/kernel.py AITER_CONFIG candidate checks, multi-node sandbox path; same files on origin/main | Ran: git merge-tree against origin/main; CR byte counts per blob at base, head and main; git merge-file with the head copies normalised to LF (0 conflicts) | Base: a56028d | Head: eb52f68
|
Root cause is upstream of the grid runner; this PR only adds the last-line guard. In session Please fix it where proposals are admitted:
Keep |
…mitted, not only in run_grid
|
Thanks @xiaofei-zheng and @chaojhou. Pushed:
On routing a "missing tuned GEMM rows" premise straight to The description is updated with the details, and the test, lint, code-metrics and import-linter runs. |
xiaofei-zheng
left a comment
There was a problem hiding this comment.
PR #1982 -- [RSI] Refuse proposals whose AITER_CONFIG file is not on disk when they are admitted; keep run_grid's check as backstop
What it does: specialist proposals that point an AITER_CONFIG_* env at a CSV no stage has written are now dropped at the specialist patch-safety gate (patch_safety.admit_payload -> drop_env_ungrounded_proposals), and the drop reaches the SEED prompt as proposal_env_file_missing through the same record_action_failure path as patches_ungrounded. run_grid keeps the same check (shared common/env_safety.ungrounded_env_file_reason) as a pre-boot backstop for variants that reach it another way.
Blocking issues: none
The earlier CRLF rewrite is reverted; the PR is mergeable and the full CI matrix is green at this head.
Checked: SpecialistRunner._finalize is the only producer of proposal_set that reaches record_specialist_result, and admission runs before specialist_done.json is written; _ADMISSION_DROPS in dispatch.record_specialist_result; unsupported_capability_reason (only caller run_grid) now checks missing files before the vllm-only probe; every AITER_CONFIG_* name aiter reads is a file env, so the prefix match is safe | Ran: the new admission/grid/dispatch tests at head (17 passed); replayed specialist_rounds 0/5/8 from session Qwen3-14B-FP8/20261014T024938Z-d210b947 through drop_env_ungrounded_proposals -- both failing proposals (aiter-tuned-gemm-k5120-shapes, aiter-tuned-gemm-fill-keyschema-fix) are dropped, the other proposals kept; ruff C901 at head (_finalize 25 -> 22, run_grid 61 unchanged, new functions under 20) | Base: adbc187 | Head: 5459375
LGTM
Problem
Config-arm variants set an
AITER_CONFIG_*env to a tuned-GEMM CSV no stage had written, and the server died in aiter's config loader (FileNotFoundError), then timed out on/health. In the Qwen3-14B-FP8 session of 10-14 both such variants came fromcandidate_discovery_specialistproposals that were two-step ("have a patch specialist author the CSV, then point the env at it"): only the env was structured, the authoring step lived in prose nothing reads, so the env-only proposal was benched as a plain config arm.Change
At admission (the specialist patch-safety gate, where a proposal's references are already grounded):
patch_safety.drop_env_ungrounded_proposalsremoves everyproposal_setentry whoseextra_envsname anAITER_CONFIG_*file (any:-joined element) that is not on disk, and says why, pointing at thegemm_tuningaction for tuned-GEMM CSVs.proposals_env_ungroundedon the specialist_done payload, anddispatch.record_specialist_resultrecords them asproposal_env_file_missinginlast_action_failures-- the same pathpatches_ungroundedtakes to the SEED prompt -- so the orchestrator learns at proposal time, not after a doomed boot. Every consumer of the payload (the orchestrator's round,FrameworkPhase's discovery candidates and config arms) reads the filteredproposal_set.Backstop:
run_grid's pre-boot fast-fail keeps the same check (unsupported_capability_reasonnow also returns the missing-file reason), for variants that reach it another way.Shared check:
common/env_safety.ungrounded_env_file_reason(envs)-- existence-only, every:-joined element,OSErrorfromPath.is_file()treated as missing. Used by both sides.Layout, no behaviour change:
runner._finalize's post-vetting bookkeeping (numeric scan, forbidden-field strip, writing the vetting results back) moves intopatch_safety.admit_payload, which also runs the new drop.runner.pywas 1525 lines on main against a 1520 baseline; it is now 1505._grid_runner.pystays at its 2089 baseline.scripts/code_metrics_baseline.txtis lowered for the entries this shrinks:runner.pymodule lines and_finalize;run_gridandrecord_specialist_resultfunction lines.Testing
proposal_set;admit_payloadbookkeeping;proposal_env_file_missing;run_gridfast-fail drops a missing-file variant on any framework..[test]):test_specialist_patch_safety_unit.py,test_grid_runner_helpers_coverage_unit.pyandtest_coordinator_async_batch2_unit.pypass (283), and the-k "patch_safety or specialist or grid_runner or env_safety or proposal or dispatch"selection over the specialists, executors and inference_optimizer tests passes (1703).ruff checkandruff format --check(0.16.2),scripts/code_metrics.py --base-ref main(exit 0) andlint-imports(7 kept, 0 broken) pass.Verification
Not verified by the auto-fix pipeline (no verification for AMD-AGI/Hyperloom yet).