Repository navigation
refactor(trace-analysis): delete the dead and repeated code in six over-gate TraceLens functions - #2069
Conversation
…ontract
Delete the local _COLL grouping tuple: it listed the same tags in the
same order as _OPMAP, so the membership test and the first-match lookup
now both read one module table, _COLLECTIVE_OPS, with the identical
match order ("reduce" still last).
Read TP_SIZE / TENSOR_PARALLEL_SIZE once instead of three times, and
replace the nested seqlen_regime ternary with a first-match lookup over
("prefill", "decode") that yields the same value.
Drop the "model_params or {}" fallback and the isinstance check on an
existing kernel_contract: the only caller passes the dict that
load_model_kernel_params always returns, and kernel_contract is only
ever written by this function, as a non-empty dict.
Co-authored-by: Cursor <cursoragent@cursor.com>
…ability
The aten:: check tested library in {"tensile", "pytorch native"}, but a
"tensile" library already returned through the _VENDOR_BACKEND_LIBRARIES
rejection above it, so the check is now one condition on {"",
"pytorch native"} with the same reason text.
The aiter device-source promotion required source_type outside the
patchable set, and the next statement accepted every source_type inside
it, so an aiter .cu/.cuh/.hip source returns True either way; the
promotion now runs first and the source_type set is tested once.
is_runtime_api_name reuses the name already read from the candidate.
The vendor-playbook comment loses its history and keeps the ordering
constraint, the verdict comment no longer repeats the one on
_RESOLVER_VERDICT_METHODS, and the cpp_itfs comment repeated its own
return message.
Co-authored-by: Cursor <cursoragent@cursor.com>
Move the per-keyword grep into _files_mentioning, which returns the existing files under one directory that contain one keyword, and an empty list when grep cannot start, times out or exits with an error; the caller only walks the benchmark directories and keywords. Delete what the loop no longer needs: - the "if source_file:" wrapper around the stem keywords: every producer of source_file writes a str, and for "" the stem is "" so neither keyword is added; - "proc.stdout or ''": text=True with capture_output always yields str; - the hand-written seen/unique loop, replaced by dict.fromkeys with the same first-seen order; - the nested _is_multigpu with its docstring, replaced by a sort key over the module tuple _MULTIGPU_PATH_TAGS; - the thirteen-line grep argv, whose include globs are now the module tuple _BENCHMARK_GREP_INCLUDES; - the history reference in the same-library comment. Co-authored-by: Cursor <cursoragent@cursor.com>
…ize_candidates
Move the shape, invocation-case and raw-arg-spec recovery from
ops_unique_args.csv into _recover_csv_invocations(item, csv_dir), so
_finalize_candidates keeps the per-candidate ids, GPU share and source
classification.
Delete the two lazily filled fused-MoE caches and their None checks.
The fused-MoE resolvers always return a fresh list, and every other
candidate in the same loop already re-reads the CSV for its own
lookups, so a fused-MoE candidate now reads the sidecar like the rest;
the rows, their order and the grafted provenance are unchanged.
Also drop two guards that could not change the result: "not _sf or"
before os.path.isfile, which is False for "", and the
item.get("kernel_repo", "") default for a key assigned a few lines
earlier. The raw-arg-spec lookup reuses the operation name instead of
reading it again.
Co-authored-by: Cursor <cursoragent@cursor.com>
…e_function The P-item prose was deduplicated while rows were added, through a _pitem_prose_seen set that had to be popped before return, then sorted and filtered after grouping. _pitem_prose(rows) now owns the whole rule: one entry per (rank, title), supplied by the first row naming it, P1 first, entries with no rank and no prose dropped. It reads the rows in arrival order, before the duration sort, so the same row wins. Delete guards the only production caller, build_task_groups, already makes unnecessary: the empty-input early return (an empty input yields no groups anyway), the isinstance check (it passes dicts only), and the try/except blocks around the P-item rank and the three metric sums (every producer writes numbers or None, as the bypass route's _build_task_groups already assumes). Every group holds at least one row, so the "if group['rows']" check before reading the primary goes too. The three identity calls share one argument dict instead of repeating four keywords each, the legacy-key merge reads the key every bucket is created with, and the module-private _safe_float alias is replaced by the safe_float it named. Co-authored-by: Cursor <cursoragent@cursor.com>
…ective injection _inject_collective_candidates (53/131/150 -> 13/24/69) loses: - the isinstance validation of its input; the only caller passes load_report_tasks' list of dicts. - the source_roots parameter, which only tests set; they now patch _aiter_csrc_root, the path production always takes. - the kernel_contract branch of the all-reduce check; write_reports stamps kernel_contract after injection, so no row carries it here. - the "dtypes" reads; nothing writes that key on a candidate. - the empty-family check; a name with an all-reduce tag always yields a non-empty family. - the local copy of the boolean env parser, replaced by hyperloom.common.env.env_flag (same tokens, same default). - the notes argument of the skip path; extraction logs nothing before the first two skips, so flushing the shared messages is the same write. Workload matching leaves the closure as _attach_collective_workloads, and the one-donor copy and the multi-donor merge become one _attach_workload. Rows, health warnings, log file text and log.warning records match base over 9,984 input combinations. The injection tests patch _aiter_csrc_root in setUp instead of passing source_roots. Co-authored-by: Cursor <cursoragent@cursor.com>
…tions cleared The six trace-analysis functions reworked on this branch are within every code-metrics limit, so their 13 baseline entries go: - tracelens_analysis.py: _enrich_kernel_contract (cognitive, lines), _finalize_candidates (cognitive, lines), _inject_collective_candidates (cyclomatic, cognitive, lines), classify_patchability (cognitive, lines), find_benchmark_files (cognitive, lines). - tracelens_skill_runner.py: aggregate_by_source_function (cognitive, lines). The module-lines entries fall to the measured lengths: tracelens_analysis.py 5347 -> 5272, tracelens_skill_runner.py 1366 -> 1308. No entry is added or raised. Co-authored-by: Cursor <cursoragent@cursor.com>
Code metrics gate: PASSEDJudged: the 6 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.
Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 1
Already over at the base (backlog, not this change's): 2
107 findings in files this change does not touchOutside this change (informational): 107
...and 82 more; the job log lists every row.
Thresholds and sources: |
…celens-gate Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # scripts/code_metrics_baseline.txt
Put back the three comments the rework dropped that say what the code cannot: that roctracer's activity buffer overflows under continuous graph replay, why a graph-replayed kernel without a cpu_op is not an attribution failure, and that the verdict's measures are reported but never compared. Comments that only narrated the code stay deleted. Co-authored-by: Cursor <cursoragent@cursor.com>
…rge with main Co-authored-by: Cursor <cursoragent@cursor.com>
ZhengGong-amd
left a comment
There was a problem hiding this comment.
Reviewed: gate numbers re-measured (ruff C901, complexipy, function lines) and match the description; each removed guard / "producer always sets it" claim checked against its producers and call sites; base vs head differential runs show no undisclosed behaviour change; tests covering the touched modules pass.
LGTM.
What this does
Six functions on the TraceLens trace-analysis path were over the code-metrics gate: five in
tracelens_analysis.pyandaggregate_by_source_functionintracelens_skill_runner.py. This PR brings each one under every limit it broke, mostly by deleting what it re-checked: guards on inputs whose only caller already guarantees them, branches no producer can reach, a parameter only tests passed, two lazily filled caches, and local copies of a shared env parser and a shared numeric coercion._inject_collective_candidatesalso hands its workload matching, which was a set of closures inside it, to named module functions.Production code: +347 / −473 (net −126) across 4 files, of which +339 / −472 is the two TraceLens modules. Tests: +6 / −10 in one file. The code-metrics baseline loses 13 entries, lowers the two module-length entries and gains none.
origin/main is merged into this branch (merge-base 958874f). The PR also carries three constraint comments that #2057 dropped and that came back after it merged: the
_TraceBuffers.graph_coveragedocstring and_finalizein_bypass_trace_reader.py, andbuild_verdictintrace_selfcert.py.This continues the complexity cleanup of #2024, #2026, #2052, #2057 and #2067. Each commit message lists what it deleted and why that is safe.
_enrich_kernel_contractclassify_patchabilityfind_benchmark_files_finalize_candidatesaggregate_by_source_function_inject_collective_candidates(also the test file)_TraceBuffers,build_verdictAll four production files are under
src/hyperloom/orchestrator/trace_analysis/.Gate numbers per function
Measured with the gate's own collectors (ruff C901, complexipy, function lines), origin/main → this branch. Limits are 20, 30 and 80; bold values were over the limit at base.
_enrich_kernel_contractclassify_patchabilityfind_benchmark_files_finalize_candidatesaggregate_by_source_function_inject_collective_candidatesNew functions, cyclomatic / cognitive / lines:
tracelens_analysis.py:_attach_collective_workloads12 / 27 / 63,_attach_workload12 / 25 / 41,_recover_csv_invocations10 / 21 / 46,_files_mentioning3 / 6 / 18,_first_shape_record7 / 14 / 14,_workload_family2 / 1 / 6,_workload_dtypes1 / 4 / 4,_has_workload1 / 1 / 3,_is_all_reduce_name1 / 1 / 3,_trace_name1 / 0 / 3.tracelens_skill_runner.py:_pitem_prose3 / 8 / 23.Deleted functions: the seven closures inside
_inject_collective_candidates(_name,_workload_dtypes,_has_workload,_is_all_reduce_workload,_workload_family,_first_shape_record,_merge_workloads) andfind_benchmark_files._is_multigpu.Across the 2 touched production files: C901 total 781 → 740, cognitive total 1,362 → 1,269, functions 150 → 153. Functions over a gate limit: cyclomatic 2 → 1, cognitive 11 → 5, length 11 → 5. What is left is outside this work:
main,write_reports,enrich_candidates_with_runtime_metadata,_apply_vendor_operator_playbook_grouping,_check_chunk_quality,_drive_sdk_streamandrun_tracelens_skill.Module lengths:
tracelens_analysis.py5,352 → 5,277,tracelens_skill_runner.py1,380 → 1,322.New names
_COLLECTIVE_OPS: the kernel-name tag → collective-op table that_enrich_kernel_contractreads for both the membership test and the lookup._BENCHMARK_GREP_INCLUDES: the grep--includeglobs offind_benchmark_files._MULTIGPU_PATH_TAGS: the path tags that sort multi-GPU benchmarks last._files_mentioning: one grep for one keyword under one directory._recover_csv_invocations: fills a candidate's shapes, invocation cases and raw arg spec fromops_unique_args.csv._pitem_prose: one prose entry per TraceLens P-item, P1 first._attach_collective_workloads: gives each source-resolved collective its traced workload and returns the rows to append._attach_workload: one donor lends its whole workload, several lend their distinct first inputs; it replaces the inline one-donor copy and_merge_workloads._trace_name,_workload_dtypes,_has_workload,_workload_family,_first_shape_record: the injector's closures moved to module level._is_all_reduce_name: whether a trace name carries all-reduce semantics; it is_is_all_reduce_workloadwithout its unreachablekernel_contractbranch.Removed
source_rootsparameter of_inject_collective_candidates, and thenotesparameter of its_skiphelper._finalize_candidates._pitem_prose_seenset that deduplicated P-item prose while rows arrived and had to be popped before return._COLLtuple that repeated_OPMAP's tags, and two of three reads of the TP size.hyperloom.common.env.env_flag) and the_safe_floatalias (nowsafe_float).tensilearm of theaten::check, the source-type condition on the aiter device-source promotion, thekernel_contractarm of the all-reduce check, thedtypesreads and the empty-family check.model_params or {}, the empty-input return, isinstance checks and try/except blocks inaggregate_by_source_function,proc.stdout or '',if source_file:andnot _sf or.dict.fromkeys), the nested_is_multigpu, and a 13-line grep argv.Why the deletions are safe
Line numbers are on this branch.
classify_patchability,tensilearm.libraryis always a lowercased string (tracelens_analysis.py:1218), and atensilelibrary returns at the vendor-backend check (:1219-1222;tensileis in_VENDOR_BACKEND_LIBRARIESat:1109) before theaten::check (:1223) can see it.not libraryandlibrary == ""are the same test on a string.classify_patchability, aiter promotion. For asource_typealready in{hip_cpp, triton, python, flydsl}the old code skipped the promotion and then returnedTrue, ""at the end (:1243-1246); the promotion (:1241-1242) now returns the same value one step earlier._inject_collective_candidates,source_rootsand input validation. The only production caller ismain(:5074-5079); it passes nosource_rootsand hands overload_report_tasksrows (:5070), which are always a list of dicts (_analysis_json.py:61-71). The roots come from_aiter_csrc_root()(:1733-1734), as they always did when no roots were passed._inject_collective_candidates,dtypesandkernel_contractreads._member_to_candidate(_analysis_json.py:90-114) writes neither key, and neither do the collective rows (_nccl_summary_candidates.py:230-256). The only writer ofkernel_contractis_enrich_kernel_contract(tracelens_analysis.py:3722), reached fromwrite_reports(:3879→:3641), whichmaincalls at:5184, after injection at:5074. Thedtypeswriters in the tree are in the bypass trace reader (_bypass_trace_reader.py:591,:724), a different route that does not reach the injector. A family key can no longer be empty: a row joins a family only if_is_all_reduce_namefinds an all-reduce tag in its name (:1580), and_workload_family(:1585-1590) never strips a name to nothing._inject_collective_candidates,_skipnow flushesmessageson every path. The first skip (:1760) runs before anything is appended. At the second (:1775),extract_collective_candidatesraisedValueError, and all of its raises (_nccl_summary_candidates.py:177-197, through_prorated_totalsat:113-151) come before its firstlog_fncall (:209,:225), somessagesis empty. The third skip (:1793) already flushed them. The log file text is the same on every path.env_flag.env_flagreads the same truthy tokens the deleted parser did: stripped, lowercased, one of1/true/yes/on; anything else is off (hyperloom/common/env.py:27,:44-61,:89-96).aggregate_by_source_function. Its only production caller isbuild_task_groups(tracelens_analysis.py:3725), which keeps only dict rows and returns[]before calling it when none is left (:3738-3741).build_task_groupsis called only fromwrite_reports(:3888), whose rows come from_finalize_candidates(:5098→:5139→:5184; the dry-run-only raw-trace path at:5162also ends in_finalize_candidates,:2399). Those rows carry numbers orNonein the fields read here:_analysis_json.py:92-94and:107,_nccl_summary_candidates.py:233-239, and_finalize_candidatesroundsduration_usand fills a missinggpu_pct(tracelens_analysis.py:2863,:2867-2868). Every group is created with the row that opens it, sorowsis never empty andlegacy_task_group_keysis always set. P-item prose still takes rows in arrival order (first row per P-item wins) and is then sorted by rank and title. The_safe_floatalias has no other reader in the tree._enrich_kernel_contract. Its only caller isenrich_candidates_with_runtime_metadata(:3641), which passes the result ofload_model_kernel_params(:3576); that function always returns a dict (:3482,:3483). The only writer ofkernel_contractis this function (:3722), and it writes a dict._COLLECTIVE_OPS(:3645) iterates in the same order the old_COLLtuple did, so the first match, and the barereducelast, are unchanged.find_benchmark_files. Its only production caller is_stamp_candidate_metadata(:2763-2765). It returns before the source-stem code whenrepo_rootis empty (:2065-2066).repo_rootis the candidate'skernel_repo, which_finalize_candidatessets withfind_repo_root(source_file)(:2872), andfind_repo_rootreturns""for an empty source file (:1822-1823). So with an emptysource_file, the code thatif source_file:guarded is never reached, andPath("").stemis empty anyway. Withtext=Trueandcapture_output=True,proc.stdoutis always a string._finalize_candidates.kernel_repois set at:2872before the FlyDSL fallback readsitem["kernel_repo"](:2882), andos.path.isfile("")is false, sonot _sf orwas redundant (:2878). The fused-MoE resolvers readops_unique_args.csvon every call and return new lists (:2573-2606,:2609-2626); nothing caches them, so no candidate shares a list with another.Observable effect
Nothing an operator sees changes. Candidate rows, task groups, the analysis JSON, trace-health warnings and log lines are the same for every input the repo's own callers produce.
The differences below are reachable only by input that no caller in the repo produces:
_inject_collective_candidates.source_roots=; only tests passed it. It always scans the aiter csrc root, as production already did.ValueError("Collective candidates must be a list of mappings"). Its only caller passesload_report_tasksrows.dtypeskey no longer counts as a workload, and a row that is all-reduce only through a pre-setkernel_contractno longer counts as all-reduce. At injection time no row carries either key.aggregate_by_source_function.Noneinput, non-dict rows, and non-numericduration_us,call_count,gpu_pctortracelens_pitem_ranknow raise. The base returned[], skipped the row or the metric, or read the rank as 0. Its only caller passes_finalize_candidatesrows, which carry those fields as numbers orNone._enrich_kernel_contract.model_params=Noneand a non-dictkernel_contractnow raise. The only caller passes the dict thatload_model_kernel_paramsreturns, and only this function writeskernel_contract.find_benchmark_files. A non-emptyrepo_rootwithsource_file=Nonenow raisesTypeError. The only caller derivesrepo_rootfrom the source file, so a missing source file means an emptyrepo_rootand an early return._finalize_candidates. Fused-MoE candidates no longer share one CSV read: each readsops_unique_args.csvitself, as every other candidate in the loop already did, and gets its own invocation-case dicts instead of shared ones. The CSV is the same file within one run, so the written JSON is identical.Checklist
ExtractCollectiveCandidatesTests.setUpintest_nccl_summary_candidates.pypatchestracelens_analysis._aiter_csrc_rootto return the test's source root (stopped inaddCleanup), and eleven injection calls drop the deletedsource_roots=argument. No assertion changed or was removed, so every contract and failure-mode assertion of these tests still runs, now through the production root lookup.test_injection_without_a_source_root_is_also_reportedstill patches_aiter_csrc_rootto""itself, so it still pins the missing-root health warning. The keyword it passed,source_roots=[], already fell back to that lookup on main.test_tracelens_csv.py:3202,"_pitem_prose_seen" not in g) still holds.tracelens_analysisfails at collection on this branch and on the base alike:tracelens_analysis.pyimports TraceLens at module level (unchanged since 2026-09), and TraceLens is not installed on that machine. For the same reason the nineExtractCollectiveCandidatesTestsfail there on this branch only, because theirsetUpnow patchestracelens_analysis._aiter_csrc_root; every other local result matches the merge-base. Linux CI on this PR, which installs TraceLens, is the test authority.[tool.coverage.run] omit(pyproject.toml:372-373), so the diff-coverage gate has no changed statements to count.classify_patchability(79 lines, cyclomatic 19, cognitive 29) andaggregate_by_source_function(79 lines) sit close to the triggers; each is one ordered decision ladder or one loop over its rows.classify_patchability(19, from 20),_inject_collective_candidates(13, from 53),_attach_collective_workloads(12),_attach_workload(12) and_enrich_kernel_contract(11, from 12). The first is the routing gate's ordered rejections; the two new ones are the one match-and-attach loop and the one-donor or many-donor copy.tracelens_analysis.py(5,277 lines) andtracelens_skill_runner.py(1,322) stay over the 1,200-line module limit the baseline carries for them; each gets shorter.source_roots=keyword belongs to a private function that only tests called with it.Known merge interactions
Trial merges used
git merge-tree --write-tree, against this branch and against origin/main alone. None of the PRs below adds a call to a function or name this PR deletes or re-signs.tracelens_skill_runner.py, and [fix] close the defects behind the MuseSpark run shortfall and delete state nothing reads #2055 sharestracelens_analysis.py. All three merge cleanly in those modules. Each conflicts in the baseline file against main alone as well, so this PR introduces no conflict with them.scripts/code_metrics_baseline.txtwith this one, among them refactor(fusion): delete the dead and repeated code in thirteen over-gate functions #2067.Gate
scripts/code_metrics.py --base-ref 958874f20(the merge-base) passes on the 6 touched files: 0 new, 0 worse and 0 out-of-date entries, no new or worse duplication or dead code, and 0 findings from the comment, English-only, repeated-literal and A1–A12 checks. Baseline: −13 entries, +0; the two TraceLens module-length entries are set to the merged lengths, 5,277 and 1,322 (main's entries are 5,347 and 1,366).The gate lists two units under Already over at the base, both untouched here:
main(898 lines) andwrite_reports(204) intracelens_analysis.py, each two lines over the 896 and 202 that main's baseline carries for them. It warns that_bypass_trace_reader.pyis 883 lines, over the 800-line warning and within the 1,200-line limit; the three restored comments add six of those lines.ruff checkandruff format --checkare clean on the touched files.