[Feature] L3 (multi-rank) swimlane: merge per-rank L2 chip swimlanes into a single cross-rank trace

[Hzfengsy]

Summary

Add an L3 (multi-rank) swimlane: one Perfetto-loadable trace that combines every
rank's L2 chip swimlane into a single file, with each rank on its own process
group and all ranks placed on a common time base.

Today a distributed run produces one independent merged_swimlane_*.json per rank
per dispatch, and there is no supported way to view them together:

<work_dir>/dfx_outputs/rank{r}/d{k}/
├── chip_swimlane_records.json
├── deps.json
├── dispatch_program.json
├── name_map.json
└── merged_swimlane_<ts>.json      # per-rank, standalone

swimlane_converter takes a single input records file and emits a single-chip
trace, so an 8-card run means opening 8 Perfetto tabs and eyeballing the
alignment by hand.

Motivation / Use Case

Every interesting question about an L3 kernel is a cross-rank question, and
none of them can be answered from per-rank files viewed separately:

• Where does the collective actually wait? AllToAll / AllReduce cost is
  "rank A finished its producer at t1, rank B only started consuming at t2".
  That interval spans two files today.
• Which rank is the straggler, and why? Benchmarking guidance is to report
  the fastest rank and treat the rest as skew or load imbalance — but telling
  start skew from a real load-balance gap needs all ranks on one timeline.
• Did the comm window overlap compute? The whole point of fusing attention
  output with AllToAll (or overlapping MoE dispatch/combine) is that one rank's
  communication hides under another rank's compute. That overlap is invisible
  when each rank is a separate trace.
• Is a cross-rank dependency the blocker? Per-rank deps.json stops at the
  card boundary, so a notify/wait or remote_store edge never shows up as an
  arrow.

Concretely, per-rank files are not even directly comparable, because the raw
cycle domains are not aligned. From an 8-card run
(dfx_outputs/rank{0..7}/d0/chip_swimlane_records.json, clock_freq_hz
50 MHz, level 4), the earliest AICore timestamp per rank:

rank0: 51908339281637   +0.000 s
rank1: 51908283781958   -1.110 s
rank2: 51908233004989   -2.126 s
rank3: 51908183305266   -3.120 s
rank4: 51908132029528   -4.145 s
rank5: 51908081665067   -5.152 s
rank6: 51908027076615   -6.244 s
rank7: 51907976948801   -7.247 s

The per-rank capture windows are 1–62 ms wide, so a monotonic ~1 s-per-rank
offset spanning 7.2 s cannot be real execution skew inside one collective. A
2-card run shows the same shape (rank0/rank1 offset ~1.02 s for ~10 ms
windows). Whatever the cause — per-die counter bases, staged capture start, or
the dump path — a naive concatenation of the per-rank JSONs would render a
badly wrong picture, which is exactly why this belongs in the tool rather than
in each user's ad-hoc script.

Proposed API / Behavior

1. Cross-rank merge in swimlane_converter. Accept a run directory (or a
   list of per-rank records files) and emit one combined trace:

   python -m simpler_setup.tools.swimlane_converter \
       <work_dir>/dfx_outputs --dispatch d0 \
       -o <work_dir>/dfx_outputs/l3_swimlane_<ts>.json

   Per-rank deps.json / name_map.json continue to be picked up as siblings,
   as they are today.

2. Namespace the lanes per rank. The converter currently hard-codes small
   process ids for its views (pid 2 = AICPU Scheduler, 3 = Scheduler View,
   4 = Worker View, 5 = Graph Execution). Those collide across ranks, so the
   merge must offset or re-key them, e.g. pid = rank * 100 + view_id, with
   process_name metadata carrying the rank (rank3 / Worker View) and
   process_sort_index grouping ranks in order.

3. Put all ranks on one time base, and say which one. Options, in
   preference order:

   • a synchronized cross-die counter if one is available;
   • the host [STRACE] CLOCK_MONOTONIC stamps, which are already comparable
     across the parent and its forked chip workers (noted in [Feature] Host swimlane (L3/L4) — the missing third DFX timeline #1708), used to
     compute a per-rank cycle→host offset;
   • failing both, an explicit anchor event (a collective all ranks provably
     participate in) with the residual error reported.

   Whichever is used, the output should record the alignment method and its
   estimated error in the trace metadata, so a reader knows whether a 5 µs
   cross-rank gap is signal or noise. An --align=none escape hatch for raw
   cycle values is fine as long as it is not the default.

4. Cross-rank dependency arrows (nice-to-have). Once ranks share a time
   base, comm edges (notify/wait, put / remote_store / remote_load) can be
   drawn as Perfetto flow events between ranks — the single highest-value thing
   the combined view unlocks beyond simple co-display.

5. Viewer support. The pypto-toolkit VSCode extension
   renders chip_swimlane_records.json natively (added in [Feature] Add native swimlane preview in PyPTO-Toolkit VSCode extension for new PyPTO traces #512). It will need a
   corresponding update to open a multi-rank trace — either by reading the new
   combined file, or by accepting a dfx_outputs/ directory and doing the merge
   itself. Filing here since simpler owns the format; the extension change is
   presumably tracked separately, but the format should be agreed before either
   side implements.

Alternatives Considered

• Open N Perfetto tabs, one per rank. What we do today. Cross-rank intervals
  have to be read off two windows and subtracted by hand, and the unaligned
  cycle domains above make that unreliable.
• Concatenate the per-rank merged_swimlane_*.json files by hand. Breaks on
  both counts named above: hard-coded pids collide, and the timestamps are not
  on a common base — it produces a plausible-looking, wrong trace.
• Per-user ad-hoc merge scripts. Already happening informally; each one
  re-derives (and re-gets-wrong) the alignment, and none of them is validated
  against the runtime that produced the records.

Additional Context

• Surfaced while tuning distributed DeepSeek-V4 decode kernels in pypto-lib
  (8-card _jit_l3_moe, 2-card _jit_l3_dspark_drafter), where AllToAll /
  comm-window overlap is the thing being optimized.
• Environment for the data above: simpler 799640e6, pypto 9561f294,
  platform a2a3, --enable-chip-swimlane 4.
• Existing per-rank layout and the level-4 requirement are documented in
  docs/dfx/chip-swimlane-profiling.md;
  §7.3 already notes that "the current implementation captures incore-level
  scope only" and that L3 composition is not visible as nested swimlane scopes.
• Related: [Tracking] DFX (Diagnostics) capability overview & roadmap #995 (DFX capability tracking), [Feature] Host swimlane (L3/L4) — the missing third DFX timeline #1708 (host swimlane — complementary:
  that one adds a host-side timeline, this one merges the existing device-side
  ones across ranks), [Feature] Add native swimlane preview in PyPTO-Toolkit VSCode extension for new PyPTO traces #512 (VSCode extension native swimlane preview).