feat(dfx): profile A5 HBG AICore scheduler

docs/dfx/args-dump.md

@@ -194,8 +194,8 @@ collector receives no records and exports no manifest — that is the deliberate
 
 The dump artifacts land under the per-task output prefix
 (`CallConfig::output_prefix`, set by
-`scene_test.py::_build_output_prefix` to
-`outputs/<ClassName>_<case>_<YYYYMMDD_HHMMSS>/` for SceneTest runs):
+`scene_test.py::build_output_prefix` to
+`outputs/<ClassName>_<case>_<YYYYMMDD_HHMMSS>_<unique>/` for SceneTest runs):
 
 ```text
 <output_prefix>/

docs/dfx/chip-swimlane-profiling.md

@@ -28,8 +28,8 @@ end-to-end runtime numbers. Two cases dominate the profiler diet:
   pinpoints the fix.
 
 chip swimlane profiling captures both: per-task `(start, end,
-dispatch, finish)` records on the AICore side, plus per-iteration
-phase records on the AICPU scheduler side and per-submit orchestrator
+dispatch, finish)` records, plus per-iteration phase records from the active
+AICPU or AICore scheduler producer and per-submit orchestrator
 envelopes. The host writes a Chrome Trace Event JSON
 that loads directly in Perfetto, and the same file feeds a
 scheduler-overhead deep-dive report when a device log is
@@ -45,7 +45,7 @@ available.
   `chip_swimlane_records.json` with `deps.json` from
   [`dep_gen`](dep-gen.md) at post-process time; see
   [§3.5](#35-dependency-arrows-from-dep_gen).
-- **AICPU scheduler phases** — per-iteration breakdown into mutually
+- **Scheduler phases** — producer-specific per-iteration breakdown. AICPU uses mutually
   time-exclusive **outer** phases (`complete` / `async_poll` / `dispatch` /
   `release` / `dummy` / `early_dispatch` / `drain` / `graph_prepare`), plus
   nested phases.
@@ -106,9 +106,9 @@ backward-compatible with the old boolean behavior).
 | Level | Collects | Notes |
 | ----- | -------- | ----- |
 | 0 | Nothing (disabled) | Default when flag is absent |
-| 1 | AICore timing only (start_time_us/end_time_us/task_id/func_id/core_type) | No AICPU timestamps |
-| 2 | + dispatch_time_us, finish_time_us | Full per-task AICPU record |
-| 3 | + scheduler phases (`aicpu_scheduler_phases[]`) | Skips orchestrator phases |
+| 1 | AICore timing only (start_time_us/end_time_us/task_id/func_id/core_type) | No Scheduler timestamps |
+| 2 | + Scheduler per-task dispatch_time_us, finish_time_us | Producer is identified as `aicpu` or `aicore` |
+| 3 | + scheduler phases (`scheduler_records`) | Skips orchestrator phases |
 | 4 | + orchestrator phases (`aicpu_orchestrator_phases[]`) | Full collection |
 
 Dependency arrows are not produced by any swimlane level — see
@@ -135,13 +135,13 @@ The flag sets `CallConfig::enable_chip_swimlane` to the chosen
 level. The host then allocates the per-core / per-thread shared
 region and publishes its base address through
 `kernel_args.chip_swimlane_data_base`. AICore writes timing into
-per-task WIP slots; AICPU commits the records on FIN. Per-task
-dispatch/finish timestamps are recorded only at level >= 2,
+per-task WIP slots; the active Scheduler records dispatch/finish timestamps.
+Per-task Scheduler timestamps are recorded only at level >= 2,
 scheduler phase records only at level >= 3, and orchestrator phase
 records only at level >= 4.
 
 The JSON output `"chip_swimlane_level"` field is the captured perf_level:
-`1` = AICore timing only, `2` = +AICPU dispatch/finish,
+`1` = AICore timing only, `2` = +Scheduler per-task dispatch/finish,
 `3` = +scheduler phases, `4` = +orchestrator phases.
 
 Chip-swimlane collection is disabled when `--rounds > 1` so benchmark
@@ -151,8 +151,8 @@ runs are not instrumented.
 
 The raw artifact lands under the per-task output prefix
 (`CallConfig::output_prefix`, set by
-`scene_test.py::_build_output_prefix` to
-`outputs/<ClassName>_<case>_<YYYYMMDD_HHMMSS>/` for SceneTest
+`scene_test.py::build_output_prefix` to
+`outputs/<ClassName>_<case>_<YYYYMMDD_HHMMSS>_<unique>/` for SceneTest
 runs):
 
 ```text
@@ -231,31 +231,51 @@ layers to be aware of:**
     "core_to_thread": [<int>, ...]     // optional; level >= 3 only
   },
 
-  // Bulk task streams — flat array of tuples. Column order is fixed.
+  // Bulk task streams. Tuple column order is fixed.
   //   aicore_tasks: [core_id, task_token_raw, reg_task_id,
   //                  start_cycles, end_cycles]
-  //   aicpu_tasks:  [core_id, reg_task_id,
-  //                  dispatch_cycles, finish_cycles]
+  //   scheduler_tasks.records: [core_id, reg_task_id,
+  //                             dispatch_cycles, finish_cycles]
   "aicore_tasks": [[...], ...],
-  "aicpu_tasks":  [[...], ...],
+  "scheduler_tasks": {
+    "schema_version": 1,
+    "producer": "<aicpu|aicore>",
+    "records": [[...], ...]
+  },
+
+  // Producer-neutral per-Scheduler streams (level >= 3 only).
+  "scheduler_records": {
+    "schema_version": 1,
+    "streams": [{
+      "platform": "<a2a3|a5>",
+      "runtime": "<host_build_graph|tensormap_and_ringbuffer>",
+      "producer": "<aicpu|aicore>",
+      "scheduler_id": <int>,
+      "worker_id": <int>,
+      "core_type": "<aicpu|aic|aiv>",
+      "physical_core_id": "<int|null>",
+      "capture": {"committed": <int>, "dropped": <int>, "truncated": <bool>},
+      "records": [{"start_cycles": <int>, "end_cycles": <int>,
+                   "loop_iter": <int>, "kind": <str>,
+                   "tasks_processed": <int>, "task_id": "<int|null>"}],
+      "metrics": [{"record_index": <int>, ...}]
+    }]
+  },
 
-  // Per-scheduler-thread arrays of objects (level >= 3 only).
-  //   sched record: {kind, start_cycles, end_cycles, loop_iter,
-  //                  tasks_processed, [pop_hit, pop_miss]}
+  // Orchestrator records (level >= 4 only).
   //   orch record:  {submit_idx, task_id, start_cycles, end_cycles}
-  // pop_hit / pop_miss are present only on Dispatch records.
-  "aicpu_scheduler_phases":    [ [ {...}, ... ], ... ],
   "aicpu_orchestrator_phases": [ [ {...}, ... ], ... ]   // level >= 4 only
 }
 ```
 
 All timestamps on disk are raw `get_sys_cnt` cycles (uint64). The
-join key between `aicore_tasks` and `aicpu_tasks` is
+join key between `aicore_tasks` and `scheduler_tasks.records` is
 `(core_id, reg_task_id)` — *not* `task_token_raw`, because SPMD
 `block_num > num_cores` and MIX cluster spread can dispatch the same
 `task_token_raw` to the same core multiple times. AICore is the
-canonical producer of `task_token_raw`; AICPU only stamps the
-dispatch / finish timestamps and the per-core join token.
+canonical producer of `task_token_raw`; the Scheduler producer stamps the
+dispatch / finish timestamps and the per-core join token. Archived raw files
+with the former `aicpu_tasks` array remain readable as `producer: "aicpu"`.
 
 #### Reader output (µs domain)
 
@@ -268,15 +288,16 @@ microseconds, downstream code sees:
 | `func_id` | Kernel function id. Always `-1` on disk; resolved post-process from `deps.json::tasks[].kernel_ids[3]` (see `swimlane_converter.resolve_func_id_from_kernel_map`) |
 | `core_id` / `core_type` | Physical core index and `"aic"` / `"aiv"` string |
 | `start_time_us` / `end_time_us` / `duration_us` | AICore execution window in microseconds |
-| `dispatch_time_us` | AICPU timestamp when this task was dispatched (filled at level >= 2; `0.0` at level 1) |
-| `finish_time_us` | AICPU timestamp when AICPU observed FIN (filled at level >= 2; `0.0` at level 1) |
+| `dispatch_time_us` | Scheduler timestamp when dispatch publication completed (filled at level >= 2) |
+| `finish_time_us` | Scheduler timestamp when completion processing began (filled at level >= 2) |
 
 Note: per-task records carry **no** fanout edges. Dependency arrows
 come from a separate `deps.json` (dep_gen) joined at convert time —
 see [§3.5](#35-dependency-arrows-from-dep_gen).
 
-Phase records (per scheduler thread, level >= 3 for
-`aicpu_scheduler_phases[]` and level >= 4 for
+Phase records (per Scheduler stream, level >= 3 in raw
+`scheduler_records`—also exposed through the legacy reader alias
+`aicpu_scheduler_phases`—and level >= 4 for
 `aicpu_orchestrator_phases[]`):
 
 | Field | Meaning |
@@ -446,13 +467,13 @@ same lane structure — the directory form repeats it once per Rank under the
 
 - **Orchestrator** (pid=1) — per-submit `orch_submit` envelope
   blocks (level >= 4).
-- **AICPU Scheduler** (pid=2) — per-iteration scheduler phase
+- **Scheduler** (pid=2) — per-iteration scheduler phase
   blocks coloured by `phase` (level >= 3). Outer phases appear as sibling bars
   on each scheduler thread's first `Sched_N` lane. TMR's nested `resolve`
   appears on an adjacent `Sched_N` sub-lane; HBG's standalone `resolve` stays
   on the P thread's first lane. `drain_prepare` and `drain_publish` nest within
   `drain`.
-- **Scheduler View** (pid=3) — task-execution overlay using AICPU
+- **Scheduler View** (pid=3) — task-execution overlay using Scheduler
   dispatch/finish timestamps (level >= 2), with the same labels
   as Worker View.
 - **Worker View** (pid=4) — one swim-lane per physical worker:

docs/dfx/dep-gen.md

@@ -134,7 +134,7 @@ runs that the converter joins back to that captured graph.
 When `--enable-dep-gen` is on with any other diagnostic flag, an
 `output_prefix` directory must be set (the runtime throws otherwise).
 The standard SceneTest path
-(`outputs/<TestName>_<case>_<timestamp>/`) handles that automatically.
+(`outputs/<TestName>_<case>_<timestamp>_<unique>/`) handles that automatically.
 
 ---
 

docs/dfx/pmu-profiling.md

@@ -71,8 +71,8 @@ rounds stay uninstrumented.
 ### 3.2 Output
 
 The PMU artifact is a CSV file under the per-task output prefix
-(`CallConfig::output_prefix`, set by `scene_test.py::_build_output_prefix`
-to `outputs/<ClassName>_<case>_<YYYYMMDD_HHMMSS>/` for SceneTest runs):
+(`CallConfig::output_prefix`, set by `scene_test.py::build_output_prefix`
+to `outputs/<ClassName>_<case>_<YYYYMMDD_HHMMSS>_<unique>/` for SceneTest runs):
 
 ```text
 <output_prefix>/pmu.csv

docs/dfx/profiling-name-map.md

@@ -173,8 +173,8 @@ python -m simpler_setup.tools.deps_viewer \
 
 Each test case writes its diagnostic artifacts under
 `CallConfig::output_prefix` (chosen by
-`scene_test.py::_build_output_prefix` as
-`outputs/<ClassName>_<case>_<YYYYMMDD_HHMMSS>/`). Filenames are fixed —
+`scene_test.py::build_output_prefix` as
+`outputs/<ClassName>_<case>_<YYYYMMDD_HHMMSS>_<unique>/`). Filenames are fixed —
 the per-case directory is the uniqueness boundary, so parallel runs
 cannot collide.
 

docs/testing.md

@@ -380,7 +380,7 @@ A single file can declare both L2 and L3 classes; they're grouped by `(runtime,
 
 ### Profiling under parallelism
 
-Each test case sets its own `CallConfig.output_prefix` (chosen by `scene_test.py::_build_output_prefix` as `outputs/<ClassName>_<case>_<YYYYMMDD_HHMMSS>/`). The C++ runtime writes all diagnostic artifacts under that prefix with fixed filenames:
+Each test case sets its own `CallConfig.output_prefix` (chosen by `scene_test.py::build_output_prefix` as `outputs/<ClassName>_<case>_<YYYYMMDD_HHMMSS>_<unique>/`). The C++ runtime writes all diagnostic artifacts under that prefix with fixed filenames:
 
 - `outputs/<case>_<ts>/chip_swimlane_records.json` — swimlane (`--enable-chip-swimlane`)
 - `outputs/<case>_<ts>/args_dump/` — args dump (`--dump-args`)

simpler_setup/runtime_builder.py

@@ -389,6 +389,11 @@ def _compile_target(target: str) -> Path:
                     pto_root = ensure_pto_isa_root(verbose=True)
                 defines["PTO_ISA_ROOT"] = pto_root
             if target == "host":
+                # host_runtime.so is built once per runtime implementation, so
+                # bake that immutable artifact identity into the shared host
+                # platform code.  Runtime code must not grow a virtual/static
+                # API merely to report the directory it was compiled from.
+                defines["SIMPLER_RUNTIME_NAME"] = name
                 if build_pto_isa_commit:
                     defines["SIMPLER_PTO_ISA_BUILD_COMMIT"] = build_pto_isa_commit
                 for opt_in_define in ("SIMPLER_ENABLE_PTO_URMA_WORKSPACE",):

simpler_setup/scene_test.py

@@ -29,6 +29,7 @@
 import os
 import platform as host_platform
 import sys
+import tempfile
 from collections.abc import Mapping
 from concurrent.futures import ThreadPoolExecutor
 from contextlib import contextmanager
@@ -1019,11 +1020,9 @@ def _outputs_dir() -> Path:
 def build_output_prefix(case_label: str) -> Path:
     """Per-case directory for diagnostic artifacts.
 
-    Each case gets its own ``outputs/<case_label>_<timestamp>/`` directory; the
+    Each invocation gets its own ``outputs/<case_label>_<timestamp>_<suffix>/`` directory; the
     runtime writes ``chip_swimlane_records.json``, ``args_dump/``, and ``pmu.csv``
-    under that root with fixed filenames. Two cases of the same name run in
-    the same second is not a contemplated scenario (parallel xdist runs differ
-    by class+method).
+    under that root with fixed filenames.
 
     The directory is created here: the dep_gen host replay (and any other
     writer) ``fopen``s ``<prefix>/<file>`` directly without an mkdir of its
@@ -1033,9 +1032,9 @@ def build_output_prefix(case_label: str) -> Path:
 
     timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
     safe_label = _sanitize_for_filename(case_label)
-    prefix = _outputs_dir() / f"{safe_label}_{timestamp}"
-    prefix.mkdir(parents=True, exist_ok=True)
-    return prefix
+    outputs = _outputs_dir()
+    outputs.mkdir(parents=True, exist_ok=True)
+    return Path(tempfile.mkdtemp(prefix=f"{safe_label}_{timestamp}_", dir=outputs))
 
 
 def _run_swimlane_converter(
@@ -1400,7 +1399,8 @@ def run_class_cases(  # noqa: PLR0913 -- shared layer-5 entry; kwargs mirror CLI
     Caller is responsible for platform/selector/manual filtering. Profiling
     snapshots wrap each case. Execution failures carry the class and case name,
     with the original exception preserved as their cause; the caller decides
-    fail-fast vs collect semantics.
+    fail-fast vs collect semantics. Returns diagnostic output prefixes keyed by
+    case name.
     """
     cls_name = type(cls_inst).__name__
     callable_spec = getattr(type(cls_inst), "CALLABLE", None)
@@ -1412,13 +1412,16 @@ def run_class_cases(  # noqa: PLR0913 -- shared layer-5 entry; kwargs mirror CLI
         or enable_scope_stats
         or enable_swimlane_overhead
     )
+    output_prefixes = {}
     for case in cases:
         case_label = f"{cls_name}_{case['name']}"
         # Per-case directory the runtime writes into. Required (non-empty) when
         # any diagnostic flag is on; CallConfig::validate() throws otherwise.
         # scope_stats writes below the per-case output prefix, so it uses the
         # same output-prefix allocation as the other diagnostics.
         prefix = build_output_prefix(case_label) if diagnostics_on else Path("")
+        if diagnostics_on:
+            output_prefixes[case["name"]] = prefix
         try:
             cls_inst._run_and_validate(
                 worker,
@@ -1447,6 +1450,7 @@ def run_class_cases(  # noqa: PLR0913 -- shared layer-5 entry; kwargs mirror CLI
                 scope_stats=enable_scope_stats,
                 swimlane_overhead=enable_swimlane_overhead,
             )
+    return output_prefixes
 
 
 def _compare_outputs(test_args, golden_args, output_names, rtol, atol):
@@ -2107,7 +2111,7 @@ def test_run(self, st_platform, st_worker, request):
         if self._st_level == 3 and chip_handles:
             callable_obj = {**chip_handles}
 
-        run_class_cases(
+        self._diagnostic_output_prefixes = run_class_cases(
             st_worker,
             self,
             matched,

simpler_setup/tools/README.md

@@ -295,7 +295,8 @@ After the test passes, the tool will:
 
 ## sched_overhead_analysis
 
-Answer **"is the AICPU scheduler the bottleneck, or is it starved?"** by
+Answer **"is the scheduler the bottleneck, or is it starved?"** for either an
+AICPU or AICore scheduler by
 measuring, dependency- and MIX-aware, how much of the makespan a free core has
 ready, undispatched work — vs. legitimately busy or dependency-limited. Full
 model: [docs/dfx/sched-overhead-model.md](../../docs/dfx/sched-overhead-model.md).
@@ -305,9 +306,10 @@ model: [docs/dfx/sched-overhead-model.md](../../docs/dfx/sched-overhead-model.md
 `sched_overhead_analysis` needs **two artifacts, captured in SEPARATE runs**
 (co-running the flags perturbs timing — `dep_gen` adds per-submit overhead):
 
-1. **Perf profiling data** (`chip_swimlane_records_*.json`, level >= 3) from a
-   `--enable-chip-swimlane` run — per-task dispatch/start/end/finish +
-   `aicpu_scheduler_phases`.
+1. **Perf profiling data** (`chip_swimlane_records_*.json`, level >= 2) from a
+   `--enable-chip-swimlane` run — per-task dispatch/start/end/finish. Level >= 3
+   also supplies `scheduler_records` for the phase breakdown (legacy artifacts
+   with `aicpu_scheduler_phases` remain readable).
 2. **`deps.json`** (the task DAG) from a separate `--enable-dep-gen` run. It
    drives `ready(C) = max(producer.end)`, which is what separates scheduler
    bubbles from dependency stalls. **Required** — the tool errors without it.
@@ -333,7 +335,7 @@ python -m simpler_setup.tools.sched_overhead_analysis \
 
 | Option | Description |
 | ------ | ----------- |
-| `--chip-swimlane-records-json` | Path to the chip_swimlane_records_*.json file (level >= 3). If omitted, the latest under outputs/ is auto-selected. |
+| `--chip-swimlane-records-json` | Path to the chip_swimlane_records_*.json file (level >= 2). If omitted, the latest under outputs/ is auto-selected. |
 | `--deps-json` | Path to deps.json from a `--enable-dep-gen` run. **Required.** Falls back to a `deps.json` sibling of the perf JSON if present. |
 
 ### Outputs
@@ -343,10 +345,11 @@ Emitted in six parts:
 - **Part 1: Overhead verdict** — per-engine overhead (idle T-core *and* a ready, undispatched T-task, MIX-aware) + system `all_overhead` / `has_overhead`, all as % of makespan. An engine with no ready work is not overhead (dependency-mandated idle, not waste).
 - **Part 2: aicore switch** — the pre-dispatched pickup gap (`dispatch < prev_end`), reported **per core** (min/mean/max, ~0.8 µs each), the overhead-vs-independent split, and the makespan switch bound `[min over cores, sum of per-engine minima]`.
 - **Part 3 / 4: Head / Tail OH distributions** — P10–P99 + mean + total (per-task pickup and detect-latency magnitude).
-- **Part 5: AICPU scheduler loop breakdown** — per-thread loops, ns/loop, complete/dispatch/idle phase ratios, pop_hit / pop_miss, fanout / fanin, + the tail-vs-loop cause analysis.
+- **Part 5: Scheduler phase breakdown** — Level >= 3 reports the producer's phases. AICPU includes per-thread loop, queue-pop, fanout/fanin, and tail-vs-loop metrics; AICore reports its bootstrap/fanin/ready/dispatch/complete/refill/resolve/idle phase totals without applying AICPU-only queue formulas. At Level 2 this section is explicitly marked unavailable while Parts 1–4 and 6 remain available.
 - **Part 6: Critical-path latency attribution** — along the makespan path, scheduler-injected µs vs compute µs ("scheduler adds X% to the critical path").
 
-The perf JSON must be captured at chip_swimlane_level >= 3 so that `aicpu_scheduler_phases` is non-empty (rerun the case with `--enable-chip-swimlane` if the tool reports the field is missing).
+The common dependency-aware analysis works at chip_swimlane_level >= 2 for
+both scheduler producers. Capture level >= 3 when phase attribution is needed.
 
 ---
 
@@ -717,9 +720,13 @@ not from the perf JSON. See [`swimlane_converter --deps-json`](#swimlane_convert
 Top-level layout depends on `chip_swimlane_level`:
 
 - All levels: `chip_swimlane_level`, `tasks[]` (per-task fields above).
-- `>= 3`: also `aicpu_scheduler_phases[]` (per-thread phase records:
-  scan / complete / dispatch / idle) and `core_to_thread[]` (core_id →
-  scheduler thread index).
+- A5 HBG `>= 2`: also `aicpu_lifecycle_records[]`; the converter renders the
+  real handshake, topology/configuration, context-publication, bootstrap-wait,
+  register-release, and exit timestamps under `AICPU Lifecycle`.
+- `>= 3`: also `scheduler_records.streams[]`. Every Record has the common
+  `start_cycles`, `end_cycles`, `loop_iter`, `kind`, `tasks_processed`, and
+  nullable `task_id` fields. Stream metadata selects the AICPU or AICore
+  interpretation; producer-specific counters live in `metrics[]`.
 - `>= 4`: also `aicpu_orchestrator_phases[]` (per-task orchestrator
   phase records).