Add: SIMPLER_TENSOR_DATA_TIMEOUT_MS to override the tensor-data wait

docs/dfx/args-dump.md

@@ -936,6 +936,15 @@ the no-progress budget without onboard-only ordering limits. CI restores the
 old fast-fail values through these env vars: 2 s scheduler, 3 s op-execute,
 and 4 s stream-sync for onboard jobs; 5 s scheduler for sim jobs.
 
+`SIMPLER_TENSOR_DATA_TIMEOUT_MS` overrides a fourth wait — the orchestration
+`get_tensor_data`/`set_tensor_data` spin, 15 s by default in
+`tensormap_and_ringbuffer` — and is deliberately outside that ordering group. It
+is a diagnostic refinement: expiring first buys the precise `TENSOR_WAIT_TIMEOUT`
+code with its producer locator, while the clean shutdown is the scheduler
+watchdog's job for every stall shape. Folding it into the ordering check would
+let a value the host cannot even see (the default is compiled into the runtime)
+invalidate an otherwise valid override set.
+
 - **Device-side graceful flush (primary).** At 20 s of no progress
   the AICPU declares the hang, runs the end-of-loop flush, *and*
   dumps the **partial output** of every task still RUNNING on a core

docs/dfx/profiling-config-naming.md

@@ -127,6 +127,7 @@ simulator platforms.
 | `SIMPLER_OP_EXECUTE_TIMEOUT_US` | host platform | µs | op-execute timeout (overrides the `platform_config.h` compile default) |
 | `SIMPLER_STREAM_SYNC_TIMEOUT_MS` | host platform | ms | stream-sync timeout |
 | `SIMPLER_SCHEDULER_TIMEOUT_MS` | host platform | ms | scheduler no-progress timeout |
+| `SIMPLER_TENSOR_DATA_TIMEOUT_MS` | host platform | ms | orchestration tensor-data wait timeout |
 
 ### Per-run DFX flag bitmask (`platform_config.h`)
 

docs/troubleshooting/device-error-codes/stall.md

@@ -17,12 +17,11 @@ the technique transfers:
 
 - To make the **scheduler** win (get a `sub_class`), squeeze it below the others:
   `SCHEDULER=2000ms < OP_EXECUTE=3s < STREAM_SYNC=4000ms` (`aicore_hang` case).
-- To let a **slow-but-alive** path finish, push the others out of the way:
-  `SCHEDULER=30000ms`, `OP_EXECUTE=30s`, `STREAM_SYNC=40000ms` — this is how the
-  `tensor_wait_timeout` case lets the 15 s tensor-data wait land code 8 instead of
-  being reaped first.
+- To make the **tensor-data wait** win (get code 8 and its producer locator),
+  squeeze `TENSOR_DATA=1000ms` below all three — it sits outside their ordering
+  group, so lowering it alone leaves them valid (`tensor_wait_timeout` case).
 
-**These three are read once, at `Worker.init()`.** Changing them between `run()`
+**These four are read once, at `Worker.init()`.** Changing them between `run()`
 calls on the same Worker does nothing — you must rebuild the Worker (or use a
 separate process) per value. Defaults and the rationale are in
 [`../local-timeout-defaults.md`](../local-timeout-defaults.md).
@@ -63,12 +62,19 @@ exist) before reading. See the "Device logs" section of
 ## Code 8, specifically
 
 Only `tensormap_and_ringbuffer` raises this code. Its tensor-data wait defaults to
-15 s (`TENSOR_DATA_TIMEOUT_MS`, frequency-scaled). It means either the producer
+15 s (`TENSOR_DATA_TIMEOUT_MS`, frequency-scaled) and is overridden per run by
+`SIMPLER_TENSOR_DATA_TIMEOUT_MS`. It means either the producer
 never completed, or a consumer never
 released its fanout reference. Check for a hung producer first (that is S1 above),
 then verify the consumer really declares the dependency and exits. If the kernel is
 merely slow, raising the timeout will prove it.
 
+Code 8 only reaches you if this wait expires before the other watchdogs reap the
+op. Its 15 s default sits below the 20 s scheduler no-progress budget but above
+the op-execute timeout CI tightens to 3 s, so under CI's values a stall reports
+code 100 or a bare 507018 instead. Lower `SIMPLER_TENSOR_DATA_TIMEOUT_MS` below
+both when you want the producer locator that code 8 carries.
+
 `host_build_graph` has no such wait: its orchestration finishes before the device
 starts, so `get_tensor_data` / `set_tensor_data` reject a tensor with a producer
 with `INVALID_ARGS` (code 5) instead of waiting.

docs/troubleshooting/local-timeout-defaults.md

@@ -37,3 +37,20 @@ stream-sync timeout > scheduler timeout + 1.5 s
 Invalid values or invalid onboard ordering are ignored with a warning and the
 compiled defaults are used instead. See [args-dump](../dfx/args-dump.md#8-faq-and-debug-guide)
 for the timeout chain and dump-recovery details.
+
+## The tensor-data wait
+
+A fourth budget bounds the `tensormap_and_ringbuffer` orchestration
+`get_tensor_data` / `set_tensor_data` spin, in which orchestration waits for a
+producer task (and, for writes, its consumers). It defaults to 15 s and is
+overridden the same way:
+
+```bash
+export SIMPLER_TENSOR_DATA_TIMEOUT_MS=1000
+```
+
+It takes no part in the ordering rules above, so setting it can never invalidate
+the other three. What the ordering costs it instead is reachability: expiring
+after the scheduler or op-execute timeout means the run is already being reaped
+and `TENSOR_WAIT_TIMEOUT` (code 8) never latches. Lower it below both when you
+want the producer locator that code 8 carries; leave it alone otherwise.

docs/user/reference/cli.md

@@ -79,10 +79,11 @@ python -m simpler_setup.tools.swimlane_converter <chip_swimlane_records_*.json>
 | `SIMPLER_OP_EXECUTE_TIMEOUT_US` | Overrides the op-execute timeout (default 45 s) |
 | `SIMPLER_STREAM_SYNC_TIMEOUT_MS` | Overrides the stream-sync timeout (default 50 s) |
 | `SIMPLER_SCHEDULER_TIMEOUT_MS` | Overrides the scheduler timeout (default 20 s) |
+| `SIMPLER_TENSOR_DATA_TIMEOUT_MS` | Overrides the orchestration tensor-data wait (default 15 s, `tensormap_and_ringbuffer` only) |
 | `ASCEND_PROCESS_LOG_PATH` | Redirects the device log into a directory you own; the directory must already exist |
 | `ASCEND_HOME_PATH` | CANN toolkit location; required for hardware platforms |
 
-The three timeouts are validated against each other at startup, and CI runs
+The first three timeouts are validated against each other at startup, and CI runs
 deliberately shorter values than the defaults — read a CI timeout against the CI
 values. Details in
 [local-timeout-defaults](../../troubleshooting/local-timeout-defaults.md).

src/a2a3/platform/include/common/kernel_args.h

@@ -143,9 +143,10 @@ static_assert(offsetof(KernelArgs, regs) == 8, "KernelArgs::regs offset drift");
  * register tables consumed on the per-run AICore path and stay in KernelArgs.
  */
 struct InitArgs {
-    uint32_t device_id{0};            // ACL device ordinal -> set_orch_device_id
-    uint32_t log_level{25};           // Threshold: DEBUG=10, INFO=20, TIMING=25, WARN=30, ERROR=40, NUL=60
-    int32_t scheduler_timeout_ms{0};  // AICPU no-progress watchdog (ms); 0 -> compile default
+    uint32_t device_id{0};              // ACL device ordinal -> set_orch_device_id
+    uint32_t log_level{25};             // Threshold: DEBUG=10, INFO=20, TIMING=25, WARN=30, ERROR=40, NUL=60
+    int32_t scheduler_timeout_ms{0};    // AICPU no-progress watchdog (ms); 0 -> compile default
+    int32_t tensor_data_timeout_ms{0};  // Orchestration tensor-data wait (ms); 0 -> compile default
     // Per-engine async-DMA workspace dev addrs -> set_dma_workspace_addr(kind, .);
     // indexed by DmaWorkspaceKind; 0 = that engine unavailable.
     uint64_t dma_workspace_addr[DMA_WORKSPACE_KIND_COUNT]{};

src/a2a3/platform/onboard/aicpu/kernel.cpp

@@ -142,6 +142,7 @@ extern "C" __attribute__((visibility("default"))) int simpler_aicpu_init(void *a
     set_log_level(static_cast<int>(init_args->log_level));
     set_orch_device_id(static_cast<int>(init_args->device_id));
     set_scheduler_timeout_ms(static_cast<int>(init_args->scheduler_timeout_ms));
+    set_tensor_data_timeout_ms(static_cast<int>(init_args->tensor_data_timeout_ms));
     for (int k = 0; k < DMA_WORKSPACE_KIND_COUNT; ++k) {
         set_dma_workspace_addr(k, init_args->dma_workspace_addr[k]);
     }

src/a2a3/platform/sim/host/device_runner.cpp

@@ -160,17 +160,25 @@ int DeviceRunner::ensure_binaries_loaded() {
             return PTO_RUNTIME_ERR_INTERNAL;
         load_optional_sym("set_orch_device_id", reinterpret_cast<void **>(&set_orch_device_id_func_));
         load_optional_sym("set_scheduler_timeout_ms", reinterpret_cast<void **>(&set_scheduler_timeout_ms_func_));
-        if (set_scheduler_timeout_ms_func_ != nullptr) {
+        load_optional_sym("set_tensor_data_timeout_ms", reinterpret_cast<void **>(&set_tensor_data_timeout_ms_func_));
+        if (set_scheduler_timeout_ms_func_ != nullptr || set_tensor_data_timeout_ms_func_ != nullptr) {
             // Per-device one-shot latch (mirrors the onboard InitArgs path):
-            // honor SIMPLER_SCHEDULER_TIMEOUT_MS once at SO load, not per run. 0 ->
-            // the scheduler keeps its compile-time default. Sim skips the
-            // op/stream ordering check (validate_runtime_timeout_order is onboard).
-            RuntimeTimeoutParseStatus sched_status;
-            RuntimeTimeoutConfig sched_cfg =
-                resolve_runtime_timeout_config(RuntimeTimeoutConfig{1, 1, 0}, &sched_status);
-            set_scheduler_timeout_ms_func_(
-                (sched_status.scheduler_env_set && sched_status.scheduler_valid) ? sched_cfg.scheduler_timeout_ms : 0
-            );
+            // honor SIMPLER_SCHEDULER_TIMEOUT_MS / SIMPLER_TENSOR_DATA_TIMEOUT_MS
+            // once at SO load, not per run. 0 -> the device keeps its
+            // compile-time default. Sim skips the op/stream ordering check
+            // (validate_runtime_timeout_order is onboard).
+            RuntimeTimeoutParseStatus status;
+            RuntimeTimeoutConfig cfg = resolve_runtime_timeout_config(RuntimeTimeoutConfig{1, 1, 0, 0}, &status);
+            const int32_t scheduler_override =
+                (status.scheduler_env_set && status.scheduler_valid) ? cfg.scheduler_timeout_ms : 0;
+            const int32_t tensor_data_override =
+                (status.tensor_data_env_set && status.tensor_data_valid) ? cfg.tensor_data_timeout_ms : 0;
+            if (set_scheduler_timeout_ms_func_ != nullptr) {
+                set_scheduler_timeout_ms_func_(scheduler_override);
+            }
+            if (set_tensor_data_timeout_ms_func_ != nullptr) {
+                set_tensor_data_timeout_ms_func_(tensor_data_override);
+            }
         }
         if (!load_sym("set_platform_dump_base", reinterpret_cast<void **>(&set_platform_dump_base_func_)))
             return PTO_RUNTIME_ERR_INTERNAL;

src/a2a3/platform/sim/host/device_runner.h

@@ -76,6 +76,7 @@ class DeviceRunner : public SimDeviceRunnerBase {
     void (*set_platform_regs_func_)(uint64_t){nullptr};
     void (*set_orch_device_id_func_)(int){nullptr};
     void (*set_scheduler_timeout_ms_func_)(int){nullptr};
+    void (*set_tensor_data_timeout_ms_func_)(int){nullptr};
     void (*set_platform_dump_base_func_)(uint64_t){nullptr};
     void (*set_platform_phase_base_func_)(uint64_t){nullptr};
     void (*set_dump_args_enabled_func_)(bool){nullptr};

src/a2a3/runtime/tensormap_and_ringbuffer/docs/SCALAR_DATA_ACCESS.md

@@ -47,7 +47,8 @@ One extra step versus get_tensor_data: wait for all consumers to finish (`fanout
 ### 3.3 Timeout
 
 - Uses cycle counter (`get_sys_cnt_aicpu()`), checked every 1024 spins
-- Threshold: `TENSOR_DATA_TIMEOUT_CYCLES` (~10 s at 1.5 GHz)
+- Threshold: `TENSOR_DATA_TIMEOUT_MS` (15 s), scaled to counter cycles at the use
+  site and overridable per run with `SIMPLER_TENSOR_DATA_TIMEOUT_MS`
 - On timeout: sets `orch.fatal = true`, preventing further task submission
 
 ## 4. Seeding a Runtime-Created Output

src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime_core.cpp

@@ -26,6 +26,7 @@
 
 #include <algorithm>
 
+#include "aicpu/aicpu_device_config.h"  // get_tensor_data_timeout_ms (per-device override)
 #include "aicpu/device_time.h"
 #include "common/platform_config.h"  // PLATFORM_PROF_SYS_CNT_FREQ (data-wait deadline)
 #include "common/unified_log.h"
@@ -43,7 +44,13 @@ __attribute__((weak, visibility("hidden"))) uint64_t get_sys_cnt_aicpu() { retur
 // that define PLATFORM_PROF_SYS_CNT_FREQ locally, so pulling the platform header into
 // it caused a redefinition conflict (#1189). Scaling MS by the counter frequency (like
 // SCHEDULER_TIMEOUT_CYCLES) keeps the data-wait wall-clock identical across arches.
-static constexpr uint64_t TENSOR_DATA_TIMEOUT_CYCLES = (TENSOR_DATA_TIMEOUT_MS * PLATFORM_PROF_SYS_CNT_FREQ) / 1000;
+// A non-zero per-device override (SIMPLER_TENSOR_DATA_TIMEOUT_MS -> InitArgs ->
+// resident-SO global, latched at worker init) replaces TENSOR_DATA_TIMEOUT_MS.
+static uint64_t tensor_data_timeout_cycles() {
+    const int32_t override_ms = get_tensor_data_timeout_ms();
+    const uint64_t timeout_ms = override_ms > 0 ? static_cast<uint64_t>(override_ms) : TENSOR_DATA_TIMEOUT_MS;
+    return (timeout_ms * PLATFORM_PROF_SYS_CNT_FREQ) / 1000;
+}
 
 // =============================================================================
 // Orchestration Ops Table (function-pointer dispatch for orchestration .so)
@@ -112,6 +119,7 @@ static bool wait_for_tensor_ready(
     const ChipTaskSlotState *seg[kSegmentCap];
     int seg_count = 0;
     bool failed = false;
+    const uint64_t timeout_cycles = tensor_data_timeout_cycles();
 
     auto wait_one_producer = [&](const ChipTaskSlotState &slot) {
         uint8_t ring_id = slot.ring_id;
@@ -126,11 +134,11 @@ static bool wait_for_tensor_ready(
                     failed = true;
                     return;
                 }
-                if (get_sys_cnt_aicpu() - t0 > TENSOR_DATA_TIMEOUT_CYCLES) {
+                if (get_sys_cnt_aicpu() - t0 > timeout_cycles) {
                     orch.report_fatal(
                         SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT, caller,
                         "Timeout (%llu cycles): producer (ring=%d, local=%d) not completed",
-                        (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, ring_id, local_id
+                        (unsigned long long)timeout_cycles, ring_id, local_id
                     );
                     failed = true;
                     return;
@@ -153,11 +161,11 @@ static bool wait_for_tensor_ready(
                     failed = true;
                     return;
                 }
-                if (get_sys_cnt_aicpu() - t0 > TENSOR_DATA_TIMEOUT_CYCLES) {
+                if (get_sys_cnt_aicpu() - t0 > timeout_cycles) {
                     orch.report_fatal(
                         SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT, caller,
                         "Timeout (%llu cycles): consumers of producer (ring=%d, local=%d) not done",
-                        (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, ring_id, local_id
+                        (unsigned long long)timeout_cycles, ring_id, local_id
                     );
                     failed = true;
                     return;

src/a5/platform/include/common/kernel_args.h

@@ -129,9 +129,10 @@ static_assert(offsetof(KernelArgs, regs) == 8, "KernelArgs::regs offset drift");
  * stays in KernelArgs.
  */
 struct InitArgs {
-    uint32_t device_id{0};            // ACL device ordinal -> set_orch_device_id
-    uint32_t log_level{25};           // Threshold: DEBUG=10, INFO=20, TIMING=25, WARN=30, ERROR=40, NUL=60
-    int32_t scheduler_timeout_ms{0};  // AICPU no-progress watchdog (ms); 0 -> compile default
+    uint32_t device_id{0};              // ACL device ordinal -> set_orch_device_id
+    uint32_t log_level{25};             // Threshold: DEBUG=10, INFO=20, TIMING=25, WARN=30, ERROR=40, NUL=60
+    int32_t scheduler_timeout_ms{0};    // AICPU no-progress watchdog (ms); 0 -> compile default
+    int32_t tensor_data_timeout_ms{0};  // Orchestration tensor-data wait (ms); 0 -> compile default
     // Per-engine async-DMA workspace dev addrs -> set_dma_workspace_addr(kind, .);
     // indexed by DmaWorkspaceKind; 0 = that engine unavailable.
     uint64_t dma_workspace_addr[DMA_WORKSPACE_KIND_COUNT]{};

src/a5/platform/onboard/aicpu/kernel.cpp

@@ -172,6 +172,7 @@ extern "C" __attribute__((visibility("default"))) int simpler_aicpu_init(void *a
     set_log_level(static_cast<int>(init_args->log_level));
     set_orch_device_id(static_cast<int>(init_args->device_id));
     set_scheduler_timeout_ms(static_cast<int>(init_args->scheduler_timeout_ms));
+    set_tensor_data_timeout_ms(static_cast<int>(init_args->tensor_data_timeout_ms));
     for (int k = 0; k < DMA_WORKSPACE_KIND_COUNT; ++k) {
         set_dma_workspace_addr(k, init_args->dma_workspace_addr[k]);
     }

src/a5/platform/sim/host/device_runner.cpp

@@ -162,17 +162,25 @@ int DeviceRunner::ensure_binaries_loaded() {
             return PTO_RUNTIME_ERR_INTERNAL;
         load_optional_sym("set_orch_device_id", reinterpret_cast<void **>(&set_orch_device_id_func_));
         load_optional_sym("set_scheduler_timeout_ms", reinterpret_cast<void **>(&set_scheduler_timeout_ms_func_));
-        if (set_scheduler_timeout_ms_func_ != nullptr) {
+        load_optional_sym("set_tensor_data_timeout_ms", reinterpret_cast<void **>(&set_tensor_data_timeout_ms_func_));
+        if (set_scheduler_timeout_ms_func_ != nullptr || set_tensor_data_timeout_ms_func_ != nullptr) {
             // Per-device one-shot latch (mirrors the onboard InitArgs path):
-            // honor SIMPLER_SCHEDULER_TIMEOUT_MS once at SO load, not per run. 0 ->
-            // the scheduler keeps its compile-time default. Sim skips the
-            // op/stream ordering check (validate_runtime_timeout_order is onboard).
-            RuntimeTimeoutParseStatus sched_status;
-            RuntimeTimeoutConfig sched_cfg =
-                resolve_runtime_timeout_config(RuntimeTimeoutConfig{1, 1, 0}, &sched_status);
-            set_scheduler_timeout_ms_func_(
-                (sched_status.scheduler_env_set && sched_status.scheduler_valid) ? sched_cfg.scheduler_timeout_ms : 0
-            );
+            // honor SIMPLER_SCHEDULER_TIMEOUT_MS / SIMPLER_TENSOR_DATA_TIMEOUT_MS
+            // once at SO load, not per run. 0 -> the device keeps its
+            // compile-time default. Sim skips the op/stream ordering check
+            // (validate_runtime_timeout_order is onboard).
+            RuntimeTimeoutParseStatus status;
+            RuntimeTimeoutConfig cfg = resolve_runtime_timeout_config(RuntimeTimeoutConfig{1, 1, 0, 0}, &status);
+            const int32_t scheduler_override =
+                (status.scheduler_env_set && status.scheduler_valid) ? cfg.scheduler_timeout_ms : 0;
+            const int32_t tensor_data_override =
+                (status.tensor_data_env_set && status.tensor_data_valid) ? cfg.tensor_data_timeout_ms : 0;
+            if (set_scheduler_timeout_ms_func_ != nullptr) {
+                set_scheduler_timeout_ms_func_(scheduler_override);
+            }
+            if (set_tensor_data_timeout_ms_func_ != nullptr) {
+                set_tensor_data_timeout_ms_func_(tensor_data_override);
+            }
         }
         if (!load_sym("set_platform_dump_base", reinterpret_cast<void **>(&set_platform_dump_base_func_)))
             return PTO_RUNTIME_ERR_INTERNAL;

src/a5/platform/sim/host/device_runner.h

@@ -77,6 +77,7 @@ class DeviceRunner : public SimDeviceRunnerBase {
     void (*set_platform_regs_func_)(uint64_t){nullptr};
     void (*set_orch_device_id_func_)(int){nullptr};
     void (*set_scheduler_timeout_ms_func_)(int){nullptr};
+    void (*set_tensor_data_timeout_ms_func_)(int){nullptr};
     void (*set_platform_dump_base_func_)(uint64_t){nullptr};
     void (*set_platform_phase_base_func_)(uint64_t){nullptr};
     void (*set_platform_pmu_base_func_)(uint64_t){nullptr};

src/a5/runtime/tensormap_and_ringbuffer/docs/SCALAR_DATA_ACCESS.md

@@ -47,7 +47,8 @@ One extra step versus get_tensor_data: wait for all consumers to finish (`fanout
 ### 3.3 Timeout
 
 - Uses cycle counter (`get_sys_cnt_aicpu()`), checked every 1024 spins
-- Threshold: `TENSOR_DATA_TIMEOUT_CYCLES` (~10 s at 1.5 GHz)
+- Threshold: `TENSOR_DATA_TIMEOUT_MS` (15 s), scaled to counter cycles at the use
+  site and overridable per run with `SIMPLER_TENSOR_DATA_TIMEOUT_MS`
 - On timeout: sets `orch.fatal = true`, preventing further task submission
 
 ## 4. Seeding a Runtime-Created Output