diff --git a/docs/dfx/args-dump.md b/docs/dfx/args-dump.md index 40e5493e5e..f192a9a43c 100644 --- a/docs/dfx/args-dump.md +++ b/docs/dfx/args-dump.md @@ -936,6 +936,15 @@ the no-progress budget without onboard-only ordering limits. CI restores the old fast-fail values through these env vars: 2 s scheduler, 3 s op-execute, and 4 s stream-sync for onboard jobs; 5 s scheduler for sim jobs. +`SIMPLER_TENSOR_DATA_TIMEOUT_MS` overrides a fourth wait — the orchestration +`get_tensor_data`/`set_tensor_data` spin, 15 s by default in +`tensormap_and_ringbuffer` — and is deliberately outside that ordering group. It +is a diagnostic refinement: expiring first buys the precise `TENSOR_WAIT_TIMEOUT` +code with its producer locator, while the clean shutdown is the scheduler +watchdog's job for every stall shape. Folding it into the ordering check would +let a value the host cannot even see (the default is compiled into the runtime) +invalidate an otherwise valid override set. + - **Device-side graceful flush (primary).** At 20 s of no progress the AICPU declares the hang, runs the end-of-loop flush, *and* dumps the **partial output** of every task still RUNNING on a core diff --git a/docs/dfx/profiling-config-naming.md b/docs/dfx/profiling-config-naming.md index 1b205671b1..403905072e 100644 --- a/docs/dfx/profiling-config-naming.md +++ b/docs/dfx/profiling-config-naming.md @@ -127,6 +127,7 @@ simulator platforms. | `SIMPLER_OP_EXECUTE_TIMEOUT_US` | host platform | µs | op-execute timeout (overrides the `platform_config.h` compile default) | | `SIMPLER_STREAM_SYNC_TIMEOUT_MS` | host platform | ms | stream-sync timeout | | `SIMPLER_SCHEDULER_TIMEOUT_MS` | host platform | ms | scheduler no-progress timeout | +| `SIMPLER_TENSOR_DATA_TIMEOUT_MS` | host platform | ms | orchestration tensor-data wait timeout | ### Per-run DFX flag bitmask (`platform_config.h`) diff --git a/docs/troubleshooting/device-error-codes/stall.md b/docs/troubleshooting/device-error-codes/stall.md index b5acf1ecf4..1edcaf512f 100644 --- a/docs/troubleshooting/device-error-codes/stall.md +++ b/docs/troubleshooting/device-error-codes/stall.md @@ -17,12 +17,11 @@ the technique transfers: - To make the **scheduler** win (get a `sub_class`), squeeze it below the others: `SCHEDULER=2000ms < OP_EXECUTE=3s < STREAM_SYNC=4000ms` (`aicore_hang` case). -- To let a **slow-but-alive** path finish, push the others out of the way: - `SCHEDULER=30000ms`, `OP_EXECUTE=30s`, `STREAM_SYNC=40000ms` — this is how the - `tensor_wait_timeout` case lets the 15 s tensor-data wait land code 8 instead of - being reaped first. +- To make the **tensor-data wait** win (get code 8 and its producer locator), + squeeze `TENSOR_DATA=1000ms` below all three — it sits outside their ordering + group, so lowering it alone leaves them valid (`tensor_wait_timeout` case). -**These three are read once, at `Worker.init()`.** Changing them between `run()` +**These four are read once, at `Worker.init()`.** Changing them between `run()` calls on the same Worker does nothing — you must rebuild the Worker (or use a separate process) per value. Defaults and the rationale are in [`../local-timeout-defaults.md`](../local-timeout-defaults.md). @@ -63,12 +62,19 @@ exist) before reading. See the "Device logs" section of ## Code 8, specifically Only `tensormap_and_ringbuffer` raises this code. Its tensor-data wait defaults to -15 s (`TENSOR_DATA_TIMEOUT_MS`, frequency-scaled). It means either the producer +15 s (`TENSOR_DATA_TIMEOUT_MS`, frequency-scaled) and is overridden per run by +`SIMPLER_TENSOR_DATA_TIMEOUT_MS`. It means either the producer never completed, or a consumer never released its fanout reference. Check for a hung producer first (that is S1 above), then verify the consumer really declares the dependency and exits. If the kernel is merely slow, raising the timeout will prove it. +Code 8 only reaches you if this wait expires before the other watchdogs reap the +op. Its 15 s default sits below the 20 s scheduler no-progress budget but above +the op-execute timeout CI tightens to 3 s, so under CI's values a stall reports +code 100 or a bare 507018 instead. Lower `SIMPLER_TENSOR_DATA_TIMEOUT_MS` below +both when you want the producer locator that code 8 carries. + `host_build_graph` has no such wait: its orchestration finishes before the device starts, so `get_tensor_data` / `set_tensor_data` reject a tensor with a producer with `INVALID_ARGS` (code 5) instead of waiting. diff --git a/docs/troubleshooting/local-timeout-defaults.md b/docs/troubleshooting/local-timeout-defaults.md index 2e17c6c40e..a644f1aabd 100644 --- a/docs/troubleshooting/local-timeout-defaults.md +++ b/docs/troubleshooting/local-timeout-defaults.md @@ -37,3 +37,20 @@ stream-sync timeout > scheduler timeout + 1.5 s Invalid values or invalid onboard ordering are ignored with a warning and the compiled defaults are used instead. See [args-dump](../dfx/args-dump.md#8-faq-and-debug-guide) for the timeout chain and dump-recovery details. + +## The tensor-data wait + +A fourth budget bounds the `tensormap_and_ringbuffer` orchestration +`get_tensor_data` / `set_tensor_data` spin, in which orchestration waits for a +producer task (and, for writes, its consumers). It defaults to 15 s and is +overridden the same way: + +```bash +export SIMPLER_TENSOR_DATA_TIMEOUT_MS=1000 +``` + +It takes no part in the ordering rules above, so setting it can never invalidate +the other three. What the ordering costs it instead is reachability: expiring +after the scheduler or op-execute timeout means the run is already being reaped +and `TENSOR_WAIT_TIMEOUT` (code 8) never latches. Lower it below both when you +want the producer locator that code 8 carries; leave it alone otherwise. diff --git a/docs/user/reference/cli.md b/docs/user/reference/cli.md index a74cdb9da6..1b9f53aabe 100644 --- a/docs/user/reference/cli.md +++ b/docs/user/reference/cli.md @@ -79,10 +79,11 @@ python -m simpler_setup.tools.swimlane_converter | `SIMPLER_OP_EXECUTE_TIMEOUT_US` | Overrides the op-execute timeout (default 45 s) | | `SIMPLER_STREAM_SYNC_TIMEOUT_MS` | Overrides the stream-sync timeout (default 50 s) | | `SIMPLER_SCHEDULER_TIMEOUT_MS` | Overrides the scheduler timeout (default 20 s) | +| `SIMPLER_TENSOR_DATA_TIMEOUT_MS` | Overrides the orchestration tensor-data wait (default 15 s, `tensormap_and_ringbuffer` only) | | `ASCEND_PROCESS_LOG_PATH` | Redirects the device log into a directory you own; the directory must already exist | | `ASCEND_HOME_PATH` | CANN toolkit location; required for hardware platforms | -The three timeouts are validated against each other at startup, and CI runs +The first three timeouts are validated against each other at startup, and CI runs deliberately shorter values than the defaults — read a CI timeout against the CI values. Details in [local-timeout-defaults](../../troubleshooting/local-timeout-defaults.md). diff --git a/src/a2a3/platform/include/common/kernel_args.h b/src/a2a3/platform/include/common/kernel_args.h index d7a07caff0..8a84a475ca 100644 --- a/src/a2a3/platform/include/common/kernel_args.h +++ b/src/a2a3/platform/include/common/kernel_args.h @@ -143,9 +143,10 @@ static_assert(offsetof(KernelArgs, regs) == 8, "KernelArgs::regs offset drift"); * register tables consumed on the per-run AICore path and stay in KernelArgs. */ struct InitArgs { - uint32_t device_id{0}; // ACL device ordinal -> set_orch_device_id - uint32_t log_level{25}; // Threshold: DEBUG=10, INFO=20, TIMING=25, WARN=30, ERROR=40, NUL=60 - int32_t scheduler_timeout_ms{0}; // AICPU no-progress watchdog (ms); 0 -> compile default + uint32_t device_id{0}; // ACL device ordinal -> set_orch_device_id + uint32_t log_level{25}; // Threshold: DEBUG=10, INFO=20, TIMING=25, WARN=30, ERROR=40, NUL=60 + int32_t scheduler_timeout_ms{0}; // AICPU no-progress watchdog (ms); 0 -> compile default + int32_t tensor_data_timeout_ms{0}; // Orchestration tensor-data wait (ms); 0 -> compile default // Per-engine async-DMA workspace dev addrs -> set_dma_workspace_addr(kind, .); // indexed by DmaWorkspaceKind; 0 = that engine unavailable. uint64_t dma_workspace_addr[DMA_WORKSPACE_KIND_COUNT]{}; diff --git a/src/a2a3/platform/onboard/aicpu/kernel.cpp b/src/a2a3/platform/onboard/aicpu/kernel.cpp index 19062868ff..3d8020b95d 100644 --- a/src/a2a3/platform/onboard/aicpu/kernel.cpp +++ b/src/a2a3/platform/onboard/aicpu/kernel.cpp @@ -142,6 +142,7 @@ extern "C" __attribute__((visibility("default"))) int simpler_aicpu_init(void *a set_log_level(static_cast(init_args->log_level)); set_orch_device_id(static_cast(init_args->device_id)); set_scheduler_timeout_ms(static_cast(init_args->scheduler_timeout_ms)); + set_tensor_data_timeout_ms(static_cast(init_args->tensor_data_timeout_ms)); for (int k = 0; k < DMA_WORKSPACE_KIND_COUNT; ++k) { set_dma_workspace_addr(k, init_args->dma_workspace_addr[k]); } diff --git a/src/a2a3/platform/sim/host/device_runner.cpp b/src/a2a3/platform/sim/host/device_runner.cpp index efb6d4c345..0c70074d61 100644 --- a/src/a2a3/platform/sim/host/device_runner.cpp +++ b/src/a2a3/platform/sim/host/device_runner.cpp @@ -160,17 +160,25 @@ int DeviceRunner::ensure_binaries_loaded() { return PTO_RUNTIME_ERR_INTERNAL; load_optional_sym("set_orch_device_id", reinterpret_cast(&set_orch_device_id_func_)); load_optional_sym("set_scheduler_timeout_ms", reinterpret_cast(&set_scheduler_timeout_ms_func_)); - if (set_scheduler_timeout_ms_func_ != nullptr) { + load_optional_sym("set_tensor_data_timeout_ms", reinterpret_cast(&set_tensor_data_timeout_ms_func_)); + if (set_scheduler_timeout_ms_func_ != nullptr || set_tensor_data_timeout_ms_func_ != nullptr) { // Per-device one-shot latch (mirrors the onboard InitArgs path): - // honor SIMPLER_SCHEDULER_TIMEOUT_MS once at SO load, not per run. 0 -> - // the scheduler keeps its compile-time default. Sim skips the - // op/stream ordering check (validate_runtime_timeout_order is onboard). - RuntimeTimeoutParseStatus sched_status; - RuntimeTimeoutConfig sched_cfg = - resolve_runtime_timeout_config(RuntimeTimeoutConfig{1, 1, 0}, &sched_status); - set_scheduler_timeout_ms_func_( - (sched_status.scheduler_env_set && sched_status.scheduler_valid) ? sched_cfg.scheduler_timeout_ms : 0 - ); + // honor SIMPLER_SCHEDULER_TIMEOUT_MS / SIMPLER_TENSOR_DATA_TIMEOUT_MS + // once at SO load, not per run. 0 -> the device keeps its + // compile-time default. Sim skips the op/stream ordering check + // (validate_runtime_timeout_order is onboard). + RuntimeTimeoutParseStatus status; + RuntimeTimeoutConfig cfg = resolve_runtime_timeout_config(RuntimeTimeoutConfig{1, 1, 0, 0}, &status); + const int32_t scheduler_override = + (status.scheduler_env_set && status.scheduler_valid) ? cfg.scheduler_timeout_ms : 0; + const int32_t tensor_data_override = + (status.tensor_data_env_set && status.tensor_data_valid) ? cfg.tensor_data_timeout_ms : 0; + if (set_scheduler_timeout_ms_func_ != nullptr) { + set_scheduler_timeout_ms_func_(scheduler_override); + } + if (set_tensor_data_timeout_ms_func_ != nullptr) { + set_tensor_data_timeout_ms_func_(tensor_data_override); + } } if (!load_sym("set_platform_dump_base", reinterpret_cast(&set_platform_dump_base_func_))) return PTO_RUNTIME_ERR_INTERNAL; diff --git a/src/a2a3/platform/sim/host/device_runner.h b/src/a2a3/platform/sim/host/device_runner.h index 2256f61593..bc0d48f70a 100644 --- a/src/a2a3/platform/sim/host/device_runner.h +++ b/src/a2a3/platform/sim/host/device_runner.h @@ -76,6 +76,7 @@ class DeviceRunner : public SimDeviceRunnerBase { void (*set_platform_regs_func_)(uint64_t){nullptr}; void (*set_orch_device_id_func_)(int){nullptr}; void (*set_scheduler_timeout_ms_func_)(int){nullptr}; + void (*set_tensor_data_timeout_ms_func_)(int){nullptr}; void (*set_platform_dump_base_func_)(uint64_t){nullptr}; void (*set_platform_phase_base_func_)(uint64_t){nullptr}; void (*set_dump_args_enabled_func_)(bool){nullptr}; diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/docs/SCALAR_DATA_ACCESS.md b/src/a2a3/runtime/tensormap_and_ringbuffer/docs/SCALAR_DATA_ACCESS.md index 08ec728d45..8bcd69a161 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/docs/SCALAR_DATA_ACCESS.md +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/docs/SCALAR_DATA_ACCESS.md @@ -47,7 +47,8 @@ One extra step versus get_tensor_data: wait for all consumers to finish (`fanout ### 3.3 Timeout - Uses cycle counter (`get_sys_cnt_aicpu()`), checked every 1024 spins -- Threshold: `TENSOR_DATA_TIMEOUT_CYCLES` (~10 s at 1.5 GHz) +- Threshold: `TENSOR_DATA_TIMEOUT_MS` (15 s), scaled to counter cycles at the use + site and overridable per run with `SIMPLER_TENSOR_DATA_TIMEOUT_MS` - On timeout: sets `orch.fatal = true`, preventing further task submission ## 4. Seeding a Runtime-Created Output diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime_core.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime_core.cpp index 4b6a349e95..7e13423fd6 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime_core.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime_core.cpp @@ -26,6 +26,7 @@ #include +#include "aicpu/aicpu_device_config.h" // get_tensor_data_timeout_ms (per-device override) #include "aicpu/device_time.h" #include "common/platform_config.h" // PLATFORM_PROF_SYS_CNT_FREQ (data-wait deadline) #include "common/unified_log.h" @@ -43,7 +44,13 @@ __attribute__((weak, visibility("hidden"))) uint64_t get_sys_cnt_aicpu() { retur // that define PLATFORM_PROF_SYS_CNT_FREQ locally, so pulling the platform header into // it caused a redefinition conflict (#1189). Scaling MS by the counter frequency (like // SCHEDULER_TIMEOUT_CYCLES) keeps the data-wait wall-clock identical across arches. -static constexpr uint64_t TENSOR_DATA_TIMEOUT_CYCLES = (TENSOR_DATA_TIMEOUT_MS * PLATFORM_PROF_SYS_CNT_FREQ) / 1000; +// A non-zero per-device override (SIMPLER_TENSOR_DATA_TIMEOUT_MS -> InitArgs -> +// resident-SO global, latched at worker init) replaces TENSOR_DATA_TIMEOUT_MS. +static uint64_t tensor_data_timeout_cycles() { + const int32_t override_ms = get_tensor_data_timeout_ms(); + const uint64_t timeout_ms = override_ms > 0 ? static_cast(override_ms) : TENSOR_DATA_TIMEOUT_MS; + return (timeout_ms * PLATFORM_PROF_SYS_CNT_FREQ) / 1000; +} // ============================================================================= // Orchestration Ops Table (function-pointer dispatch for orchestration .so) @@ -112,6 +119,7 @@ static bool wait_for_tensor_ready( const ChipTaskSlotState *seg[kSegmentCap]; int seg_count = 0; bool failed = false; + const uint64_t timeout_cycles = tensor_data_timeout_cycles(); auto wait_one_producer = [&](const ChipTaskSlotState &slot) { uint8_t ring_id = slot.ring_id; @@ -126,11 +134,11 @@ static bool wait_for_tensor_ready( failed = true; return; } - if (get_sys_cnt_aicpu() - t0 > TENSOR_DATA_TIMEOUT_CYCLES) { + if (get_sys_cnt_aicpu() - t0 > timeout_cycles) { orch.report_fatal( SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT, caller, "Timeout (%llu cycles): producer (ring=%d, local=%d) not completed", - (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, ring_id, local_id + (unsigned long long)timeout_cycles, ring_id, local_id ); failed = true; return; @@ -153,11 +161,11 @@ static bool wait_for_tensor_ready( failed = true; return; } - if (get_sys_cnt_aicpu() - t0 > TENSOR_DATA_TIMEOUT_CYCLES) { + if (get_sys_cnt_aicpu() - t0 > timeout_cycles) { orch.report_fatal( SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT, caller, "Timeout (%llu cycles): consumers of producer (ring=%d, local=%d) not done", - (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, ring_id, local_id + (unsigned long long)timeout_cycles, ring_id, local_id ); failed = true; return; diff --git a/src/a5/platform/include/common/kernel_args.h b/src/a5/platform/include/common/kernel_args.h index 5dfaa76ca3..4a5306329b 100644 --- a/src/a5/platform/include/common/kernel_args.h +++ b/src/a5/platform/include/common/kernel_args.h @@ -129,9 +129,10 @@ static_assert(offsetof(KernelArgs, regs) == 8, "KernelArgs::regs offset drift"); * stays in KernelArgs. */ struct InitArgs { - uint32_t device_id{0}; // ACL device ordinal -> set_orch_device_id - uint32_t log_level{25}; // Threshold: DEBUG=10, INFO=20, TIMING=25, WARN=30, ERROR=40, NUL=60 - int32_t scheduler_timeout_ms{0}; // AICPU no-progress watchdog (ms); 0 -> compile default + uint32_t device_id{0}; // ACL device ordinal -> set_orch_device_id + uint32_t log_level{25}; // Threshold: DEBUG=10, INFO=20, TIMING=25, WARN=30, ERROR=40, NUL=60 + int32_t scheduler_timeout_ms{0}; // AICPU no-progress watchdog (ms); 0 -> compile default + int32_t tensor_data_timeout_ms{0}; // Orchestration tensor-data wait (ms); 0 -> compile default // Per-engine async-DMA workspace dev addrs -> set_dma_workspace_addr(kind, .); // indexed by DmaWorkspaceKind; 0 = that engine unavailable. uint64_t dma_workspace_addr[DMA_WORKSPACE_KIND_COUNT]{}; diff --git a/src/a5/platform/onboard/aicpu/kernel.cpp b/src/a5/platform/onboard/aicpu/kernel.cpp index 21891ac45e..4eec202d0b 100644 --- a/src/a5/platform/onboard/aicpu/kernel.cpp +++ b/src/a5/platform/onboard/aicpu/kernel.cpp @@ -172,6 +172,7 @@ extern "C" __attribute__((visibility("default"))) int simpler_aicpu_init(void *a set_log_level(static_cast(init_args->log_level)); set_orch_device_id(static_cast(init_args->device_id)); set_scheduler_timeout_ms(static_cast(init_args->scheduler_timeout_ms)); + set_tensor_data_timeout_ms(static_cast(init_args->tensor_data_timeout_ms)); for (int k = 0; k < DMA_WORKSPACE_KIND_COUNT; ++k) { set_dma_workspace_addr(k, init_args->dma_workspace_addr[k]); } diff --git a/src/a5/platform/sim/host/device_runner.cpp b/src/a5/platform/sim/host/device_runner.cpp index 9c6d9484dd..ba5212d868 100644 --- a/src/a5/platform/sim/host/device_runner.cpp +++ b/src/a5/platform/sim/host/device_runner.cpp @@ -162,17 +162,25 @@ int DeviceRunner::ensure_binaries_loaded() { return PTO_RUNTIME_ERR_INTERNAL; load_optional_sym("set_orch_device_id", reinterpret_cast(&set_orch_device_id_func_)); load_optional_sym("set_scheduler_timeout_ms", reinterpret_cast(&set_scheduler_timeout_ms_func_)); - if (set_scheduler_timeout_ms_func_ != nullptr) { + load_optional_sym("set_tensor_data_timeout_ms", reinterpret_cast(&set_tensor_data_timeout_ms_func_)); + if (set_scheduler_timeout_ms_func_ != nullptr || set_tensor_data_timeout_ms_func_ != nullptr) { // Per-device one-shot latch (mirrors the onboard InitArgs path): - // honor SIMPLER_SCHEDULER_TIMEOUT_MS once at SO load, not per run. 0 -> - // the scheduler keeps its compile-time default. Sim skips the - // op/stream ordering check (validate_runtime_timeout_order is onboard). - RuntimeTimeoutParseStatus sched_status; - RuntimeTimeoutConfig sched_cfg = - resolve_runtime_timeout_config(RuntimeTimeoutConfig{1, 1, 0}, &sched_status); - set_scheduler_timeout_ms_func_( - (sched_status.scheduler_env_set && sched_status.scheduler_valid) ? sched_cfg.scheduler_timeout_ms : 0 - ); + // honor SIMPLER_SCHEDULER_TIMEOUT_MS / SIMPLER_TENSOR_DATA_TIMEOUT_MS + // once at SO load, not per run. 0 -> the device keeps its + // compile-time default. Sim skips the op/stream ordering check + // (validate_runtime_timeout_order is onboard). + RuntimeTimeoutParseStatus status; + RuntimeTimeoutConfig cfg = resolve_runtime_timeout_config(RuntimeTimeoutConfig{1, 1, 0, 0}, &status); + const int32_t scheduler_override = + (status.scheduler_env_set && status.scheduler_valid) ? cfg.scheduler_timeout_ms : 0; + const int32_t tensor_data_override = + (status.tensor_data_env_set && status.tensor_data_valid) ? cfg.tensor_data_timeout_ms : 0; + if (set_scheduler_timeout_ms_func_ != nullptr) { + set_scheduler_timeout_ms_func_(scheduler_override); + } + if (set_tensor_data_timeout_ms_func_ != nullptr) { + set_tensor_data_timeout_ms_func_(tensor_data_override); + } } if (!load_sym("set_platform_dump_base", reinterpret_cast(&set_platform_dump_base_func_))) return PTO_RUNTIME_ERR_INTERNAL; diff --git a/src/a5/platform/sim/host/device_runner.h b/src/a5/platform/sim/host/device_runner.h index 2f5f48d3a6..0ca6a24922 100644 --- a/src/a5/platform/sim/host/device_runner.h +++ b/src/a5/platform/sim/host/device_runner.h @@ -77,6 +77,7 @@ class DeviceRunner : public SimDeviceRunnerBase { void (*set_platform_regs_func_)(uint64_t){nullptr}; void (*set_orch_device_id_func_)(int){nullptr}; void (*set_scheduler_timeout_ms_func_)(int){nullptr}; + void (*set_tensor_data_timeout_ms_func_)(int){nullptr}; void (*set_platform_dump_base_func_)(uint64_t){nullptr}; void (*set_platform_phase_base_func_)(uint64_t){nullptr}; void (*set_platform_pmu_base_func_)(uint64_t){nullptr}; diff --git a/src/a5/runtime/tensormap_and_ringbuffer/docs/SCALAR_DATA_ACCESS.md b/src/a5/runtime/tensormap_and_ringbuffer/docs/SCALAR_DATA_ACCESS.md index 08ec728d45..8bcd69a161 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/docs/SCALAR_DATA_ACCESS.md +++ b/src/a5/runtime/tensormap_and_ringbuffer/docs/SCALAR_DATA_ACCESS.md @@ -47,7 +47,8 @@ One extra step versus get_tensor_data: wait for all consumers to finish (`fanout ### 3.3 Timeout - Uses cycle counter (`get_sys_cnt_aicpu()`), checked every 1024 spins -- Threshold: `TENSOR_DATA_TIMEOUT_CYCLES` (~10 s at 1.5 GHz) +- Threshold: `TENSOR_DATA_TIMEOUT_MS` (15 s), scaled to counter cycles at the use + site and overridable per run with `SIMPLER_TENSOR_DATA_TIMEOUT_MS` - On timeout: sets `orch.fatal = true`, preventing further task submission ## 4. Seeding a Runtime-Created Output diff --git a/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime_core.cpp b/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime_core.cpp index 4b6a349e95..7e13423fd6 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime_core.cpp +++ b/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime_core.cpp @@ -26,6 +26,7 @@ #include +#include "aicpu/aicpu_device_config.h" // get_tensor_data_timeout_ms (per-device override) #include "aicpu/device_time.h" #include "common/platform_config.h" // PLATFORM_PROF_SYS_CNT_FREQ (data-wait deadline) #include "common/unified_log.h" @@ -43,7 +44,13 @@ __attribute__((weak, visibility("hidden"))) uint64_t get_sys_cnt_aicpu() { retur // that define PLATFORM_PROF_SYS_CNT_FREQ locally, so pulling the platform header into // it caused a redefinition conflict (#1189). Scaling MS by the counter frequency (like // SCHEDULER_TIMEOUT_CYCLES) keeps the data-wait wall-clock identical across arches. -static constexpr uint64_t TENSOR_DATA_TIMEOUT_CYCLES = (TENSOR_DATA_TIMEOUT_MS * PLATFORM_PROF_SYS_CNT_FREQ) / 1000; +// A non-zero per-device override (SIMPLER_TENSOR_DATA_TIMEOUT_MS -> InitArgs -> +// resident-SO global, latched at worker init) replaces TENSOR_DATA_TIMEOUT_MS. +static uint64_t tensor_data_timeout_cycles() { + const int32_t override_ms = get_tensor_data_timeout_ms(); + const uint64_t timeout_ms = override_ms > 0 ? static_cast(override_ms) : TENSOR_DATA_TIMEOUT_MS; + return (timeout_ms * PLATFORM_PROF_SYS_CNT_FREQ) / 1000; +} // ============================================================================= // Orchestration Ops Table (function-pointer dispatch for orchestration .so) @@ -112,6 +119,7 @@ static bool wait_for_tensor_ready( const ChipTaskSlotState *seg[kSegmentCap]; int seg_count = 0; bool failed = false; + const uint64_t timeout_cycles = tensor_data_timeout_cycles(); auto wait_one_producer = [&](const ChipTaskSlotState &slot) { uint8_t ring_id = slot.ring_id; @@ -126,11 +134,11 @@ static bool wait_for_tensor_ready( failed = true; return; } - if (get_sys_cnt_aicpu() - t0 > TENSOR_DATA_TIMEOUT_CYCLES) { + if (get_sys_cnt_aicpu() - t0 > timeout_cycles) { orch.report_fatal( SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT, caller, "Timeout (%llu cycles): producer (ring=%d, local=%d) not completed", - (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, ring_id, local_id + (unsigned long long)timeout_cycles, ring_id, local_id ); failed = true; return; @@ -153,11 +161,11 @@ static bool wait_for_tensor_ready( failed = true; return; } - if (get_sys_cnt_aicpu() - t0 > TENSOR_DATA_TIMEOUT_CYCLES) { + if (get_sys_cnt_aicpu() - t0 > timeout_cycles) { orch.report_fatal( SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT, caller, "Timeout (%llu cycles): consumers of producer (ring=%d, local=%d) not done", - (unsigned long long)TENSOR_DATA_TIMEOUT_CYCLES, ring_id, local_id + (unsigned long long)timeout_cycles, ring_id, local_id ); failed = true; return; diff --git a/src/common/platform/include/aicpu/aicpu_device_config.h b/src/common/platform/include/aicpu/aicpu_device_config.h index 4b9a65a658..b7b0f7a123 100644 --- a/src/common/platform/include/aicpu/aicpu_device_config.h +++ b/src/common/platform/include/aicpu/aicpu_device_config.h @@ -50,6 +50,19 @@ void set_scheduler_timeout_ms(int timeout_ms); /** Get the scheduler watchdog timeout override in ms (0 if unset). */ int get_scheduler_timeout_ms(); +/** + * Set the orchestration tensor-data wait timeout (ms). Latched once per device + * by simpler_aicpu_init (from InitArgs.tensor_data_timeout_ms); read by + * get_tensor_data / set_tensor_data when they block on a producer or its + * consumers. 0 means "no override" — the wait keeps its compile-time + * TENSOR_DATA_TIMEOUT_MS. Consumed by tensormap_and_ringbuffer only; + * host_build_graph resolves tensor data on the host and never waits. + */ +void set_tensor_data_timeout_ms(int timeout_ms); + +/** Get the tensor-data wait timeout override in ms (0 if unset). */ +int get_tensor_data_timeout_ms(); + /** * Set the device address of the per-device async-DMA workspace for one engine * kind (see DmaWorkspaceKind). Published by simpler_aicpu_init (from diff --git a/src/common/platform/include/host/runtime_timeout_config.h b/src/common/platform/include/host/runtime_timeout_config.h index 4d51f3116f..3504b63bf8 100644 --- a/src/common/platform/include/host/runtime_timeout_config.h +++ b/src/common/platform/include/host/runtime_timeout_config.h @@ -23,6 +23,7 @@ constexpr const char *SIMPLER_OP_EXECUTE_TIMEOUT_US_ENV = "SIMPLER_OP_EXECUTE_TIMEOUT_US"; constexpr const char *SIMPLER_STREAM_SYNC_TIMEOUT_MS_ENV = "SIMPLER_STREAM_SYNC_TIMEOUT_MS"; constexpr const char *SIMPLER_SCHEDULER_TIMEOUT_MS_ENV = "SIMPLER_SCHEDULER_TIMEOUT_MS"; +constexpr const char *SIMPLER_TENSOR_DATA_TIMEOUT_MS_ENV = "SIMPLER_TENSOR_DATA_TIMEOUT_MS"; // Covers the host stream-sync window before the AICPU scheduler no-progress // timer is armed: cold kernel registration, orchestration SO dlopen, runtime @@ -35,6 +36,11 @@ struct RuntimeTimeoutConfig { uint64_t op_execute_timeout_us; int32_t stream_sync_timeout_ms; int32_t scheduler_timeout_ms; + // Orchestration tensor-data wait (ms). Unlike its three siblings this one + // has no host-side default to seed: the budget lives in the runtime's + // TENSOR_DATA_TIMEOUT_MS, which no host translation unit can include, so a + // caller passes 0 here and 0 survives to mean "no override". + int32_t tensor_data_timeout_ms; }; struct HostRuntimeTimeoutConfig { @@ -44,6 +50,12 @@ struct HostRuntimeTimeoutConfig { // override" — the device falls back to its compile-time default // (SCHEDULER_TIMEOUT_CYCLES). Latched once per device into InitArgs. int32_t scheduler_timeout_ms{0}; + // Orchestration tensor-data wait override (ms), same 0-means-default + // convention and the same per-device InitArgs latch. It takes no part in + // the ordering validation below: an unreachable value costs the precise + // SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT classification, never the clean + // shutdown, which the scheduler watchdog owns for every stall shape. + int32_t tensor_data_timeout_ms{0}; }; struct RuntimeTimeoutParseStatus { @@ -53,6 +65,8 @@ struct RuntimeTimeoutParseStatus { bool stream_sync_valid{true}; bool scheduler_env_set{false}; bool scheduler_valid{true}; + bool tensor_data_env_set{false}; + bool tensor_data_valid{true}; }; enum class RuntimeTimeoutOrderStatus { @@ -171,9 +185,28 @@ resolve_runtime_timeout_config(const RuntimeTimeoutConfig &defaults, RuntimeTime ); if (status != nullptr) status->scheduler_valid = ok; } + const char *tensor_env = std::getenv(SIMPLER_TENSOR_DATA_TIMEOUT_MS_ENV); + if (tensor_env != nullptr) { + if (status != nullptr) status->tensor_data_env_set = true; + bool ok = apply_runtime_timeout_override( + SIMPLER_TENSOR_DATA_TIMEOUT_MS_ENV, tensor_env, 1, + static_cast(std::numeric_limits::max()), &cfg.tensor_data_timeout_ms + ); + if (status != nullptr) status->tensor_data_valid = ok; + } return cfg; } +// Whether an overridden tensor-data wait can still latch +// SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT: it must expire before STARS reaps the op. +// Advisory — the caller warns rather than rejecting, because losing that code +// degrades the diagnosis to a generic 507018 without changing what the run does. +// Meaningless for an unset (0) budget, whose real value is compiled into the +// runtime and unknown here. +inline bool tensor_data_timeout_can_latch(const RuntimeTimeoutConfig &cfg) { + return static_cast(cfg.tensor_data_timeout_ms) * 1000 < cfg.op_execute_timeout_us; +} + inline RuntimeTimeoutOrderStatus validate_runtime_timeout_order(const RuntimeTimeoutConfig &cfg) { uint64_t scheduler_timeout_us = static_cast(cfg.scheduler_timeout_ms) * 1000; uint64_t stream_sync_timeout_us = static_cast(cfg.stream_sync_timeout_ms) * 1000; diff --git a/src/common/platform/onboard/host/device_runner_base.cpp b/src/common/platform/onboard/host/device_runner_base.cpp index dc738bee54..aef3f0b13d 100644 --- a/src/common/platform/onboard/host/device_runner_base.cpp +++ b/src/common/platform/onboard/host/device_runner_base.cpp @@ -66,8 +66,10 @@ extern "C" const char *const *runtime_extra_aicpu_symbols(size_t *count); namespace { HostRuntimeTimeoutConfig resolve_onboard_timeout_config() { + // The tensor-data budget seeds as 0: its default lives in the runtime, out + // of this file's reach, and 0 already means "device keeps that default". RuntimeTimeoutConfig order_defaults{ - PLATFORM_OP_EXECUTE_TIMEOUT_US, PLATFORM_STREAM_SYNC_TIMEOUT_MS, PLATFORM_SCHEDULER_TIMEOUT_MS + PLATFORM_OP_EXECUTE_TIMEOUT_US, PLATFORM_STREAM_SYNC_TIMEOUT_MS, PLATFORM_SCHEDULER_TIMEOUT_MS, 0 }; RuntimeTimeoutParseStatus parse_status; RuntimeTimeoutConfig cfg = resolve_runtime_timeout_config(order_defaults, &parse_status); @@ -96,6 +98,11 @@ HostRuntimeTimeoutConfig resolve_onboard_timeout_config() { ); } + if (parse_status.tensor_data_env_set && !parse_status.tensor_data_valid) { + const char *tensor_env = std::getenv(SIMPLER_TENSOR_DATA_TIMEOUT_MS_ENV); + LOG_WARN("%s=%s invalid, ignored", SIMPLER_TENSOR_DATA_TIMEOUT_MS_ENV, tensor_env); + } + bool host_timeout_env_set = parse_status.op_execute_env_set || parse_status.stream_sync_env_set || parse_status.scheduler_env_set; RuntimeTimeoutOrderStatus order_status = validate_runtime_timeout_order(cfg); @@ -107,17 +114,35 @@ HostRuntimeTimeoutConfig resolve_onboard_timeout_config() { order_status == RuntimeTimeoutOrderStatus::OK) ? cfg.scheduler_timeout_ms : 0; + // The tensor-data override rides the same InitArgs latch but stands outside + // the ordering group, so a rejected group still forwards it. + int32_t tensor_data_override = + (parse_status.tensor_data_env_set && parse_status.tensor_data_valid) ? cfg.tensor_data_timeout_ms : 0; + HostRuntimeTimeoutConfig resolved{ + cfg.op_execute_timeout_us, cfg.stream_sync_timeout_ms, scheduler_override, tensor_data_override + }; if (host_timeout_env_set && order_status != RuntimeTimeoutOrderStatus::OK) { LOG_WARN( "Ignoring timeout env overrides: %s (scheduler=%d ms, op_execute=%llu us, stream_sync=%d ms)", runtime_timeout_order_status_name(order_status), cfg.scheduler_timeout_ms, (unsigned long long)cfg.op_execute_timeout_us, cfg.stream_sync_timeout_ms ); - return HostRuntimeTimeoutConfig{ - order_defaults.op_execute_timeout_us, order_defaults.stream_sync_timeout_ms, scheduler_override - }; + resolved.op_execute_timeout_us = order_defaults.op_execute_timeout_us; + resolved.stream_sync_timeout_ms = order_defaults.stream_sync_timeout_ms; + } + if (tensor_data_override != 0) { + RuntimeTimeoutConfig effective = cfg; + effective.op_execute_timeout_us = resolved.op_execute_timeout_us; + if (!tensor_data_timeout_can_latch(effective)) { + LOG_WARN( + "%s=%d ms is not below op_execute=%llu us; the op is reaped before the tensor-data wait can report " + "SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT", + SIMPLER_TENSOR_DATA_TIMEOUT_MS_ENV, tensor_data_override, + (unsigned long long)resolved.op_execute_timeout_us + ); + } } - return HostRuntimeTimeoutConfig{cfg.op_execute_timeout_us, cfg.stream_sync_timeout_ms, scheduler_override}; + return resolved; } } // namespace @@ -571,6 +596,8 @@ int DeviceRunnerBase::ensure_aicpu_init_launched() { // Per-device scheduler watchdog override, resolved once at attach into // timeout_config_. 0 -> the AICPU scheduler keeps its compile-time default. init_args.scheduler_timeout_ms = timeout_config_.scheduler_timeout_ms; + // Same latch for the orchestration tensor-data wait; 0 -> compile-time default. + init_args.tensor_data_timeout_ms = timeout_config_.tensor_data_timeout_ms; // Publish the provisioned async-DMA workspace addresses (all-zero unless the // Worker opted into SDMA). ensure_dma_workspace_provisioned() runs first, so // this single launch carries them; the AICPU SO stays resident, and the diff --git a/src/common/platform/shared/aicpu/aicpu_device_config.cpp b/src/common/platform/shared/aicpu/aicpu_device_config.cpp index 77c1c4424d..f9a2767be2 100644 --- a/src/common/platform/shared/aicpu/aicpu_device_config.cpp +++ b/src/common/platform/shared/aicpu/aicpu_device_config.cpp @@ -17,6 +17,7 @@ namespace { // survives per-task launches because the inner SO stays dlopen'd. int g_orch_device_id = 0; int g_scheduler_timeout_ms = 0; +int g_tensor_data_timeout_ms = 0; unsigned long long g_dma_workspace_addr[DMA_WORKSPACE_KIND_COUNT] = {0}; } // namespace @@ -28,6 +29,10 @@ void set_scheduler_timeout_ms(int timeout_ms) { g_scheduler_timeout_ms = timeout int get_scheduler_timeout_ms() { return g_scheduler_timeout_ms; } +void set_tensor_data_timeout_ms(int timeout_ms) { g_tensor_data_timeout_ms = timeout_ms; } + +int get_tensor_data_timeout_ms() { return g_tensor_data_timeout_ms; } + void set_dma_workspace_addr(int kind, unsigned long long addr) { if (kind < 0 || kind >= DMA_WORKSPACE_KIND_COUNT) return; g_dma_workspace_addr[kind] = addr; diff --git a/src/common/runtime_status/error_names.h b/src/common/runtime_status/error_names.h index 9810da9687..33f6352dfe 100644 --- a/src/common/runtime_status/error_names.h +++ b/src/common/runtime_status/error_names.h @@ -154,7 +154,7 @@ static inline const char *error_hint(int32_t code) { "(total AIV count, or total cluster count for MIX/AIC), or drop the sync-start request"; case SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT: return "find the producing kernel and check it for a hang (see SCHEDULER_TIMEOUT sub_class S1); " - "verify the consumer declares the dependency and exits; raise TENSOR_DATA_TIMEOUT_MS " + "verify the consumer declares the dependency and exits; raise SIMPLER_TENSOR_DATA_TIMEOUT_MS " "to tell a slow kernel apart from a stuck one"; case SIMPLER_ERROR_EXPLICIT_ORCH_FATAL: return "self-inflicted -- follow the message passed to the rt_report_fatal() call site"; diff --git a/tests/st/runtime_fatal_codes/kernels/orchestration/tensor_wait_timeout_orch.cpp b/tests/st/runtime_fatal_codes/kernels/orchestration/tensor_wait_timeout_orch.cpp index b19820fac3..3939122c11 100644 --- a/tests/st/runtime_fatal_codes/kernels/orchestration/tensor_wait_timeout_orch.cpp +++ b/tests/st/runtime_fatal_codes/kernels/orchestration/tensor_wait_timeout_orch.cpp @@ -15,13 +15,13 @@ * Submits an AIC kernel that spins forever, then reads its output with * get_tensor_data. Because the output has a producer in the TensorMap, * get_tensor_data spin-waits for that producer to complete — which never - * happens — so after TENSOR_DATA_TIMEOUT_CYCLES (15e9 cycles == 15 s at the - * 1 GHz AICPU counter) the orchestrator latches TENSOR_WAIT_TIMEOUT. + * happens — so once the tensor-data wait expires the orchestrator latches + * TENSOR_WAIT_TIMEOUT. * * Onboard only (the hang kernel would spin the simulator forever). The test - * raises the AICPU scheduler / STARS op / host stream-sync timeouts all above - * 15 s so the tensor-data wait wins the race; otherwise the no-progress watchdog - * (code 100) or STARS reaps the hang first. + * lowers SIMPLER_TENSOR_DATA_TIMEOUT_MS below every other watchdog so that wait + * wins the race; otherwise the no-progress watchdog (code 100) or STARS reaps + * the hang first. */ #include diff --git a/tests/st/runtime_fatal_codes/test_runtime_fatal_codes.py b/tests/st/runtime_fatal_codes/test_runtime_fatal_codes.py index ea63310559..657ef01544 100644 --- a/tests/st/runtime_fatal_codes/test_runtime_fatal_codes.py +++ b/tests/st/runtime_fatal_codes/test_runtime_fatal_codes.py @@ -164,16 +164,12 @@ def _wait_for_host_log(capfd, markers: tuple[str, ...], dropped_before: int, tim kernel="aic/kernel_hang.cpp", kernel_core="aic", onboard_only=True, # a while(true) kernel would hang the simulator - # The data-wait timeout is 15 s on both arches now that it is frequency- - # scaled (TENSOR_DATA_TIMEOUT_MS, #1189) -- before that it was 15 s on - # a5 but 300 s on a2a3, so this case used to be a5-only. Raise every other - # watchdog above 15 s so the tensor-data wait wins the race and latches - # code 8 before they reap the hung core. - env={ - "SIMPLER_SCHEDULER_TIMEOUT_MS": 30000, - "SIMPLER_OP_EXECUTE_TIMEOUT_US": 30000000, - "SIMPLER_STREAM_SYNC_TIMEOUT_MS": 40000, - }, + # Drop the data wait to 1 s so it beats every other watchdog to the hung + # core -- including the 2 s scheduler timeout the onboard CI jobs set, + # which would otherwise latch code 100 first. The other three keep + # whatever the job/platform sets: the tensor-data budget is not part of + # their ordering group, so lowering it alone cannot invalidate them. + env={"SIMPLER_TENSOR_DATA_TIMEOUT_MS": 1000}, marker="orch_error_code=8", explain="TENSOR_WAIT_TIMEOUT", ), diff --git a/tests/ut/cpp/common/test_runtime_timeout_config.cpp b/tests/ut/cpp/common/test_runtime_timeout_config.cpp index 13a308a3e4..b316cbad25 100644 --- a/tests/ut/cpp/common/test_runtime_timeout_config.cpp +++ b/tests/ut/cpp/common/test_runtime_timeout_config.cpp @@ -20,10 +20,12 @@ namespace { +// The tensor-data budget seeds as 0 ("no override"): its default is compiled +// into the runtime, which no host translation unit can see. constexpr RuntimeTimeoutConfig kDefaults{ - PLATFORM_OP_EXECUTE_TIMEOUT_US, PLATFORM_STREAM_SYNC_TIMEOUT_MS, PLATFORM_SCHEDULER_TIMEOUT_MS + PLATFORM_OP_EXECUTE_TIMEOUT_US, PLATFORM_STREAM_SYNC_TIMEOUT_MS, PLATFORM_SCHEDULER_TIMEOUT_MS, 0 }; -constexpr RuntimeTimeoutConfig kCiTightTimeouts{3000000, 4000, 2000}; +constexpr RuntimeTimeoutConfig kCiTightTimeouts{3000000, 4000, 2000, 0}; void set_env_var(const char *name, const char *value) { #if defined(_WIN32) @@ -47,15 +49,18 @@ class ScopedUnsetTimeoutEnv { save(SIMPLER_OP_EXECUTE_TIMEOUT_US_ENV, op_); save(SIMPLER_STREAM_SYNC_TIMEOUT_MS_ENV, stream_); save(SIMPLER_SCHEDULER_TIMEOUT_MS_ENV, scheduler_); + save(SIMPLER_TENSOR_DATA_TIMEOUT_MS_ENV, tensor_data_); unset_env_var(SIMPLER_OP_EXECUTE_TIMEOUT_US_ENV); unset_env_var(SIMPLER_STREAM_SYNC_TIMEOUT_MS_ENV); unset_env_var(SIMPLER_SCHEDULER_TIMEOUT_MS_ENV); + unset_env_var(SIMPLER_TENSOR_DATA_TIMEOUT_MS_ENV); } ~ScopedUnsetTimeoutEnv() { restore(SIMPLER_OP_EXECUTE_TIMEOUT_US_ENV, op_); restore(SIMPLER_STREAM_SYNC_TIMEOUT_MS_ENV, stream_); restore(SIMPLER_SCHEDULER_TIMEOUT_MS_ENV, scheduler_); + restore(SIMPLER_TENSOR_DATA_TIMEOUT_MS_ENV, tensor_data_); } private: @@ -83,6 +88,7 @@ class ScopedUnsetTimeoutEnv { SavedValue op_; SavedValue stream_; SavedValue scheduler_; + SavedValue tensor_data_; }; } // namespace @@ -94,6 +100,7 @@ TEST(RuntimeTimeoutConfig, UnsetEnvKeepsDefaults) { EXPECT_EQ(cfg.op_execute_timeout_us, 45000000u); EXPECT_EQ(cfg.stream_sync_timeout_ms, 50000); EXPECT_EQ(cfg.scheduler_timeout_ms, 20000); + EXPECT_EQ(cfg.tensor_data_timeout_ms, 0); EXPECT_EQ(validate_runtime_timeout_order(cfg), RuntimeTimeoutOrderStatus::OK); } @@ -102,6 +109,7 @@ TEST(RuntimeTimeoutConfig, ValidEnvOverridesDefaults) { set_env_var(SIMPLER_OP_EXECUTE_TIMEOUT_US_ENV, "5000000"); set_env_var(SIMPLER_STREAM_SYNC_TIMEOUT_MS_ENV, "7000"); set_env_var(SIMPLER_SCHEDULER_TIMEOUT_MS_ENV, "3000"); + set_env_var(SIMPLER_TENSOR_DATA_TIMEOUT_MS_ENV, "2500"); RuntimeTimeoutParseStatus status; RuntimeTimeoutConfig cfg = resolve_runtime_timeout_config(kDefaults, &status); @@ -109,13 +117,17 @@ TEST(RuntimeTimeoutConfig, ValidEnvOverridesDefaults) { EXPECT_EQ(cfg.op_execute_timeout_us, 5000000u); EXPECT_EQ(cfg.stream_sync_timeout_ms, 7000); EXPECT_EQ(cfg.scheduler_timeout_ms, 3000); + EXPECT_EQ(cfg.tensor_data_timeout_ms, 2500); EXPECT_TRUE(status.op_execute_env_set); EXPECT_TRUE(status.stream_sync_env_set); EXPECT_TRUE(status.scheduler_env_set); + EXPECT_TRUE(status.tensor_data_env_set); EXPECT_TRUE(status.op_execute_valid); EXPECT_TRUE(status.stream_sync_valid); EXPECT_TRUE(status.scheduler_valid); + EXPECT_TRUE(status.tensor_data_valid); EXPECT_EQ(validate_runtime_timeout_order(cfg), RuntimeTimeoutOrderStatus::OK); + EXPECT_TRUE(tensor_data_timeout_can_latch(cfg)); } TEST(RuntimeTimeoutConfig, InvalidEnvKeepsDefaultAndReportsStatus) { @@ -187,3 +199,29 @@ TEST(RuntimeTimeoutConfig, SimPlatformSkipsOnboardOrdering) { RuntimeTimeoutOrderStatus::SCHEDULER_NOT_BELOW_OP_EXECUTE ); } + +TEST(RuntimeTimeoutConfig, TensorDataInvalidEnvLeavesNoOverride) { + ScopedUnsetTimeoutEnv env; + set_env_var(SIMPLER_TENSOR_DATA_TIMEOUT_MS_ENV, "15s"); + + RuntimeTimeoutParseStatus status; + RuntimeTimeoutConfig cfg = resolve_runtime_timeout_config(kDefaults, &status); + + EXPECT_EQ(cfg.tensor_data_timeout_ms, 0); + EXPECT_TRUE(status.tensor_data_env_set); + EXPECT_FALSE(status.tensor_data_valid); +} + +// The tensor-data wait stands outside the ordering group: an unreachable value +// is reported by tensor_data_timeout_can_latch, never by rejecting the group, +// which would silently drop the op/stream overrides that were valid. +TEST(RuntimeTimeoutConfig, TensorDataStaysOutOfOrderingVerdict) { + RuntimeTimeoutConfig cfg = kCiTightTimeouts; + cfg.tensor_data_timeout_ms = 15000; + + EXPECT_EQ(validate_runtime_timeout_order(cfg), RuntimeTimeoutOrderStatus::OK); + EXPECT_FALSE(tensor_data_timeout_can_latch(cfg)); + + cfg.tensor_data_timeout_ms = 1000; + EXPECT_TRUE(tensor_data_timeout_can_latch(cfg)); +}