Add: SIMPLER_TENSOR_DATA_TIMEOUT_MS to override the tensor-data wait - #2114
Draft
ChaoZheng109 wants to merge 1 commit into
Draft
Add: SIMPLER_TENSOR_DATA_TIMEOUT_MS to override the tensor-data wait#2114ChaoZheng109 wants to merge 1 commit into
ChaoZheng109 wants to merge 1 commit into
Conversation
|
Warning Review limit reachedNext included review available in 12 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (25)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The orchestration get_tensor_data / set_tensor_data spin in tensormap_and_ringbuffer is bounded by TENSOR_DATA_TIMEOUT_MS, compiled into the runtime at 15 s. Changing it needed a rebuild, so callers in other repos could not tell a slow producer from a stuck one without reinstalling simpler, and the tensor_wait_timeout ST had to raise the other three watchdogs above 15 s (an env set the ordering check rejects outright, since 30000 ms is not below a 30 s op-execute) and then wait out the full 15 s on hardware. Route the budget through the path the scheduler watchdog already uses: the host parses the env var, forwards a validated value in InitArgs, the AICPU entry latches it into the resident SO, and wait_for_tensor_ready reads it once per call. 0 keeps the compiled default, so an unset variable changes nothing. Only tensormap_and_ringbuffer consumes it; host_build_graph resolves tensor data on the host and never waits. The budget stays out of the scheduler/op-execute/stream-sync ordering group. That group is validated as a unit and a failure discards every override in it, so folding in a value the host cannot even see — the default lives in the runtime — would let it invalidate an otherwise valid set. What ordering buys instead is reachability: a wait that expires after the scheduler or op-execute timeout never latches TENSOR_WAIT_TIMEOUT, because the run is already being reaped. tensor_data_timeout_can_latch() reports that as a warning, and the clean shutdown remains the scheduler watchdog's job for every stall shape. The ST case now sets the wait to 1 s instead of pushing three other timeouts out of the way: it latches code 8 in 7.9 s instead of 21 s, and the device log confirms the override reaches the AICPU (Timeout (50000000 cycles) at a2a3's 50 MHz counter, against 750000000 for the default).
ChaoZheng109
marked this pull request as draft
September 3, 2026 09:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tensormap_and_ringbufferorchestration tensor-data wait(
get_tensor_data/set_tensor_data,TENSOR_DATA_TIMEOUT_MS, 15 s)overridable per run with
SIMPLER_TENSOR_DATA_TIMEOUT_MS, so a caller cantell a slow producer from a stuck one without reinstalling simpler.
→
InitArgs.tensor_data_timeout_ms→ resident-SO global latched bysimpler_aicpu_init→ read once perwait_for_tensor_ready()call. Simlatches the same value through the existing optional-symbol path.
means "keep the compiled default".
scheduler/op-execute/stream-sync ordering group: that group is validated as a
unit and a failure discards every override in it, so folding in a value the
host cannot see (the default is compiled into the runtime) would let it
invalidate an otherwise valid set. An unreachable value is reported by
tensor_data_timeout_can_latch()as a warning instead.host_build_graphis untouched: it resolves tensor data on the host and hasno such wait.
tensor_wait_timeoutST now lowers this one budget to 1 s instead ofraising the other three above 15 s. The old env set was rejected by the
ordering check anyway (
SCHEDULER=30000msis not belowOP_EXECUTE=30s), sothe case was passing on the compiled defaults and burning the full 15 s.
Testing
ctest -LE requires_hardware— 132/132 pass (extendstest_runtime_timeout_configwith the new field, an invalid-value case,and a case pinning "tensor-data stays out of the ordering verdict")
tests/st/runtime_fatal_codes --platform a2a3simtests/st/runtime_fatal_codes -k tensor_wait_timeout --platform a2a3undertask-submit: PASS in 7.9 s (was 21 s), and thedevice log shows the override actually reaching the AICPU —
FATAL(code=8): Timeout (50000000 cycles)= 1 s at a2a3's 50 MHz counter,against
750000000(15 s) without it.check_english_only.py,check_headers.py, markdownlint cleanNote for review
This adds one behavior-gating environment variable, which
.claude/rules/env-macro-gating.mdrequires be called out explicitly. It is thefourth in the timeout family (
SIMPLER_SCHEDULER_TIMEOUT_MS/SIMPLER_OP_EXECUTE_TIMEOUT_US/SIMPLER_STREAM_SYNC_TIMEOUT_MS, #1112), opt-inand default-off in the same sense: unset means the compiled default.