[Bug][sim] Simulator rejects any async-DMA workspace request, blocking 4 prefetch-using model cases at worker init with -1001

[Crystal-wzy]

simpler — simulator host C-API shim. simpler_init on simulation rejects enable_sdma, so a Worker that
asks for an async-DMA workspace cannot come up. Four deepseek_v4_flash_mtp model cases die at runtime init
on both simulator targets, even though the simulator's prefetch path is a no-op by construction and the
kernels are numerically correct without a real engine.

-1001 is PTO_RUNTIME_ERR_UNSUPPORTED: src/common/worker/runtime_c_api.h:96-104 defines
PTO_RUNTIME_LATCHED_CODE_MAX = 999, PTO_RUNTIME_ERR_BASE = -(PTO_RUNTIME_LATCHED_CODE_MAX + 1) = -1000,
and PTO_RUNTIME_ERR_UNSUPPORTED = PTO_RUNTIME_ERR_BASE - 1. Its comment reads "The request names a
capability this platform/runtime does not implement."

The chain, verified at every link:

1. models/deepseek_v4_flash_mtp/decode_csa.py:227-237 calls pl.prefetch.make_context() and
   pl.prefetch.async_prefetch(...) nine times — fire-and-forget, no session, no wait.
2. pypto codegen records that a kernel consumes the runtime-owned SDMA workspace
   (pypto/backend/pto_backend.py:990). Confirmed on the emitted artifact:

   # <work_dir>/kernel_config.py
   RUNTIME_CONFIG = {"runtime": "tensormap_and_ringbuffer", "aicpu_thread_num": 0, "enable_sdma": True}

3. pypto/runtime/runner.py:1503 reads it and builds the Worker with enable_sdma=True;
   simpler/worker.py forwards it into ChipWorker.init, which reaches simpler_init.
4. On simulation, src/common/platform/sim/host/c_api_shared.cpp:396 rejects it outright:

   // Opting into SDMA fails here rather than at the first kernel read, so such
   // a Worker cannot come up on sim at all.
   if (enable_sdma != 0) return PTO_RUNTIME_ERR_UNSUPPORTED;

   Onboard instead records the request and provisions later
   (src/common/platform/onboard/host/c_api_shared.cpp:440,
   runner->set_dma_workspace_request(enable_sdma != 0, std::move(warmup_vec))).

The rejection is deliberate, and its reasoning is sound. It is pinned by
tests/ut/py/test_worker/test_dma_workspace_sim.py, whose docstring states the contract: such a Worker must
fail to come up "rather than reaching its first run reading a zero address it expected to be live — the
rejection is the only thing standing between an unsupported request and a silent wrong answer."

That risk is real, and this repo contains the proof. examples/a2a3/tensormap_and_ringbuffer/prefetch_async_demo
declares SDMA mandatory and treats a null workspace as broken injection:

__gm__ uint8_t *sdma_workspace = get_dma_workspace(args, DMA_WORKSPACE_SDMA);
if (sdma_workspace == nullptr) {
    pipe_barrier(PIPE_ALL);
    return;                      // out is never written
}

Simply deleting the enable_sdma rejection and running that demo on a2a3sim turns a clear
simpler_init failed with code -1001 into Golden mismatch on 'out': max_diff=15.875 — exactly the silent
wrong answer the contract exists to prevent. So the defect is not the rejection itself; it is that sim
offers no way to satisfy the request.

What sim can honestly provide. Under CPU simulation TPREFETCH_ASYNC is a no-op
(pto-isa/include/pto/cpu/TPrefetchAsync.hpp — AsyncEvent::Wait/Test return true, the impl ignores its
context), and the async transfers that would actually read a workspace, TGET_ASYNC / TPUT_ASYNC, have no
CPU-sim implementation at all. Nothing on sim dereferences the workspace — but a kernel may still test it
for null, as the demo does. A live-but-inert block therefore satisfies every consumer that exists.

Source

• Parent CI run: https://github.com/hw-native-sys/pypto-lib/actions/runs/33453529066
• Workflow: Daily CI #187 (scheduled 2026-09-01 00:08 UTC)
• PyPTO-Lib branch / commit: main / 0e7e07aa5e42db6e08c68fd5bfe1def571492b53
• simpler: main @ 5cb790c0
• Jobs: model-tests-sim (a2a3sim), model-tests-sim (a5sim)

Affected cases — exactly the ones that reach pl.prefetch; no other model in the 159-file corpus uses it:

case	uses prefetch	a2a3sim	a5sim
mtp/decode_csa.py	directly (9 calls)	-1001	-1001
mtp/decode_hca.py	directly (6 calls)	-1001	-1001
mtp/decode_swa.py	directly (6 calls)	-1001	-1001
mtp/decode_layer.py	via import	-1001	-1001

Failure Details From Logs

Compilation, input generation and the golden computation all succeed; the failure is in worker init:

[RUN] compile done (3.44s)
[RUN] generate inputs done (0.84s)
[RUN] compute golden done (1.18s)
[RUN] runtime ...
Traceback (most recent call last):
  ...
  File ".../pypto/runtime/device_runner.py", line 831, in execute_on_device
    worker.init(prewarm_config=cfg)
  File ".../simpler/worker.py", line 7586, in _init_level2
    self._chip_worker.init(
  File ".../simpler/task_interface.py", line 1349, in init
    self._impl.init(
RuntimeError: simpler_init failed with code -1001

The same failure reproduces from simpler's own tree, after adding a2a3sim to the demo case's platforms:

pytest examples/a2a3/tensormap_and_ringbuffer/prefetch_async_demo --platform a2a3sim
# -> RuntimeError: simpler_init failed with code -1001

Reproduction

python models/deepseek_v4_flash_mtp/decode_csa.py -p a2a3sim
# -> RuntimeError: simpler_init failed with code -1001

Compilation alone is fine — the same case passes with compile_only=True in 3.3 s. Any kernel using
pl.prefetch.async_prefetch reproduces it; no special shape or size is needed.

Expected Behavior

• A kernel that hints a prefetch runs on the CPU simulator.
• A kernel that reads get_dma_workspace(args, DMA_WORKSPACE_SDMA) sees a live address on sim, as it does
  onboard, so the null-workspace branch keeps meaning "injection is broken" on every platform.
• Onboard is unchanged: a5 onboard and builds without the a2a3 PTO-SDMA provider still refuse.
• A workspace kind with no sim implementation behind it is still refused, so a real gap cannot be masked.

Actual Behavior

• Simulation rejects any enable_sdma request at worker init, so four model cases get no functional coverage
  on either simulator target.

Fix

Provision an inert workspace on sim rather than removing the gate, so the contract the rejection was
protecting still holds. Implemented on main @ 5cb790c0; 6 files, +119/-29.

file	change
src/common/platform/sim/host/c_api_shared.cpp	drop the rejection; runner->set_dma_workspace_request(enable_sdma != 0), mirroring onboard's :440
src/common/platform/sim/host/device_runner_base.h	the request flag, kSimDmaWorkspaceBytes = 16 * 1024, the block pointer
src/a2a3/platform/sim/host/device_runner.cpp	one-shot latch in ensure_binaries_loaded
src/a5/platform/sim/host/device_runner.cpp	same
examples/.../prefetch_async_demo/test_prefetch_async_demo.py	case runs on a2a3sim too; docstring updated
tests/ut/py/test_worker/test_dma_workspace_sim.py	rewritten to pin the new contract

The latch sits beside the existing set_scheduler_timeout_ms one, which already describes itself as
mirroring the onboard InitArgs path, and publishes through set_dma_workspace_addr — the same entry point
onboard feeds, already exported by the sim libaicpu_kernel.so:

if (dma_workspace_requested_) {
    using SetDmaWorkspaceAddrFunc = void (*)(int, unsigned long long);
    SetDmaWorkspaceAddrFunc set_dma_workspace_addr_func = nullptr;
    if (!load_sym("set_dma_workspace_addr", reinterpret_cast<void **>(&set_dma_workspace_addr_func)))
        return PTO_RUNTIME_ERR_INTERNAL;
    if (dma_workspace_block_ == nullptr) {
        dma_workspace_block_ = mem_alloc_.alloc(kSimDmaWorkspaceBytes);
        if (dma_workspace_block_ == nullptr) { /* ... */ return PTO_RUNTIME_ERR_INTERNAL; }
        std::memset(dma_workspace_block_, 0, kSimDmaWorkspaceBytes);
    }
    set_dma_workspace_addr_func(
        DMA_WORKSPACE_SDMA, static_cast<unsigned long long>(reinterpret_cast<uintptr_t>(dma_workspace_block_))
    );
}

16 KB matches what the a2a3 onboard SdmaWorkspaceManager::Init provisions, so a kernel sized against the
onboard budget stays in bounds. Only DMA_WORKSPACE_SDMA is published; DMA_WORKSPACE_URMA keeps its zero
address, because no sim implementation stands behind it. The block is tracked by mem_alloc_ and freed with
every other allocation.

test_dma_workspace_sim.py is rewritten, not deleted: the contract it guards — a kernel never reads a
zero address where a live one was expected — is preserved by this fix, so the test now pins that an
SDMA-enabled Worker comes up, with the no-SDMA Worker as the control.

Validation

All results below are from simpler main with the fix applied.

suite	result
ctest --test-dir tests/ut/cpp/build -LE requires_hardware	128/128 passed
pytest tests/ut	2075 passed / 15 failed — all 15 reproduce on a clean stashed tree (test_callable_identity.py remote-session cases, test_remote_zero_residual.py); pre-existing and unrelated
pytest examples tests/st --platform a2a3sim --device 0-15	72 passed, 0 failed
pytest examples tests/st --platform a5sim --device 0-15	70 passed, 0 failed
prefetch_async_demo on a2a3sim	PASS — the case that yields Golden mismatch: max_diff=15.875 if the gate is merely deleted
rewritten test_dma_workspace_sim.py	2 passed

--collect-only confirms TestPrefetchAsyncDemo::test_run is collected by the a2a3sim sweep:
_st-sim-a2a3.yml runs it without -m 'not sdma' (the SDMA quarantine applies only to the a2a3 hardware
sweep in _st-npu-a2a3.yml:102), so the sim case is a live CI barrier rather than dead weight.

Provenance of the pypto-lib numbers. Before this fix existed, the four affected cases were measured on
the CI-pinned simpler snapshot (dbdd041e, whose sim shim still used the older mask-based
simpler_provision_dma_workspace) with the gate removed: all four passed on both a2a3sim and a5sim with
golden validation. That establishes the symptom, and that nothing else blocks those models — it is not a
measurement of the code shipped here, whose verification is simpler's own suites above. Re-running the
pypto-lib corpus against a main-based runtime is the remaining end-to-end check.

Environment

Component	Version
simpler	main @ 5cb790c0
pypto-lib CI commit	0e7e07aa5e42db6e08c68fd5bfe1def571492b53
pypto CI commit	cb49d101aa770b72c0113f6a9ee32301f38464ff
simpler CI commit at the time of #187	dbdd041e957420ea15b03e878400dd4de5e9c34c
pto-isa	39782974cea3b8068b428b43df8f0aaceb0925e4
PTOAS	v0.57 (aarch64)
Runtime	tensormap_and_ringbuffer
Targets	a2a3sim, a5sim (both fail before the fix); a2a3 onboard passes
Host platform	Linux (aarch64)

Additional Context

• a5 onboard is a separate gap, out of scope. src/a5/platform/onboard/host/comm_hccl.cpp:762 returns
  0 from dma_workspace_supported_mask() ("Callable-declared workspace injection is not available on a5
  yet. Its URMA workspace is sized per communication domain (rank count), not per device"). These cases
  therefore still fail on real A5 hardware, with the same -1001. That needs the a5 URMA per-domain provider.
• sdma_async_completion_demo stays hardware-only on both a2a3 and a5. It uses TGET_ASYNC /
  TPUT_ASYNC, which have no CPU-sim implementation, so it would fail at the unimplemented instruction —
  the right place to fail — rather than at worker init.
• Alternative considered: have pypto omit enable_sdma when the target platform has no SDMA provider,
  lowering prefetch to nothing at compile time. That would also cover a5 onboard, but it moves the runtime's
  capability matrix into the compiler and makes the sim artifact differ from the device artifact.
• The workspace does not reach the kernel as a user argument. The generated prefetch_attn_w.cpp takes ten
  parameters while the orchestrator binds only nine inputs; the trailing one is the runtime-injected
  workspace.
• The affected models use prefetch fire-and-forget: no pl.prefetch.session, no pl.prefetch.wait, and the
  generated kernels contain no BuildAsyncSession or .Wait( call. Even if they did, the CPU stubs return
  true unconditionally.

[Crystal-wzy]

SIM_SDMA_WORKSPACE.patch