NPU firmware hang during gpt-oss:20b prefill GEMM — deterministic TDR timeout (qds_device::wait unexpected command state)

[jwp2987]

NPU firmware hang during gpt-oss:20b prefill GEMM — deterministic TDR timeout after a few requests

Summary

Running gpt-oss:20b on a Ryzen AI NPU with FastFlowLM, the NPU firmware
deadlocks during a prefill matrix-multiply after a small number of independent
requests. The firmware control processor wedges at a fixed program counter
(0x28b060ad) without ever dispatching work to the DPU array and without
raising a fault. The kernel driver's TDR (timeout detection & recovery) reaps
the stuck command ~2 s later, and FLM surfaces it as:

qds_device::wait() unexpected command state

This reproduces on stock upstream FLM fd37140 with the shipped xclbins —
it is not specific to any local modification. The trigger is request count /
scheduling, not prompt content or context length.

Environment

Component	Version
FLM	upstream fd37140 (also reproduced on a fork 20 commits ahead)
Model	gpt-oss:20b (Q4NX, GPT-OSS-20B-NPU2), shipped xclbins
NPU	AMD Ryzen AI 7 445 ("Gorgon Point"), XDNA2 / AIE2P, PCI 1022:17f0 rev 0x20
Kernel driver	amdxdna 2.26.0_20260720 (srcversion 9212AE883D8BFBE70F758C6)
XRT	2.26 (built from xdna-driver, matching shim)
tdr_timeout_ms	default (2000 ms)
OS / kernel	Ubuntu, Linux 7.1.0+

Reproduced identically on:

• stock libgpt_oss_npu.so + shipped xclbins (upstream binary)
• a locally-built driver + XRT 2.26 (upgraded from 2.25) — no change

Reproduction

15 independent (non-concurrent) chat requests, each a distinct short prompt
("Name three ."), FLM_SKIP_KV_PROBE=1, gpt-oss:20b, streaming off.

Upstream fd37140

req1 [colors]  (24.0s): ok
req2 [metals]  (26.1s): ok
req3 [planets] (36.8s): ok
req4 [rivers]  (36.0s): ERROR  qds_device::wait() unexpected command state
req5..15               : server no longer serves (all EMPTY, 0.0s)
qds_device errors: 1

The upstream server does not recover from the fault — every subsequent
request returns empty.

Fork (20 commits ahead), same xclbins/driver

req1..5   ok
req6  [trees]       : ERROR qds_device::wait() unexpected command state
req7..11  ok
req12 [instruments] : ERROR qds_device::wait() unexpected command state
req13..15 ok
server ALIVE   (per-request recovery)
qds_device errors: 2

Same underlying fault; the fork happens to survive it per-request rather than
wedging the whole server. The fork's changes are therefore not the cause — if
anything they add resilience.

Firmware state capture (from amdxdna TDR)

aie2_tdr_detect: TDR timeout detected
aie2_set_cmd_timeout: Firmware timeout state capture:
   Version: 2.0
   Report size: 0x230
   Context ID: 12
   DPU PC: 0xffffffff          <- DPU array never dispatched
   TXN OP ID: 0xffffffff       <- no transaction op in flight on the DPU
   Context PC: 0x28b060ad      <- control processor wedged here (FIXED across all faults)
   Fatal error type: 0x0       <- NOT a crash/exception
   Fatal error exception type: 0x0
   Fatal error exception PC: 0x0
   Timed out sub command ID: 0

Interpretation: the firmware control processor is hung, not crashed. It
spins at a constant PC during prefill GEMM and never dispatches the DPU
(DPU PC = 0xffffffff), so it never posts the mailbox completion. The submit
mailbox message (opcode 0x18, size 24) is sent; the matching size-12 completion
never arrives, and TDR aborts the command.

Host-side hang location (gdb)

The FLM host thread blocks inside XRT waiting on that command:

qds_device::wait()
  <- kernel_command::wait()
  <- run_impl::wait()
  <- gpt_oss_npu::Impl::_prefill_with_mm()
  <- prefill()
  <- AutoModel::_chunked_insert()
  <- _shared_insert()

So the stuck command is a prefill matrix-multiply submitted from the
gpt-oss engine's _prefill_with_mm path.

What was ruled out

Layer	Ruled out how
FLM host C++ / request path	reproduces on stock upstream fd37140
KV-cache truncation probe	FLM_SKIP_KV_PROBE=1, and reproduces on upstream w/o the patch
Prompt content / context length	faults on varied subjects; long-prompt run faulted at same request index, not earlier (trigger is request count, not cumulative tokens)
Power mode	reproduces in default and turbo
Kernel driver	reproduces on freshly built driver (37 commits newer)
XRT base + xdna shim	reproduces on full XRT 2.26 (upgraded from 2.25)
Model weights	deterministic PC, DPU never runs — not a data-dependent numeric issue

Everything above the firmware is eliminated. The remaining suspect is the
firmware / xclbin GEMM kernel used by _prefill_with_mm for gpt-oss, both
of which ship as opaque binaries.

Trigger characteristics

• Timing/scheduling dependent: fault index shifts with load (req6 normally;
  req3 under dynamic-debug overhead). Points to a race in the firmware
  command/queue handling, not a fixed input.
• gpt-oss-specific: gpt-oss uses more distinct NPU contexts (6) than e.g.
  qwen3:8b (4); qwen3:8b under the same harness does not fault.
• Deterministic wedge PC (0x28b060ad) every time.

Asks

1. Can AMD/FLM map Context PC 0x28b060ad to a location in the gpt-oss prefill
   GEMM firmware/xclbin? A fixed hang PC with the DPU never dispatched strongly
   suggests a control-processor spin-wait / queue-handling race.
2. Should the upstream server recover from a TDR'd command the way the fork
   does, instead of wedging all subsequent requests?
3. Is there a known-good firmware/xclbin revision for gpt-oss on XDNA2
   Gorgon Point?