[Bug][Jetson Orin][v0.9.1] SM87 GDN CuTeDSL artifact targets CUDA 13.2 and silently corrupts output on JetPack 6.2.1

[ljandshp]

Summary

TensorRT Edge-LLM v0.9.1 ships an aarch64/sm_87 CuTe DSL artifact whose metadata reports CUDA 13.2.78. On JetPack 6.2.1 / CUDA 12.6, the Gated Delta Net (GDN) AOT modules fail to initialize/load with cudaErrorUnknown, but inference continues and produces plausible-looking garbage tokens instead of failing.

Regenerating the same GDN kernels natively for SM87 with CUDA 12.6 fixes both the upstream GDN unit tests and end-to-end Qwen3.5 inference. This appears to be a release-artifact compatibility issue combined with a fail-open loader path.

Environment

• Device: NVIDIA Jetson Orin NX 16GB (SM87)
• L4T: R36.4.4
• OS: Ubuntu 22.04.5 LTS
• JetPack: 6.2.1
• CUDA toolkit: 12.6
• TensorRT package: 10.3.0.30
• TensorRT Edge-LLM: v0.9.1, commit 7f061f21f0a581ba234a1e233c9315b89d8e47d6
• Model: stock dense Qwen3.5-0.8B, FP16 export
• No external weights or custom quantization are involved in this reproducer.

Artifact mismatch

The artifact selected on Orin is:

cpp/kernels/cuteDSLArtifact/aarch64/sm_87/metadata.json

It contains:

{
  "arch": "aarch64",
  "artifact_tag": "sm_87",
  "gpu_arch": "sm_87",
  "cuda_version": "13.2.78",
  "cutlass_dsl_version": "4.6.0"
}

The target JetPack environment provides CUDA 12.6.

Reproduction

1. Build v0.9.1 on the Orin NX using the bundled aarch64/sm_87 artifact.
2. Export stock Qwen3.5-0.8B to FP16 ONNX.
3. Build an engine with batch size 1, max input length 64, and max KV capacity 192.
4. Run greedy inference with:

{
  "batch_size": 1,
  "temperature": 0.0,
  "top_p": 1.0,
  "top_k": 1,
  "max_generate_length": 4,
  "apply_chat_template": true,
  "enable_thinking": false,
  "requests": [{
    "messages": [{
      "role": "user",
      "content": "Answer with only the number: 2 + 2 ="
    }]
  }]
}

Observed

The log prints exactly eight instances of:

Got Cuda Error cudaErrorUnknown: unknown error

The engine still loads and inference completes, returning:

chettebung个头enses

Expected

Either:

• the engine returns 4; or
• the GDN plugin fails initialization clearly and prevents inference from running.

Continuing after the AOT module load failure is a silent correctness problem.

Control experiment

I regenerated only the GDN artifact natively on the same Orin NX with CUDA 12.6:

python kernelSrcs/build_cutedsl.py \
  --kernels gdn \
  --gpu_arch sm_87 \
  --arch aarch64 \
  --output_dir <output-dir> \
  --clean -j 2

The regenerated metadata reports CUDA 12.6.68 and includes all six GDN variants.

After rebuilding the plugin and engine against that artifact:

• test_gated_delta_net_plugin.py::test_decode[bs1]: PASS
• test_gated_delta_net_plugin.py::test_prefill[seq16-bs1]: PASS
• Same FP16 ONNX and same inference input: output is exactly 4
• Finish reason: end-of-sequence
• cudaErrorUnknown count: 0

The upstream tests failed with the bundled CUDA 13.2 artifact and passed with the CUDA 12.6 artifact.

Suggested fixes

1. Ship an SM87/aarch64 CuTe DSL artifact built for the CUDA version supported by JetPack 6.2/6.2.1, or select artifacts by both SM and CUDA compatibility.
2. Validate artifact metadata against the runtime/toolkit compatibility range.
3. Make GDN AOT loading fail closed:
   • propagate initialization/module-load errors;
   • do not mark the runner as loaded after a failed load;
   • make plugin construction or enqueue return a hard error.
4. Add a small dense Qwen3.5 JetPack 6.2.1 correctness smoke test to release validation.

I can provide the complete sanitized logs and a minimal fail-closed patch if useful.

[nvluxiaoz]

Thanks. CUDA 12.x should regenerate the artifacts and we will support Jetpack 7.2 only in the future for Jetson Orin. JP 6 support will be dropped in the future once Jetpack 7.2 is widely adopted. That is why we stopped shipping CUDA 12.x Orin tarballs.