Segfault in libggml-base at [5/7] FlexiDualGrid shape decode on Vulkan (NVIDIA proprietary); CPU backend completes the same stage

[3DJ77]

Summary

trellis-server segfaults in libggml-base.so at stage [5/7] FlexiDualGrid shape decode on the Vulkan backend with the NVIDIA proprietary driver. Reproduced twice on the same host with the same input. The identical stage completes normally on the CPU backend on that host, so this looks specific to the Vulkan path rather than to the input or the model weights.

VRAM is not the constraint: the device reports 8438 MB and peak usage during the run was ~1.8 GB.

Environment

• GPU: NVIDIA T1000 8GB, driver 550.163.01, Vulkan API 1.3.277
• OS: Debian-based, kernel 6.12.95
• Build: -DGGML_VULKAN=ON -DGGML_OPENMP=ON, no CUDA backend
• Models: TRELLIS.2 GGUF set, 1024 cascade

Repro

trellis-server --models <dir> --host 127.0.0.1 --port 8080 --gpu 1 --seed 42 --require-gpu
curl -f -X POST -F image=@input.png http://127.0.0.1:8080/generate -o out.glb

Input is a single centered object on a plain background, 1024x1024 PNG.

What happens

Stages 1-4 complete normally, including the full sparse-structure flow (12 steps, 20 forwards, ~503s). The crash lands immediately after the HR SLAT stats line:

      [flow] 12 steps, 20 forwards, 502.5s
      [stats] HR slat n=178688 mean=-0.0505 std=5.5947 min=-24.335 max=28.489 nan/inf=0
[5/7] FlexiDualGrid shape decode -> mesh @res1024
[trellis] using Vulkan1 (8438 MB)

Then nothing further is logged, the HTTP connection closes (curl: (52) empty reply), and the kernel logs:

trellis-server[585614]: segfault at 0 ip 00007fd0fbdb6598 sp 00007fd0f6ffad08 error 4 in libggml-base.so.0.15.1[1f598,7fd0fbdad000+8e000]
trellis-server[634397]: segfault at 0 ip 00007f2ae5acd598 sp 00007f2ae... error 4 in libggml-base.so.0.15.1

segfault at 0 with error 4 is a null-pointer read. Both runs crashed at the same stage with the same signature; wall time to crash was 947s and 932s.

Note the stats line reports nan/inf=0, so the tensor feeding the decode looks finite.

Contrast: the CPU backend gets through it

Same host, same image, same seed, --gpu -1: the pipeline completes all seven stages including [5/7] FlexiDualGrid shape decode and writes a valid textured GLB (8962s end to end). So the decode stage itself works on this input; only the Vulkan path faults.

Other data points from the same host

• Intel UHD 630 (Mesa, UMA) as the selected device: the run dies earlier, at [3/6], with vk::Device::waitForFences: ErrorDeviceLost and a kernel-level i915 GPU HANG. Different failure, likely unrelated, mentioned only in case the two share a root cause in Vulkan resource handling.
• GTX 1650 4GB via NVK: fails at stage 1 with Requested buffer size exceeds device buffer size limit on a 2.4 GB allocation for BiRefNet. That one is a genuine device limit, not a bug.

Happy to run a debug build with symbols, capture a backtrace, or test a patch — the repro is deterministic here.

[pwilkin]

Aight, could you possibly provide a backtrace? That would help narrow this one down.

[3DJ77]

Backtrace captured — and it reshapes the report: the crash is intermittent, not deterministic (3 of 4 runs crash; details below).

Setup: RelWithDebInfo build of current main (1324775), -DGGML_VULKAN=ON -DGGML_OPENMP=ON, same host/driver/input/seed as the report, server run under gdb.

Faulting thread (run 1, crashed at ~930s, same signature as the original two)

Thread 2 "trellis-server" received signal SIGSEGV, Segmentation fault.
0x00007ffff7d3a564 in ggml_dup_tensor (ctx=0x7fffe40c9c30, src=src@entry=0x0)
    at thirdparty/ggml/src/ggml.c:1859
1859	    return ggml_new_tensor(ctx, src->type, GGML_MAX_DIMS, src->ne);

#0  ggml_dup_tensor (ctx=0x7fffe40c9c30, src=src@entry=0x0) at ggml/src/ggml.c:1859
#1  ggml_norm_impl (ctx=<optimized out>, a=0x0, eps=<optimized out>, inplace=false) at ggml/src/ggml.c:3093
#2  ggml_norm (ctx=<optimized out>, a=0x0, eps=9.99999997e-07) at ggml/src/ggml.c:3107
#3  trellis::sparse_c2s (m=..., prefix="blocks.0.4", feats_in=<len 5695488>, Cin=1024,
        coords=<len 5562>, Cout=512, ext_subdiv=0x0) at src/sparse.cpp:293
#4  trellis::decode_unet (m=..., latent=<len 177984>, out_ch=7, coords=..., guide_subs=0x0, ...)
        at src/shape_decoder.cpp:109
#5  trellis::shape_decode (m=..., latent=<len 177984>, coords0=<len 5562>, resolution=1024)
        at src/shape_decoder.cpp:122
#6  trellis_run (cfg=...) at src/trellis_cli.cpp:252
#7  operator() (req=..., res=...) at src/trellis-server.cpp:125
    ... (httplib dispatch / thread-pool frames below)

Registers agree with the frame: rsi = 0 (the null src), rax = 0x15ba = 5562 = N.

What the null is

sparse.cpp:293 is the norm2 call ggml_norm(c, hraw, 1e-6f). hraw is only ever seeded inside the chunk loop, and every chunk was skipped by if (m1 == m0) continue; — i.e. M = 0: not one octant of the 5562 input voxels survived the subdivision gate. With ext_subdiv == nullptr the gate is subdiv[o + 8*i] > 0.0f on the output of graph 1 (the to_subdiv matmul + bias) computed on the Vulkan backend and read back via tensor_to_f32. So for the crash to happen, all 44,496 logits came back ≤ 0 (NaNs also take that branch, since NaN > 0.0f is false). libggml-base is just where the null lands.

The surprise: an instrumented re-run completed

Re-ran the identical command with TRELLIS_DBG_SUBDIV=1. That run passed the crash point and completed all seven stages, HTTP 200, valid 14.3 MB GLB — first time this pipeline has ever finished on the NVIDIA card here. Every sparse_c2s call had healthy logits, including the call that crashed in run 1:

[c2s] blocks.0.4    N=1377    subdiv>0: 5617/11016     (50.99%)  logit[min=-169.767 max=288.561]
[c2s] blocks.1.16   N=5617    subdiv>0: 22843/44936    (50.83%)  logit[min=-186.734 max=251.529]
[c2s] blocks.2.8    N=22843   subdiv>0: 92935/182744   (50.86%)  logit[min=-143.373 max=105.488]
[c2s] blocks.3.4    N=92935   subdiv>0: 376710/743480  (50.67%)  logit[min=-118.739 max=104.313]
[c2s] blocks.0.4    N=5601    subdiv>0: 34610/44808    (77.24%)  logit[min=-200.937 max=294.243]   <- the call that crashed in run 1
[c2s] blocks.1.16   N=34610   subdiv>0: 98697/276880   (35.65%)  logit[min=-174.114 max=283.220]
[c2s] blocks.2.8    N=98697   subdiv>0: 176392/789576  (22.34%)  logit[min=-83.824 max=30.864]
[c2s] blocks.3.4    N=176392  subdiv>0: 486015/1411136 (34.44%)  logit[min=-43.691 max=28.755]

So across four runs of the same command on this host: crash, crash, crash, pass. Two more observations:

• The Vulkan path is not run-to-run deterministic even with --seed 42: the shape-decode entry call saw N=5562 in the crashing run vs N=5601 in the passing one, so upstream float results already differ between runs (the CPU backend with the same seed gave a different-but-valid mesh too).
• When the pipeline passes, the logits at every gate are nowhere near degenerate (22–77% positive, wide finite ranges). An all-≤0 result at one call is not a near-threshold flip — it looks like the graph-1 output readback occasionally returning wholesale wrong data (zeros or garbage) on NVIDIA proprietary Vulkan, i.e. a sync/race flavor of bug rather than deterministic bad math.

Two separable issues, then:

1. trellis-side: sparse_c2s doesn't handle M == 0 — a degenerate subdiv output turns into a null-tensor segfault deep in ggml instead of an error. A check after the gate loop would make this fail loud.
2. backend-side: whatever intermittently produces the degenerate graph-1 result on this driver. I'm re-running with the instrumentation in place to try to catch a crashing run's actual logit line (subdiv>0: 0/… plus min/max would distinguish all-zeros from NaN) and will follow up when I land one.

Full gdb logs (bt full, all threads, registers) saved locally — happy to attach if useful, or to test a patch; each run is ~15 min here.

[3DJ77]

Caught a crashing run with the instrumentation live. The degenerate case is all-NaN, not all-zeros:

      [stats] HR slat n=212288 mean=-0.0448 std=5.5972 min=-24.335 max=28.489 nan/inf=0
[5/7] FlexiDualGrid shape decode -> mesh @res1024
      [c2s] blocks.0.4                 N=6634  subdiv>0: 0/53072 (0.00%)  logit[min=nan max=nan]
Thread 2 "trellis-server" received signal SIGSEGV, Segmentation fault.

Same backtrace as before (ggml_norm(a=0x0) from sparse.cpp:293). So the sequence on a crashing run is:

• the HR latent handed to shape_decode is fully finite (nan/inf=0 from the stats pass — that's a host-side scan);
• the shape decoder's first block then produces a to_subdiv readback where every one of the 53,072 logits is NaN;
• subdiv[i] > 0.0f is false for NaN → M=0 → null hraw → segfault.

For contrast, in the run that passed, the same call read back subdiv>0: 34610/44808 (77.24%) logit[min=-200.937 max=294.243], and the four sparse-structure-stage calls are healthy in every run, crashing or not (60.95%/49.72%/48.15%/51.18% this run). The poisoning is specific to the HR shape-decode graphs and intermittent.

One more datum: N at the crashing call was 5562, 5601, and 6634 across the three crashing/passing runs — with --seed 42 each time — so the NVIDIA Vulkan path shows substantial run-to-run numeric divergence well before the failure, in the flow stages upstream.

The NaN could enter in the decoder's input-projection graphs or in the subdiv graph itself — feats_in is host-resident at the call but I didn't have a scan on it, and gdb elided the vector contents, so I can't discriminate post-mortem. Happy to re-run with a NaN check on feats_in (or with GGML_VULKAN_CHECK_RESULTS) if that would help narrow which graph mints the NaNs.