memory: bound dense 1024 decode and host postprocess allocations

[danthemighty316-jpg]

Summary

This patch addresses several memory-scaling failures encountered during dense 1024 shape generation on a 16 GB GPU.

The goal is to bound peak allocations in sparse decode, DiT/shape decoding, subdivision, and host-side mesh postprocessing without reducing generation resolution, imposing a token cap, or disabling postprocessing stages.

Changes

C2S sparse convolution

The existing chunked C2S path could still retain/upload the full sparse input feature table for each chunk.

This patch makes the working set more local by:

• collecting only feature columns referenced by the current chunk's neighbour indices
• building a compact host feature table
• remapping neighbour indices to the compact local table
• running each chunk as a short-lived graph and stitching the results on the host

DiT

Large FlashAttention padding-mask/query handling could produce a very large allocation at high token counts.

The affected work is chunked so the full allocation is no longer required at once.

Shape decoder / ConvNeXt

Large sparse ConvNeXt stages are streamed in chunks rather than processing the entire stage as one large allocation.

C2S subdivision

to_subdiv contained another full [Cin, N] F32 pre-pass which became the next memory wall after the earlier C2S changes.

This path is now chunked and its 8-channel output is stitched on the host.

Host mesh postprocessing

A dense decode can produce a very large intermediate mesh, exposing host-side allocation pressure after GPU decode succeeds.

Changes include:

• fill_small_holes no longer keeps multiple giant full-mesh edge maps alive simultaneously
• the small boundary set is extracted before releasing the large edge-use map
• dead intermediate feature buffers are released earlier
• BVH allocation is sized from the actual balanced-tree node requirement rather than reserving approximately 2 * F
• remesh candidate-bitset worker count is capped to keep aggregate CPU memory bounded

Validation

The patch was originally developed and runtime-tested from commit 16f3109.

A former DiT allocation failure case reached:

• 37,312 HR tokens
• all 12 HR sampling steps completed successfully

A former host-side bad allocation reproducer reached approximately:

• 30.9 million vertices
• 143.9 million faces before later mesh processing

It then completed:

• hole filling
• BVH construction
• remeshing
• Vulkan decimation
• xatlas UV generation
• texture generation
• final textured GLB output

That generation completed in approximately 3718 seconds.

A subsequent continuous 1024/xatlas soak completed 18 consecutive server generations with:

• 18/18 completed generations
• zero generation failures
• maximum 33,235 HR tokens during the soak
• no observed C2S allocation failure
• no bad allocation

Test hardware

• Windows
• AMD Radeon RX 9070 XT
• Vulkan backend
• 16,304 MB VRAM reported by ggml
• 1024 cascade generation
• xatlas UV generation

Current upstream

The tested patch was preserved separately and then cherry-picked onto current main at 2516c48.

The cherry-pick applied without conflicts and the resulting branch builds successfully with the Vulkan backend.

The full runtime soak above was performed on the original 16f3109 base; the 2516c48 integration has currently been compile-tested rather than having the full soak repeated.

Caveat

I previously encountered an AMD display-driver TDR/BSOD during separate heavy testing. No BSOD occurred during the captured 18-generation soak, but I am treating the driver-level issue as separate and am not claiming this patch resolves it.

I'm happy to split this into smaller changes if that would make review easier.