Skip to content

[Windows - AMD - Vulkan] Trellis Studio Portable - generate failed: birefnet alloc #35

Description

@Karhunkaatolaulu
  • Windows 11
  • AMD Radeon RX 9060 XT 16GB
  • Trellis-studio-windows-x64-portable (v0.5.4)
  • Trellis-vulkan-windows-x64 (v0.5.4)
  • Model: Q8-quant

The Trellis Studio program starts, but does not work. A few seconds after generation starts, an error appears and the process stops. I am providing the contents of the log file. On this same PC in the Manjaro Linux KDE operating system, the Trellis Studio Portable program works without errors. But there are minor problems with text readability in drop-down lists when the dark interface theme is active (screenshot).

[08:08:49] [studio] Trellis Studio server log — 2026-08-13 08:08:49 UTC
[08:08:49] [studio] config: bin=D:\trellis-studio-windows-x64-portable\runtime\trellis-server.exe models=D:\trellis-studio-windows-x64-portable\models gpu=0 backend=unknown host=127.0.0.1 port=8080
[08:08:49] [studio] launched D:\trellis-studio-windows-x64-portable\runtime\trellis-server.exe on 127.0.0.1:8080
[08:08:49] [trellis-server] models=D:\trellis-studio-windows-x64-portable\models gpu=0 listening on http://127.0.0.1:8080
[08:09:22] [trellis-server] generate: 128215-byte image, seed 42, res 512, bg auto, uv xatlas
[08:09:22] [trellis] generating model with seed 42
[08:09:22] [1/6] preprocess C:\Users\Karhunkaatolaulu\AppData\Local\Temp\trellis-req-0.png (BiRefNet bg removal, 512)
[08:09:23] ggml_vulkan: Found 1 Vulkan devices:
[08:09:23] ggml_vulkan: 0 = AMD Radeon RX 9060 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
[08:09:23] [trellis] using Vulkan0 (16304 MB)
[08:09:28] [deform_vk] using AMD Radeon RX 9060 XT
[08:09:35] ggml_vulkan: Device memory allocation of size 2415919104 failed.
[08:09:35] ggml_vulkan: Requested buffer size exceeds device buffer size limit: ErrorOutOfDeviceMemory
[08:09:35] ggml_gallocr_reserve_n_impl: failed to allocate Vulkan0 buffer of size 2684354560
[08:09:35] [trellis-server] generate failed: birefnet alloc

Image

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions