Skip to content

Audio Model Support.md

Latest commit

 

History

History
80 lines (69 loc) · 5.11 KB

Audio Model Support.md

File metadata and controls

80 lines (69 loc) · 5.11 KB
Raw

Audio Model Type Support In SwarmUI

Model Year Author Scale Type Quality/Status
Ace Step 1.5 2026 StepFun 2B DiT Music Modern, Fast, Decent Quality
MiniMax Music 3 2026 Hailuo & MiniMax 2B DiT Music Modern, fast-ish, Decent Quality
YuE2 2026 Multimodal Art Projection 3B DiT Music Modern, super fast, good

Support for image models and technical formats is documented in the Model Support doc, as well as explanation of the table columns above.

Audio models vary in intention and purpose. Some examples include:

  • Sound effect models are used to create general purpose sound effects, useful to adjust videos with (see eg mmaudio)
  • Music models create full songs. Often these support instruments, styles, and lyrics (see eg ACE-Step)
  • Speech models create speech. Often these take in voice references and text prompts for what to say (see eg VibeVoice)

Ace Step 1.5

  • Ace Step 1.5 is the first audio model to be natively supported in SwarmUI!
  • Download it from Comfy-Org/ace_step_1.5
    • save in diffusion_models
  • It uses twin small Qwen text encoders, they will be autodownloaded.
  • The audio VAE will be autodownloaded.
  • Parameters:
    • Prompt: Write the lyrics of the song, you can add section hint labels like [Intro] or [Chorus - Pop Explosion]
    • Text2Audio Parameter Group will appear on the sidebar after the model is selected:
      • Audio Duration: defaults to 120 seconds (2 minutes), but designed to short any duration from a few seconds to up to 10 minutes
      • Audio Style: write a short description of the music style.
    • CFG Scale: For the turbo, use 1
    • Steps: For the turbo, use 8
    • Sigma Shift: Defaults to 3

MiniMax Music 3

  • MiniMax Music 3 is supported in SwarmUI!
  • Download it from Comfy-Org/MiniMax-Music-3
    • save in diffusion_models
  • It uses a dedicated 8B LLM for building the song and a 0.6B mini-LLM for audio property encoding. This will be auto-downloaded.
  • The audio VAE will be autodownloaded.
  • Parameters:
    • Prompt: Write the lyrics of the song
    • Text2Audio Parameter Group will appear on the sidebar after the model is selected:
      • Audio Duration: defaults to 120 seconds (2 minutes), but designed to support any duration from a few seconds to up to 5 minutes
        • Will be used as a max duration, the LLM song builder may choose to make the song shorter
      • Audio Style: write a short description of the music style.
    • CFG Scale: 1.7 is the official reference
    • Steps: Normal steps range eg 20, 30 is the official reference

YuE2

  • YuE2 is supported in SwarmUI!
  • Download it from Comfy-Org/YuE2
    • save in Models/Stable-Diffusion
  • The file contains its own Audio VAE and text encoder
  • It works internally by having one model generate a 'full plan' (melody, chords, etc) from your inputs, then a separate model makes music from the plan
  • They also have a separate tool to transcribe songs into their own format called 'SheetSage2', not implemented in Swarm currently
    • Use on Comfy tab with a short workflow of: Load Audio Encoder, Load Audio, SheetSage2 Audio to ABC, Preview Any
  • Parameters:
    • Prompt: Write the lyrics of the song
      • Note: Think of the input here like an old/dumb TTS, it won't know how to pronounce uncommon words so you'll have to guide it.
      • For example, don't write "SwarmUI", write "Swarm-you-eye".
      • You can do blocks like:
      [verse]
some song lyrics here!
I am singing a song!

[chorus]
Wow what a song!
Such a singable song!
      • Note that it may slightly hallucinate some additional lyrics, eg adding a 'yeah' or duplicating a few words
    • Text2Audio Style: Write a short description of the music style.
      • This is a freeform LLM input, though how it interprets it will be somewhat arbitrary. Major genre names are understood, but complex instructions may confuse it.
    • Audio Duration: defaults to 300 seconds (5 minutes), can be up to 900 (15 minutes)
      • Will be used as a max duration, the LLM song builder may choose to make the song shorter
    • CFG Scale: use 1
    • Steps: Normal steps range eg 20