[Model Support] Cosmos3 (4B, 16B, 64B pretrained models)

[MeiYi-dev]

Needs JSON style prompts like Ideogram 4.0
OpenMDW1.1 license (similar to MIT license)

https://github.com/NVIDIA/cosmos-framework supports training these models.
NVIDIA/cosmos-framework#11 This PR allows for training only the DiT seperately.

People are severely underestimating these models. These models are pretrained models without any post training done (similar to Krea2 Raw) and is a great base for post training research.

Edit: Removed the lowres videos. Added some high resolution versions #2849 (comment)

[MeiYi-dev]

Let me know if you want to see more prompts, I can generate more demos, just give me the prompts. People are severely underestimating these models. With CREPA (with Qwen-VL features) and TREAD it will be very fast to post train it on a large multi-shot dataset.

[bghira]

yikes, am i missing something? those look kinda terrible, like LTX Video v0.9.x

[MeiYi-dev]

@bghira Found a native ComfyUI implementation https://github.com/RyukoMatoiFan/ComfyUI-Cosmos3

Below are some 720x480 2s videos using the above ComfyUI implementation with the 8B model. Going to add some more soon.

Cosmos3-native_00011_.mp4

Cosmos3-native_00008_.mp4

Cosmos3-native_00007_.mp4

Cosmos3-native_00004_.mp4

Cosmos3-native_00020_.mp4

Cosmos3-native_00014_.mp4

Cosmos3-native_00012_.mp4

Cosmos3-native_00021_.mp4

Cosmos3-native_00024_.mp4

Cosmos3-native_00031_.mp4

Cosmos3-native_00030_.mp4

Cosmos3-native_00026_.mp4

Cosmos3-native_00037_.mp4

Cosmos3-native_00035_.mp4

[bghira]

@copilot can you implement this under simpletuner/helpers/models/cosmos3 and reuse cosmos2 components extended with the comfyui implementation details linked in an earlier comment here?