Training speed?

[antithing]

Hi, and thank you for this amazing work!

I am training on an RTX 4090 card and I see around 30 mins per frame for training time. Is this the expected time? Are there any ways to speed this up?

Thanks again!

[moqiyinlun]

This is closely dependent on the number of initial point clouds and the input data. We recommend referring to our paper for the specific number of joint Gaussians and skin Gaussians. In addition, complex data scenarios may also lead to slower training. Using our DualGS/HiFi4G dataset, the training time is approximately 10 minutes per frame.

[antithing]

Thank you for getting back to me. I am using

python train.py \ -s <path to HiFi4G dataset> -m <output path> \ --frame_st 0 --frame_ed 100 \ --iterations 30000 --subseq_iters 15000 \ --training_mode 0 \ --parallel_load \ -r 2

And I see between 30 and 40 mins per frame on a 4090. Do I need to make changes to speed up training?

Thanks again!

[Guochch]

It looks like there’s nothing wrong with your command. What kind of data are you testing on? Our method is specifically designed for human-centric data, so if you're using a dataset with complex background scenes—like the N3DV dataset—it could lead to an excessive number of points, which may slow down training. In our tests with the HiFi4G and DualGS datasets, the number of SkinGS points typically doesn't exceed 250K. Also, what operating system are you using? We ran our tests on Linux with a 3090 GPU, and it took about 12 minutes per frame. We did observe obviously slower performance on Windows before, but that was when the environment variable CUDA_LAUNCH_BLOCKING=1 was set.

[antithing]

I am on Windows, and using the HiFi4G dataset. I will check for CUDA_LAUNCH_BLOCKING=1, but I do not think I have it set.

[antithing]

Confirming the var CUDA_LAUNCH_BLOCKING is not set to 1. Is there anything else I can look at to speed up training?

I do see:

[06:35:13] Could not load dynamic_training point cloud, creating from scratch                            __init__.py:108 [07/07 06:35:13]
           Number of points at initialization :  100000    

Is that the expected point number?

Thanks!

[Guochch]

If you're training the first frame, then seeing the message
Could not load dynamic_training point cloud, creating from scratch
is expected — the system initializes from points3d.ply in the dataset or a random point cloud (or point cloud you specify with --ply_path) for the first frame.

Does the ~30 minutes of training time occur for every frame, or is it only slow for the first frame? Also, have you checked whether your training speed is normal when running the original 3DGS implementation?

Currently, our code still uses the rasterizer from a very old version of 3DGS (maybe it led to compatibility or performance issues on some systems?) We're planning to replace it with the taming-3DGS(https://github.com/humansensinglab/taming-3dgs) rasterizer in the coming two weeks. In addition, we’ll also be switching to fused-SSIM to accelerate.

In our tests, turning off SSIM for frames after the first one did not significantly affect the final result quality, so you could try commenting out the SSIM calculation as a quick workaround — it might help reduce training time.

Hope this helps!

[antithing]

Hi @gcccccccccccc12345 sorry to bother you, did you make these additions to the code? Will they be made available soon?

Thank you!

[Guochch]

Hi! The code is now up to date and available. Thank you for checking in! Just a heads-up, you'll need to reinstall the taming 3dgs rasterizer and fused-ssim.

[antithing]

Thank you!
A couple more questions...

Does DualGS work equally well on multi-human data? Eg, 2 people at once?

Would there be any disadvantages in splitting the training over multiple machines? Eg:

Machine A:

python train.py -s C:\\image_white_undistortion -m C:\\output6 --frame_st 0 --frame_ed 10 --iterations 30000 --subseq_iters 15000 --training_mode 0 --parallel_load -r 2

Machine B:

python train.py -s C:\\image_white_undistortion -m C:\\output6 --frame_st 11 --frame_ed 20 --iterations 30000 --subseq_iters 15000 --training_mode 0 --parallel_load -r 2

[Guochch]

For the first question, we have tested DualGS on simple multi-human scenes, and it works reasonably well. However, it's important to avoid topological changes like objects appearing, disappearing or splitting. In areas with severe occlusions, the quality may also degrade.

For the second question, regarding distributed training across multiple machines, one disadvantage is that the training is not continuous between segments. This may lead to noticeable jitter between groups and affect the compression rate of the entire sequence.