Support iterable / webdataset

[richardrl]

What would be required to make this work for very large video sets? Would it be some integration with web dataset?

[RaivoKoot]

Depends how large. in general your dataset can be as big as you want. I've used the code with a video dataset where the videos totalled 1.5 terrabyte. However, those 1.5 terrabyte did fit on my local disk. Once your dataset is too big to fit on a single machine or if you want to do multi-node training you would have to make some changes to the code to support this somehow.

[ZuowenWang0000]

Disk space indeed can be a factor, especially when it comes to SSD size.
Plus, loading inflated .jpg frames puts higher IO demands than loading .mp4 files and decoding on the fly. Also decoding on the fly offers more flexibility for researching hyperparameters.
Wish there were some study on the IO time vs. decoding time..

[richardrl]

@ZuowenWang0000 In my experience, the best combination is mp4 plus extreme CPU parallelization for decoding. I haven't seen off the shelf dataloaders implementing this well though, I had to do it myself for my own dataset.