Create large hub dataset

[TSLUAN]

Hi,

1. Thanks for the great tool for managing datasets.

2. I have got an issue when doing some test to create a local hub dataset from a local large dataset. The process stopped and said Killed.

3. I tried to upload the dataset in batches, i.e. within with ds: only appending a certain number of data, so that it fits in memory.

4. Then call this with ds: iteratively until the whole dataset is uploaded.

5. However, after the first batch (1800 samples with multiple tensors) is written, the program was killed during the second batch.

6. I noticed this might be an indication of running out of resource. But don't know where the issue could be.

7. If I can have any suggestion from you, it would be very appreciated.

Many thanks
TS

[AbhinavTuli]

Hey @TSLUAN! Which version of Hub are you using? We pushed a fix for this recently in v2.2.2, if you're on an earlier version, could you update and try again?

[TSLUAN]

I was using v2.2.0. And after updating it works! Thanks a lot.

Can I ask what the issue was btw?

[AbhinavTuli]

Hey @TSLUAN, missed this somehow.
The issue was that the size of certain hub objects wasn't getting updated properly in the cache, leading to the cache filling up with more items than the allocated memory limit.