are datasets automatically migrated when updating python hub package?

[hoshimura]

So I have created some datasets with some version of hub and then I update the python hub package.
Will the new version automatically restructure(or migrate, call it what you like) the datasets I have, on access so that the internal structure is compliant with the possibly new ways of the newer package?

[davidbuniat]

Hi @hoshimura, thanks for the question! We have committed to backward compatibility. This means that datasets created with previous version should be fully functional in new hub versions. We do run tests on every PR to make sure it is the case https://github.com/activeloopai/buH.

[hoshimura]

Excellent, backwards compatibility is good! But a stale storage will eventually lead to a point were breaking changes might be necessary. So a strategy where data is updated on access or migrations similar to django would ensure that data never grows old.

[davidbuniat]

Very fair point @hoshimura, for breaking changes we are expecting to change a major release version and provide tooling around migration of the existing datasets.

Would love to learn your thoughts on more advanced strategies that would be convenient for your use case?

[hoshimura]

I am just curious how these matters are meant to be handled. I have no specific use case in mind more than the ordinary longtime storage of datasets so I marked your first answer.