Loading all map data causes high memory usage and easily lead to crashes

[sixshine]

Due to loading all map information at once, the memory usage of the page at https://mathartbang.com/deca/hp/map.html
has reached 2500 MB.
Can store the map data in a compressed form and only retrieve the corresponding map information after loading the saved data ?

[sixshine]

I tested changing the original code:

areas[reserve_id] = areas_reserve;
const reserve_area = areas[reserve_id];

to store and retrieve it like this:

areas[reserve_id] = JSON.stringify(areas_reserve);
const reserve_area = JSON.parse(areas[reserve_id]);

The loading time increased by 1–2 seconds, but memory usage can be reduced by more than half.

[kk49]

Thanks for the suggestion! I saw that you had also suggested using compression (and then simplified it to this) The files are huge so I will also sort out downloading a compressed version of the files to save bandwidth and download time as well

[sixshine]

Thank you very much for your reply.
I changed the solution I suggested because, in my testing, I found that compressing such large objects on the client side is extremely time-consuming, making loading particularly slow and also more prone to crashes.
The server-side compression approach you mentioned is indeed much better; temporarily constructing the area information when loading a save each time maybe take less than one second.
I started investigating the cause because I kept encountering crashes on my own Windows handheld. If you could release a simple workaround version first, that would be ideal.
Additionally, would it be better to automatically jump to the corresponding map page after loading a new save?

[kk49]

@sixshine I pushed the simple change, it takes a few minute to filter through the github servers.

[kk49]

As for jumping to the corresponding map, let me take a look, I didn't realize people where loading just one population file, but jumping to the map for that case makes sense

[sixshine]

After the update, the memory usage has indeed decreased.
I’ve always been loading only one file, and since it worked, I assumed that was the correct way to use it.
I admit I didn’t really read the instructions carefully. I didn’t know it was possible to load everything at once.

[kk49]

It makes sense too that someone one would do one reserve at a time, I don't play COTW so I'm not sure how much people bounce between reserves.

I loaded an update to use compressed reserve json files, IDK how big of an improvement it will actually be since, it seems like at least firefox is using a compressed transport protocol. So the data transfer seems to be the same either way. I will leave it like this for now. It could eventually set the stage for doing more dynamic loading based on which reserve is being viewed.

[sixshine]

Being able to load all the maps at once is certainly more convenient, so I don’t think it’s necessary to further compress the memory used by the areas data, as that would make it too slow in this scenario.

I noticed that the latest update doesn’t actually reduce bandwidth and may slightly increase loading time. The original JSON files are already transmitted using gzip, which the browser automatically decompresses, so there’s no need to manually compress them into zip files. I would suggest reverting to using the original JSON files and, if needed, minifying them by removing whitespace—this could cut the file size by about two-thirds and reduce gzip transfer by roughly 10%.

[kk49]

Reverted to normal json and removed white spaces. Thanks for all the work/suggestions!

[kk49]

I separated out the single reserve file drop -> display reserve issue here #81

[sixshine]

Sorry, this may require another update.

I did some further testing today and found that using JSON.stringify as a workaround significantly slows down the processing time when loading the entire Save directory, increasing it to around 6–7 seconds.

I then tried a different approach: moving the area analysis step (which generates extremely large objects) into the Save file parsing phase, and only analyzing area IDs that actually exist in the Save files. With this change, the initial loading is much faster compared to the workaround version, memory usage is reduced by several hundred MB, and loading the entire Save directory now takes only 2–3 seconds.

I’ve attached the modified and tested HTML file here, in case it’s helpful for you.

map_better.html

[sixshine]

By the way, the file corresponding to Silver Ridge Peaks is animal_population_9, and it is processed last when loading the entire Save folder. You can load the Save folder on this page and estimate the overall processing time by watching how the page refreshes. I didn’t realize this earlier, so I hadn’t noticed how slow the workaround actually was.
Lastly, I really appreciate you making this tool—it’s been extremely helpful to me.