perf(media): support multi-million-file MiSTer libraries

[rittwage]

Original failure

A MiSTer library containing roughly 2.5 million files produced a media.db around 1.5 GB. Initial indexing took approximately 4.5 hours, interrupted reindexing could exhaust memory, and opening systems failed with BrowseRouteCounts context-deadline errors. The reporter supplied the database for development and hardware testing.

Indexing cannot be disabled: Core’s indexed media identity is the foundation for browsing, folder routing, launch resolution, metadata, favorites, and history. A filename-only frontend mode would create a separate incompatible browse stack.

Resolution

ZaparooProject/zaparoo-core#1040 hardened extreme-library behavior:

• SQL scan staging instead of loading scanner resume state into RAM
• Bounded interrupted-index recovery
• Smaller commit batches and WAL checkpoint handling
• Cached media counts, systems, tags, and browse routes
• Browse-cache healing and rebuild support
• Fresh-database query-planner statistics
• Low-priority and gameplay-aware background work
• Per-system indexing progress
• Hardware verification against the supplied large database

Current Core #1239 and frontend #377 add further Core-owned browse fast paths and bounded frontend pagination/cache behavior.

[wizzomafizzo]

Indexing is required for browsing, it's the foundation of every feature so I'm afraid there's no way around this.

I'm surprised to hear you have too many file because we have users with over a million files and it does complete (though yes it takes a while). When it appears to be frozen for you, can you please download your log file and upload it here.

Instructions: https://zaparoo.org/support/#collecting-logs

[wizzomafizzo]

Your media.db file in the zaparoo folder on the SD card would also be extremely helpful to fix this but I'm guessing it's quite large. I'll see what we can figure out from the logs

[rittwage]

Thanks, I am going to delete everything and try again and see. If it happens again I will upload the results.

[rittwage]

I deleted and reinstalled it on my MiSTer, then let it completely index everything. It took about 4.5 hours to complete.

It does work now, but it is very sluggish when you select any game system. It will say "Loading games" and then get an error that says "Failed to load error getting system route counts: browse route counts media scan: context deadline exceeded".

I think I just have too much stuff. I understand.

I tried to attach the media.db file for you, but it's almost 1.5GB. Zipped, it's 200M, which github also won't allow me to upload. Probably too much for the poor little ARM CPU in the MiSTer.

I didn't realize the logs were deleted when the system is restarted, so I didn't get the logs of the indexing. It locked up at some point and I had to power cycle.

However, I uploaded the log after launch of the system and selecting Atari 2600 and getting the above error.
https://logs.zaparoo.org/12niu8yH.log

[wizzomafizzo]

@rittwage wow!!! 1.5GB you must have like.. 2.5mil files???

Let me see what I can come up with but yeah this is a tricky one. I'd love to support it though.

Is it possible you can upload the db to dropbox or something? It would be extremely valuable for testing! I can try figure something out if you're not signed up for anything with bigger upload limits

[rittwage]

https://drive.google.com/file/d/1QXbkCkQHeA-Rsx5_g74GY4rGHyevuFRN/view?usp=drive_link

Here you go!

[wizzomafizzo]

@rittwage please give me permission: callan@wizzo.au

thank you!!

[rittwage]

Done. Sorry about that.

…

On Tue, Jun 30, 2026 at 9:41 PM Callan Barrett ***@***.***> wrote: *wizzomafizzo* left a comment (ZaparooProject/zaparoo-frontend#292) <#292 (comment)> @rittwage <https://github.com/rittwage> please give me permission: ***@***.*** thank you!! — Reply to this email directly, view it on GitHub <#292?email_source=notifications&email_token=A5GKMSHPVXUL32LL2QZOLGL5CRT4TA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBUHE2DQNJRGY2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4849485164>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5GKMSH4XJ7AFQJPCGY5U435CRT4TAVCNFSNUABGKJSXA33TNF2G64TZHMYTEMJWHAZDSNZSG45US43TOVSTWNBXGU4TAMBYGY3THILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/A5GKMSDIYOW6ESNUEQWAIXL5CRT4TA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBUHE2DQNJRGY2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/A5GKMSDG55SPSL2HAJILUXL5CRT4TA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBUHE2DQNJRGY2KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

-- - Peter Rittwage

[wizzomafizzo]

zaparoo-mister_arm-71fa917e-dev.zip

@rittwage give this a shot! replace your current zaparoo.sh in Scripts folder, power cycle

you'll still have to wait a bit while it reindexes and settles, but it should eventually work itself out. i was able to use your db to browse on my mister and it was running very well :)

EDIT: Actually, nevermind! It was working because I was skipping the reindex step. It will run out of memory for you at the moment. I will post a new binary when that's fixed up

[rittwage]

It does work much much better. I have not tried to reindex again, and won't for now.

Edit: Oh well, it eventually starts to reindex on it's own...

[wizzomafizzo]

Closing as resolved by Core #1040’s released large-database indexing and browse hardening. Indexing remains required; current Core #1239/frontend #377 further improve browse performance.