Can not search in older releases

[flip111]

Reproduce

• Go to https://www.stackage.org/
• Click on LTS 14.27 for ghc-8.6.5
• In search box type exceptions and click Search

Note the following:

• URL is set to: https://www.stackage.org/lts-14.27/hoogle?q=exceptions
• On the page there is the text Within [LTS Haskell 22.11 (ghc-9.6.4)](https://www.stackage.org/lts-22.11)

Continue

• Click on package exceptions

Note the following:
Getting package information for exceptions in LTS-22.11

[alaendle]

Yeah, a little bit misleading - but I think this is a duplicate of #305.

[flip111]

You are right it's a duplicate

[chreekat]

Let's call this "not a duplicate" because searching in older snapshots would be rad. (#305 was about showing the wrong information in the search UI as opposed to not having search in older snapshots.) I'll move the issue to stackage-server, though, since that's where it's implemented.

[chreekat]

fwiw we have a bunch of disk space we could use for caching hoogle databases now. But we'd still need some sort of LRU cache rather than keeping all of them local.

[flip111]

@chreekat that sounds interesting! Where do you have the disk space, on a server? How big of a storage is needed? Does the LRU cache need to be designed or can we just plug in some kind of proxy caching app?

[chreekat]

@flip111 yeah, Stackage runs on a server provided by the Haskell Foundation. Right now it doesn't have much in the way of constraints, but I still don't want unbounded growth. We could be pretty liberal with limits to get started.

All the Hoogle databases for all Stackage snapshots are stored in an S3-compatible bucket. But the app is designed to fetch those databases to the local filesystem and then use them when people do searches. That's where the problem used to come in.

In the old infrastructure, Stackage was run in a Kubernetes cluster on small containers. In those times, the app was modified to only search in the most recent LTS and Nightly snapshots. Thus it only ever would fetch the Hoogle databases for those snapshots. (Ironically, it would still run out of space, because when the snapshots advance, the new Hoogle dbs are downloaded, but the old ones were left behind. But that doesn't matter in k8s, of course, because the pods would just self-immolate and get born anew with fresh disks.)

I have not done any research into LRU stuff. To be clear, my naive plan is to remove the oldest Hoogle DB after some limit is hit. This could even almost be done with logrotate or something. I have simply not looked into it yet.

[chreekat]

One could even imagine other solutions where the database is never even brought onto the local disk. I imagine it might be way too slow to always fetch from the bucket, of course. But these servers also have a lot of memory. The LRU could even exist in ram?

Notwithstanding these musings, I suspect it'll still be better to use the local disk as the cache.

[flip111]

I looked through the code and build the hoogle database locally. What i saw was this:

stack hoogle --rebuild
Minimum version is hoogle-5.0. Found acceptable hoogle-5.0.18.4 in your index, requiring its installation.

this uses lts-23.17 (what was configured in my global project)

I'm assuming (but this might be incorrect) that the hoogle database being built on everything available in the stackage release.

la -B .stack/global-project/.stack-work/hoogle/x86_64-linux-tinfo6/592d9a894609f023ea179dffa09efa4564e9ac2b2c5f10da24af78b72616d817/9.8.4/database.hoo

This gives 9,776,531 bytes (9.8 MB)

The database is not read as a file, de-serialized and then converted into data structures. Instead it's directly mapped into the memory and the search is build to read directly from the memory stream with de-serializing or further moving data around https://github.com/ndmitchell/hoogle/blob/e83646ee1b62eb724def6c838a34017b60fc519a/src/General/Store.hs#L170

When i look at all of the versions on the front-page of stackage https://www.stackage.org/ there are 33 listed versions (excluding latest nightly). Of course there are more versions into existence, but those 33 are a good beginning and probably the most important ones. So to have that all mapped into memory is 322,625,523 bytes (323MB). Maybe the server capacity is large enough to keep them loaded in memory permanently. But anyway when using a least recently used cache in practice it would probably be those same 33 versions anyway, so i don't know if it makes sense to keep any version of the hoogle database pinned permanently.

For the least recently used cache i saw these two packages:

• https://github.com/chowells79/lrucache
• https://github.com/hverr/haskell-cache

they are pretty old though, and probably the LRU mechanism itself doesn't have to be very optimized, since it's more about loading and unloading the files to memory anyway.

there is also the option of Amazon ElastiCache with Redis, i imagine that way there is more RAM available, or it can be shared between different pods. However i see some tradeoffs as well

• increased complexity in infrastructure setup
• Application becomes tied to specific AWS stuff. For example through this library https://hackage.haskell.org/package/amazonka-elasticache
• Potential of cost-lowering. If there are more instances of the application then a shared ElastiCache can save costs. Potential cost-increase due to paying for ElastiCache. So needs some study (extra effort) to find out if that is worth it.
• Though potentially easier scaling through the AWS dashboards rather than changing the configuration file of an application

Then there could also be a second mechanism where the file is copied from the S3 to the local file storage since S3 can be slow. I guess if the space is there anyway and it's unused then this can be used. But of course for the most pleasant experience for users you want to have things loaded into a cache anyway.

Directly loading from S3 to memory skipping local files is possible but also requires some modification of the stackage-server code as it requires a local path currently.

I think it should work in such a way that if you just start the stackage-server app normally that it just memory maps that one single database as it does now. But it should take some optional configuration file that lets you choose the have an LRU cache of either up to X versions of a database or up to Y megabytes of RAM. This might require some modifications in the hoogle library as well, or we have to copy over code into stackage-server that now lives in hoogle to modify it. Also configuration to have local files LRU with a max versions / max file-space would be cool too.

It's hard to develop these functionalities with an AWS sandbox though since some functionality is specifically designed to work with AWS such as S3 buckets.

[chreekat]

Sadly the actual size of the full-snapshot hoogle database is more like 291MB:

# find -type f | xargs ls -lh
-rw-rw-rw- 1 stackage-server stackage-server 220M Mar 25 03:58 ./lts-23.16/5.0.18.4.hoo
-rw-rw-rw- 1 stackage-server stackage-server 219M Mar 29 20:34 ./lts-23.17/5.0.18.4.hoo
-rw-rw-rw- 1 stackage-server stackage-server 219M Apr  6 17:22 ./lts-23.18/5.0.18.4.hoo
<snip>

(Don't ask me what's up with the permissions.. I should figure out why they are so lax)

We can definitely fit a couple of those in memory, but I think the simplest thing to do would still be to just re-enable the old behavior, which was to read from the local disk, fetching it first if necessary. The question just becomes one of dropping the 500th oldest (and all older) databases, or something.

(Well, honestly, how many snapshots are there, anyway...)

[flip111]

@chreekat what are the specs (or name so we can lookup the specs) of the AWS EC2 instances and how many of them are there?

[chreekat]

@flip111 it's not on AWS. :) The new box is a beefy piece of hardware living in a colo. There's just one of them reserved for Stackage, but it's way more than we currently need.

And the "S3 compatible" object storage I mentioned earlier is actually Cloudflare's R2 service.

[flip111]

@chreekat what's the best plan of attack here? a pull request to stackage-server with caching mechanisms and whatnot?