Cache configured version-pinned paths of generic upstreams

[gilesw]

The generic handler treats only GitHub release asset paths ({owner}/{repo}/releases/download/{tag}/{asset}) as immutable. Any other download through a generic upstream, such as Maven from archive.apache.org (mise's aqua backend fetches apache/maven from there), goes through the metadata route: streamed uncached by default, or with cache_metadata kept under _metadata/ and revalidated, and never listed as a package.

This adds upstream.generic_artifacts, mapping a generic upstream to regular expressions for its version-pinned paths:

upstream:
  generic:
    apache: "https://archive.apache.org"
  generic_artifacts:
    apache:
      - 'dist/(?P<name>maven)/maven-3/(?P<version>[^/]+)/binaries/(?P<file>[^/]+)'

• A pattern must match the whole path after /generic/{name}/; it is wrapped in ^(?:...)$.
• It needs version and file named groups, each a single non-empty path segment. An optional name group names the package; it defaults to the upstream name.
• Matching downloads take the same route as GitHub release assets: artifact cache, served while the upstream is down, listed under the generic ecosystem. Patterns are checked before the built-in GitHub pattern.
• Config validation rejects patterns that don't compile, lack the groups, or name an upstream missing from upstream.generic.
• upstream.generic is unchanged, so existing configs behave the same.

Tested with go test ./... and golangci-lint run ./... (v2.13.1, 0 issues), and by running the proxy against archive.apache.org: the first fetch of apache-maven-3.9.9-bin.tar.gz took 18s, the second 11ms from cache, and /api/search?ecosystem=generic lists maven.

[gilesw]

I don't profess to understand this well but what happened was a download of maven from archive.apache.org ended up in the metadata area. i took the metadata file and renamed it to maven.tar.gz and extracted apache-maven-3.9.9.

[andrew]

Two failures in the configured artifact path need fixing. Both were reproduced through the HTTP handler with a real upstream test server.

[andrew]

Distinct matching paths can have the same captured name, version and file, so they share cached bytes. With releases/(?P<name>tool)/(?P<version>[^/]+)/(linux|darwin)/(?P<file>[^/]+), requesting releases/tool/1.0.0/linux/tool.tar.gz followed by releases/tool/1.0.0/darwin/tool.tar.gz returned the Linux bytes for both requests. Include the full upstream path in the cache identity and add a handler-level regression test for platform-specific paths with identical filenames.

[andrew]

Paths newly matched by generic_artifacts lose query parameters because downloadURL contains only the upstream base and path. The previous metadata route forwards r.URL.RawQuery. In a handler-level reproduction, a download requiring ?token=example returned HTTP 200 without a pattern and HTTP 502 after adding one. Preserve the query when fetching configured artifacts and account for it in cache identity, with regression coverage.

[andrew]

The earlier path-collision and query-forwarding findings are fixed. The new cache identity introduces an upgrade regression for existing GitHub release downloads.

[andrew]

This also changes the cache filename for existing built-in GitHub release paths, even when no generic_artifacts patterns are configured. I reproduced the upgrade through both HTTP handlers using the same database and storage: prime a release with the previous handler, stop the upstream, and request it again. The previous handler returns the cached bytes with HTTP 200; this handler returns HTTP 502 because it only checks the new hashed key. Preserve access to legacy entries when their recorded upstream URL matches the requested download, while keeping distinct paths and queries isolated, and add an upgrade regression test.

[gilesw]

Fixed in 140b4b3. handleArtifact now checks the old {repository}/{asset} key before the hashed one. It serves that entry only when its recorded upstream_url equals the current download URL. The old code never forwarded queries, so a request with a query can't match, and a different path or a repointed upstream has a different URL. New entries are still written only under the hashed key.

TestGenericHandler_ServesReleaseAssetCachedUnderUnhashedKey fills the cache the way the old handler did and takes the upstream down. It then checks that the same path returns 200 with the cached bytes, and that the path with a query or a repointed upstream returns 502. Without the fix, the test fails with the 502 you reproduced.

[gilesw]

I've kept the fallback as you asked, but I'm not sure it's worth having long term. The generic handler has only been out since v0.8.0 (#302, early September). Without the fallback, the cost is that each cached release asset is downloaded once more after the upgrade, and that only fails if the upstream is down at that moment. The fallback won't remove itself either: serving an old entry counts as a hit, so those entries never get evicted and the code stays until we delete it.

If you'd rather keep one key scheme, I'm happy to drop it and add a changelog note that existing generic artifacts are re-fetched once. Otherwise we could plan to remove it in a later release.