Repository navigation
Conversation
Every fetch of an artifact wrote one path, so fetches with different URLs
or digests, which do not share a coalescing key, could overwrite each
other's bytes, serve the wrong ones, or delete them on a digest mismatch
while the other was about to open them.
storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename},
with a random id per fetch. Most ecosystems learn the digest only after
the fetch, and storage has no rename, so the id is the one rule that fits
all of them. Existing records keep their paths and stay readable. On a
file:// bucket, Delete now removes an emptied fetch directory, which
fileblob leaves behind, and no other directory, since another fetch may
be creating its own inside it.
A path a record stops pointing at is queued in pending_deletes rather
than deleted, since a request that read the record may still open it. A
loop deletes queued paths after max(1h, direct_serve_ttl), whether or not
max_size is set. UpsertArtifact and clears apply only if the record still
holds the path the caller read, so racing commits each queue the path
they replaced and a clear never orphans a newer commit. Eviction still
deletes inline, and counts space as freed only when the record still
pointed at what it deleted.
Eviction deleted an object before checking its record still pointed at it. If a refetch had moved the record, that object was queued and a request that read the record earlier could still be opening it. Eviction now clears first, skips the delete if the record moved, and queues the object when the delete fails after a clear. Reclaim queues a failed delete again, which moves it behind the rest, so objects the backend keeps refusing cannot fill every batch. Also drop the clearArtifactCache passthrough and ClearCachedArtifact's unused context, note that only tests call ArtifactPath, and document that max_size does not count objects waiting to be reclaimed.
A truncated relay panics with http.ErrAbortHandler, which skipped the request log line, the request metrics and the access log entry.
Give each fetch its own storage path
…ay-326 fix: centralize pass-through response relaying
Every cached download wrote its hit in its own UPDATE. SQLite runs on one connection, so each download was a write transaction the rest queued behind, holding a warm npm cache near 1,000 requests a second with the CPU two thirds idle. Hits are now counted in memory and written in one transaction every database.hit_flush_interval, "1s" by default, "0" for a write per hit. A failed write keeps the hits for the next flush, and Close writes what is pending. The same test then served about 2,200 requests a second. Closes git-pkgs#322.
fix pnpm audit
- Map /apk to alpine, matching what the handler writes to the database - Report /helm, /homebrew, /generic and /swift under their own names - Test every one of the 23 mounts, so the next route added cannot quietly fall through to "other" Part 1 of 6 splitting git-pkgs#381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ndons TestRelayRoutes fails intermittently on CI with "write: connection reset by peer" at the upstream handler's final Flush, on /hex/packages/demo and /gem/info/demo. Those two routes are the only ones in the table that fan out a second upstream request: with cooldown enabled, the hex and gem handlers fetch version timestamps concurrently with the artifact. The test server answers that sidecar request with the same 64 KiB chunked 502, but the handler returns as soon as it sees a non-200 and closes the body unread, so the proxy drops the connection while the test server is still writing to it. Whether the reset lands before or after the Flush is a scheduling race, which is why it only surfaces on loaded runners. The flush error says nothing about the response under test, which is asserted on the downstream side, so stop reporting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test(handler): fix TestRelayRoutes flake on /hex/packages and /gem/info
- Add GetEcosystemStats, aggregating packages, versions, artifacts, cache size, downloads and downloaded bytes per ecosystem - Publish six proxy_ecosystem_* gauges from it on the existing one-minute cache-stats tick - Set and selectively delete rather than Reset, so no scrape lands on a half-populated vector - Report artifacts with no package row under "unattributed" rather than dropping them, so the figures still add up Part 2 of 6 splitting git-pkgs#381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Ring of download size by ecosystem, cache figures, per-ecosystem table, vulnerability overview and a Runtime card - Add metrics.Gather so the page can render counters that were never in the database - Add proxy_response_bytes_total and response-writer byte counting - Serve a retained snapshot behind a staleness banner when the aggregation fails, rather than rendering old figures as current - Extract the security overview into a shared component Part 3 of 6 splitting git-pkgs#381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Add downloaded_bytes, downloads and an ecosystems array, served from the snapshot the gauges and the page already share - Add stats_unavailable so a failed aggregation is distinguishable from an idle proxy - Regenerate the OpenAPI spec Part 4 of 6 splitting git-pkgs#381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- 33 panels, no hardcoded data source UID - Three ecosystem variables, because the label means three different things across /metrics: the package record, the request path, and the handler's own name Part 5 of 6 splitting git-pkgs#381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Track per-caller request and byte counts in a bounded table - Export proxy_client_requests_total and proxy_client_response_bytes_total over a closed label set; addresses are never labels - Gate the sources card behind ui_request_sources, off by default, since /ui carries no authentication of its own - Move access_log.trust_forwarded_for to a top-level key - Keep remote as the TCP peer and add remote_ip alongside it Part 6 of 6 splitting git-pkgs#381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update ecosystems-go to v0.5.0 * Update enrichment to v0.7.2
With proxies sharing a database, a batch can flush after a newer hit is already written, and overwriting last_accessed_at and updated_at with its older time could reorder LRU eviction. Both now keep the later of the stored and incoming times. The count still adds up.
Bumps alpine from 3.24.1 to 3.24.2. --- updated-dependencies: - dependency-name: alpine dependency-version: 3.24.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…it-pkgs#387) * Add storage.passthrough to serve artifacts without storing them With storage.passthrough enabled the proxy streams every artifact from upstream to the client and never writes it to storage or the cache database. Metadata filtering, cooldown and the denylist work as before, so the proxy can sit behind another cache (e.g. an Artifactory remote) purely as a policy layer without holding a second copy of every package. Artifacts whose digest is known up front (OCI, Swift, Helm) are verified while streaming. Their responses are sent chunked and the connection is aborted on a mismatch, so a client never receives a tampered artifact as a complete response. Passthrough is rejected together with scanning, direct_serve and mirror_api, and the mirror command refuses to run with it, since all of them depend on stored artifacts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review: abort truncated streams, guard every cache read - serveArtifact aborts the response on any body read error, and when fewer bytes than the declared size were written, not only on a digest mismatch. A failed or short upstream body in passthrough mode no longer reaches the client as a complete 200. - Streamed upstream read failures are logged and counted as stream_failed upstream errors before the response is aborted. - checkCache reports a miss when artifact caching is off, so every cache read path, including Swift archive HEAD requests, ignores entries stored before the mode was enabled. Denylisted versions are still rejected. - Rename storage.passthrough to storage.cache_artifacts (default true, PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes. Tests cover an OCI blob shorter than its Content-Length and an npm tarball whose chunked response ends without the final chunk, both served through the HTTP handlers, plus a Swift HEAD request against a cached archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rename Proxy.Passthrough to StreamArtifacts Match the storage.cache_artifacts option: the field says what changes (artifacts are streamed instead of stored) and its zero value keeps the usual caching behaviour for every Proxy built without the server config, such as the mirror command and tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Batch cache hit writes
…rage [1/6] Name every mounted route in requestEcosystem
…-stats # Conflicts: # README.md
…stats [2/6] Per-ecosystem cache and download statistics
[3/6] Add the /ui/analytics page
…kdown [4/6] Report the per-ecosystem breakdown from GET /stats
[5/6] Add a Grafana dashboard
The flag exists to withhold caller addresses. The by-client table carries none, and the same figures are already public at /metrics, so gating it left two metrics with no tile on the page in the default config.
…ribution [6/6] Attribute requests to a caller and a client tool
Concurrent cache misses for one artifact already share a single upstream fetch (git-pkgs#329), but metadata misses did not: every request for a package's metadata went to the registry, even when the same document was already being fetched. In CI that is the common case. Composer resolves versions on every job, since MediaWiki core and its extensions commit no composer.lock, and each package is one /p2/ document, so a burst of jobs fetched each document once per job. Metadata misses now follow the same pattern as artifacts: the first caller fetches, the rest wait for its result, and the caller that takes the key rechecks the cache first so a fetch that just finished is not repeated. The key includes the Accept and Accept-Encoding headers, which change the bytes upstream returns (npm's abbreviated and full documents), and whether the caller validates the response. One difference from artifacts: the shared fetch is detached from the first caller's cancellation. Artifact fetches keep it for scanning and mirroring, neither of which applies to metadata, and otherwise one job disconnecting would fail every job waiting on the same document. runScan detaches the same way, and the HTTP client's timeout still bounds the fetch. This covers every ecosystem that goes through fetchOrCacheMetadata (Composer, npm, PyPI, Maven, Cargo, Swift, Pub, Helm, NuGet), with metadata caching on or off. Streamed metadata with caching off is not coalesced. On an 8 vCPU test instance with caching off, 10 concurrent clients requesting the npm metadata of 1,263 packages went from 12,630 upstream requests to 1,898. With cache_metadata on, 20 clients requesting 1,538 packages made exactly 1,538.
Owner
Author
|
CI passed on all jobs; closing as planned. The change is going upstream. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft PR inside the fork, only to run CI on this branch before proposing it upstream. Will be closed once CI finishes.