diff --git a/README.md b/README.md index 6fb3e8ab..07781143 100644 --- a/README.md +++ b/README.md @@ -1351,10 +1351,13 @@ The proxy exposes Prometheus metrics at `GET /metrics`. All metric names are pre |--------|------|--------|-------------| | `proxy_requests_total` | counter | `ecosystem`, `status` | Proxy responses by package ecosystem and HTTP status | | `proxy_request_duration_seconds` | histogram | `ecosystem`, `status` | Proxy request duration | -| `proxy_cache_hits_total` | counter | `ecosystem` | Cache hits | -| `proxy_cache_misses_total` | counter | `ecosystem` | Cache misses | +| `proxy_cache_hits_total` | counter | `ecosystem`, `kind` | Cache hits. `kind` is `artifact` or `metadata`. | +| `proxy_cache_misses_total` | counter | `ecosystem`, `kind` | Cache misses. `kind` is `artifact` or `metadata`; metadata misses are counted only while `cache_metadata` is on. | | `proxy_cache_size_bytes` | gauge | | Total size of cached artifacts | | `proxy_cached_artifacts_total` | gauge | | Number of cached artifacts | +| `proxy_metadata_cache_served_bytes_total` | counter | `ecosystem` | Metadata bytes served from the metadata cache. | +| `proxy_metadata_cache_size_bytes` | gauge | `ecosystem` | Size of cached metadata documents per ecosystem. | +| `proxy_metadata_cache_entries` | gauge | `ecosystem` | Number of cached metadata documents per ecosystem. | | `proxy_artifacts_evicted_total` | counter | `reason`, `ecosystem` | Artifacts evicted by the size limit (`lru`) or by age (`retention`) | | `proxy_upstream_fetch_duration_seconds` | histogram | `ecosystem` | Time spent fetching from upstream | | `proxy_upstream_errors_total` | counter | `ecosystem`, `error_type` | Upstream fetch failures | @@ -1414,11 +1417,17 @@ For history, trends and alerting, scrape `/metrics` with Prometheus. That is the The same figures are available as JSON from `GET /stats`, which reports `downloaded_bytes`, `downloads` and an `ecosystems` array carrying the per-ecosystem breakdown, served from the same 60-second snapshot the page and the gauges read. When the aggregation fails with no snapshot to fall back on, the response carries `stats_unavailable: true` rather than passing zeros off as a count -- the endpoint keeps answering with the artifact count and cache size either way. +#### Artifact and metadata caches + +The proxy keeps two caches. Artifacts (tarballs, wheels, crates and so on) live in storage and are what `proxy_cache_size_bytes`, `proxy_ecosystem_downloaded_bytes` and the other artifact figures describe. Registry metadata is cached separately when `cache_metadata` is on. It is stored persistently in the `metadata_cache` table rather than held in memory. + +The `kind` label on `proxy_cache_hits_total` and `proxy_cache_misses_total` keeps the two apart, so an artifact hit rate is `sum(rate(proxy_cache_hits_total{kind="artifact"}[5m]))` over the same expression with misses added. Metadata has its own size and entry gauges plus `proxy_metadata_cache_served_bytes_total`. That one is a process counter rather than a database total, because metadata hits are not recorded per row. On `/ui/analytics` metadata appears in its own card and in the Runtime card's metadata lookups, never mixed into the artifact figures. The card and the two gauges read one 60-second snapshot, like the artifact figures. The bundled Grafana dashboard's hit ratio covers artifacts only, while its hits and misses panel shows each kind as its own series. + #### Three ecosystem label sets `ecosystem` means three slightly different things across `/metrics`, and queries that join across them need to know which. -**From the package record, normalized.** The six `proxy_ecosystem_*` gauges, `proxy_cache_hits_total`, `proxy_cache_misses_total`, `proxy_artifacts_evicted_total`, `proxy_integrity_failures_total` and the scan metrics. Aliases collapse here: `gem` reads as `rubygems`, `composer` as `packagist`, `go` as `golang`. +**From the package record, normalized.** The six `proxy_ecosystem_*` gauges, `proxy_cache_hits_total`, `proxy_cache_misses_total`, the three `proxy_metadata_cache_*` metrics, `proxy_artifacts_evicted_total`, `proxy_integrity_failures_total` and the scan metrics. Aliases collapse here: `gem` reads as `rubygems`, `composer` as `packagist`, `go` as `golang`. **From the request path.** `proxy_requests_total`, `proxy_request_duration_seconds` and `proxy_response_bytes_total`. The names mostly coincide with the normalized ones -- these also report `rubygems`, `packagist` and `golang` -- but the Debian route reports `debian` where the package record says `deb`, and any path that is not a package endpoint reports `other`, which corresponds to no ecosystem at all. diff --git a/deploy/grafana/git-pkgs-proxy.json b/deploy/grafana/git-pkgs-proxy.json index e875469b..2592625e 100644 --- a/deploy/grafana/git-pkgs-proxy.json +++ b/deploy/grafana/git-pkgs-proxy.json @@ -365,13 +365,13 @@ "uid": "${datasource}" }, "editorMode": "code", - "expr": "sum(increase(proxy_cache_hits_total{job=~\"$job\", ecosystem=~\"$ecosystem\"}[$__range]))\n/\n(\n sum(increase(proxy_cache_hits_total{job=~\"$job\", ecosystem=~\"$ecosystem\"}[$__range]))\n +\n sum(increase(proxy_cache_misses_total{job=~\"$job\", ecosystem=~\"$ecosystem\"}[$__range]))\n)", + "expr": "sum(increase(proxy_cache_hits_total{job=~\"$job\", ecosystem=~\"$ecosystem\", kind=\"artifact\"}[$__range]))\n/\n(\n sum(increase(proxy_cache_hits_total{job=~\"$job\", ecosystem=~\"$ecosystem\", kind=\"artifact\"}[$__range]))\n +\n sum(increase(proxy_cache_misses_total{job=~\"$job\", ecosystem=~\"$ecosystem\", kind=\"artifact\"}[$__range]))\n)", "range": true, "instant": false, "refId": "A" } ], - "title": "Cache hit ratio", + "title": "Artifact cache hit ratio", "type": "stat" }, { @@ -981,11 +981,11 @@ "uid": "${datasource}" }, "editorMode": "code", - "expr": "sum(rate(proxy_cache_hits_total{job=~\"$job\", ecosystem=~\"$ecosystem\"}[$__rate_interval]))", + "expr": "sum by (kind) (rate(proxy_cache_hits_total{job=~\"$job\", ecosystem=~\"$ecosystem\"}[$__rate_interval]))", "range": true, "instant": false, "refId": "A", - "legendFormat": "hits" + "legendFormat": "{{kind}} hits" }, { "datasource": { @@ -993,11 +993,11 @@ "uid": "${datasource}" }, "editorMode": "code", - "expr": "sum(rate(proxy_cache_misses_total{job=~\"$job\", ecosystem=~\"$ecosystem\"}[$__rate_interval]))", + "expr": "sum by (kind) (rate(proxy_cache_misses_total{job=~\"$job\", ecosystem=~\"$ecosystem\"}[$__rate_interval]))", "range": true, "instant": false, "refId": "B", - "legendFormat": "misses" + "legendFormat": "{{kind}} misses" } ], "title": "Cache hits and misses", diff --git a/internal/database/metadata_stats_test.go b/internal/database/metadata_stats_test.go new file mode 100644 index 00000000..e66e4074 --- /dev/null +++ b/internal/database/metadata_stats_test.go @@ -0,0 +1,60 @@ +package database + +import ( + "database/sql" + "testing" +) + +func TestGetMetadataCacheStats(t *testing.T) { + db := newTestDB(t) + + stats, err := db.GetMetadataCacheStats() + if err != nil { + t.Fatalf("GetMetadataCacheStats on an empty cache: %v", err) + } + if len(stats) != 0 { + t.Fatalf("empty cache returned %+v, want no rows", stats) + } + + rows := []struct { + ecosystem, name string + size sql.NullInt64 + }{ + {"npm", "lodash", sql.NullInt64{Int64: 300, Valid: true}}, + {"npm", "react", sql.NullInt64{Int64: 200, Valid: true}}, + // gem and rubygems are one ecosystem and must share a row. + {"gem", "rails", sql.NullInt64{Int64: 1000, Valid: true}}, + {"rubygems", "rack", sql.NullInt64{Int64: 24, Valid: true}}, + // A row written before sizes were recorded still counts as an entry. + {"pypi", "requests", sql.NullInt64{}}, + } + for _, r := range rows { + if err := db.UpsertMetadataCache(&MetadataCacheEntry{ + Ecosystem: r.ecosystem, + Name: r.name, + StoragePath: "_metadata/" + r.ecosystem + "/" + r.name, + Size: r.size, + }); err != nil { + t.Fatalf("UpsertMetadataCache(%s/%s): %v", r.ecosystem, r.name, err) + } + } + + stats, err = db.GetMetadataCacheStats() + if err != nil { + t.Fatalf("GetMetadataCacheStats: %v", err) + } + + want := []MetadataCacheStats{ + {Ecosystem: "rubygems", Entries: 2, Size: 1024}, + {Ecosystem: "npm", Entries: 2, Size: 500}, + {Ecosystem: "pypi", Entries: 1, Size: 0}, + } + if len(stats) != len(want) { + t.Fatalf("got %+v, want %+v", stats, want) + } + for i := range want { + if stats[i] != want[i] { + t.Errorf("row %d = %+v, want %+v", i, stats[i], want[i]) + } + } +} diff --git a/internal/database/queries.go b/internal/database/queries.go index cd7c51ce..84ba1b4c 100644 --- a/internal/database/queries.go +++ b/internal/database/queries.go @@ -1123,6 +1123,62 @@ func (db *DB) CountCachedPackages(ecosystem string) (int64, error) { // Metadata cache queries +// MetadataCacheStats is one ecosystem's share of the metadata cache. +type MetadataCacheStats struct { + Ecosystem string + Entries int64 + Size int64 +} + +// GetMetadataCacheStats returns the number and total size of cached metadata +// documents per ecosystem, largest first. Ecosystem names are normalized so +// aliases such as "gem" and "rubygems" share one row. Rows written before +// sizes were recorded count as entries with no size. +func (db *DB) GetMetadataCacheStats() ([]MetadataCacheStats, error) { + hasTable, err := db.HasTable("metadata_cache") + if err != nil || !hasTable { + return nil, err + } + + rows, err := db.Query(`SELECT ecosystem, COUNT(*), COALESCE(SUM(size), 0) FROM metadata_cache GROUP BY ecosystem`) + if err != nil { + return nil, err + } + defer func() { _ = rows.Close() }() + + byEcosystem := make(map[string]*MetadataCacheStats) + for rows.Next() { + var ecosystem string + var entries, size int64 + if err := rows.Scan(&ecosystem, &entries, &size); err != nil { + return nil, err + } + key := purl.NormalizeEcosystem(ecosystem) + row, ok := byEcosystem[key] + if !ok { + row = &MetadataCacheStats{Ecosystem: key} + byEcosystem[key] = row + } + row.Entries += entries + row.Size += size + } + if err := rows.Err(); err != nil { + return nil, err + } + + stats := make([]MetadataCacheStats, 0, len(byEcosystem)) + for _, row := range byEcosystem { + stats = append(stats, *row) + } + sort.Slice(stats, func(i, j int) bool { + if stats[i].Size != stats[j].Size { + return stats[i].Size > stats[j].Size + } + return stats[i].Ecosystem < stats[j].Ecosystem + }) + return stats, nil +} + func (db *DB) GetMetadataCache(ecosystem, name string) (*MetadataCacheEntry, error) { var entry MetadataCacheEntry query := db.Rebind(` diff --git a/internal/handler/arch.go b/internal/handler/arch.go index 6706dae4..6579b093 100644 --- a/internal/handler/arch.go +++ b/internal/handler/arch.go @@ -198,7 +198,7 @@ func (h *ArchHandler) handleDatabase(w http.ResponseWriter, r *http.Request, rep } d := h.database(repository, upstreamURL, path) if _, hit := p.cachedMetadataState(r.Context(), archEcosystem, d.dbKey, nil); hit != nil { - metrics.RecordCacheHit(archEcosystem) + metrics.RecordMetadataCacheHit(archEcosystem, len(hit.body)) h.writeMetadata(w, r, d.dbKey, *hit) return } @@ -217,7 +217,7 @@ func (h *ArchHandler) handleDatabaseSignature(w http.ResponseWriter, r *http.Req } d := h.database(repository, upstreamURL, strings.TrimSuffix(path, archSignatureSuffix)) if res, ok := h.cachedSignature(r.Context(), d); ok { - metrics.RecordCacheHit(archEcosystem) + metrics.RecordMetadataCacheHit(archEcosystem, len(res.body)) h.writeMetadata(w, r, d.sigKey, res) return } diff --git a/internal/handler/go_sumdb.go b/internal/handler/go_sumdb.go index 020a126e..37f58606 100644 --- a/internal/handler/go_sumdb.go +++ b/internal/handler/go_sumdb.go @@ -348,7 +348,7 @@ func (h *GoHandler) serveCachedSumDBTile(w http.ResponseWriter, r *http.Request, if check != nil && check(body) != nil { return false, true } - metrics.RecordCacheHit(goEcosystem) + metrics.RecordMetadataCacheHit(goEcosystem, len(body)) if entry.ETag.Valid { w.Header().Set(headerETag, entry.ETag.String) diff --git a/internal/handler/gradle.go b/internal/handler/gradle.go index e2512092..cd3e78a5 100644 --- a/internal/handler/gradle.go +++ b/internal/handler/gradle.go @@ -106,11 +106,11 @@ func (h *GradleBuildCacheHandler) handleGetOrHead(w http.ResponseWriter, r *http return } if !exists { - metrics.RecordCacheMiss("gradle") + metrics.RecordArtifactCacheMiss("gradle") http.NotFound(w, r) return } - metrics.RecordCacheHit("gradle") + metrics.RecordArtifactCacheHit("gradle") sizeStart := time.Now() size, err := h.proxy.Storage.Size(r.Context(), storagePath) @@ -130,7 +130,7 @@ func (h *GradleBuildCacheHandler) handleGetOrHead(w http.ResponseWriter, r *http metrics.RecordStorageOperation("read", time.Since(readStart)) if err != nil { if errors.Is(err, storage.ErrNotFound) { - metrics.RecordCacheMiss("gradle") + metrics.RecordArtifactCacheMiss("gradle") http.NotFound(w, r) return } @@ -140,7 +140,7 @@ func (h *GradleBuildCacheHandler) handleGetOrHead(w http.ResponseWriter, r *http return } defer func() { _ = reader.Close() }() - metrics.RecordCacheHit("gradle") + metrics.RecordArtifactCacheHit("gradle") w.WriteHeader(http.StatusOK) _, _ = io.Copy(w, reader) diff --git a/internal/handler/gradle_test.go b/internal/handler/gradle_test.go index a05d07e5..a4b95c28 100644 --- a/internal/handler/gradle_test.go +++ b/internal/handler/gradle_test.go @@ -237,8 +237,8 @@ func TestGradleBuildCacheHandler_RecordsMetrics(t *testing.T) { srv := httptest.NewServer(h.Routes()) defer srv.Close() - hitsBefore := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("gradle")) - missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("gradle")) + hitsBefore := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("gradle", metrics.CacheKindArtifact)) + missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("gradle", metrics.CacheKindArtifact)) key := "metrics-key" putReq, err := http.NewRequest(http.MethodPut, srv.URL+"/"+key, strings.NewReader("payload")) @@ -273,8 +273,8 @@ func TestGradleBuildCacheHandler_RecordsMetrics(t *testing.T) { } _ = missResp.Body.Close() - hitsAfter := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("gradle")) - missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("gradle")) + hitsAfter := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("gradle", metrics.CacheKindArtifact)) + missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("gradle", metrics.CacheKindArtifact)) if diff := hitsAfter - hitsBefore; diff != 2 { t.Fatalf("cache hits delta = %.0f, want 2", diff) diff --git a/internal/handler/handler.go b/internal/handler/handler.go index c3a9ea0f..6b8d962a 100644 --- a/internal/handler/handler.go +++ b/internal/handler/handler.go @@ -303,7 +303,7 @@ func (p *Proxy) GetOrFetchArtifact(ctx context.Context, ecosystem, name, version } else if cached != nil { return cached, nil } - metrics.RecordCacheMiss(ecosystem) + metrics.RecordArtifactCacheMiss(ecosystem) key := artifactCoalesceKey(versionPURL, filename, "", "", "") recheck := func() (artifacts.Artifact, string, bool) { @@ -474,7 +474,7 @@ func rewriteSignedURLHost(signed, baseURL string) string { func (p *Proxy) recordCacheHit(ecosystem, versionPURL, filename string) { _ = p.DB.RecordArtifactHit(versionPURL, filename) - metrics.RecordCacheHit(ecosystem) + metrics.RecordArtifactCacheHit(ecosystem) } func (p *Proxy) rejectUnusableCacheRecord(artifact *database.CachedArtifact, versionPURL, filename string, cause error) { @@ -1266,7 +1266,7 @@ func (p *Proxy) lookupMetadata(ctx context.Context, ecosystem, cacheKey, upstrea // Serve from cache if within TTL (skip upstream entirely) if _, hit := p.cachedMetadataState(ctx, ecosystem, cacheKey, validate); hit != nil { - metrics.RecordCacheHit(ecosystem) + metrics.RecordMetadataCacheHit(ecosystem, len(hit.body)) return *hit } p.recordMetadataCacheMiss(ecosystem) @@ -1477,7 +1477,7 @@ func (p *Proxy) readCachedMetadata(ctx context.Context, entry *database.Metadata func (p *Proxy) recordMetadataCacheMiss(ecosystem string) { if p.CacheMetadata { - metrics.RecordCacheMiss(ecosystem) + metrics.RecordMetadataCacheMiss(ecosystem) } } @@ -1843,7 +1843,7 @@ func (p *Proxy) getOrFetchArtifactFromURLWithCachePURLs(ctx context.Context, eco } else if cached != nil { return cached, nil } - metrics.RecordCacheMiss(ecosystem) + metrics.RecordArtifactCacheMiss(ecosystem) return p.coalescedFetchFromURL(ctx, ecosystem, name, version, filename, pkgPURL, versionPURL, downloadURL, headers, upstreamHash, declared) } diff --git a/internal/handler/handler_test.go b/internal/handler/handler_test.go index ffc0d5f2..5599af1a 100644 --- a/internal/handler/handler_test.go +++ b/internal/handler/handler_test.go @@ -451,7 +451,7 @@ func assertQueuedForDeletion(t *testing.T, db *database.DB, path string) { func TestGetOrFetchArtifact_CacheMiss_NoPackage(t *testing.T) { proxy, _, _, fetcher := setupTestProxy(t) - missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("npm")) + missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("npm", metrics.CacheKindArtifact)) // The resolver will fail because "nonexistent" isn't a real package, // but we're testing that it tries to fetch (doesn't return from cache). @@ -461,7 +461,7 @@ func TestGetOrFetchArtifact_CacheMiss_NoPackage(t *testing.T) { if err == nil { t.Fatal("expected error for uncached package") } - missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("npm")) + missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("npm", metrics.CacheKindArtifact)) if diff := missesAfter - missesBefore; diff != 1 { t.Errorf("cache misses delta = %.0f, want 1", diff) } @@ -905,7 +905,7 @@ func newRangeTestResult(payload string) *CacheResult { func TestGetOrFetchArtifactFromURL_CacheHit(t *testing.T) { proxy, db, store, fetcher := setupTestProxy(t) seedPackage(t, db, store, "pypi", "requests", "2.28.0", "requests-2.28.0.tar.gz", "pypi content") - missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("pypi")) + missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("pypi", metrics.CacheKindArtifact)) result, err := proxy.GetOrFetchArtifactFromURL(context.Background(), "pypi", "requests", "2.28.0", "requests-2.28.0.tar.gz", "https://pypi.org/files/requests-2.28.0.tar.gz") if err != nil { @@ -919,7 +919,7 @@ func TestGetOrFetchArtifactFromURL_CacheHit(t *testing.T) { if fetcher.fetchCalled { t.Error("fetcher should not be called on cache hit") } - missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("pypi")) + missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("pypi", metrics.CacheKindArtifact)) if diff := missesAfter - missesBefore; diff != 0 { t.Errorf("cache misses delta = %.0f, want 0", diff) } @@ -927,7 +927,7 @@ func TestGetOrFetchArtifactFromURL_CacheHit(t *testing.T) { func TestGetOrFetchArtifactFromURL_CacheMiss(t *testing.T) { proxy, db, store, fetcher := setupTestProxy(t) - missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("pypi")) + missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("pypi", metrics.CacheKindArtifact)) fetchesBefore := histogramSampleCount(t, metrics.UpstreamFetchDuration.WithLabelValues("pypi")) writesBefore := histogramSampleCount(t, metrics.StorageOperationDuration.WithLabelValues("write")) readsBefore := histogramSampleCount(t, metrics.StorageOperationDuration.WithLabelValues("read")) @@ -975,7 +975,7 @@ func TestGetOrFetchArtifactFromURL_CacheMiss(t *testing.T) { if _, ok := store.files[storagePath]; !ok { t.Error("artifact was not stored in storage") } - missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("pypi")) + missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("pypi", metrics.CacheKindArtifact)) if diff := missesAfter - missesBefore; diff != 1 { t.Errorf("cache misses delta = %.0f, want 1", diff) } @@ -1305,8 +1305,10 @@ func TestProxyCached_NoValidators_OmitsHeaders(t *testing.T) { } func TestFetchOrCacheMetadata_TTL_ServesFreshFromCache(t *testing.T) { - hitsBefore := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("test")) - missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("test")) + hitsBefore := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("test", metrics.CacheKindMetadata)) + missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("test", metrics.CacheKindMetadata)) + artifactHitsBefore := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("test", metrics.CacheKindArtifact)) + servedBefore := testutil.ToFloat64(metrics.MetadataServedBytes.WithLabelValues("test")) upstreamHits := 0 upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { upstreamHits++ @@ -1333,10 +1335,10 @@ func TestFetchOrCacheMetadata_TTL_ServesFreshFromCache(t *testing.T) { if upstreamHits != 1 { t.Fatalf("expected 1 upstream hit, got %d", upstreamHits) } - if diff := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("test")) - missesBefore; diff != 1 { + if diff := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("test", metrics.CacheKindMetadata)) - missesBefore; diff != 1 { t.Errorf("cache misses delta after first request = %.0f, want 1", diff) } - if diff := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("test")) - hitsBefore; diff != 0 { + if diff := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("test", metrics.CacheKindMetadata)) - hitsBefore; diff != 0 { t.Errorf("cache hits delta after first request = %.0f, want 0", diff) } @@ -1351,16 +1353,22 @@ func TestFetchOrCacheMetadata_TTL_ServesFreshFromCache(t *testing.T) { if upstreamHits != 1 { t.Errorf("expected upstream to still be hit only once, got %d", upstreamHits) } - if diff := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("test")) - hitsBefore; diff != 1 { + if diff := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("test", metrics.CacheKindMetadata)) - hitsBefore; diff != 1 { t.Errorf("cache hits delta after second request = %.0f, want 1", diff) } - if diff := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("test")) - missesBefore; diff != 1 { + if diff := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("test", metrics.CacheKindMetadata)) - missesBefore; diff != 1 { t.Errorf("cache misses delta after second request = %.0f, want 1", diff) } + if diff := testutil.ToFloat64(metrics.MetadataServedBytes.WithLabelValues("test")) - servedBefore; diff != float64(len(body)) { + t.Errorf("metadata served bytes delta = %.0f, want %d (only the cached response)", diff, len(body)) + } + if diff := testutil.ToFloat64(metrics.CacheHits.WithLabelValues("test", metrics.CacheKindArtifact)) - artifactHitsBefore; diff != 0 { + t.Errorf("artifact cache hits delta = %.0f, want 0 for a metadata lookup", diff) + } } func TestFetchOrCacheMetadata_TTL_Zero_AlwaysRevalidates(t *testing.T) { - missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("test")) + missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("test", metrics.CacheKindMetadata)) upstreamHits := 0 upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { upstreamHits++ @@ -1389,7 +1397,7 @@ func TestFetchOrCacheMetadata_TTL_Zero_AlwaysRevalidates(t *testing.T) { if upstreamHits != 2 { t.Errorf("expected 2 upstream hits with TTL=0, got %d", upstreamHits) } - missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("test")) + missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues("test", metrics.CacheKindMetadata)) if diff := missesAfter - missesBefore; diff != 2 { t.Errorf("cache misses delta = %.0f, want 2", diff) } @@ -1407,16 +1415,16 @@ func TestFetchOrCacheMetadata_CacheDisabledDoesNotRecordMetrics(t *testing.T) { proxy, _, _, _ := setupTestProxy(t) proxy.HTTPClient = upstream.Client() - hitsBefore := testutil.ToFloat64(metrics.CacheHits.WithLabelValues(ecosystem)) - missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues(ecosystem)) + hitsBefore := testutil.ToFloat64(metrics.CacheHits.WithLabelValues(ecosystem, metrics.CacheKindMetadata)) + missesBefore := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues(ecosystem, metrics.CacheKindMetadata)) _, _, err := proxy.FetchOrCacheMetadata(context.Background(), ecosystem, "pkg", upstream.URL+"/pkg") if err != nil { t.Fatalf("fetch metadata: %v", err) } - hitsAfter := testutil.ToFloat64(metrics.CacheHits.WithLabelValues(ecosystem)) - missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues(ecosystem)) + hitsAfter := testutil.ToFloat64(metrics.CacheHits.WithLabelValues(ecosystem, metrics.CacheKindMetadata)) + missesAfter := testutil.ToFloat64(metrics.CacheMisses.WithLabelValues(ecosystem, metrics.CacheKindMetadata)) if diff := hitsAfter - hitsBefore; diff != 0 { t.Errorf("cache hits delta = %.0f, want 0", diff) } diff --git a/internal/handler/npm_rewrite_test.go b/internal/handler/npm_rewrite_test.go index d9a787eb..d0a5c3e7 100644 --- a/internal/handler/npm_rewrite_test.go +++ b/internal/handler/npm_rewrite_test.go @@ -12,6 +12,8 @@ import ( "time" "github.com/git-pkgs/cooldown" + "github.com/git-pkgs/proxy/internal/metrics" + "github.com/prometheus/client_golang/prometheus/testutil" ) // TestNPMRewriteMetadataKeepsUpstreamBytes checks that only tarball URLs @@ -270,3 +272,43 @@ func TestNPMStoredRewriteSkippedWhenStale(t *testing.T) { t.Error("cooldown on: expected no stored rewrite") } } + +// TestNPMStoredRewriteRecordsMetadataHit checks the digest fast path counts as +// a metadata cache hit with the bytes it serves, never as an artifact hit. +func TestNPMStoredRewriteRecordsMetadataHit(t *testing.T) { + proxy, _, _, _ := setupTestProxy(t) + proxy.CacheMetadata = true + proxy.MetadataTTL = time.Hour + proxy.SetMetadataRewriteCacheSize(1 << 20) + + body := []byte(`{"name":"demo"}`) + proxy.cacheMetadataBlob(t.Context(), "npm", "hit-demo", metadataStoragePath("npm", "hit-demo"), &upstreamMetadata{body: body}) + key := rewriteCacheKey("npm", "http://proxy", "hit-demo", body) + if _, _, err := proxy.rewrites.rewrite(t.Context(), key, body, func(b []byte) ([]byte, error) { return b, nil }); err != nil { + t.Fatal(err) + } + + metadataHits := func() float64 { + return testutil.ToFloat64(metrics.CacheHits.WithLabelValues("npm", metrics.CacheKindMetadata)) + } + artifactHits := func() float64 { + return testutil.ToFloat64(metrics.CacheHits.WithLabelValues("npm", metrics.CacheKindArtifact)) + } + served := func() float64 { return testutil.ToFloat64(metrics.MetadataServedBytes.WithLabelValues("npm")) } + metadataBefore, artifactBefore, servedBefore := metadataHits(), artifactHits(), served() + + m, ok := proxy.storedRewrite("npm", "hit-demo", "http://proxy", "hit-demo") + if !ok { + t.Fatal("expected a stored rewrite") + } + + if diff := metadataHits() - metadataBefore; diff != 1 { + t.Errorf("metadata hits delta = %.0f, want 1", diff) + } + if diff := artifactHits() - artifactBefore; diff != 0 { + t.Errorf("artifact hits delta = %.0f, want 0", diff) + } + if diff := served() - servedBefore; diff != float64(len(m.body)) { + t.Errorf("metadata served bytes delta = %.0f, want %d", diff, len(m.body)) + } +} diff --git a/internal/handler/rewrite_cache.go b/internal/handler/rewrite_cache.go index 42273c3b..6cc21a0f 100644 --- a/internal/handler/rewrite_cache.go +++ b/internal/handler/rewrite_cache.go @@ -312,7 +312,7 @@ func (p *Proxy) storedRewrite(ecosystem, cacheKey, proxyURL, name string) (rewri return rewrittenMetadata{}, false } v.lastModified = p.metadataDate(ecosystem, proxyURL, name, v.etag) - metrics.RecordCacheHit(ecosystem) + metrics.RecordMetadataCacheHit(ecosystem, len(out)) return rewrittenMetadata{out, v}, true } diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go index 7567ff8e..a919a34e 100644 --- a/internal/metrics/metrics.go +++ b/internal/metrics/metrics.go @@ -35,15 +35,23 @@ var ( CacheHits = prometheus.NewCounterVec( prometheus.CounterOpts{ Name: "proxy_cache_hits_total", - Help: "Total number of cache hits by ecosystem", + Help: "Total number of cache hits by ecosystem and kind (artifact or metadata)", }, - []string{"ecosystem"}, + []string{"ecosystem", "kind"}, ) CacheMisses = prometheus.NewCounterVec( prometheus.CounterOpts{ Name: "proxy_cache_misses_total", - Help: "Total number of cache misses by ecosystem", + Help: "Total number of cache misses by ecosystem and kind (artifact or metadata)", + }, + []string{"ecosystem", "kind"}, + ) + + MetadataServedBytes = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "proxy_metadata_cache_served_bytes_total", + Help: "Total metadata bytes served from the metadata cache, by ecosystem", }, []string{"ecosystem"}, ) @@ -198,6 +206,24 @@ var ( []string{"ecosystem"}, ) + // Metadata cache gauges, derived from the metadata_cache table on the + // same tick as the per-ecosystem gauges above. + MetadataCacheSize = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "proxy_metadata_cache_size_bytes", + Help: "Size of cached metadata documents per ecosystem in bytes", + }, + []string{"ecosystem"}, + ) + + MetadataCacheEntries = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "proxy_metadata_cache_entries", + Help: "Number of cached metadata documents per ecosystem", + }, + []string{"ecosystem"}, + ) + ResponseBytes = prometheus.NewCounterVec( prometheus.CounterOpts{ Name: "proxy_response_bytes_total", @@ -256,6 +282,7 @@ func init() { RequestDuration, CacheHits, CacheMisses, + MetadataServedBytes, CacheSize, CachedArtifacts, ArtifactsEvicted, @@ -274,6 +301,8 @@ func init() { EcosystemCachedArtifacts, EcosystemPackages, EcosystemVersions, + MetadataCacheSize, + MetadataCacheEntries, ResponseBytes, ClientRequests, ClientResponseBytes, @@ -313,9 +342,27 @@ func RecordResponse(ecosystem, client string, bytes int64) { ClientResponseBytes.WithLabelValues(client).Add(float64(bytes)) } -// RecordCacheHit increments cache hit counter. -func RecordCacheHit(ecosystem string) { - CacheHits.WithLabelValues(purl.NormalizeEcosystem(ecosystem)).Inc() +// Values of the kind label on proxy_cache_hits_total and +// proxy_cache_misses_total. Artifacts and metadata are separate caches with +// very different costs, so a hit on one says nothing about the other. +const ( + CacheKindArtifact = "artifact" + CacheKindMetadata = "metadata" +) + +// RecordArtifactCacheHit counts an artifact served from storage. +func RecordArtifactCacheHit(ecosystem string) { + CacheHits.WithLabelValues(purl.NormalizeEcosystem(ecosystem), CacheKindArtifact).Inc() +} + +// RecordMetadataCacheHit counts a metadata document served from the metadata +// cache, along with the bytes it served. +func RecordMetadataCacheHit(ecosystem string, bytes int) { + ecosystem = purl.NormalizeEcosystem(ecosystem) + CacheHits.WithLabelValues(ecosystem, CacheKindMetadata).Inc() + if bytes > 0 { + MetadataServedBytes.WithLabelValues(ecosystem).Add(float64(bytes)) + } } // RecordArtifactEvicted counts one evicted artifact. ecosystem is the @@ -328,9 +375,14 @@ func RecordArtifactEvicted(reason, ecosystem string) { ArtifactsEvicted.WithLabelValues(reason, purl.NormalizeEcosystem(ecosystem)).Inc() } -// RecordCacheMiss increments cache miss counter. -func RecordCacheMiss(ecosystem string) { - CacheMisses.WithLabelValues(purl.NormalizeEcosystem(ecosystem)).Inc() +// RecordArtifactCacheMiss counts an artifact that had to be fetched upstream. +func RecordArtifactCacheMiss(ecosystem string) { + CacheMisses.WithLabelValues(purl.NormalizeEcosystem(ecosystem), CacheKindArtifact).Inc() +} + +// RecordMetadataCacheMiss counts a metadata lookup the cache could not serve. +func RecordMetadataCacheMiss(ecosystem string) { + CacheMisses.WithLabelValues(purl.NormalizeEcosystem(ecosystem), CacheKindMetadata).Inc() } // RecordUpstreamFetch tracks upstream fetch duration. @@ -414,6 +466,8 @@ type EcosystemStats struct { var ( publishedMu sync.Mutex publishedEcosystems = map[string]bool{} + // publishedMetadataEcosystems does the same for the metadata cache gauges. + publishedMetadataEcosystems = map[string]bool{} ) // UpdateEcosystemStats republishes the per-ecosystem gauges from a fresh snapshot. @@ -471,6 +525,48 @@ func UpdateEcosystemStats(stats []EcosystemStats) { } } +// MetadataCacheStats is one ecosystem's share of the metadata cache. +type MetadataCacheStats struct { + Ecosystem string + Entries int64 + Size int64 +} + +// UpdateMetadataCacheStats republishes the metadata cache gauges from a fresh +// snapshot. Like UpdateEcosystemStats it sums rows by normalized label, sets +// each series in place and deletes only labels that have disappeared. +func UpdateMetadataCacheStats(stats []MetadataCacheStats) { + totals := make(map[string]MetadataCacheStats, len(stats)) + for _, s := range stats { + ecosystem := purl.NormalizeEcosystem(s.Ecosystem) + t := totals[ecosystem] + t.Entries += s.Entries + t.Size += s.Size + totals[ecosystem] = t + } + + for ecosystem, t := range totals { + MetadataCacheEntries.WithLabelValues(ecosystem).Set(float64(t.Entries)) + MetadataCacheSize.WithLabelValues(ecosystem).Set(float64(t.Size)) + } + + publishedMu.Lock() + defer publishedMu.Unlock() + + for ecosystem := range publishedMetadataEcosystems { + if _, still := totals[ecosystem]; still { + continue + } + MetadataCacheEntries.DeleteLabelValues(ecosystem) + MetadataCacheSize.DeleteLabelValues(ecosystem) + } + + publishedMetadataEcosystems = make(map[string]bool, len(totals)) + for ecosystem := range totals { + publishedMetadataEcosystems[ecosystem] = true + } +} + // UpdateCircuitBreakerState updates circuit breaker state gauge. // state: 0=closed, 1=half-open, 2=open func UpdateCircuitBreakerState(registry string, state int) { diff --git a/internal/metrics/metrics_test.go b/internal/metrics/metrics_test.go index cca39c89..dc19c80d 100644 --- a/internal/metrics/metrics_test.go +++ b/internal/metrics/metrics_test.go @@ -21,10 +21,11 @@ func TestRecordRequest(t *testing.T) { } func TestRecordCacheOperations(t *testing.T) { - RecordCacheHit("npm") - RecordCacheHit("npm") - RecordCacheMiss("npm") - RecordCacheMiss("pypi") + RecordArtifactCacheHit("npm") + RecordArtifactCacheHit("npm") + RecordArtifactCacheMiss("npm") + RecordMetadataCacheHit("pypi", 128) + RecordMetadataCacheMiss("pypi") // No panics = success } @@ -113,9 +114,9 @@ func TestMetricsAreRegistered(t *testing.T) { func TestCacheHitRatio(t *testing.T) { // Reset by recording some hits and misses - RecordCacheHit("test") - RecordCacheHit("test") - RecordCacheMiss("test") + RecordArtifactCacheHit("test") + RecordArtifactCacheHit("test") + RecordArtifactCacheMiss("test") // Collect metrics hits := getMetricValue(t, CacheHits, "test") @@ -178,7 +179,7 @@ func getMetricValue(t *testing.T, collector prometheus.Collector, labelValue str func TestMetricsEndpointOutput(t *testing.T) { // Record some test metrics RecordRequest("npm", 200, 50*time.Millisecond) - RecordCacheHit("npm") + RecordArtifactCacheHit("npm") UpdateCacheStats(1024*1024, 10) // Get the handler and check it produces output @@ -206,7 +207,7 @@ func TestMetricsLabeling(t *testing.T) { for _, eco := range ecosystems { RecordRequest(eco.input, 200, 10*time.Millisecond) - RecordCacheHit(eco.input) + RecordArtifactCacheHit(eco.input) } // Verify each ecosystem has metrics @@ -219,18 +220,93 @@ func TestMetricsLabeling(t *testing.T) { } func TestCacheMetricLabelsAreNormalized(t *testing.T) { - rubyHitsBefore := testutil.ToFloat64(CacheHits.WithLabelValues("rubygems")) - composerMissesBefore := testutil.ToFloat64(CacheMisses.WithLabelValues("packagist")) + rubyHitsBefore := testutil.ToFloat64(CacheHits.WithLabelValues("rubygems", CacheKindArtifact)) + composerMissesBefore := testutil.ToFloat64(CacheMisses.WithLabelValues("packagist", CacheKindArtifact)) + goMetadataHitsBefore := testutil.ToFloat64(CacheHits.WithLabelValues("golang", CacheKindMetadata)) + goServedBefore := testutil.ToFloat64(MetadataServedBytes.WithLabelValues("golang")) - RecordCacheHit("gem") - RecordCacheMiss("composer") + RecordArtifactCacheHit("gem") + RecordArtifactCacheMiss("composer") + RecordMetadataCacheHit("go", 10) - if diff := testutil.ToFloat64(CacheHits.WithLabelValues("rubygems")) - rubyHitsBefore; diff != 1 { + if diff := testutil.ToFloat64(CacheHits.WithLabelValues("rubygems", CacheKindArtifact)) - rubyHitsBefore; diff != 1 { t.Errorf("rubygems cache hits delta = %.0f, want 1", diff) } - if diff := testutil.ToFloat64(CacheMisses.WithLabelValues("packagist")) - composerMissesBefore; diff != 1 { + if diff := testutil.ToFloat64(CacheMisses.WithLabelValues("packagist", CacheKindArtifact)) - composerMissesBefore; diff != 1 { t.Errorf("packagist cache misses delta = %.0f, want 1", diff) } + if diff := testutil.ToFloat64(CacheHits.WithLabelValues("golang", CacheKindMetadata)) - goMetadataHitsBefore; diff != 1 { + t.Errorf("golang metadata hits delta = %.0f, want 1", diff) + } + if diff := testutil.ToFloat64(MetadataServedBytes.WithLabelValues("golang")) - goServedBefore; diff != 10 { + t.Errorf("golang metadata served bytes delta = %.0f, want 10", diff) + } +} + +// TestCacheKindsAreCountedApart checks an artifact hit and a metadata hit land +// in different series. Only metadata hits add served bytes. +func TestCacheKindsAreCountedApart(t *testing.T) { + const eco = "kind-test" + artifactHits := func() float64 { return testutil.ToFloat64(CacheHits.WithLabelValues(eco, CacheKindArtifact)) } + metadataHits := func() float64 { return testutil.ToFloat64(CacheHits.WithLabelValues(eco, CacheKindMetadata)) } + artifactMisses := func() float64 { return testutil.ToFloat64(CacheMisses.WithLabelValues(eco, CacheKindArtifact)) } + metadataMisses := func() float64 { return testutil.ToFloat64(CacheMisses.WithLabelValues(eco, CacheKindMetadata)) } + served := func() float64 { return testutil.ToFloat64(MetadataServedBytes.WithLabelValues(eco)) } + + aHits, mHits, aMisses, mMisses, bytes := artifactHits(), metadataHits(), artifactMisses(), metadataMisses(), served() + + RecordArtifactCacheHit(eco) + RecordMetadataCacheHit(eco, 300) + RecordMetadataCacheHit(eco, 0) + RecordMetadataCacheMiss(eco) + + checks := []struct { + name string + got, want float64 + }{ + {"artifact hits", artifactHits() - aHits, 1}, + {"metadata hits", metadataHits() - mHits, 2}, + {"artifact misses", artifactMisses() - aMisses, 0}, + {"metadata misses", metadataMisses() - mMisses, 1}, + {"metadata served bytes", served() - bytes, 300}, + } + for _, c := range checks { + if c.got != c.want { + t.Errorf("%s delta = %.0f, want %.0f", c.name, c.got, c.want) + } + } +} + +func TestUpdateMetadataCacheStats(t *testing.T) { + UpdateMetadataCacheStats([]MetadataCacheStats{ + {Ecosystem: "gem", Entries: 2, Size: 100}, + {Ecosystem: "rubygems", Entries: 3, Size: 50}, + {Ecosystem: "npm", Entries: 1, Size: 10}, + }) + + if got := testutil.ToFloat64(MetadataCacheEntries.WithLabelValues("rubygems")); got != 5 { + t.Errorf("rubygems entries = %.0f, want 5 (aliases summed)", got) + } + if got := testutil.ToFloat64(MetadataCacheSize.WithLabelValues("rubygems")); got != 150 { + t.Errorf("rubygems size = %.0f, want 150 (aliases summed)", got) + } + if got := testutil.CollectAndCount(MetadataCacheSize); got != 2 { + t.Errorf("size series = %d, want 2", got) + } + + UpdateMetadataCacheStats([]MetadataCacheStats{{Ecosystem: "npm", Entries: 4, Size: 40}}) + + if got := testutil.CollectAndCount(MetadataCacheSize); got != 1 { + t.Errorf("size series after rubygems disappeared = %d, want 1", got) + } + if got := testutil.CollectAndCount(MetadataCacheEntries); got != 1 { + t.Errorf("entry series after rubygems disappeared = %d, want 1", got) + } + if got := testutil.ToFloat64(MetadataCacheSize.WithLabelValues("npm")); got != 40 { + t.Errorf("npm size = %.0f, want 40", got) + } + + UpdateMetadataCacheStats(nil) } func TestMetricNames(t *testing.T) { @@ -240,6 +316,9 @@ func TestMetricNames(t *testing.T) { "proxy_request_duration_seconds", "proxy_cache_hits_total", "proxy_cache_misses_total", + "proxy_metadata_cache_served_bytes_total", + "proxy_metadata_cache_size_bytes", + "proxy_metadata_cache_entries", "proxy_upstream_fetch_duration_seconds", "proxy_upstream_errors_total", "proxy_storage_operation_duration_seconds", diff --git a/internal/metrics/snapshot_test.go b/internal/metrics/snapshot_test.go index 1a20906d..5b6c7fd4 100644 --- a/internal/metrics/snapshot_test.go +++ b/internal/metrics/snapshot_test.go @@ -125,7 +125,7 @@ func TestSnapshotMissingMetric(t *testing.T) { // come back. This is the check that a newly added metric is reachable by the UI. func TestGatherSeesRegisteredMetrics(t *testing.T) { isolateRecentFailures(t) - RecordCacheHit("npm") + RecordArtifactCacheHit("npm") RecordScanError("npm", "clamav", "timeout", FailureDetail{}) snap, err := Gather() diff --git a/internal/server/analytics.go b/internal/server/analytics.go index b7b4a837..2e0e0ced 100644 --- a/internal/server/analytics.go +++ b/internal/server/analytics.go @@ -21,6 +21,7 @@ type AnalyticsData struct { Donut DonutView EnrichmentStats EnrichmentStatsView Ecosystems []EcosystemRow + Metadata MetadataCacheView Runtime RuntimeView // StatsFailed records that the per-ecosystem query errored, so the page // can say the figures are unavailable instead of claiming an empty cache. @@ -66,6 +67,54 @@ type EcosystemRow struct { SharePct string } +// MetadataCacheView summarises the metadata cache, which is stored separately +// from artifacts and so appears in none of the artifact figures. +type MetadataCacheView struct { + // Enabled reports whether cache_metadata is on. Entries can remain from + // an earlier run with it on, so both are shown independently. + Enabled bool + // Failed records that the query errored, so the card can say so rather + // than claim the cache is empty. + Failed bool + Entries string + Size string + Rows []MetadataCacheRow +} + +// MetadataCacheRow is one ecosystem's share of the metadata cache. +type MetadataCacheRow struct { + Ecosystem string + Entries string + Size string + SharePct string +} + +// metadataCacheView turns per-ecosystem metadata cache rows into the card's +// totals and table. +func metadataCacheView(stats []database.MetadataCacheStats, enabled bool) MetadataCacheView { + var entries, size int64 + for _, e := range stats { + entries += e.Entries + size += e.Size + } + + v := MetadataCacheView{ + Enabled: enabled, + Entries: formatCount(entries), + Size: formatSize(size), + Rows: make([]MetadataCacheRow, 0, len(stats)), + } + for _, e := range stats { + v.Rows = append(v.Rows, MetadataCacheRow{ + Ecosystem: e.Ecosystem, + Entries: formatCount(e.Entries), + Size: formatSize(e.Size), + SharePct: formatPercent(e.Size, size), + }) + } + return v +} + func formatCount(n int64) string { s := strconv.FormatInt(n, 10) neg := strings.HasPrefix(s, "-") @@ -165,6 +214,16 @@ func (s *Server) handleAnalytics(w http.ResponseWriter, r *http.Request) { data.Totals, data.Ecosystems = analyticsView(ecosystems) data.Donut = donutView(ecosystems, data.Totals.DownloadedBytes, data.Totals.Downloaded) + // The same 60-second snapshot the metadata cache gauges are published + // from, so a page load costs no query of its own. A failed read keeps the + // last good rows, which the card shows in preference to the error. + metadataStats, err := s.metaStats.Get(s.db) + if err != nil { + s.logger.Error("failed to get metadata cache stats", "error", err) + } + data.Metadata = metadataCacheView(metadataStats, s.cfg != nil && s.cfg.CacheMetadata) + data.Metadata.Failed = err != nil + // Process-lifetime counters come from the Prometheus registry rather than // the database; a failure here must not take the page down with it. snap, err := metrics.Gather() diff --git a/internal/server/analytics_coverage_test.go b/internal/server/analytics_coverage_test.go index 12cc8959..d353c89f 100644 --- a/internal/server/analytics_coverage_test.go +++ b/internal/server/analytics_coverage_test.go @@ -24,6 +24,8 @@ var metricSurface = map[string]string{ "proxy_ecosystem_versions": "breakdown table", "proxy_cache_size_bytes": "Cache size tile", "proxy_cached_artifacts_total": "Cached artifacts tile", + "proxy_metadata_cache_size_bytes": "Metadata cache card", + "proxy_metadata_cache_entries": "Metadata cache card", // Request-path counters. "proxy_response_bytes_total": "Runtime: Served", @@ -34,8 +36,9 @@ var metricSurface = map[string]string{ "proxy_requests_total": "Runtime: Requests + Responses by status", "proxy_request_duration_seconds": "Runtime: Mean latency", "proxy_active_requests": "Runtime: In flight", - "proxy_cache_hits_total": "Runtime: Cache lookups + hit rate", - "proxy_cache_misses_total": "Runtime: Cache lookups + hit rate", + "proxy_cache_hits_total": "Runtime: Artifact and metadata cache lookups + artifact hit rate", + "proxy_cache_misses_total": "Runtime: Artifact and metadata cache lookups + artifact hit rate", + "proxy_metadata_cache_served_bytes_total": "Runtime: Metadata cache lookups -- served from cache", "proxy_upstream_fetch_duration_seconds": "Runtime: Upstream fetches + mean fetch", "proxy_upstream_errors_total": "Runtime: Upstream errors", "proxy_storage_operation_duration_seconds": "Runtime: Storage operations", @@ -59,8 +62,10 @@ func TestEveryMetricIsSurfaced(t *testing.T) { // a vector with no observed label values gathers as nothing at all. metrics.RecordRequest("npm", 200, 0) metrics.RecordResponse("npm", "npm", 1) - metrics.RecordCacheHit("npm") - metrics.RecordCacheMiss("npm") + metrics.RecordArtifactCacheHit("npm") + metrics.RecordArtifactCacheMiss("npm") + metrics.RecordMetadataCacheHit("npm", 1) + metrics.RecordMetadataCacheMiss("npm") metrics.RecordUpstreamFetch("npm", 0) metrics.RecordUpstreamError("npm", "fetch_failed", metrics.FailureDetail{}) metrics.RecordStorageOperation("get", 0) @@ -74,6 +79,7 @@ func TestEveryMetricIsSurfaced(t *testing.T) { metrics.UpdateCacheStats(1, 1) metrics.RecordArtifactEvicted("retention", "npm") metrics.UpdateEcosystemStats([]metrics.EcosystemStats{{Ecosystem: "npm", DownloadedBytes: 1}}) + metrics.UpdateMetadataCacheStats([]metrics.MetadataCacheStats{{Ecosystem: "npm", Entries: 1, Size: 1}}) snap, err := metrics.Gather() if err != nil { diff --git a/internal/server/analytics_runtime.go b/internal/server/analytics_runtime.go index 8e5d0d17..621670fd 100644 --- a/internal/server/analytics_runtime.go +++ b/internal/server/analytics_runtime.go @@ -15,14 +15,15 @@ import ( type RuntimeView struct { Available bool - Requests string - ActiveRequests string - RequestMean string - StatusClasses []LabelledCount - CacheHits string - CacheMisses string - CacheHitRatio string - HasCacheTraffic bool + Requests string + ActiveRequests string + RequestMean string + StatusClasses []LabelledCount + // Artifact and metadata lookups are counted apart: they are separate + // caches. Only artifacts feed the database figures above. + ArtifactCache CacheLookups + MetadataCache CacheLookups + MetadataServed string UpstreamFetches string UpstreamFetchMean string @@ -58,6 +59,28 @@ type RuntimeView struct { HasFailures bool } +// CacheLookups is one cache's hits and misses since the process started. +type CacheLookups struct { + Hits string + Misses string + HitRatio string + HasTraffic bool +} + +// cacheLookups reads one kind's share of the hit and miss counters. +func cacheLookups(hits, misses map[string]float64, kind string) CacheLookups { + h, m := hits[kind], misses[kind] + c := CacheLookups{ + Hits: formatCount(int64(h)), + Misses: formatCount(int64(m)), + HasTraffic: h+m > 0, + } + if c.HasTraffic { + c.HitRatio = strconv.FormatFloat(h/(h+m)*100, 'f', 1, 64) //nolint:mnd // percent + } + return c +} + // ClientRow is one client tool's share of requests and bytes. type ClientRow struct { Client string @@ -121,14 +144,11 @@ func runtimeView(snap *metrics.Snapshot, failureDetails bool) RuntimeView { v.RequestMean = formatDuration(snap.Mean("proxy_request_duration_seconds")) v.StatusClasses = statusClasses(snap) - hits := snap.Sum("proxy_cache_hits_total") - misses := snap.Sum("proxy_cache_misses_total") - v.CacheHits = formatCount(int64(hits)) - v.CacheMisses = formatCount(int64(misses)) - v.HasCacheTraffic = hits+misses > 0 - if v.HasCacheTraffic { - v.CacheHitRatio = strconv.FormatFloat(hits/(hits+misses)*100, 'f', 1, 64) //nolint:mnd // percent - } + hits := snap.SumBy("proxy_cache_hits_total", "kind") + misses := snap.SumBy("proxy_cache_misses_total", "kind") + v.ArtifactCache = cacheLookups(hits, misses, metrics.CacheKindArtifact) + v.MetadataCache = cacheLookups(hits, misses, metrics.CacheKindMetadata) + v.MetadataServed = formatSize(int64(snap.Sum("proxy_metadata_cache_served_bytes_total"))) v.UpstreamFetches = formatCount(int64(snap.Count("proxy_upstream_fetch_duration_seconds"))) v.UpstreamFetchMean = formatDuration(snap.Mean("proxy_upstream_fetch_duration_seconds")) diff --git a/internal/server/analytics_runtime_test.go b/internal/server/analytics_runtime_test.go index 5bff0f36..8493e766 100644 --- a/internal/server/analytics_runtime_test.go +++ b/internal/server/analytics_runtime_test.go @@ -40,9 +40,9 @@ func TestRuntimeViewShapesRegistry(t *testing.T) { scanErrors := prometheus.NewCounterVec( prometheus.CounterOpts{Name: "proxy_scan_errors_total", Help: "t"}, []string{"ecosystem", "scanner", "error_type"}) hits := prometheus.NewCounterVec( - prometheus.CounterOpts{Name: "proxy_cache_hits_total", Help: "t"}, []string{"ecosystem"}) + prometheus.CounterOpts{Name: "proxy_cache_hits_total", Help: "t"}, []string{"ecosystem", "kind"}) misses := prometheus.NewCounterVec( - prometheus.CounterOpts{Name: "proxy_cache_misses_total", Help: "t"}, []string{"ecosystem"}) + prometheus.CounterOpts{Name: "proxy_cache_misses_total", Help: "t"}, []string{"ecosystem", "kind"}) breaker := prometheus.NewGaugeVec( prometheus.GaugeOpts{Name: "proxy_circuit_breaker_state", Help: "t"}, []string{"registry"}) storage := prometheus.NewHistogramVec( @@ -53,8 +53,8 @@ func TestRuntimeViewShapesRegistry(t *testing.T) { requests.WithLabelValues("npm", "304").Add(5) requests.WithLabelValues("npm", "500").Add(5) scanErrors.WithLabelValues("npm", "clamav", "timeout").Add(3) - hits.WithLabelValues("npm").Add(80) - misses.WithLabelValues("npm").Add(20) + hits.WithLabelValues("npm", metrics.CacheKindArtifact).Add(80) + misses.WithLabelValues("npm", metrics.CacheKindArtifact).Add(20) breaker.WithLabelValues("registry.npmjs.org").Set(0) breaker.WithLabelValues("static.crates.io").Set(2) storage.WithLabelValues("get").Observe(0.002) @@ -71,8 +71,8 @@ func TestRuntimeViewShapesRegistry(t *testing.T) { if v.Requests != "100" { t.Errorf("Requests = %q, want %q", v.Requests, "100") } - if v.CacheHitRatio != "80.0" || !v.HasCacheTraffic { - t.Errorf("CacheHitRatio = %q (traffic=%v), want 80.0", v.CacheHitRatio, v.HasCacheTraffic) + if a := v.ArtifactCache; a.HitRatio != "80.0" || !a.HasTraffic { + t.Errorf("ArtifactCache = %+v, want an 80.0%% hit rate", a) } // Status codes collapse to classes; 5xx is flagged as a failure row. @@ -178,6 +178,48 @@ func TestRuntimeViewDropsZeroCounters(t *testing.T) { } } +// Artifact and metadata lookups are separate caches, so the view reports each +// on its own rather than pooling them into one hit rate. +func TestRuntimeViewSplitsCacheKinds(t *testing.T) { + reg := prometheus.NewRegistry() + hits := prometheus.NewCounterVec( + prometheus.CounterOpts{Name: "proxy_cache_hits_total", Help: "t"}, []string{"ecosystem", "kind"}) + misses := prometheus.NewCounterVec( + prometheus.CounterOpts{Name: "proxy_cache_misses_total", Help: "t"}, []string{"ecosystem", "kind"}) + served := prometheus.NewCounterVec( + prometheus.CounterOpts{Name: "proxy_metadata_cache_served_bytes_total", Help: "t"}, []string{"ecosystem"}) + reg.MustRegister(hits, misses, served) + + hits.WithLabelValues("npm", metrics.CacheKindArtifact).Add(80) + misses.WithLabelValues("npm", metrics.CacheKindArtifact).Add(20) + hits.WithLabelValues("pypi", metrics.CacheKindMetadata).Add(20) + hits.WithLabelValues("npm", metrics.CacheKindMetadata).Add(10) + misses.WithLabelValues("pypi", metrics.CacheKindMetadata).Add(10) + served.WithLabelValues("pypi").Add(2048) + + snap, err := metrics.GatherFrom(reg) + if err != nil { + t.Fatalf("GatherFrom: %v", err) + } + v := runtimeView(snap, false) + + want := map[string]struct { + got CacheLookups + want CacheLookups + }{ + "artifact": {v.ArtifactCache, CacheLookups{Hits: "80", Misses: "20", HitRatio: "80.0", HasTraffic: true}}, + "metadata": {v.MetadataCache, CacheLookups{Hits: "30", Misses: "10", HitRatio: "75.0", HasTraffic: true}}, + } + for kind, c := range want { + if c.got != c.want { + t.Errorf("%s lookups = %+v, want %+v", kind, c.got, c.want) + } + } + if v.MetadataServed != "2.0 KB" { + t.Errorf("MetadataServed = %q, want %q", v.MetadataServed, "2.0 KB") + } +} + // An empty registry must still render, reporting nothing rather than dividing // by zero or showing a hit rate of NaN. func TestRuntimeViewEmptyRegistry(t *testing.T) { @@ -190,11 +232,13 @@ func TestRuntimeViewEmptyRegistry(t *testing.T) { if !v.Available { t.Error("an empty registry is still a valid snapshot") } - if v.HasCacheTraffic { - t.Error("no traffic recorded, so HasCacheTraffic must be false") - } - if v.CacheHitRatio != "" { - t.Errorf("CacheHitRatio = %q, want empty when nothing was recorded", v.CacheHitRatio) + for name, c := range map[string]CacheLookups{"artifact": v.ArtifactCache, "metadata": v.MetadataCache} { + if c.HasTraffic { + t.Errorf("%s: no traffic recorded, so HasTraffic must be false", name) + } + if c.HitRatio != "" { + t.Errorf("%s: HitRatio = %q, want empty when nothing was recorded", name, c.HitRatio) + } } if v.Requests != "0" { t.Errorf("Requests = %q, want %q", v.Requests, "0") diff --git a/internal/server/analytics_test.go b/internal/server/analytics_test.go index 3ce5289a..540408b0 100644 --- a/internal/server/analytics_test.go +++ b/internal/server/analytics_test.go @@ -1,6 +1,7 @@ package server import ( + "database/sql" "net/http/httptest" "strings" "testing" @@ -275,3 +276,95 @@ func TestEcosystemMetricsMapping(t *testing.T) { t.Errorf("fields did not map across: %+v", m) } } + +// The metadata cache is stored apart from artifacts, so the page reports it in +// its own card rather than leaving it out of every figure. +func TestAnalyticsPageShowsMetadataCache(t *testing.T) { + ts := newTestServer(t) + defer ts.close() + ts.server.cfg.CacheMetadata = true + + for _, e := range []database.MetadataCacheEntry{ + {Ecosystem: "pypi", Name: "requests", StoragePath: "_metadata/pypi/requests", Size: sql.NullInt64{Int64: 3 << 10, Valid: true}}, + {Ecosystem: "pypi", Name: "urllib3", StoragePath: "_metadata/pypi/urllib3", Size: sql.NullInt64{Int64: 1 << 10, Valid: true}}, + } { + if err := ts.db.UpsertMetadataCache(&e); err != nil { + t.Fatalf("UpsertMetadataCache: %v", err) + } + } + + body := ts.getOK(t, "/ui/analytics") + for _, want := range []string{"Metadata cache", "Cached documents", "4.0 KB", "100.0%"} { + if !strings.Contains(body, want) { + t.Errorf("rendered page missing %q", want) + } + } + if strings.Contains(body, "Metadata caching is off") { + t.Error("the off-state note rendered with cache_metadata on") + } +} + +func TestAnalyticsPageNotesMetadataCachingOff(t *testing.T) { + ts := newTestServer(t) + defer ts.close() + + body := ts.getOK(t, "/ui/analytics") + for _, want := range []string{"Metadata cache", "Metadata caching is off", "No metadata cached yet"} { + if !strings.Contains(body, want) { + t.Errorf("rendered page missing %q", want) + } + } +} + +func TestMetadataCacheView(t *testing.T) { + v := metadataCacheView([]database.MetadataCacheStats{ + {Ecosystem: "npm", Entries: 3, Size: 3000}, + {Ecosystem: "pypi", Entries: 1, Size: 1000}, + }, true) + + if !v.Enabled || v.Entries != "4" || v.Size != "3.9 KB" { + t.Errorf("totals = %+v, want enabled, 4 entries, 3.9 KB", v) + } + if len(v.Rows) != 2 || v.Rows[0].SharePct != "75.0" || v.Rows[1].SharePct != "25.0" { + t.Errorf("rows = %+v, want npm 75.0%% then pypi 25.0%%", v.Rows) + } + + empty := metadataCacheView(nil, false) + if empty.Enabled || len(empty.Rows) != 0 || empty.Entries != "0" || empty.Size != "0 B" { + t.Errorf("empty view = %+v", empty) + } +} + +// The page reads the metadata card from the snapshot the gauges publish, so a +// page load issues no query of its own and the two always agree. +func TestAnalyticsPageMetadataCardUsesSnapshot(t *testing.T) { + ts := newTestServer(t) + defer ts.close() + ts.server.cfg.CacheMetadata = true + + seed := func(name string, size int64) { + t.Helper() + if err := ts.db.UpsertMetadataCache(&database.MetadataCacheEntry{ + Ecosystem: "pypi", Name: name, StoragePath: "_metadata/pypi/" + name, + Size: sql.NullInt64{Int64: size, Valid: true}, + }); err != nil { + t.Fatalf("UpsertMetadataCache: %v", err) + } + } + + seed("requests", 1<<10) + ts.server.updateMetadataCacheStats() + if !strings.Contains(ts.getOK(t, "/ui/analytics"), "1.0 KB") { + t.Fatal("page did not show the snapshot the metrics tick read") + } + + seed("urllib3", 2<<10) + if body := ts.getOK(t, "/ui/analytics"); !strings.Contains(body, "1.0 KB") || strings.Contains(body, "3.0 KB") { + t.Error("a page load re-queried the database instead of serving the snapshot") + } + + ts.server.updateMetadataCacheStats() + if !strings.Contains(ts.getOK(t, "/ui/analytics"), "3.0 KB") { + t.Error("page did not pick up the refreshed snapshot") + } +} diff --git a/internal/server/ecosystem_cache.go b/internal/server/ecosystem_cache.go index 83e8c0d7..d12d3a2c 100644 --- a/internal/server/ecosystem_cache.go +++ b/internal/server/ecosystem_cache.go @@ -17,20 +17,23 @@ const ecosystemStatsTTL = time.Minute // that is down is not queried once per request. const ecosystemStatsErrorTTL = 5 * time.Second -// ecosystemStatsCache memoizes GetEcosystemStats. +// statsCache memoizes one of the grouped statistics queries behind the +// analytics page and the metrics gauges. // -// The query is three grouped aggregations over the whole artifact table, which -// is fine once a minute for the metrics gauges but not once per page load: the -// UI is unauthenticated, so without this a client reloading /ui/analytics would -// keep the database busy for as long as it cared to. +// Each query is a grouped aggregation over a whole table, which is fine once a +// minute for the metrics gauges but not once per page load: the UI is +// unauthenticated, so without this a client reloading /ui/analytics would keep +// the database busy for as long as it cared to. The page and the gauges read +// the same snapshot, so they never disagree. // -// The zero value is usable, so a Server assembled as a struct literal -- as +// L selects the query. It is a zero-size type rather than a stored function so +// the zero value stays usable: a Server assembled as a struct literal -- as // tests do -- needs no constructor and cannot end up with a nil cache. -type ecosystemStatsCache struct { +type statsCache[E any, L statsLoader[E]] struct { ttl time.Duration mu sync.Mutex - stats []database.EcosystemStats + stats []E err error lastAt time.Time // goodAt is when stats was last read successfully, which is not lastAt @@ -38,15 +41,39 @@ type ecosystemStatsCache struct { goodAt time.Time } +// statsLoader runs the query a statsCache memoizes, returning one row per +// ecosystem. +type statsLoader[E any] interface { + load(db *database.DB) ([]E, error) +} + +type ecosystemStatsLoader struct{} + +func (ecosystemStatsLoader) load(db *database.DB) ([]database.EcosystemStats, error) { + return db.GetEcosystemStats() +} + +type metadataStatsLoader struct{} + +func (metadataStatsLoader) load(db *database.DB) ([]database.MetadataCacheStats, error) { + return db.GetMetadataCacheStats() +} + +// ecosystemStatsCache memoizes GetEcosystemStats. +type ecosystemStatsCache = statsCache[database.EcosystemStats, ecosystemStatsLoader] + +// metadataStatsCache memoizes GetMetadataCacheStats. +type metadataStatsCache = statsCache[database.MetadataCacheStats, metadataStatsLoader] + // SnapshotAt reports when the rows currently held were last read successfully. // The zero time means no read has ever succeeded. -func (c *ecosystemStatsCache) SnapshotAt() time.Time { +func (c *statsCache[E, L]) SnapshotAt() time.Time { c.mu.Lock() defer c.mu.Unlock() return c.goodAt } -func (c *ecosystemStatsCache) interval() time.Duration { +func (c *statsCache[E, L]) interval() time.Duration { if c.err != nil { return ecosystemStatsErrorTTL } @@ -60,7 +87,7 @@ func (c *ecosystemStatsCache) interval() time.Duration { // // A failed read returns the last good snapshot alongside the error, so a // caller can choose to serve stale figures rather than fail outright. -func (c *ecosystemStatsCache) Get(db *database.DB) ([]database.EcosystemStats, error) { +func (c *statsCache[E, L]) Get(db *database.DB) ([]E, error) { c.mu.Lock() defer c.mu.Unlock() @@ -72,14 +99,15 @@ func (c *ecosystemStatsCache) Get(db *database.DB) ([]database.EcosystemStats, e // Refresh forces a read, bypassing the TTL. The metrics loop uses it so its own // tick is never served a snapshot that is about to expire. -func (c *ecosystemStatsCache) Refresh(db *database.DB) ([]database.EcosystemStats, error) { +func (c *statsCache[E, L]) Refresh(db *database.DB) ([]E, error) { c.mu.Lock() defer c.mu.Unlock() return c.refreshLocked(db) } -func (c *ecosystemStatsCache) refreshLocked(db *database.DB) ([]database.EcosystemStats, error) { - stats, err := db.GetEcosystemStats() +func (c *statsCache[E, L]) refreshLocked(db *database.DB) ([]E, error) { + var loader L + stats, err := loader.load(db) c.lastAt = time.Now() c.err = err if err != nil { diff --git a/internal/server/ecosystem_cache_test.go b/internal/server/ecosystem_cache_test.go index 5323ad1b..9356dc49 100644 --- a/internal/server/ecosystem_cache_test.go +++ b/internal/server/ecosystem_cache_test.go @@ -1,6 +1,7 @@ package server import ( + "database/sql" "path/filepath" "testing" "time" @@ -197,3 +198,49 @@ func TestEcosystemStatsCacheSnapshotAtTracksSuccessOnly(t *testing.T) { t.Errorf("SnapshotAt moved on a failed read: %v, want %v", got, at) } } + +// The metadata cache figures go through the same memo as the artifact ones. +// Rows added inside the TTL stay out of the snapshot until a Refresh, which is +// what the metrics tick calls. +func TestMetadataStatsCacheServesSnapshot(t *testing.T) { + db := cacheTestDB(t) + c := metadataStatsCache{ttl: time.Hour} + + seed := func(name string, size int64) { + t.Helper() + if err := db.UpsertMetadataCache(&database.MetadataCacheEntry{ + Ecosystem: "npm", + Name: name, + StoragePath: "_metadata/npm/" + name, + Size: sql.NullInt64{Int64: size, Valid: true}, + }); err != nil { + t.Fatalf("UpsertMetadataCache: %v", err) + } + } + entries := func(stats []database.MetadataCacheStats) int64 { + var n int64 + for _, s := range stats { + n += s.Entries + } + return n + } + + seed("lodash", 100) + stats, err := c.Get(db) + if err != nil || entries(stats) != 1 { + t.Fatalf("first Get = %+v, %v; want 1 entry", stats, err) + } + + seed("react", 50) + if stats, _ := c.Get(db); entries(stats) != 1 { + t.Errorf("Get inside the TTL = %+v; want the 1-entry snapshot", stats) + } + + stats, err = c.Refresh(db) + if err != nil || entries(stats) != 2 { + t.Fatalf("Refresh = %+v, %v; want 2 entries", stats, err) + } + if stats, _ := c.Get(db); entries(stats) != 2 { + t.Errorf("Get after Refresh = %+v; want the refreshed snapshot", stats) + } +} diff --git a/internal/server/server.go b/internal/server/server.go index 2907f04c..ccc78f42 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -118,6 +118,7 @@ type Server struct { ecr *ecrTokens breakers *breakerMonitor ecoStats ecosystemStatsCache + metaStats metadataStatsCache sources sourceTracker // browseLimits overrides defaultBrowseLimits when set; tests use it. @@ -567,12 +568,30 @@ func (s *Server) updateCacheStatsMetrics() { // Update once immediately s.updateCacheStats() + s.updateMetadataCacheStats() for range ticker.C { s.updateCacheStats() + s.updateMetadataCacheStats() } } +// updateMetadataCacheStats republishes the metadata cache gauges from a fresh +// read of the snapshot the analytics page also serves. It runs separately +// from updateCacheStats so a failure in one leaves the other's gauges current. +func (s *Server) updateMetadataCacheStats() { + stats, err := s.metaStats.Refresh(s.db) + if err != nil { + s.logger.Warn("failed to get metadata cache stats for metrics", "error", err) + return + } + out := make([]metrics.MetadataCacheStats, 0, len(stats)) + for _, e := range stats { + out = append(out, metrics.MetadataCacheStats{Ecosystem: e.Ecosystem, Entries: e.Entries, Size: e.Size}) + } + metrics.UpdateMetadataCacheStats(out) +} + func (s *Server) updateCacheStats() { stats, err := s.db.GetCacheStats() if err != nil { diff --git a/internal/server/templates/components/runtime_metrics.html b/internal/server/templates/components/runtime_metrics.html index 9804b475..a2bee450 100644 --- a/internal/server/templates/components/runtime_metrics.html +++ b/internal/server/templates/components/runtime_metrics.html @@ -25,8 +25,8 @@

Runtime

{{.RequestMean}}
-
Cache hit rate
-
{{if .HasCacheTraffic}}{{.CacheHitRatio}}%{{else}}—{{end}}
+
Artifact hit rate
+
{{if .ArtifactCache.HasTraffic}}{{.ArtifactCache.HitRatio}}%{{else}}—{{end}}
Upstream fetches
@@ -41,16 +41,14 @@

Runtime

{{template "runtime_counts" dict "Title" "Responses by status" "Rows" .StatusClasses "Empty" "No requests yet"}} + {{template "runtime_cache" dict "Title" "Artifact cache lookups" "Lookups" .ArtifactCache}} +
-

Cache lookups

-
-
-
hits
-
{{.CacheHits}}
-
+ {{template "runtime_cache" dict "Title" "Metadata cache lookups" "Lookups" .MetadataCache}} +
-
misses
-
{{.CacheMisses}}
+
served from cache
+
{{.MetadataServed}}
@@ -267,3 +265,23 @@

{{.Title}}

{{end}}
{{end}} + +{{define "runtime_cache"}} +
+

{{.Title}}

+
+
+
hits
+
{{.Lookups.Hits}}
+
+
+
misses
+
{{.Lookups.Misses}}
+
+
+
hit rate
+
{{if .Lookups.HasTraffic}}{{.Lookups.HitRatio}}%{{else}}—{{end}}
+
+
+
+{{end}} diff --git a/internal/server/templates/pages/analytics.html b/internal/server/templates/pages/analytics.html index da97a7ce..cda81bb4 100644 --- a/internal/server/templates/pages/analytics.html +++ b/internal/server/templates/pages/analytics.html @@ -215,6 +215,58 @@

Per-ecosystem breakdown

{{end}}
+ +
+
+

Metadata cache

+

+ Cached registry metadata, kept apart from artifacts: the download, cache size and + artifact figures above do not include it. + {{if not .Metadata.Enabled}}Metadata caching is off; set cache_metadata to enable it.{{end}} +

+
+ {{if .Metadata.Rows}} +
+
+
Cached documents
+
{{.Metadata.Entries}}
+
+
+
Metadata size
+
{{.Metadata.Size}}
+
+
+
+ + + + + + + + + + + {{range .Metadata.Rows}} + + + + + + + {{end}} + +
EcosystemSizeShareDocuments
{{template "ecosystem_badge" .Ecosystem}}{{.Size}}{{.SharePct}}%{{.Entries}}
+
+ {{else if .Metadata.Failed}} +
+ The metadata cache query failed, so these figures are unavailable. Check the proxy logs. +
+ {{else}} +
No metadata cached yet
+ {{end}} +
+ {{if .EnrichmentStats.HasVulns}} {{template "security_overview" .EnrichmentStats}} {{end}} diff --git a/internal/server/templates_test.go b/internal/server/templates_test.go index 276af6e4..708f02c5 100644 --- a/internal/server/templates_test.go +++ b/internal/server/templates_test.go @@ -63,19 +63,18 @@ func TestTemplatesRenderAllPages(t *testing.T) { {Ecosystem: "cargo", CacheSize: 120, Downloads: 3004, DownloadedBytes: 500_000_000}, }, 1_500_000_000, "1.4 GB"), Runtime: RuntimeView{ - Available: true, - Requests: "12,004", - ActiveRequests: "2", - RequestMean: "8.1 ms", - CacheHits: "9,000", - CacheMisses: "1,000", - CacheHitRatio: "90.0", - HasCacheTraffic: true, - StatusClasses: []LabelledCount{{Label: "2xx", Count: "11,900"}, {Label: "5xx", Count: "104", Bad: true}}, - StorageOps: []LabelledStat{{Label: "get", Count: "9,000", Mean: "412 µs"}}, - ScanErrors: []LabelledCount{{Label: "npm · clamav · timeout", Count: "3", Bad: true}}, - ScanningOn: true, - Breakers: []BreakerRow{{Registry: "registry.npmjs.org", State: "closed"}}, + Available: true, + Requests: "12,004", + ActiveRequests: "2", + RequestMean: "8.1 ms", + ArtifactCache: CacheLookups{Hits: "9,000", Misses: "1,000", HitRatio: "90.0", HasTraffic: true}, + MetadataCache: CacheLookups{Hits: "4,000", Misses: "500", HitRatio: "88.9", HasTraffic: true}, + MetadataServed: "12.0 MB", + StatusClasses: []LabelledCount{{Label: "2xx", Count: "11,900"}, {Label: "5xx", Count: "104", Bad: true}}, + StorageOps: []LabelledStat{{Label: "get", Count: "9,000", Mean: "412 µs"}}, + ScanErrors: []LabelledCount{{Label: "npm · clamav · timeout", Count: "3", Bad: true}}, + ScanningOn: true, + Breakers: []BreakerRow{{Registry: "registry.npmjs.org", State: "closed"}}, }, }}, {"analytics", AnalyticsData{}},