Add opt-in memory stability soak to the perf suite

.gitignore

@@ -10,6 +10,9 @@ coverage/
 .pnpm-store/
 test/perf/last-results.json
 test/perf/last-profile.json
+test/perf/last-memory-results.json
+test/perf/last-memory-report.md
+test/perf/last-memory-profile.json
 test/perf/*-results.json
 profiles/
 package-lock.json

AGENTS.md

@@ -20,7 +20,8 @@ This is `couch-auth-proxy`: a TypeScript (Hono) reverse proxy that enforces per-
 - `pnpm lint`, `pnpm fmt:check`, `pnpm typecheck`, `pnpm test` (unit) need no running services.
 - `pnpm test:integration` requires the docker stack up first (`docker compose up -d --build`); it hits the proxy at `http://127.0.0.1:8000`.
 - `pnpm test:perf` is the ACL load harness (multi-client Pouch sync + HTTP r/w ops/sec). Prefer the dev overlay so direct Couch is on `5985` for overhead compare: `docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build`. Not part of CI.
-- Request phase profiling: set `PROFILE=true` (compose overlay `docker-compose.profile.yml`, or `pnpm docker:up:profile`). Scrapes `GET /_couch-auth-proxy/profile` (auth / acl / aclMiss / upstream / filter). Convenience: `pnpm test:perf:profile`. Host CPU profiles: build then `pnpm start:profile` (writes under `profiles/`).
+- Request phase profiling: set `PROFILE=true` (compose overlay `docker-compose.profile.yml`, or `pnpm docker:up:profile`). Scrapes `GET /_couch-auth-proxy/profile` (auth / acl / aclMiss / upstream / filter + process memory / resource sizes). Convenience: `pnpm test:perf:profile`. Host CPU profiles: build then `pnpm start:profile` (writes under `profiles/`).
+- Memory stability soak (opt-in): `pnpm test:perf:memory` brings up the profile overlay (`PROFILE=true`, `NODE_OPTIONS=--expose-gc`), runs a multi-minute steady-state ACL load while scraping heap/rss trends, and writes `test/perf/last-memory-report.md`. Tunable via `PERF_MEMORY_DURATION_SEC` (default 300).
 
 ### Gotchas
 

docker-compose.profile.yml

@@ -1,11 +1,15 @@
-# Profiling overlay: enables request phase timers on the proxy.
+# Profiling overlay: enables request phase timers + memory probe on the proxy.
 # Usage:
 #   docker compose -f docker-compose.yml -f docker-compose.dev.yml -f docker-compose.profile.yml up -d --build
 #   pnpm test:perf:profile
+#   pnpm test:perf:memory
 services:
   couch-auth-proxy:
     environment:
       PROFILE: "true"
+      # Expose V8 GC for optional POST /_couch-auth-proxy/profile/gc samples used
+      # by the memory-stability soak. Harmless when unused; still opt-in via this overlay.
+      NODE_OPTIONS: "--expose-gc"
       # Session principal cache defaults to 5000ms in the app. Override here to
       # attribute auth vs ACL under an alternate TTL (0 = re-resolve every request).
       # SESSION_CACHE_TTL_MS: "0"

docs/memory-stability.md

@@ -0,0 +1,70 @@
+# Memory stability report
+
+Generated: 2026-07-24T22:24:16.881Z  
+Branch / harness: `feat/memory-stability-perf-38c4` via `pnpm test:perf:memory`  
+Proxy: compose profile overlay (`PROFILE=true`, `NODE_OPTIONS=--expose-gc`)
+
+## Verdict
+
+**STABLE — no evidence of a memory leak**
+
+- V8 `heapUsed` was flat-to-down over a 5-minute steady-state ACL load (net **−1.55 MiB**).
+- In-process ACL map size plateaued (`acl_rows` **504 → 504**), so cache growth does not explain any residual process size change.
+- Linear heap slope after warmup was **~0.45 KiB/s** with weak correlation (**r = 0.094**) — consistent with GC sawtooth, not a leak.
+- Median heap in the last third of the steady window was only **~200 KiB** above the first third.
+- RSS rose modestly (**+9.6 MiB**, ~48 KiB/s, r = 0.988) while heap declined. That pattern matches native allocator retention / freelist growth under sustained HTTP, not retained JavaScript objects. Slope remained under the soak threshold (128 KiB/s).
+
+## Load (steady-state)
+
+| Metric                  | Value                                      |
+| ----------------------- | ------------------------------------------ |
+| duration                | 302.8 s                                    |
+| clients                 | 6                                          |
+| seed docs               | 300                                        |
+| samples                 | 150 (2 s interval; 25% warmup discarded)   |
+| ops                     | 185,926 (613.9 ops/s)                      |
+| docs read / written     | 600,998 / 37,188                           |
+| error rate              | 0.00%                                      |
+| latency p50 / p95 / p99 | 7.0 / 24.8 / 31.2 ms                       |
+| forced GC samples       | yes (`POST /_couch-auth-proxy/profile/gc`) |
+
+Workload reuses rotating document slots and a fixed mixed-ACL corpus so the ACL row count plateaus while still exercising auth, ACL lookup, `_bulk_get` filtering, and writes.
+
+## Memory trend (steady state)
+
+| Signal                      | Value                             |
+| --------------------------- | --------------------------------- |
+| heap_used first → last      | 16.48 → 14.93 MiB (Δ −1.55 MiB)   |
+| heap_used slope             | 0.45 KiB/s (r = 0.094)            |
+| heap median 1st → 3rd third | 16.27 → 16.47 MiB                 |
+| rss first → last            | 217.10 → 226.73 MiB (Δ +9.63 MiB) |
+| rss slope                   | 47.88 KiB/s (r = 0.988)           |
+| acl_rows first → last       | 504 → 504 (Δ 0)                   |
+
+## Method
+
+1. Seed a fixed mixed-ACL corpus through the proxy.
+2. Run concurrent HTTP readers/writers that reuse rotating document slots so the in-memory ACL map plateaus.
+3. While load runs, scrape `GET /_couch-auth-proxy/profile` (opt-in `PROFILE=true`) for `process.memoryUsage()` plus ACL/session resource sizes.
+4. Optionally `POST /_couch-auth-proxy/profile/gc` each sample when the proxy was started with `--expose-gc` (profile compose overlay).
+5. Discard the leading warmup fraction, fit heap/rss vs time, and compare median heap in the first vs last third of the steady window. Expected ACL-row growth is budgeted; unexplained growth fails the assessment.
+
+## Thresholds used
+
+| Threshold                         | Value       |
+| --------------------------------- | ----------- |
+| max heap slope                    | 64.0 KiB/s  |
+| max rss slope                     | 128.0 KiB/s |
+| max unexplained heap median shift | 48.0 MiB    |
+| heap budget per new ACL row       | 2048 B      |
+| min steady samples                | 8           |
+
+## How to reproduce
+
+```bash
+pnpm test:perf:memory
+# longer soak:
+PERF_MEMORY_DURATION_SEC=600 pnpm test:perf:memory
+```
+
+Artifacts (gitignored): `test/perf/last-memory-results.json`, `test/perf/last-memory-report.md`, `test/perf/last-memory-profile.json`.

package.json

@@ -16,6 +16,7 @@
     "test:integration": "vitest run --config vitest.integration.config.ts",
     "test:perf": "vitest run --config vitest.perf.config.ts",
     "test:perf:profile": "bash scripts/perf-profile.sh",
+    "test:perf:memory": "bash scripts/perf-memory.sh",
     "docker:up": "docker compose up -d --build",
     "docker:up:dev": "docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build",
     "docker:up:profile": "docker compose -f docker-compose.yml -f docker-compose.dev.yml -f docker-compose.profile.yml up -d --build",

scripts/perf-memory.sh

@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+# Bring up the compose stack with PROFILE=true (+ optional --expose-gc), run the
+# long-running memory stability soak, and write JSON + Markdown reports.
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT"
+
+COMPOSE=(docker compose -f docker-compose.yml -f docker-compose.dev.yml -f docker-compose.profile.yml)
+PROXY_URL="${COUCH_AUTH_PROXY_URL:-http://127.0.0.1:8000}"
+RESULTS_PATH="${PERF_MEMORY_RESULTS_PATH:-test/perf/last-memory-results.json}"
+REPORT_PATH="${PERF_MEMORY_REPORT_PATH:-test/perf/last-memory-report.md}"
+# Default 5-minute soak; override with PERF_MEMORY_DURATION_SEC.
+export PERF_MEMORY_DURATION_SEC="${PERF_MEMORY_DURATION_SEC:-300}"
+
+die() {
+  echo "error: $*" >&2
+  exit 1
+}
+
+echo "==> starting stack with PROFILE=true (memory probe + optional expose-gc)"
+"${COMPOSE[@]}" up -d --build
+
+echo "==> waiting for ready"
+for i in $(seq 1 90); do
+  if curl -sf "${PROXY_URL}/_couch-auth-proxy/ready" >/dev/null; then
+    break
+  fi
+  if [[ "$i" -eq 90 ]]; then
+    "${COMPOSE[@]}" logs couch-auth-proxy couchdb || true
+    die "proxy not ready at ${PROXY_URL}"
+  fi
+  sleep 2
+done
+
+echo "==> checking PROFILE memory probe"
+curl -sf "${PROXY_URL}/_couch-auth-proxy/profile" >/dev/null ||
+  die "profile endpoint unavailable — is PROFILE=true on the proxy?"
+
+echo "==> running memory soak (${PERF_MEMORY_DURATION_SEC}s)"
+PERF_MEMORY_RESULTS_PATH="${RESULTS_PATH}" \
+  PERF_MEMORY_REPORT_PATH="${REPORT_PATH}" \
+  PERF_MEMORY_REQUIRED=1 \
+  pnpm exec vitest run --config vitest.perf.memory.config.ts
+
+echo "==> final profile snapshot"
+curl -sf "${PROXY_URL}/_couch-auth-proxy/profile" | tee test/perf/last-memory-profile.json >/dev/null
+echo
+echo "Wrote ${RESULTS_PATH}"
+echo "Wrote ${REPORT_PATH}"
+echo "Wrote test/perf/last-memory-profile.json"
+if [[ -f "${REPORT_PATH}" ]]; then
+  echo
+  sed -n '1,80p' "${REPORT_PATH}"
+fi

src/acl/cache.ts

@@ -110,6 +110,32 @@ export class AclCache {
     return this.dbs.values();
   }
 
+  /**
+   * Compact size counters for opt-in PROFILE memory scrapes.
+   * Does not walk row contents — O(dbs).
+   */
+  resourceStats(): {
+    aclDbs: number;
+    aclRows: number;
+    aclTombstones: number;
+    aclInflightEnsures: number;
+    aclInflightRefreshes: number;
+  } {
+    let aclRows = 0;
+    let aclTombstones = 0;
+    for (const state of this.dbs.values()) {
+      aclRows += state.acl.size;
+      aclTombstones += state.tombstones?.size ?? 0;
+    }
+    return {
+      aclDbs: this.dbs.size,
+      aclRows,
+      aclTombstones,
+      aclInflightEnsures: this.inflight.size,
+      aclInflightRefreshes: this.refreshInflight.size,
+    };
+  }
+
   /**
    * Read only the bucket policy needed by `/_all_dbs`.
    *

src/app.ts

@@ -20,6 +20,7 @@ import { bodyLimit } from "./middleware/bodyLimit.js";
 import { registerRoutes } from "./routes/register.js";
 import { jsonResponse } from "./proxy/forward.js";
 import { createLogger } from "./util/log.js";
+import { captureProcessMemory, tryForceGc, type ResourceStats } from "./util/memory.js";
 import { ProfileAggregator } from "./util/profile.js";
 
 const log = createLogger("app");
@@ -154,15 +155,22 @@ export function createApp(services: AppServices): Hono<AppEnv> {
   });
 
   /**
-   * Scrapeable phase-timing snapshot when `PROFILE=true`.
+   * Scrapeable phase-timing + memory snapshot when `PROFILE=true`.
    * Returns 404 when profiling is off so probes stay non-sensitive by default.
    */
   app.get("/_couch-auth-proxy/profile", (c) => {
     const agg = c.get("profileAggregator");
     if (!c.get("config").server.profile || !agg) {
       return jsonResponse({ error: "not_found", reason: "Profiling disabled" }, 404);
     }
-    return jsonResponse(agg.snapshot());
+    const acl = c.get("aclCache").resourceStats();
+    const session = c.get("sessions").resourceStats();
+    const resources: ResourceStats = { ...acl, ...session };
+    return jsonResponse({
+      ...agg.snapshot(),
+      memory: captureProcessMemory(),
+      resources,
+    });
   });
 
   /** Reset aggregated profile counters (load harness between phases). */
@@ -175,6 +183,22 @@ export function createApp(services: AppServices): Hono<AppEnv> {
     return jsonResponse({ ok: true });
   });
 
+  /**
+   * Best-effort V8 GC when the process was started with `--expose-gc`.
+   * Opt-in only (`PROFILE=true`); used by the memory-stability harness between samples.
+   */
+  app.post("/_couch-auth-proxy/profile/gc", (c) => {
+    if (!c.get("config").server.profile || !c.get("profileAggregator")) {
+      return jsonResponse({ error: "not_found", reason: "Profiling disabled" }, 404);
+    }
+    const ran = tryForceGc();
+    return jsonResponse({
+      ok: true,
+      gc: ran,
+      memory: captureProcessMemory(),
+    });
+  });
+
   registerRoutes(app, services.accessPolicy);
 
   // Keyed ACL view failures may happen after the DB gate. Keep their response

src/auth/session.ts

@@ -38,6 +38,17 @@ export class SessionResolver {
     this.cache = new LruMap(config.couch.sessionCacheMaxEntries);
   }
 
+  /**
+   * Compact size counters for opt-in PROFILE memory scrapes.
+   * Session cache is LRU-bounded (`SESSION_CACHE_MAX`); inflight should stay near 0 at rest.
+   */
+  resourceStats(): { sessionCacheEntries: number; sessionInflight: number } {
+    return {
+      sessionCacheEntries: this.cache.size,
+      sessionInflight: this.inflight.size,
+    };
+  }
+
   /**
    * Resolve identity from incoming request headers.
    * Missing credentials → anonymous. Couch 401 → anonymous (upstream may still reject).

src/util/memory.ts

@@ -0,0 +1,88 @@
+/**
+ * Opt-in process memory snapshots for `/_couch-auth-proxy/profile`.
+ *
+ * Only consulted when `PROFILE=true`. Cheap (`process.memoryUsage()`); no
+ * allocators or heap dumps — meant for long-running load harness scrapes.
+ */
+
+/** Bytes from `process.memoryUsage()`. */
+export type ProcessMemorySnapshot = {
+  rss: number;
+  heapTotal: number;
+  heapUsed: number;
+  external: number;
+  arrayBuffers: number;
+};
+
+/** Bounded in-process structure sizes correlated with expected heap growth. */
+export type ResourceStats = {
+  aclDbs: number;
+  aclRows: number;
+  aclTombstones: number;
+  aclInflightEnsures: number;
+  aclInflightRefreshes: number;
+  sessionCacheEntries: number;
+  sessionInflight: number;
+};
+
+/** Capture current process memory counters. */
+export function captureProcessMemory(): ProcessMemorySnapshot {
+  const m = process.memoryUsage();
+  return {
+    rss: m.rss,
+    heapTotal: m.heapTotal,
+    heapUsed: m.heapUsed,
+    external: m.external,
+    arrayBuffers: m.arrayBuffers,
+  };
+}
+
+/** Format bytes for harness / console output. */
+export function formatBytes(bytes: number): string {
+  const sign = bytes < 0 ? "-" : "";
+  const abs = Math.abs(bytes);
+  if (abs < 1024) {
+    const whole = Number.isInteger(abs) ? String(abs) : abs.toFixed(1);
+    return `${sign}${whole}B`;
+  }
+  if (abs < 1024 * 1024) return `${sign}${(abs / 1024).toFixed(1)}KiB`;
+  if (abs < 1024 * 1024 * 1024) return `${sign}${(abs / (1024 * 1024)).toFixed(2)}MiB`;
+  return `${sign}${(abs / (1024 * 1024 * 1024)).toFixed(2)}GiB`;
+}
+
+/** Human-readable memory + resource lines for harness logs. */
+export function formatMemorySnapshot(
+  memory: ProcessMemorySnapshot,
+  resources?: ResourceStats,
+  label = "process memory",
+): string {
+  const lines = [
+    `=== ${label} ===`,
+    `rss:           ${formatBytes(memory.rss)}`,
+    `heap_used:     ${formatBytes(memory.heapUsed)}`,
+    `heap_total:    ${formatBytes(memory.heapTotal)}`,
+    `external:      ${formatBytes(memory.external)}`,
+    `array_buffers: ${formatBytes(memory.arrayBuffers)}`,
+  ];
+  if (resources) {
+    lines.push(
+      `acl_dbs:       ${resources.aclDbs}`,
+      `acl_rows:      ${resources.aclRows}`,
+      `acl_tombstones:${resources.aclTombstones}`,
+      `session_cache: ${resources.sessionCacheEntries}`,
+      `session_inflight:${resources.sessionInflight}`,
+    );
+  }
+  return lines.join("\n");
+}
+
+/**
+ * Request a V8 GC when the process was started with `--expose-gc`.
+ * Returns false when GC is unavailable (typical production / default PROFILE).
+ */
+export function tryForceGc(): boolean {
+  const gc = (globalThis as { gc?: () => void }).gc;
+  if (typeof gc !== "function") return false;
+  gc();
+  return true;
+}

src/util/profile.ts

@@ -4,11 +4,14 @@
  * When enabled, middleware installs an AsyncLocalStorage request profile and
  * hot-path helpers accumulate wall time for auth / ACL / upstream / filter.
  * Aggregated stats are exposed via `/_couch-auth-proxy/profile` for the perf
- * harness; per-request phase ms are also attached to structured access logs.
+ * harness (including process memory + resource sizes); per-request phase ms
+ * are also attached to structured access logs.
  *
  * Disabled by default — zero ALS / timer cost on the hot path when off.
  */
 import { AsyncLocalStorage } from "node:async_hooks";
+import type { ProcessMemorySnapshot, ResourceStats } from "./memory.js";
+export type { ProcessMemorySnapshot, ResourceStats } from "./memory.js";
 
 /** Timed phases on the ACL proxy hot path. */
 export const PROFILE_PHASES = ["auth", "acl", "aclMiss", "upstream", "filter"] as const;
@@ -45,6 +48,12 @@ export type ProfileSnapshot = {
    * overlap or when other work is unattributed — use as a relative signal).
    */
   phaseShareOfMean: Record<ProfilePhase, number>;
+  /**
+   * Present on scrape responses from the HTTP probe (not on bare aggregator
+   * snapshots). Process `memoryUsage()` + in-process cache sizes.
+   */
+  memory?: ProcessMemorySnapshot;
+  resources?: ResourceStats;
 };
 
 type PhaseAccum = {