From 31db9df90073eaa139d9649ea58749c5cf0f14b3 Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Mon, 7 Sep 2026 15:30:02 -0400 Subject: [PATCH 01/19] docs: add prebuilt Caddy/CrowdSec toolchain image spec Extract the caddy-builder and crowdsec-builder Dockerfile stages into a digest-pinned, multi-arch ghcr.io/wikid82/charon-toolchain image so the ~14-min xcaddy compile runs once per security-relevant change instead of on every CI image build (currently forced via --no-cache-filter in 7 places, the root cause of the PR #1298 build-job timeout cancellations). Rev 2 incorporates the supervisor review: alternatives decision record, corrected security-refresh baseline, withdrawn overstated recurrence claim, pinned xcaddy plugins, guard-never-inert commit slicing, fork-reachable timeout preservation, and a failure-closed freshness guard. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- docs/plans/current_spec.md | 1836 ++++++++++++++---------------------- 1 file changed, 713 insertions(+), 1123 deletions(-) diff --git a/docs/plans/current_spec.md b/docs/plans/current_spec.md index 3c044f6a4..17fe72b44 100644 --- a/docs/plans/current_spec.md +++ b/docs/plans/current_spec.md @@ -1,10 +1,18 @@ -# Technical Spec — Uptime Monitoring at Scale +# Technical Spec — Prebuilt Caddy + CrowdSec Toolchain Image (CI Docker-build timeout fix) -**Status:** Draft for review -**Branch:** `feat/uptime-monitoring-scale` -**Delivery model:** One feature = one PR, sliced into ordered logical commits (see [Commit Slicing Strategy](#commit-slicing-strategy)). +**Status:** Revision 2 — for supervisor re-review +**Branch:** `feat/prebuilt-toolchain-image` +**Delivery model:** One feature = one PR, sliced into ordered logical commits (see [Commit Slicing Strategy](#12-commit-slicing-strategy)). **Author:** Planning (Principal Architect) -**Date:** 2026-08-27 +**Date:** 2026-09-07 +**Supersedes on merge:** the previous `current_spec.md` (Uptime Monitoring at Scale — already delivered). + +### Revision history + +| Rev | Date | Change | +|---|---|---| +| 1 | 2026-09-07 | Initial draft. | +| **2** | **2026-09-07** | **Supervisor "APPROVE WITH CHANGES (major)" — resolved 7 blocking items:** B1 added §2.6 Alternatives Considered (decision record); B2 corrected the true current recurrence baseline (daily via nightly, not weekly) and pulled a **daily** `schedule` toolchain rebuild into committed scope (§3.4.1, §3.8.1–3.8.2; workflow lands in Commit 1, security-rebuild reroute in Commit 5); B3 rewrote the overstated "fresh `go mod tidy` MVS" claim in §3.8/R3 to state accurately what the forced rebuild catches (base-image + pin-bump drift only); B4 pin the two unpinned xcaddy plugins + feed them to the key (§2.2, §3.2.1, §3.4.2, Commit 1); B5 reworked the Commit Slicing Strategy so the CVE recurrence guard is never inert — the `--no-cache-filter` is retargeted to `caddy-inline`/`crowdsec-inline` inside Commit 1 and every commit gate proves the guard is live; B6 reconciled §3.9 timeouts with §3.7 fork path (fork-reachable CVE-gate jobs stay at 20 min); B7 made `verify-toolchain-pin.sh` failure-closed on same-repo PRs. Folded non-blocking N1–N11. | --- @@ -12,1314 +20,896 @@ ### 1.1 Overview -Charon's uptime subsystem was built for a handful of monitors. It degrades non-linearly as monitor count grows: the user runs ~100 monitors today and wants comfortable headroom to **500**. Under load, every monitor's latency number inflates together (checks queue behind each other and behind the single DB connection), and the Uptime page is slow to load because each monitor card issues its own history request. +Every CI workflow that builds the Charon container image recompiles a **custom Caddy v2 binary** (via `xcaddy`, with in-place source patching of transitive dependencies) and a **custom CrowdSec agent** (`crowdsec` + `cscli`) **from source, from scratch, on every run**. The two Dockerfile stages that do this — `caddy-builder` (`Dockerfile:302`) and `crowdsec-builder` (`Dockerfile:577`) — are explicitly excluded from all layer caching by `--no-cache-filter` / `no-cache-filters` in six workflows plus the shared composite action. -This spec replaces the uptime **execution model** (global ticker → per-monitor scheduler + bounded worker pool), the **write model** (synchronous per-check DB writes → a buffered ingester mirroring `StatsIngester`), and the **read model** (N per-card history queries → one cached batch endpoint). It adds **heartbeat retention** (unbounded growth → hourly chunked pruner) and a small **admin config surface**. +Measured on the PR #1298 amd64 run: -### 1.2 Objectives & Goals (ranked — locked with user) - -1. **Uptime page + heartbeat history UI loads fast at 100+ monitors.** Target: `GET /api/v1/uptime/monitors/summary` p95 < 300 ms at 500 monitors; the page issues **one** history request regardless of monitor count. -2. **Faster detection via per-monitor configurable intervals.** The stored `Interval` field becomes authoritative, with a **30 s hard floor** and an admin global default for new/legacy monitors. -3. **Latency numbers stay stable under load.** A check's measured latency reflects the target's real response time, not scheduler/queue backlog. Achieved via a bounded worker pool, a shared keep-alive HTTP client, and moving DB writes off the check's critical path. -4. **Throughput (supporting requirement):** all due checks for 500 monitors complete within their interval under normal conditions, and degrade *gracefully* (some checks delayed, metric incremented) rather than collapsing when many targets are slow/down. - -### 1.3 Non-goals - -- Scaling beyond 500 monitors, or moving off SQLite. -- Heartbeat downsampling / rollup tables (retention is hard-delete only). -- Changing the notification *content* or provider routing (only *when* transition detection fires relative to buffered writes). -- Distributed / multi-node checking. -- Touching `SetMaxOpenConns(1)` or the main GORM pool (explicitly out of scope — see §2.2). - ---- - -## 2. Research Findings - -### 2.1 Existing architecture (as-is) - -| Concern | Current implementation | File / line | -|---|---|---| -| Scheduler | One global `time.NewTicker(1 * time.Minute)` in a `go func()` that also `time.Sleep(30s)` on boot; calls `SyncMonitors()` + `CheckAll()` every tick. `for range ticker.C` — **ignores `ctx`**, no graceful shutdown. | `backend/internal/api/routes/routes.go` ~652–690 | -| Per-monitor `Interval` | Stored on `models.UptimeMonitor.Interval` (seconds), default 60. **Read nowhere in the check path** — every monitor checks every 60 s. | `backend/internal/models/uptime.go:20` | -| Check fan-out | `CheckAll()` groups monitors by `UptimeHost`, then `go s.checkMonitor(m)` per monitor — **unbounded goroutines**, no pool. | `uptime_service.go:419–479` | -| Host TCP pre-check | `checkHost()` retries a dial `MaxRetries` (2) times with `time.Sleep(2 * time.Second)` between attempts — **blocks its goroutine up to ~4 s** on a down host, redundant with the `FailureThreshold` (2) cross-cycle debounce. | `uptime_service.go:524–645` | -| HTTP client | `network.NewSafeHTTPClient(...)` constructed **fresh per check** (`uptime_service.go:850`). `DisableKeepAlives: true`, `MaxIdleConns: 1` — full DNS + TCP + TLS every check. | `network/safeclient.go:410` | -| URL validation | `security.ValidateExternalURL()` does its own `net.Resolver{}.LookupIP` (Layer 1); `safeDialer` re-resolves at connect time (Layer 2, DNS-rebinding guard). Two lookups per HTTP check. | `security/url_validator.go:190,270` | -| DB | `SetMaxOpenConns(1)` — the whole backend is serialized through one SQLite connection (WAL, `busy_timeout=5000`, `synchronous=NORMAL`, `cache_size=-64000`). Every check does `s.DB.Create(&heartbeat)` + `s.DB.Save(&monitor)` synchronously, contending with all API traffic. | `database/database.go:140–145` | -| Heartbeats | `models.UptimeHeartbeat` — one row per monitor per check. Indexes on `MonitorID`, `Status`, `CreatedAt`, plus composite `idx_heartbeat_lookup (monitor_id, status, created_at)`. **No retention/pruning** — rows deleted only in `DeleteMonitor()` (`uptime_service.go:1319`). Unbounded growth. | `models/uptime.go:38–45` | -| History UI (N+1) | `frontend/src/pages/Uptime.tsx:24` — each `MonitorCard` runs its own `useQuery(['uptimeHistory', id])` → `GET /uptime/monitors/:id/history?limit=60` with `refetchInterval: 60000`. The monitor list refetches every 30 s. N monitors ⇒ N history requests/min, each a separate query through the one connection. `GetHistory` handler applies **no cap** to `limit`. | `Uptime.tsx:24`, `uptime_handler.go:58–69` | +| Stage | Cold build time | +|---|---| +| `caddy-builder` (xcaddy build + patch + rebuild) | **748 s (12.5 min)** | +| `crowdsec-builder` (clone + patch + 2× `xx-go build`) | **~330 s** combined | +| GHA cache export (`type=gha,mode=max`) | ~90 s | -### 2.2 Reference precedent — the Stats subsystem (mirror this) +`build-amd64` has `timeout-minutes: 15` with a nested `nick-fields/retry` `timeout_minutes: 15` (`docker-build.yml:403`, `:441`). The ~14-minute cold compile plus cache export blows the 15-minute budget; the integration jobs (`timeout-minutes: 20`) run out of budget once test work is stacked on top of the same cold compile. **PR #1298's four "failed" checks (`build-amd64`, `CrowdSec Bouncer Integration`, `Trivy Binary Scan`, `Cerberus Security Stack Integration`) were all CI job-timeout cancellations on the image build — not test or assertion failures.** -`ARCHITECTURE.md` §"Stats Subsystem" (~lines 356–362) and §"Database (SQLite + GORM)" (~line 596) establish the pattern this feature must copy: +The `--no-cache-filter` guards are deliberate (commits `5c046238`, `8cbc71f2`): the two builder stages patch pinned transitive dependencies **inside** the stage (`go get pkg@fixed`), and a build-arg bump does not reliably invalidate the GHA layer-cache key of a stage that only *consumes* that arg, so a restored stale layer keeps shipping a superseded, still-vulnerable dependency (this is exactly what produced the CVE-2026-45135 and 2026-09-04 grpc-go v1.83.0 recurrences). Removing the guards without another mechanism would reintroduce that class of silent regression. -- **`StatsIngester`** (`backend/internal/services/stats_ingester.go`, 157 lines): - - Non-blocking buffered channel `ingestCh` (`channelBufferSize = 1000`). - - `Send()` does a `select { case ch <- e: default: droppedCount.Add(1); log.Warn(...) }` — **drop-on-full**, counter exposed. - - `Run(ctx)` batches: flush every `flushInterval = 500ms` **or** when `batchSize = 100` rows accumulate, via `db.CreateInBatches(batch, batchSize)`. - - On `ctx.Done()` it **drains** the channel then flushes (crash/shutdown safety); `Stop()` is a second drain for post-`Run` cleanup. -- **`StatsService`** (`stats_service.go`, 303 lines): read-side aggregation with a `summaryCache` struct — `sync.Mutex` + `cachedValue` + `expiresAt`, `summaryCacheTTL = 30 * time.Second`. `GetSummary` checks cache first, runs grouped/windowed SQL, sets cache. -- **Health metric:** `StatsHandler.GetStatsHealth` → `GET /api/stats/health` → `gin.H{"dropped_count": n}` (`stats_handler.go:164`). -- **Wiring:** `routes.go` ~801–808 — `statsIngester := services.NewStatsIngester(db); go statsIngester.Run(ctx)` using the `ctx` from `Register(ctx, ...)`. +### 1.2 Objectives & Goals (ranked) -The uptime feature adds a directly analogous `UptimeIngester`, an `UptimeSummaryService` with a 30 s TTL cache, and a `GET /api/v1/uptime/health` endpoint. +1. **CI image builds stop timing out.** The `xcaddy` / CrowdSec compile must **not** run on the hot path of an ordinary app image build. Target: warm `build-amd64` completes in **< 8 min**; integration jobs **< 12 min**. +2. **The CVE-2026-84304-class recurrence guarantee is preserved or strengthened.** "Upstream ships a security fix, no repo pin changes" must still be caught on a defined cadence with an explicit alert path. +3. **Multi-arch is preserved.** `linux/amd64` and `linux/arm64` images keep getting a correctly cross-compiled Caddy/CrowdSec binary. +4. **A bumped pin can never silently ship an old toolchain.** A stale digest in the Dockerfile against a newer pin must fail a PR fast. +5. **Fork PRs, first-run bootstrap, and local `docker build` still work** without `packages: write` and without a published toolchain image. +6. **One source of truth for the build logic.** The `xcaddy` / CrowdSec build recipe must not be duplicated between the app Dockerfile and a separate toolchain Dockerfile. -### 2.3 Runtime & infra facts confirmed +### 1.3 Non-goals -- SQLite driver: `github.com/glebarez/sqlite` (pure-Go, wraps `modernc.org/sqlite`, SQLite 3.4x). **Window functions supported.** `DELETE ... LIMIT` is **not** compiled in — the pruner must use the `WHERE id IN (SELECT id ... LIMIT n)` subquery form. -- API base path: `/api/v1`; uptime routes live in the `management` group (`routes.go` ~610). E2E specs address them as `**/api/v1/uptime/...`. -- `Register(ctx context.Context, router, db, cfg)` (`routes.go:72`) — `ctx` is already threaded and used for `go statsIngester.Run(ctx)` / `go statsWSHub.Run(ctx)`. Uptime background goroutines will use the same `ctx`. -- Config: generic key/value `models.Setting` (`Key`, `Value`, `Type`, `Category`). `GET /api/v1/settings` returns a `map[key]value`; `POST /api/v1/settings` (`SettingsHandler.UpdateSetting`, admin-gated) writes one key. New `uptime.*` keys work through this endpoint with no new route; key-specific validation is added in `UpdateSetting` (precedent: `backup.*` rejection, `security.admin_whitelist` validation at `settings_handler.go:143–156`). -- Feature flag `feature.uptime.enabled` (bool Setting) already gates the subsystem and is `FirstOrCreate`d in `routes.go:641`. -- E2E: `tests/monitoring/uptime-monitoring.spec.ts` and `tests/a11y/uptime.a11y.spec.ts` use **`page.route` interception** (mock JSON), not a live backend. New E2E specs follow the same mock-response style. -- Frontend consumers of uptime data: `frontend/src/pages/Uptime.tsx` (list + per-card history), `frontend/src/components/UptimeWidget.tsx` (list only — `getMonitors`, no history; unaffected by the N+1 fix but may adopt the summary endpoint opportunistically). +- Changing *which* Caddy plugins or CrowdSec version are shipped, or any dependency pin values. +- Changing the runtime image contents, entrypoint, ports, or `internal/caddy` / `internal/cerberus` behavior. +- Reworking the `arm64` QEMU split in `docker-build.yml` (already done in a prior spec). +- Moving off GHCR or introducing a second registry. -### 2.4 External dependencies +### 1.4 EARS-style requirements -None added. All work is in existing packages (`net/http`, `context`, `time`, `sync`, GORM). One small **additive** option is added to `backend/internal/network/safeclient.go` (`WithKeepAlive(...)`) — no new module. +| # | Requirement (EARS) | +|---|---| +| R1 | **When** an app image build runs in CI or locally with the default build-args, the system **shall** obtain the Caddy and CrowdSec binaries by `COPY --from` a digest-pinned prebuilt toolchain image, **without** invoking `xcaddy` or compiling CrowdSec. | +| R2 | **When** any security-relevant toolchain input changes on a PR (the two builder-stage bodies, their consumed version ARGs incl. the two now-pinned xcaddy plugins, `xx` pin, the digest-pinned `golang`/`alpine` builder bases, or `.trivyignore`), the toolchain-image workflow **shall** run and the freshness-guard check **shall** fail until the Dockerfile's pinned toolchain digest matches the newly published image for those inputs. | +| R3 | **While** no repo pin has changed, the scheduled **daily** toolchain rebuild **shall** rebuild the toolchain image with `--no-cache --pull` and scan it with Trivy; it **catches base-image drift** (new `golang`/`alpine`/plugin-base CVEs picked up via `--pull` and the digest re-resolve) **and pin-bump drift**, and — if a new digest or a new CRITICAL/HIGH finding results — **shall** open a bot PR bumping the pinned digest and alert via a GitHub issue on failure. It does **not** independently discover upstream security fixes to *unpinned transitive* Go dependencies (see §3.8 — that gap exists identically today and is closed only by an explicit pin bump). | +| R4 | **Where** the builder lacks `packages: write` or the toolchain image is unavailable (fork PR, first bootstrap, offline local build), the system **shall** fall back to compiling the `caddy-inline` / `crowdsec-inline` stages from source, producing an equivalent binary. | +| R5 | **When** the toolchain image is built, it **shall** be published as a multi-arch manifest list covering `linux/amd64` and `linux/arm64`, each entry carrying the correctly cross-compiled binary. | +| R6 | **When** `--no-cache-filter caddy-builder` / `crowdsec-builder` (and the `no-cache-filters` input) are removed from all six workflows and the composite action, normal `type=gha` layer caching **shall** cover every remaining stage. | --- -## 3. Technical Specifications +## 2. Research Findings -### 3.0 Component map +### 2.1 Current build graph (verified) ``` - ┌────────────────────────────────────────────────────────┐ - │ Register(ctx, ...) — routes.go │ - │ (replaces the global 60s ticker go-func) │ - └───────┬───────────────┬───────────────┬────────────────┘ - go .Run(ctx) │ │ │ - ┌───────────────▼──┐ ┌────────▼────────┐ ┌───▼──────────────┐ - │ UptimeScheduler │ │ UptimeSyncLoop │ │ UptimePruner │ - │ tick ~5s │ │ tick ~5m + event│ │ tick 1h │ - │ monSchedule map │ │ SyncMonitors() │ │ chunked DELETE │ - │ hostSchedule map │ └─────────────────┘ │ + deferred INDEX │ - │ reads hostState ◄─────────────┐ └──────────────────┘ - └───────┬──────────┘ │ (RLock read: "is this host down?") - enqueue│ UptimeJob{Kind: Monitor|Host} (non-blocking, bounded chan cap 512) - ┌───────▼────────────────────────────────────────┐ - │ UptimeWorkerPool (fixed N, default 30) │ - │ - shared SSRF-safe keep-alive *http.Client │ ── HTTP/TCP/orthrus monitor check - │ - single-dial host TCP check (Kind==Host) │ - │ - monState map {status,failCount,lastChange, │ ── AUTHORITATIVE debounce state - │ lastNotifiedDown} (seeded from DB, the │ (NOT the scheduler DB snapshot) - │ source of truth for transition detection) │ - │ - hostState map {status,failCount,lastChange} │ ── written here, read by scheduler - │ - transition detection + notifications (SYNC, on the worker, before enqueue) - │ - host→down: synthesize child `down` CheckResults for the host's TCP monitors (SYNC) - └───────┬────────────────────────────────────────┘ - result │ CheckResult / HostCheckResult (non-blocking chan, drop-on-full) - │ pool is the SOLE sender → pool closes this channel at shutdown - ┌───────▼──────────────────────────────┐ - │ UptimeIngester (dumb writer only) │ - │ - batch INSERT uptime_heartbeats │ 500ms / 100 rows - │ - coalesced UPDATE uptime_monitors │ (status,latency,last_check,failure_count,…) - │ - coalesced UPDATE uptime_hosts │ — pure column copy from pre-computed result; - │ - DroppedCount() metric │ NO transition logic, NO fan-out - └──────────────────────────────────────┘ - - Read path: GET /api/v1/uptime/monitors/summary ──► UptimeSummaryService (30s TTL cache) - GET /api/v1/uptime/monitors/:id/history (paginated, capped) ──► UptimeService.GetMonitorHistory - GET /api/v1/uptime/health ──► UptimeIngester.DroppedCount() + pool queue depths + pool size +Dockerfile stages (944 lines total): + + xx (tonistiigi/xx:1.9.0, Dockerfile:73) ── cross-compile helper + │ + ├─► gosu-builder (:80, COPY --from=xx) + ├─► frontend-builder (:134, node:24) + ├─► backend-builder (:178, COPY --from=xx) + │ + ├─► caddy-builder (:302) FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine + │ NO `COPY --from=xx`. Pure Go cross-compile: + │ go install xcaddy → `xcaddy build` (Stage 1, generates go.mod) + │ → ~20× `go get pkg@fixed` security patches (Stage 2) + │ → module-cache source patches (celmatcher.go, bouncer) + │ → GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o /usr/bin/caddy ← 748 s + │ → embeds-version assertions (cel-go v0.29.x, grpc v1.83.1) + │ + ├─► crowdsec-builder (:577) FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine + │ COPY --from=xx / / (:578). CGO cross-compile: + │ xx-apk add gcc musl-dev musl → git clone crowdsec vX.Y + │ → ~20× `go get pkg@fixed` → sed patch debugger.go + │ → CGO_ENABLED=1 xx-go build crowdsec + cscli ← ~330 s + │ → xx-verify + │ + ├─► crowdsec-fallback (:713) FROM ${ALPINE_IMAGE} ── DEAD CODE (see note below) + │ + └─► final runtime (:751) FROM ${ALPINE_IMAGE} + COPY --from=caddy-builder /usr/bin/caddy /usr/bin/caddy (:807) + COPY --from=crowdsec-builder /crowdsec-out/crowdsec /usr/local/bin/crowdsec (:814) + COPY --from=crowdsec-builder /crowdsec-out/cscli /usr/local/bin/cscli (:815) + COPY --from=crowdsec-builder /crowdsec-out/config /etc/crowdsec.dist (:817) ``` -Two in-memory maps live on `UptimeWorkerPool` and are the crux of B2/B3: +Key finding: **`caddy-builder` does not use `xx`** — it is a plain `$BUILDPLATFORM` golang image doing `GOOS/GOARCH` cross-compilation with `CGO` disabled. `crowdsec-builder` **does** use `xx` because CrowdSec needs `CGO_ENABLED=1` (sqlite). **Both stages are `FROM --platform=$BUILDPLATFORM`**, so on `docker-build.yml`'s arm64 leg the *builder* stages have always run natively on the amd64 host and cross-compiled — **QEMU has only ever emulated the final arm64 stage's `RUN` lines**, never the Caddy/CrowdSec compile. This is what makes a native-amd64, no-QEMU multi-arch toolchain build possible (see §3.5). -- **`monState`** — authoritative per-monitor debounce state (`status`, `failureCount`, `lastStatusChange`, `lastNotifiedDown`). Seeded from `uptime_monitors` at pool start, then read-modify-written **synchronously by the worker** on every check result. Transition detection and notification dispatch read/write this map, **never** the scheduler's DB snapshot. The ingester's DB write of `status`/`failure_count` is a best-effort persistence *mirror* (used only to reseed on restart) — a dropped `CheckResult` cannot suppress a transition. -- **`hostState`** — authoritative per-host connectivity state (`status`, `failureCount`, `lastStatusChange`). Written synchronously by the worker running a `Kind==Host` job; read (RLock) by the scheduler to decide whether to skip enqueueing a down host's TCP monitors. The ingester's `uptime_hosts` write is likewise a mirror. +**Correction (was wrong in Rev 1): `crowdsec-fallback` (`:713`) is dead code.** Verified: no `COPY --from=crowdsec-fallback`, no `FROM crowdsec-fallback`, and no `--target crowdsec-fallback` anywhere in the repo. It is never built and never consumed. The final stage copies unconditionally from `crowdsec-builder` (`:814-817`). Rev 1's §3.2.3 claim that "`crowdsec-fallback` is selected by the existing arch logic" was incorrect. **Action:** Commit 1 deletes the `crowdsec-fallback` stage (`:713-748`). `CROWDSEC_RELEASE_SHA256` (`:22`, re-declared at `:586`) is *only* used by the tarball `sha256sum -c` inside that stage — after deletion the global `ARG` and the `:586` re-declaration are dead too and are removed in the same commit. `crowdsec-inline` clones from git (`git clone --branch "v${CROWDSEC_VERSION}"`), so it is unaffected. **Verify at implementation time** whether `CROWDSEC_RELEASE_SHA256` has its own updater workflow (grep `.github/workflows/` for `CROWDSEC_RELEASE_SHA256`); if so, delete it in the same commit. (Per CLAUDE.md "delete dead code immediately". If the reviewer prefers to keep `crowdsec-fallback` as a deliberate escape hatch, the fallback position is: leave it untouched and simply exclude it from the toolchain key — it does not affect the shipped binary. Planning's recommendation is deletion.) -New files (all `backend/internal/services/` unless noted): - -| File | Contents | -|---|---| -| `uptime_scheduler.go` | `UptimeScheduler` — due-selection loop over **two** in-memory maps (`monSchedule`, `hostSchedule`), jittered cold-start backfill, batched `next_check_at` write-back (monitors only), host-down short-circuit consult against `pool.hostState`, `Rehydrate()` for post-restore resync. | -| `uptime_worker_pool.go` | `UptimeWorkerPool` — fixed worker set, bounded job channel, shared keep-alive client, `Kind`-discriminated jobs (monitor / host), the authoritative `monState` + `hostState` maps (seeded from DB), synchronous transition detection + notification + host-down child fan-out, `Enqueue`/`TryEnqueue`, shutdown WaitGroup + closes `results`. | -| `uptime_ingester.go` | `UptimeIngester` — buffered heartbeat writes + coalesced monitor/host column updates (mirrors `StatsIngester`). **Dumb writer**: pure column copy from pre-computed results, no transition logic, no fan-out. Receives on a channel the pool owns and closes. | -| `uptime_pruner.go` | `UptimePruner` — hourly chunked retention delete (wider first-pass pause) + periodic `PRAGMA optimize` + deferred `CREATE INDEX IF NOT EXISTS idx_heartbeat_monitor_created`, retried at the end of every clean+caught-up pass until it lands. | -| `uptime_summary_service.go` | `UptimeSummaryService` — one windowed query for recent beats + monitor metadata, 30 s TTL cache. | -| `uptime_check.go` | Extracted pure check logic: `runCheck(job, client) rawResult` and `runHostCheck(job, dialer) rawResult` (no DB writes, no client construction, no state-map access) — refactored out of `uptime_service.go:checkMonitor` / `checkHost`. Debounce + transition + fan-out live in the worker (`uptime_worker_pool.go`), which calls these. | -| `backend/internal/network/safeclient.go` | **edit** — add `WithKeepAlive(maxIdle, perHost int, idleTimeout time.Duration) Option`. | -| `backend/internal/models/uptime.go` | **edit** — add `NextCheckAt` field (+ index) to `UptimeMonitor` **only**. `UptimeHeartbeat` tags unchanged (index created lazily by the pruner). | -| `backend/internal/services/uptime_service.go` | **edit** — `SyncAndCheckForRemoteServer` / `SyncMonitorForRemoteServer`; check path routed through ingester; `checkHost` de-blocked; `GetMonitorHistory` gains `before` + cap. | -| `backend/internal/api/handlers/uptime_handler.go` | **edit** — `Summary`, `Health` handlers; interval-floor validation in `Create`/`Update`; cap + `before` on `GetHistory`; `CheckMonitor` enqueues via pool. | -| `backend/internal/api/handlers/remote_server_handler.go` | **edit** — `NewRemoteServerHandler` gains nil-guarded `*services.UptimeService`; create/update/delete drive targeted monitor sync + cleanup. | -| `backend/internal/services/backup_service.go` | **edit** — `RestoreBackupSafe` reconcile step calls `UptimeScheduler.Rehydrate()` after a live DB restore (§3.9 / S6). | -| `backend/cmd/api/main.go` | **edit** — `migrate` CLI: warning log + eager unconditional `CREATE INDEX IF NOT EXISTS idx_heartbeat_monitor_created` (S7). | -| `backend/internal/api/routes/routes.go` | **edit** — replace ticker go-func with the background components (scheduler, sync loop, worker pool, ingester, pruner); register `/uptime/monitors/summary` + `/uptime/health`; seed 3 `uptime.*` Settings defaults; pass `uptimeService` into `NewRemoteServerHandler`. | -| `frontend/src/api/uptime.ts` | **edit** — `getMonitorsSummary(beats = 30)`, `MonitorSummary`/`BeatDTO` types, `before` on history; `interval` already present. | -| `frontend/src/pages/Uptime.tsx` | **edit** — list page owns one summary query; `MonitorCard` reads from props; `BEAT_BAR_SLOTS = 30`; interval field in create/edit forms with 30 s floor. | -| `frontend/src/pages/SystemSettings.tsx` | **edit** — new admin "Uptime Monitoring" card (3 `uptime.*` fields, bounds validation, restart note, feature-flag gated). | -| `frontend/src/hooks/useUptimeSummary.ts` | **new** — React Query hook wrapping `getMonitorsSummary(30)`. | +### 2.2 Version ARGs consumed by the builder stages (verified line numbers) ---- +| ARG | Global default (line) | Re-declared in | +|---|---|---| +| `GO_VERSION` | `1.27.1` (`:13`) | base of both stages (`FROM golang:${GO_VERSION}-alpine` — **moving tag, see N4 below**) | +| `ALPINE_IMAGE` | `alpine:3.24.1@sha256:28bd…` (`:16`) | toolchain-runtime base (already digest-pinned) | +| `CROWDSEC_VERSION` | `1.8.1` (`:20`) | caddy `:318`, crowdsec `:585`, fallback `:720` | +| `CROWDSEC_RELEASE_SHA256` | `deae1f43…` (`:22`) | crowdsec `:586`, fallback `:721` | +| `EXPR_LANG_VERSION` | `1.17.8` (`:26`) | caddy `:313`, crowdsec `:587` | +| `XNET_VERSION` | `0.58.0` (`:28`) | caddy `:314`, crowdsec `:588` | +| `XCRYPTO_VERSION` | `0.56.0` (`:33`) | caddy `:315`, crowdsec `:589` | +| `KLAUSPOST_COMPRESS_VERSION` | `1.20.0` (`:38`) | caddy `:316`, crowdsec `:590` | +| `GRPC_VERSION` | `1.83.1` (`:44`) | caddy `:317`, crowdsec `:591` | +| `CADDY_VERSION` | `2.11.4` (`:56`) | caddy `:305` | +| `CADDY_CANDIDATE_VERSION` | `2.11.4` (`:58`) | caddy `:306` | +| `CADDY_USE_CANDIDATE` | `0` (`:59`) | caddy `:307` | +| `CADDY_PATCH_SCENARIO` | `B` (`:60`) | caddy `:308` | +| `CADDY_SECURITY_VERSION` | `1.1.64` (`:62`) | caddy `:309` | +| `CORAZA_CADDY_VERSION` | `2.6.0` (`:64`) | caddy `:310` | +| `XCADDY_VERSION` | `0.4.7` (declared inside stage, `:~311`) | caddy only | +| `xx` image | `tonistiigi/xx:1.9.0@sha256:c64defb9…` (`:73`) | crowdsec `:578` | +| **`CADDY_GEOIP2_VERSION`** | **NEW — see B4** | caddy `:391` — currently `--with github.com/zhangjiayin/caddy-geoip2` with **no `@version`** | +| **`CADDY_RATELIMIT_VERSION`** | **NEW — see B4** | caddy `:392` — currently `--with github.com/mholt/caddy-ratelimit` with **no `@version`** | + +**B4 — two xcaddy plugins are unpinned (`Dockerfile:391-392`).** `--with github.com/zhangjiayin/caddy-geoip2` and `--with github.com/mholt/caddy-ratelimit` carry no `@version`, so `xcaddy` resolves "latest" at build time. The literal Dockerfile text never changes when those projects tag a release, so `toolchain-key.sh` would not notice, and a stale toolchain image would be reused when an upstream plugin fix actually warrants a rebuild. **Fix (Commit 1, preferred):** add `ARG CADDY_GEOIP2_VERSION=` and `ARG CADDY_RATELIMIT_VERSION=` near `:64` with `# renovate: datasource=go` annotations, and change lines `:391-392` to `--with github.com/zhangjiayin/caddy-geoip2@v${CADDY_GEOIP2_VERSION}` / `--with github.com/mholt/caddy-ratelimit@v${CADDY_RATELIMIT_VERSION}`. Resolve the current versions at implementation time via `xcaddy`'s build log or `go list -m` in the existing `caddy-inline` module cache. Both ARGs join the §3.4.2 key input set. + +**N4 — `golang:${GO_VERSION}-alpine` is a moving tag.** Only `GO_VERSION` (the minor, e.g. `1.27.1`) feeds the key; the underlying `-alpine` digest floats and, post-change, the app hot path no longer `--pull`s it (only the toolchain workflow does). A silent `golang:1.27.1-alpine` rebuild upstream (new Alpine base, patched toolchain) would not change the key. **Fix (Commit 1):** digest-pin both builder-stage bases — `FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine@sha256: AS caddy-inline` (and `crowdsec-inline`) — with a `# renovate: datasource=docker depName=golang` annotation, and include the pinned digest line in the key input set. The **daily** toolchain rebuild's `--pull` + Renovate digest bumps then keep it fresh; the app build inherits it transitively through the pinned toolchain image — the intended daily-cadence refresh path for builder-base drift (the app hot path deliberately does not re-pull it). + +Also inside the stages: many *literal* pinned versions in `go get` lines (e.g. `go-jose/v3@v3.0.5`, `cel-go@v0.29.2`, `quic-go@v0.60.0`, `golang.org/x/mod@v0.40.0`, `ipstore@v0.4.0`). Because these are literals in the stage body, the **content hash of the stage text** (not just the ARG list) must feed the toolchain tag key (§3.4). + +### 2.3 `--no-cache-filter` / `no-cache-filters` occurrences (verified — full removal list) + +| # | File:line | Form | Job / context | +|---|---|---|---| +| 1 | `.github/workflows/docker-build.yml:463-464` | raw `--no-cache-filter caddy-builder` / `crowdsec-builder` | `build-amd64` (`nick-fields/retry` → raw `docker buildx build`) | +| 2 | `.github/workflows/docker-build.yml:549-550` | raw `--no-cache-filter …` | `build-arm64` | +| 3 | `.github/workflows/security-pr.yml:164` | `no-cache-filters: caddy-builder,crowdsec-builder` (composite input) | `Build Docker image (Local)` step, job `timeout-minutes: 20` (`:32`) | +| 4 | `.github/workflows/supply-chain-pr.yml:261` | `no-cache-filters:` (composite input) | `Build Docker image (Local)` step, job `timeout-minutes: 20` (`:34`) | +| 5 | `.github/workflows/e2e-tests-split.yml:224` | `no-cache-filters:` on `docker/build-push-action` (`:215`) | `build` job | +| 6 | `.github/workflows/nightly-build.yml:243` | `no-cache-filters:` on `docker/build-push-action` (`:229`) | `Build and push Docker image` (multi-arch) | +| 7 | `.github/actions/build-charon-image/action.yml:15` (input decl) + `:52` (passthrough) | `no-cache-filters` composite input, default `''` | consumed by `cerberus-integration.yml:34`, `crowdsec-integration.yml:34`, `waf-integration.yml:34`, `rate-limit-integration.yml:34` (none of those four override it today) | -### 3.1 Component A — Real per-monitor scheduler +The composite action's own doc comment (`action.yml:16-33`) instructs CVE-scan callers to set `no-cache-filters: caddy-builder,crowdsec-builder`; that comment must be rewritten (§3.6). -#### 3.1.1 Model change +### 2.4 Existing patterns to reuse -`models.UptimeMonitor` gains: +- **Weekly security rebuild:** `.github/workflows/security-weekly-rebuild.yml` — `schedule: '0 12 * * 2'` (Tue 12:00 UTC) + `workflow_dispatch{force_rebuild}`, `timeout-minutes: 60`, `no-cache: ${{ schedule || force_rebuild }}`, `pull: true`, publishes `ghcr.io/wikid82/charon:security-scan-YYYYMMDD`, Trivy CRITICAL/HIGH gate + SARIF upload + JSON artifact + failure `::warning::`. **This spec repurposes this workflow to rebuild the *toolchain* image** rather than a throwaway app image (it currently scans an image nobody consumes). +- **Bot-PR-bumps-a-pin:** `.github/workflows/update-geolite2.yml` — weekly cron + `workflow_dispatch`, downloads upstream, `sed -i` the `ARG …_SHA256=` line in the Dockerfile, `docker build --check` syntax gate, `peter-evans/create-pull-request@v8` targeting `base: development`, `branch: bot/update-geolite2-checksum`, labels `dependencies/automated/docker`, failure → `actions/github-script` opens an issue. Commits `15ca90b8` / `94b93fdf` are live examples. **Reuse verbatim structure for the digest-bump bot (§3.4.3).** +- **Toolchain-bump scripts:** `scripts/update-go-toolchain.sh`, `scripts/update-node-toolchain.sh`, `scripts/caddy-compat-matrix.sh` — house style for a `scripts/*.sh` helper invoked by CI. +- **Renovate regex managers** already track every `ARG` above via `# renovate:` annotations — must be preserved (§3.3). -```go -// NextCheckAt is the wall-clock time this monitor is next due for a check. -// Zero value ⇒ "due now" (legacy rows and freshly-created monitors). -NextCheckAt time.Time `json:"next_check_at" gorm:"index"` -``` +### 2.5 Constraints from `CLAUDE.md` / `ARCHITECTURE.md` -Migration: `&models.UptimeMonitor{}` is already in the `db.AutoMigrate(...)` list (`routes.go:118`). GORM adds the nullable column + index automatically. `uptime_monitors` is small (≤ 500 rows) so this is sub-millisecond. No data backfill needed — a zero `NextCheckAt` is treated as "due". - -#### 3.1.2 `UptimeScheduler` - -```go -type UptimeScheduler struct { - db *gorm.DB - pool *UptimeWorkerPool - cfg *uptimeConfig // hot-reloading snapshot (see §3.6) - tick time.Duration // default 5s - monSchedule map[string]time.Time // monitorID -> next due (in-memory source of truth) - hostSchedule map[string]time.Time // hostID -> next due (in-memory only; NOT persisted) - hostMinInt map[string]int // hostID -> min(enabled child monitor intervals), clamped - writeback map[string]time.Time // pending uptime_monitors.next_check_at persists - known map[string]struct{} // monitorIDs already hydrated (for the new-monitor re-scan) - mu sync.Mutex - now func() time.Time // injectable clock for tests -} +- All frontend in `frontend/`, backend in `backend/` — unaffected (this is CI/build only). +- Conventional commits; `(security)` scope only for genuine security work, subject line vague. The digest-bump and freshness-guard commits *are* security-relevant — use `feat(security):` / `fix(security):` with vague subjects (e.g. `feat(security): pin bundled proxy toolchain to a scanned prebuilt image`). +- Weekly `nightly → main` promotion PRs merge via **merge commit**. This feature's PR targets `development` (normal flow) — **confirmed it does not touch `weekly-nightly-promotion.yml`** and imposes no new constraint on the promotion merge method. (`weekly-nightly-promotion.yml` carries the app image through unchanged; the toolchain digest pin travels with the Dockerfile like any other line.) +- `ARCHITECTURE.md` §"Deployment Architecture / Multi-Stage Dockerfile" (`:1082`), §"Infrastructure" table (`:158`), §"Directory Structure" (`:286`), §"Layer 2: CrowdSec Integration" (`:780`) must be updated (§9). +- **Ignore-file check (CLAUDE.md "Ignore Files"):** the new files are `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh`, `scripts/tests/toolchain-key.bats`, `.github/workflows/toolchain-image.yml`, `docs/ci/toolchain-image.md`. `.dockerignore` already excludes `.github/`, `docs/`, `scripts/` is not copied into the image context by any `COPY` (the Dockerfile only `COPY`s `backend/`, `frontend/`, `.docker/`) → **no `.dockerignore` change needed**. `.gitignore` — these are source files that must be committed; none matches an existing ignore glob (`scripts/tests/` is new, not ignored) → **no `.gitignore` change needed**. `.codecov.yml` — shell/bats and YAML carry no Go/TS coverage; not in any coverage path → **no `.codecov.yml` change needed**. This is recorded explicitly per CLAUDE.md. -func (s *UptimeScheduler) Run(ctx context.Context) // launched as `go s.Run(ctx)` -func (s *UptimeScheduler) Rehydrate() // re-runs cold-start hydration; called after a live DB restore (§3.9) -``` +--- -**Cold-start hydration (`hydrate()`, run once at `Run` entry and again on `Rehydrate()`):** +## 2.6 Alternatives Considered (Decision Record) -1. **Monitors** — `SELECT id, interval, enabled, next_check_at, uptime_host_id FROM uptime_monitors WHERE enabled = true`. - - For each: `effInterval = clampInterval(interval, cfg)` (§3.6.2); assign `monSchedule[id]` with **jittered backfill**: - - `next_check_at` in the future → keep it. - - past or zero → `due = now + rand(0s, min(effInterval, backfillWindow))`, `backfillWindow = 60s`. - - This spreads past-due monitors uniformly over the first 60 s after boot (≈ `monitors/60` enqueues/s) instead of a 500-wide first tick. - - Persist the backfilled monitor due-times in one batched write (see write-back below). -2. **Hosts** — `SELECT uptime_host_id AS id, MIN(interval) AS min_interval FROM uptime_monitors WHERE enabled = true AND uptime_host_id IS NOT NULL GROUP BY uptime_host_id`. - - `hostMinInt[id] = clampInterval(min_interval, cfg)`. - - `hostSchedule[id] = now + rand(0s, min(hostMinInt[id], backfillWindow))`. - - Host due-times are **in-memory only** — no column is added to `uptime_hosts`, no write-back. Hosts are few (one per distinct upstream), so a cold-start host-check wave is trivially small; jitter still applies for tidiness. -3. `known` = the set of hydrated monitor IDs. +The user has confirmed **approach A (prebuilt toolchain image)**. This section records the lighter alternative that was weighed against it, why that alternative is genuinely viable, and the concrete grounds on which A was still chosen — so the decision is auditable rather than assumed. -`Rehydrate()` re-runs `hydrate()` under `s.mu`, discarding stale in-memory schedule entries and rebuilding from the (restored) DB. It also calls `pool.ReseedState()` (§3.2.1) so the debounce maps re-sync. See §3.9. +### Alternative B — Keep the two stages inline; replace `--no-cache-filter` with a content-hash-keyed buildx GHA cache scope -**Per-tick loop (`ticker := time.NewTicker(s.tick)`):** +**Mechanism.** Leave `caddy-builder` / `crowdsec-builder` exactly where they are in the Dockerfile. Delete every `--no-cache-filter caddy-builder,crowdsec-builder`. In its place, give the two expensive stages their own dedicated GHA cache scope whose key is the content hash of the pin set (the same `scripts/toolchain-key.sh` output proposed for approach A): ``` -select { -case <-ctx.Done(): - flushWriteback() // best-effort final persist - return // STOP enqueuing — first link of the teardown chain (§3.1.4) -case <-ticker.C: - if !featureEnabled() { continue } // feature.uptime.enabled, cached 60s - - // (a) HOST pass — connectivity pre-checks - hostDue := hostIDs where hostSchedule[id] <= now() (cap 200/tick) - for _, hid := range hostDue: - host := loadHostSnapshot(hid) // batched SELECT, one query - if s.pool.TryEnqueue(UptimeJob{Kind: JobHostCheck, Host: host}): - hostSchedule[hid] = now() + durSecs(hostMinInt[hid]) - // else: leave due, retried next tick - - // (b) MONITOR pass - monDue := monitorIDs where monSchedule[id] <= now() (sorted by due asc, cap 200/tick) - snaps := loadJobSnapshots(monDue) // one batched SELECT WHERE id IN (...) - for _, job := range snaps: - // host-down short-circuit: skip TCP-type monitors whose host is known-down. - // hostState is written by the host-check worker; scheduler only reads (RLock). - if job.Monitor.Type == "tcp" && job.Monitor.UptimeHostID != nil { - if st, ok := s.pool.HostState(*job.Monitor.UptimeHostID); ok && st.Status == "down" { - next := now() + durSecs(clampInterval(job.Monitor.Interval, cfg)) - monSchedule[job.Monitor.ID] = next; writeback[job.Monitor.ID] = next - continue // no enqueue; the host check drives recovery. Synthetic - // `down` heartbeat was already written at the transition. - } - } - if s.pool.TryEnqueue(UptimeJob{Kind: JobMonitorCheck, Monitor: job.Monitor}): - next := now() + durSecs(clampInterval(job.Monitor.Interval, cfg)) - monSchedule[job.Monitor.ID] = next; writeback[job.Monitor.ID] = next - // else: leave due, retried next tick; pool.EnqueueDropped()++ - - // (c) new-monitor / disabled reconcile — every 6th tick (~30s) - if tickCount%6 == 0 { rescan() } - - flushWriteback() // one batched UPDATE per tick (grouped by value) -} +KEY=$(scripts/toolchain-key.sh) # caddy-crowdsec- +docker buildx build \ + --cache-from type=gha,scope=charon-app \ + --cache-to type=gha,mode=max,scope=charon-app \ + --cache-from type=gha,scope=builders-${KEY} \ + --cache-to type=gha,mode=max,scope=builders-${KEY} \ + ... ``` -- **`loadJobSnapshots`**: one `SELECT ... FROM uptime_monitors WHERE id IN ()` per tick. The snapshot supplies the worker's **static** fields (`Type`, `URL`, `MaxRetries`, `UptimeHostID`, `UpstreamHost`, `Interval`, `Enabled`). It also carries the dynamic columns (`Status`, `FailureCount`, `LastStatusChange`, `LastNotifiedDown`) but the worker **does not** use them for debounce — those come from the pool's authoritative `monState` map (§3.2.1 / §3.3.3). The snapshot's dynamic columns are only a diagnostic breadcrumb. -- **`loadHostSnapshot`**: one batched `SELECT ... FROM uptime_hosts WHERE id IN ()` per tick; supplies the host's identity and the ports its monitors dial (via a joined `uptime_monitors`/`proxy_hosts` lookup, exactly as `checkHost` does today). -- **Write-back batching** (monitors only): `flushWriteback()` groups `writeback` entries by identical `next` value and emits `UPDATE uptime_monitors SET next_check_at = ? WHERE id IN (?)` per group in one transaction — 1–3 statements per 5 s tick regardless of monitor count. On failure it logs and retries next tick (in-memory `monSchedule` is the runtime truth; a lost write-back risks one duplicate check after a crash, absorbed by cold-start jitter). -- **Manual check** (`POST /uptime/monitors/:id/check`) bypasses the schedule: handler calls `pool.Enqueue(ctx, job)` (blocking, 2 s timeout → 503) directly, does not touch `next_check_at`. It goes through the same `monState` lock as scheduled checks (§3.3.3), so a manual-vs-scheduled race no longer double-counts or under-counts the failure streak — whichever worker takes the lock first increments; the second sees the updated count. -- **New / re-enabled monitor**: `CreateMonitor` and `UpdateMonitor(enabled=true)` set `NextCheckAt = now()`; `rescan()` (every ~30 s) picks up `WHERE enabled = true AND id NOT IN (known)`, hydrates them (jittered), recomputes affected `hostMinInt`, and calls `pool.EnsureMonitorState(id)` to seed a `monState` entry. -- **Disabled monitor**: dropped from `monSchedule` / `known` on `rescan()`; the per-tick due scan is in-memory so it simply stops being enqueued. The worker also re-checks `job.Monitor.Enabled` and emits nothing if false (guards the ≤ 30 s race window). +When a pin moves, `KEY` changes, the `builders-` scope is a guaranteed miss, and the stage recompiles exactly once; every subsequent build on that `KEY` restores the layer. When a pin does **not** move, the layer is restored and no compile happens. + +**What Alternative B genuinely delivers — stated fairly:** -#### 3.1.3 `SyncMonitors` off the hot path — `UptimeSyncLoop` +- It **does** fix the #1298 timeouts in the common case: after the first build on a given `KEY`, every PR/CI build restores the `caddy-builder` / `crowdsec-builder` layers from `builders-` and skips the ~14 min compile. +- It **preserves the exact pin-bump recurrence guarantee**: the cache key is derived from the pin content, so a bumped `CADDY_VERSION` (or any tracked ARG, or any edit to the stage body) forces a clean recompile — the same property approach A's freshness guard enforces, achieved without a guard because the key *is* the cache identity. +- It is **~half the work**: no new image, no new registry package, no `toolchain-image.yml`, no digest-bump bot, no `verify-toolchain-pin.sh`, no fork-fallback selector stages, no `packages: read` cross-workflow plumbing. Roughly Commits 1, 3 and 6 of approach A's plan, and no new failure surface (GHCR availability, private-package permissions, bot-PR merge latency). +- Local `docker build` is unaffected — no image pull, no login. -- Its own goroutine, `time.NewTicker(5 * time.Minute)`, `ctx`-aware. -- Also invoked opportunistically on mutation: - - **Proxy hosts** (existing): `ProxyHostHandler` calls `go uptimeService.SyncAndCheckForHost(hostID)` on create, `SyncMonitorForHost(hostID)` on update, and iterates `WHERE proxy_host_id = ?` → `DeleteMonitor` on delete. - - **Remote servers** (new, this PR): mirror the proxy-host pattern exactly. `UptimeService` gains three methods, analogous to the existing `SyncAndCheckForHost` / `SyncMonitorForHost` / delete cleanup: - - `SyncAndCheckForRemoteServer(remoteServerID uint)` — ensure a monitor exists for the remote server (create if missing, using the same target-type/URL derivation `SyncMonitors` already does for `RemoteServer` rows — `tcp` host:port, or `http(s)://` / `orthrus` per `ConnectionType`), then run an immediate check. Per-server mutex via the existing `hostMutexes` map (key `remote-`). Feature-flag gated like `SyncAndCheckForHost`. - - **Orthrus remote servers with a not-yet-bound agent UUID:** when `ConnectionType == ConnectionTypeOrthrus` and `OrthrusAgentUUID` is `nil`/empty at create time, `SyncAndCheckForRemoteServer` **returns silently — no error, no monitor row created** (mirrors `SyncMonitors`'s existing `continue` at `uptime_service.go:300`). The `UptimeSyncLoop` (below) creates the monitor on a later pass once the agent connects and the UUID is persisted. This is the decided behavior, not a placeholder. - - `SyncMonitorForRemoteServer(remoteServerID uint) error` — update the linked monitor's `Name`/`Type`/`URL`/`Enabled`/`UpstreamHost` from current `RemoteServer` fields; no-op (nil) if no monitor exists. - - Auto-created monitors (proxy-host and remote-server alike) are created with the interval resolved from `uptime.default_interval_seconds` at write time, **not** a hardcoded 60 — see §3.6.3 (S3). - - Delete cleanup runs inline in the handler (mirrors `ProxyHostHandler.Delete` at `proxy_host_handler.go:755-761`): `uptimeService.DB.Where("remote_server_id = ?", id).Find(&monitors)` → `DeleteMonitor(m.ID)` for each. - - **Wiring:** `RemoteServerHandler` currently has no `UptimeService` reference. `NewRemoteServerHandler(service, ns)` (`remote_server_handler.go:24`) gains a third param `uptimeService *services.UptimeService` (nil-guarded, same as `ProxyHostHandler`). Call sites: `routes.go:897`. `RemoteServerHandler.Create` → `go h.uptimeService.SyncAndCheckForRemoteServer(server.ID)`; `.Update` → `go h.uptimeService.SyncMonitorForRemoteServer(server.ID)` (log on error); `.Delete` → inline monitor cleanup before `h.service.Delete(...)`. - - The 5-minute `UptimeSyncLoop` remains the backstop for any mutation path that misses the targeted hook (e.g. direct DB edits, Orthrus agent-UUID late-binding). -- `CleanupStaleFailureCounts()` runs once at boot (kept, via the existing `runInitialUptimeBootstrap` path, minus `CheckAll()`). +**Why approach A is still chosen — concrete grounds:** -#### 3.1.4 Graceful shutdown — explicit teardown handshake +1. **GHA cache eviction makes Alternative B's timeout fix unreliable.** GitHub Actions caches (`type=gha`) share a **10 GB per-repository LRU budget**. This repo already runs `gh_cache_cleanup.yml` and its existing timeout comments explicitly cite "cold GHA cache (first run / post-eviction) is a full ~10–14 m image build" (`security-pr.yml:32`, `supply-chain-pr.yml:34`, the integration workflows' `:29`). A `mode=max` multi-stage image cache for Charon is large; the `builders-` scope competes with `docker-build-amd64`, `docker-build-arm64`, `charon-integration-image`, `charon-app`, npm, Go build caches, and the e2e image tarball for that 10 GB. On a busy week the `builders-` entry is evicted between runs and the **next** PR eats a cold ~14 min compile again — i.e. Alternative B reduces the *frequency* of timeout-class builds but does not *eliminate* them, which is the actual acceptance bar (§5 AC #2: no timeout across 3 consecutive runs, and none thereafter). A digest-pinned image in GHCR's package store is **not** subject to the Actions cache LRU — it is pulled, not cache-restored — so approach A removes the cold-build possibility entirely rather than making it rarer. -Sharing `ctx` is not enough — an in-flight worker that `emit`s a `CheckResult` **after** the ingester has already returned on `ctx.Done()` loses that result's persistence (and if it was a transition, the in-memory `monState` has it but the DB never will, so a later restart reseeds slightly stale — bounded, see §3.9). The teardown is therefore an **ordered chain enforced by channel ownership**, not five components independently reacting to `ctx`: +2. **Trivy scans a small, stable, isolated artifact.** With approach A, the weekly/daily security scan targets `ghcr.io/wikid82/charon-toolchain` — two binaries plus an Alpine base, a stable surface whose findings map directly to the bundled Caddy/CrowdSec supply chain. With Alternative B there is no separate artifact: every scan re-derives bundled-binary findings from the full application image on every run, mixed with app-layer and base-image findings, and there is no way to pin/attest "the bundled toolchain that was scanned green on date X" independently of the app image. -1. **`UptimeScheduler`** sees `ctx.Done()` first thing in its select, does a final `flushWriteback()`, and returns. **No further enqueues happen after this.** -2. **`UptimeSyncLoop`** sees `ctx.Done()`, returns. (Independent; nothing depends on its ordering.) -3. **`UptimeWorkerPool.Run`** sees `ctx.Done()`: stops pulling from `jobs`, then `workerWG.Wait()` blocks until every worker goroutine has finished the check it was mid-flight on. Each check is bounded by the per-check hard deadline (`hardCap`, default 20 s — see §3.2.1), and the worker still `emit`s that final result. When `workerWG` is drained the pool **closes `results`** (the pool is the *sole* sender, so closing is safe) and returns. -3a. In-flight results emitted during step 3 land in `results` (or drop-on-full → `DroppedCount`, same as steady state — acceptable). -4. **`UptimeIngester.Run`** is structured as `for r := range results { buffer; flush on tick-or-count }`. It does **not** terminate on `ctx.Done()` — only on `results` being **closed** by the pool. `ctx.Done()` only stops its periodic flush *ticker* early (so the loop tightens to drain-and-final-flush). When `results` closes, it does one final `flush()` and returns. This guarantees every result emitted in step 3 is persisted. -5. **`UptimePruner`** sees `ctx.Done()` via its `ctx.Err()` check between chunks, aborts the current pass, returns. Independent. +3. **The recompile cost is paid out-of-band.** Under approach A the ~14–30 min compile only ever runs in `toolchain-image.yml` (45 min budget) or `security-weekly-rebuild.yml` (60 min budget) — never on a contributor's PR or on `docker-build.yml`'s tight per-arch budgets. Under Alternative B the first build on every new `KEY` (every pin bump — routine, Renovate opens several a week) pays the full compile *on whatever PR happens to bump the pin*, on that PR's normal timeout budget. B6/§3.9 shows those budgets are already close to the edge. -**Grace-period requirement:** the process-level shutdown timeout (in `server.Run(ctx)` / the `http.Server.Shutdown` path) must be **≥ `hardCap` (20 s) + ~2 s** for the final ingester flush. **Verify** the existing server shutdown grace during implementation — if it is shorter than ~25 s, either raise it for this path or lower the uptime `hardCap`. (Documented as a C5 implementation check.) +4. **Eviction-immunity also fixes the arm64 leg.** `docker-build.yml`'s `build-arm64` runs under QEMU; today it emulates the *fast* stages plus the final-stage `RUN` lines around a cold-or-warm builders layer. Under approach A the arm64 app build does a `COPY --from` of a pre-cross-compiled binary out of the pinned image's arm64 child — no dependence on an arm64-scoped GHA cache entry surviving. Alternative B's `builders-` scope for arm64 is a separate, separately-evictable entry. -**Test (C5):** start the full pipeline; enqueue a monitor check whose mock target blocks ~2 s; cancel `ctx` immediately; assert the resulting heartbeat row **is** written (no result loss for an in-flight check) and all goroutines exit. +**Residual point in Alternative B's favour, acknowledged:** approach A adds GHCR as a hard build dependency and a private-package permission surface (N8), needs a fork fallback (§3.7), and is more moving parts to operate. The mitigations are in §3.10 (retry wrap, documented inline fallback, `imagetools` platform assertion) and the operator runbook (`docs/ci/toolchain-image.md`, §9). On balance the eviction-immunity (point 1) is decisive: it is the difference between "timeouts become rarer" and "timeouts cannot happen", and the latter is the stated goal. -#### 3.1.5 Removed / retired +### Alternative C — Bake binaries into a committed build artifact / Git LFS -- The `go func(){ time.Sleep(30s); ...; ticker := time.NewTicker(1*time.Minute); for range ticker.C {...} }()` block in `routes.go`. -- `UptimeService.CheckAll()` and `checkAllHosts()` **as the scheduling mechanism** — host connectivity checks are now scheduled by `UptimeScheduler`'s per-tick **host pass** (§3.1.2 step (a)), enqueued as `UptimeJob{Kind: JobHostCheck}` on the same bounded queue with the same drop-on-full semantics as monitor jobs. `CheckAll()` is **kept as an exported method** (used by `POST /api/v1/system/uptime/check` and tests) but re-implemented to *enqueue every enabled host + monitor into the pool* and return `(enqueued, dropped int)` (see §3.7 / N5) rather than spawning goroutines directly. -- `runInitialUptimeBootstrap` loses its `CheckAll()` call (the scheduler's jittered backfill covers the "no blind window on boot" goal; backfill window 60 s < old 90 s blind window). +Rejected without deep analysis: storing compiled multi-arch binaries in the repo (or LFS) defeats reproducibility, bloats history, has no scan/attestation story, and still needs a refresh mechanism. Strictly worse than A on every axis that matters here. --- -### 3.2 Component B — Bounded worker pool + shared HTTP client +## 3. Technical Specifications -#### 3.2.1 `UptimeWorkerPool` +### 3.1 Target architecture -```go -type UptimeJobKind uint8 -const ( - JobMonitorCheck UptimeJobKind = iota - JobHostCheck -) +Extract the two expensive stages' *outputs* into a **separately-versioned, independently-scanned multi-arch prebuilt image** — `ghcr.io/wikid82/charon-toolchain` — so the `xcaddy` / CrowdSec compile happens on a **daily schedule and whenever a tracked pin moves**, not once per app build. -type UptimeJob struct { - Kind UptimeJobKind - Monitor models.UptimeMonitor // populated for JobMonitorCheck - Host models.UptimeHost // populated for JobHostCheck - Manual bool // true for POST /:id/check -} +**Single source of truth:** the build recipe stays in the **main `Dockerfile`**. The existing stage bodies are renamed `caddy-builder → caddy-inline` and `crowdsec-builder → crowdsec-inline`. A new thin `toolchain-runtime` stage assembles their outputs into a publishable image. The toolchain workflow builds `--target toolchain-runtime`; the app build selects between the prebuilt image and the inline stages via a build-arg. No recipe duplication. -// monStateEntry / hostStateEntry are the AUTHORITATIVE debounce state (B2/B3). -type monStateEntry struct { - status string - failureCount int - lastStatusChange time.Time - lastNotifiedDown time.Time -} -type hostStateEntry struct { - status string - failureCount int - lastStatusChange time.Time -} +#### Build graph — BEFORE -type UptimeWorkerPool struct { - db *gorm.DB - jobs chan UptimeJob // bounded, cap = queueCapacity (default 512) - results chan any // CheckResult | HostCheckResult; cap = 2048; pool is sole sender & closes it - ingester *UptimeIngester // Send target - size int // worker count (default 30) - httpClient *http.Client // shared, keep-alive, SSRF-safe - hostDialer *net.Dialer // shared, 3s timeout, for JobHostCheck + TCP monitors - notifier *UptimeService // for queueDownNotification / sendRecoveryNotification (SYNC) - - monMu sync.Mutex // guards monState (short RMW critical sections; see note) - monState map[string]monStateEntry - hostMu sync.RWMutex // guards hostState (scheduler reads via RLock) - hostState map[string]hostStateEntry - - workerWG sync.WaitGroup - enqDropped atomic.Int64 -} - -func (p *UptimeWorkerPool) SeedState(ctx context.Context) error // one-time DB→map seed; called before Run -func (p *UptimeWorkerPool) ReseedState() error // re-seed after a live DB restore (§3.9) -func (p *UptimeWorkerPool) EnsureMonitorState(id string) // add a zero entry for a newly-created monitor -func (p *UptimeWorkerPool) Run(ctx context.Context) // seeds (if not seeded), spawns p.size workers, owns teardown -func (p *UptimeWorkerPool) TryEnqueue(j UptimeJob) bool // non-blocking; false + metric on full -func (p *UptimeWorkerPool) Enqueue(ctx, j UptimeJob) error // blocking with 2s timeout (manual checks) -func (p *UptimeWorkerPool) QueueDepth() int // len(p.jobs) -func (p *UptimeWorkerPool) EnqueueDropped() int64 -func (p *UptimeWorkerPool) HostState(hostID string) (hostStateEntry, bool) // RLock; used by the scheduler ``` - -- **State seeding (`SeedState`, once before `Run`):** - - `monState`: `SELECT id, status, failure_count, last_status_change, last_notified_down FROM uptime_monitors WHERE enabled = true` → one entry per monitor. - - `hostState`: `SELECT id, status, failure_count, last_status_change FROM uptime_hosts` → one entry per host. - - This is the debounce **source of truth** for the process lifetime. The ingester's later DB writes of these same columns are a persistence *mirror* consulted only by the next process's `SeedState`. -- **Worker loop:** `for j := range p.jobs { p.handle(ctx, j) }`, wrapped in `p.workerWG`. `handle` dispatches on `j.Kind`: - - **`JobMonitorCheck`** → `raw := runCheck(ctx, j, p.httpClient)` (pure: HTTP/TCP/orthrus probe, no state) → **worker** takes `p.monMu`, reads `monState[id]`, applies the existing debounce (`success ⇒ up + failCount=0`; `fail ⇒ failCount++`, `down` at `failCount >= MaxRetries`), computes `StatusChanged`, writes the entry back, releases `monMu` → if `StatusChanged`, dispatch notification **synchronously** (§3.3.3) → `p.emit(CheckResult{...pre-computed...})`. - - **`JobHostCheck`** → `raw := runHostCheck(ctx, j, p.hostDialer)` (pure: single TCP dial to any child-monitor port, 3 s) → **worker** takes `p.hostMu` (write), reads `hostState[hostID]`, applies the `FailureThreshold = 2` debounce, computes host `StatusChanged`, writes back, releases → **if host → `down`** (transition): for each of that host's `tcp`-type child monitors whose `monState` is not already `down`, the worker synthesizes a `CheckResult{HeartbeatStatus:"down", Latency:0, Message:"Host unreachable"}`, runs it through the same `monMu` debounce (so the child's `failureCount` maxes and `StatusChanged` is computed per child), fires the **consolidated** down-notification once via `notifier.queueDownNotification(...)` (the existing 30 s batch window coalesces the fan-out into one alert), and `p.emit`s each child result → **if host → `up`**: just update `hostState`; child TCP monitors resume on their next scheduler tick → `p.emit(HostCheckResult{...})` for the `uptime_hosts` row. -- **`p.emit`** is a non-blocking send to `p.results`; on full, `p.ingester.noteDropped()` increments the drop counter (§3.3). The ingester never distinguishes synthetic from real results — all are pre-computed column copies. -- **`monMu` contention:** a single `sync.Mutex` is adequate — the scheduler enqueues at most one job per monitor per cycle (and advances `next_check_at`), so per-monitor RMW is effectively serial; the only real concurrency is *different* monitors' workers contending for the map lock, and each critical section is ~5 field assignments. If profiling ever shows contention, shard by `fnv(monitorID) % 64` — a mechanical change, not a design one. Noted, not pre-optimized. -- **Queue capacity 512:** headroom for a cold-start thundering herd (500 monitors + their hosts) without unbounded memory; each `UptimeJob` ≈ 400 B ⇒ ~200 KB worst case. When full, `TryEnqueue` returns false and the scheduler retries that monitor/host next tick (graceful degradation: check delayed by ≤ 5 s per retry, not lost silently — `enqDropped` is exposed at `/uptime/health`). -- **Shutdown:** `Run` owns steps 3–3a of §3.1.4 — on `ctx.Done()` it stops pulling from `jobs`, `workerWG.Wait()`s, then `close(p.results)`. -- **Worker count default 30**, admin-configurable via `uptime.worker_pool_size` (§3.6). **Sizing guidance** (documented in `docs/features/uptime-monitoring.md`): - `poolSize ≳ ceil( monitors / minIntervalSeconds × worstCaseCheckSeconds )`. - For 500 monitors @ 30 s floor with a 5 s worst-case failing check ⇒ ≈ 83 — but that is the pathological "every target slow-failing simultaneously" case. Normal steady state (checks ≈ 50–300 ms) needs < 10 workers for 500 monitors. Default **30** covers normal operation with margin; 500-monitor deployments with many chronically-down targets should raise to **60–90**. Restart required to apply (pool sized at construction). -- **Per-check deadline:** `ctx, cancel := context.WithTimeout(parent, checkBudget)` where `checkBudget = min(clampInterval(interval), hardCap)`, `hardCap` default **20 s**. Connect timeout **3 s**; TLS handshake 10 s (unchanged); response-header timeout = remaining budget. - -#### 3.2.2 Shared SSRF-safe HTTP client - -Add to `backend/internal/network/safeclient.go` (additive, no behavior change to existing callers): - -```go -// WithKeepAlive enables connection pooling on the SSRF-safe client. -// maxIdle: total idle conns kept; perHost: idle conns per host; idleTimeout: how long to keep them. -// The safeDialer, redirect policy, and all timeouts are unchanged — only -// DisableKeepAlives/MaxIdleConns/MaxIdleConnsPerHost/IdleConnTimeout are overridden. -func WithKeepAlive(maxIdle, perHost int, idleTimeout time.Duration) Option + ┌─────────────────────────────┐ + every app build ───────►│ caddy-builder (748 s) │──┐ + (CI: --no-cache-filter) │ xcaddy build + patch + build│ │ + └─────────────────────────────┘ │ COPY --from + ┌─────────────────────────────┐ ├──► final runtime image + every app build ───────►│ crowdsec-builder (330 s) │──┘ + (CI: --no-cache-filter) │ clone + patch + xx-go build │ + └─────────────────────────────┘ + cold compile on EVERY: docker-build (amd64+arm64), nightly, security-pr, + supply-chain-pr, e2e-tests-split, 4× integration workflows ``` -Implementation: sets `cfg.keepAlive = true` and the three ints; in `NewSafeHTTPClient` the `http.Transport` fields become conditional on `cfg.keepAlive` (`DisableKeepAlives: !cfg.keepAlive`, `MaxIdleConns: cfg.maxIdle`, `MaxIdleConnsPerHost: cfg.perHost`, `IdleConnTimeout: cfg.idleTimeout`). Default (option not passed) is byte-for-byte the current behavior. +#### Build graph — AFTER -The pool constructs **one** shared client at startup: - -```go -p.httpClient = network.NewSafeHTTPClient( - network.WithTimeout(20*time.Second), // hard ceiling; per-request ctx is tighter - network.WithDialTimeout(3*time.Second), - network.WithMaxRedirects(0), - network.WithAllowLocalhost(), // parity with today's per-check client - network.WithAllowRFC1918(), // parity with today's per-check client - network.WithKeepAlive(100, 4, 30*time.Second), // idleTimeout 30s — see below -) ``` - -- Security parity: today **every** uptime HTTP check already passes `WithAllowLocalhost()` + `WithAllowRFC1918()` and `WithMaxRedirects(0)`, so a single shared client with the same options is not a regression. `safeDialer` still validates the resolved IP at connect time on every **new** connection (DNS-rebinding / TOCTOU guard preserved). Link-local (169.254/16), cloud-metadata, and other reserved ranges remain blocked at both layers. -- **`idleTimeout` = 30 s** (not 90 s): a pooled idle connection skips Layer-2 re-resolution for its lifetime, so bounding that window to 30 s bounds the staleness (an *established* TCP connection cannot be re-bound to a new IP, so there is no actual SSRF here — R11 stays Low — but 30 s is the tighter, defensible choice). `safeclient_test.go` gains a case asserting a connection older than `idleTimeout` is **not** reused. -- **`MaxIdleConnsPerHost: 4` / `MaxIdleConns: 100`** assume meaningful per-host reuse. With ~500 distinct target hosts the idle pool churns and the keep-alive win shrinks — but repeat checks of the *same* host within 30 s still reuse the connection (the common case: each monitor re-checks its one host every ≥ 30 s), so it stays net-positive. Not tuned further; noted. -- `security.ValidateExternalURL(...)` is still called per HTTP check (Layer 1) with the same options as today. -- Keep-alive win: repeat checks of the same host reuse the TCP + TLS connection — the dominant cost at scale — so measured latency drops to the target's actual response time and stops absorbing handshake variance. - -#### 3.2.3 Host TCP pre-check — scheduling + de-blocking - -The `checkHost()` inner `for retry := 0; retry <= MaxRetries; retry++ { ... time.Sleep(2*time.Second) ... }` loop is **removed**, and host checks become first-class scheduled jobs. - -**Scheduling (B1).** `UptimeScheduler` maintains `hostSchedule` alongside `monSchedule` (§3.1.2): every `UptimeHost` row is hydrated at cold start with an in-memory due-time (**not persisted** — no column added to `uptime_hosts`), `due = min(clamped intervals of its enabled child monitors)`, jittered over the first 60 s. The per-tick **host pass** (§3.1.2 step (a)) selects due hosts and enqueues `UptimeJob{Kind: JobHostCheck, Host: }` on the same bounded queue as monitor jobs, with the same `TryEnqueue` drop-on-full semantics. `hostMinInt` is recomputed on the ~30 s `rescan()` when child monitors are added/removed/re-intervalled. - -**De-blocking.** `runHostCheck` does **a single TCP dial** (connect timeout 3 s, via the pool's shared `hostDialer`) to any one child-monitor port — no `time.Sleep` retry. The consecutive-failure debounce is **unchanged in outcome**: the worker applies `FailureThreshold = 2` against the authoritative `hostState` entry, so the host flips to `down` only after 2 consecutive failed host-check cycles. Dropping the sleep-retry removes up to ~4 s of blocked worker time per down host with no change to detection semantics. - -**Host-down short-circuit — single owner: the worker (B2).** There is **no** ingester back-channel. When `runHostCheck` + debounce produces a host `up→down` transition, the **worker** (synchronously, exactly like a monitor transition in §3.3.3): -1. writes the new `down` state into the pool's `hostState` map; -2. for each of that host's `tcp`-type child monitors whose `monState` is not already `down`, synthesizes a `CheckResult{HeartbeatStatus:"down", Latency:0, Message:"Host unreachable"}`, runs it through the normal `monMu` debounce (child `failureCount` → max, per-child `StatusChanged` computed), and `emit`s it into the normal result stream — so the ingester writes those synthetic `down` heartbeats + column updates as ordinary dumb column copies; -3. fires **one** consolidated down-notification via `notifier.queueDownNotification(...)` — the existing 30 s batch window coalesces the fan-out into a single "N services down on host X" alert (unchanged behavior from `markHostMonitorsDown` today). - -While the host stays `down`, the **scheduler** reads `pool.HostState(hostID)` (RLock) in its monitor pass and **skips enqueueing** that host's TCP monitors (advancing their `next_check_at` so they resume cleanly on recovery) — no new heartbeats are written for them, which is correct (nothing changed). On the host `down→up` transition the worker clears the `hostState` entry to `up`; the scheduler stops skipping and the TCP monitors resume normal checks on their next due tick, each re-evaluating its own status from its first real result. - -HTTP / HTTPS / orthrus monitors are **never** short-circuited (URL-truth authoritative — unchanged). - -The ingester remains a dumb writer throughout: it copies pre-computed `status` / `failure_count` / `last_status_change` / heartbeat rows for both real and synthetic results and never inspects a transition. - -#### 3.2.4 Redundant second DNS lookup - -**Accepted as-is, documented.** `ValidateExternalURL`'s `LookupIP` (Layer 1) and `safeDialer`'s connect-time resolution (Layer 2) are *deliberately* independent — collapsing them re-opens the DNS-rebinding TOCTOU window that Layer 2 exists to close. The cost is one extra `getaddrinfo` per HTTP check; with the OS resolver cache and (post-change) keep-alive amortizing connection setup, this is negligible relative to the check itself. A shared in-process DNS cache was considered and **deferred** (adds rebinding risk for marginal gain). Note added to `docs/features/uptime-monitoring.md` and code comment at the call site. - ---- - -### 3.3 Component C — Heartbeat ingester (mirror `StatsIngester`) - -#### 3.3.1 `CheckResult` (worker → ingester) - -```go -type CheckResult struct { - MonitorID string - HostID string // UptimeHostID, "" if none - HeartbeatStatus string // "up" | "down" (raw check outcome) - Latency int64 // ms - Message string - CheckedAt time.Time - - // Pre-computed by the worker against the authoritative monState map - // (§3.2.1 / §3.3.3), NOT the scheduler's DB snapshot — so the ingester only writes: - NewMonitorStatus string // resolved status after debounce ("up"|"down"|unchanged) - FailureCount int // post-check failure counter (from monState, authoritative) - StatusChanged bool - StatusChangedAt time.Time // set iff StatusChanged - Synthetic bool // true for host-down child fan-out results (§3.2.3) — informational only -} - -type HostCheckResult struct { - HostID string - Status string // resolved after FailureThreshold debounce - FailureCount int - Latency int64 - Message string - CheckedAt time.Time - StatusChanged bool - StatusChangedAt time.Time -} + ┌──────────────────────── toolchain image lifecycle (rare) ─────────────────────────┐ + │ trigger: daily cron | workflow_dispatch | PR touching toolchain inputs │ + │ │ + │ docker buildx build --target toolchain-runtime │ + │ --platform linux/amd64,linux/arm64 --no-cache --pull (cron/dispatch) │ + │ caddy-inline (cross-compile, no QEMU) ─┐ │ + │ crowdsec-inline (xx cross-compile, no QEMU) ─┤ │ + │ toolchain-runtime: FROM alpine; COPY both ─┘ │ + │ → push ghcr.io/wikid82/charon-toolchain:caddy-crowdsec- (+ :latest, │ + │ + :) → Trivy CRITICAL/HIGH gate → SARIF │ + │ → if new digest: bot PR bumps ARG CHARON_TOOLCHAIN_DIGEST in Dockerfile │ + └───────────────────────────────────────────────────────────────────────────────────┘ + │ digest pin (one ARG line in Dockerfile) + ▼ + ┌──────────────────────── every app build (hot path) ──────────────────────────────┐ + │ FROM ${CHARON_TOOLCHAIN_IMAGE}@${CHARON_TOOLCHAIN_DIGEST} AS toolchain-prebuilt │ + │ FROM ${CADDY_BUILDER_SRC} AS caddy-builder (default → toolchain-prebuilt) │ + │ FROM ${CROWDSEC_BUILDER_SRC} AS crowdsec-builder (default → toolchain-prebuilt) │ + │ COPY --from=caddy-builder /usr/bin/caddy ... (UNCHANGED) │ + │ COPY --from=crowdsec-builder /crowdsec-out/crowdsec ... (UNCHANGED) │ + │ normal type=gha layer cache covers every stage; NO --no-cache-filter │ + │ │ + │ fallback (fork PR / bootstrap / offline): │ + │ --build-arg CADDY_BUILDER_SRC=caddy-inline │ + │ --build-arg CROWDSEC_BUILDER_SRC=crowdsec-inline → compiles from source │ + └───────────────────────────────────────────────────────────────────────────────────┘ ``` -The pool sends both types on one `chan any`; the ingester type-switches to route each to the `uptime_monitors` or `uptime_hosts` coalescing map. Neither carries any instruction the ingester acts on beyond "copy these columns". - -#### 3.3.2 `UptimeIngester` - -Structure mirrors `stats_ingester.go` almost line-for-line: - -```go -const ( - uptimeChannelBufferSize = 2048 // 500 monitors * ~4 in-flight cycles - uptimeBatchSize = 100 - uptimeFlushInterval = 500 * time.Millisecond -) - -type UptimeIngester struct { - db *gorm.DB - results <-chan any // CheckResult | HostCheckResult; created in routes.go, OWNED & CLOSED by the pool - droppedCount atomic.Int64 -} - -func NewUptimeIngester(db *gorm.DB, results <-chan any) *UptimeIngester -func (i *UptimeIngester) noteDropped() // called by the pool's emit() when the channel is full -func (i *UptimeIngester) DroppedCount() int64 -func (i *UptimeIngester) Run(ctx context.Context) // for r := range results { ... }; returns when results is CLOSED -func (i *UptimeIngester) Stop() // test-only: drain + flush for an instance whose Run isn't driven +### 3.2 `Dockerfile` changes + +#### 3.2.1 New ARGs (add near the pinned-toolchain block, `:11`) + +```dockerfile +# ---- Prebuilt Caddy + CrowdSec toolchain image ---- +# Built by .github/workflows/toolchain-image.yml from the caddy-inline / +# crowdsec-inline stages below. Bumped by the open-bump-pr job (bot PR) when a +# security-relevant input moves OR the DAILY --no-cache --pull rebuild produces +# a new digest. The freshness-guard CI check (scripts/verify-toolchain-pin.sh) +# fails any PR where TAG/DIGEST is stale for the current pins. +ARG CHARON_TOOLCHAIN_IMAGE=ghcr.io/wikid82/charon-toolchain +# NOT Renovate-tracked (content-hash tag has no series to follow, N7) — the +# open-bump-pr bot in toolchain-image.yml owns these two lines. +ARG CHARON_TOOLCHAIN_TAG=caddy-crowdsec-0000000000000000 +ARG CHARON_TOOLCHAIN_DIGEST=sha256: + +# Stage selector — default uses the prebuilt image; fork PRs / bootstrap / +# offline builds pass `--build-arg CADDY_BUILDER_SRC=caddy-inline +# --build-arg CROWDSEC_BUILDER_SRC=crowdsec-inline` to compile from source. +ARG CADDY_BUILDER_SRC=toolchain-prebuilt +ARG CROWDSEC_BUILDER_SRC=toolchain-prebuilt ``` -**Channel ownership (differs from `StatsIngester`).** `StatsIngester` owns `ingestCh`; here the `results` channel is created in `routes.go`, given to the pool as `chan<- any` and to the ingester as `<-chan any`. The **pool is the sole sender and closes it** at shutdown (§3.1.4 step 3). `Run` is `for r := range results { buffer; flush on 500 ms-tick or 100-count }` and returns **only when `results` is closed** — `ctx.Done()` merely tightens the flush ticker so the tail drains fast; it does not end the loop. This is what guarantees no in-flight result is lost at shutdown. - -**Flush (on 500 ms tick or 100 buffered results):** - -1. **Heartbeat inserts** — `[]models.UptimeHeartbeat` from every buffered `CheckResult` (real *and* synthetic host-down children), `db.CreateInBatches(rows, uptimeBatchSize)`. -2. **Coalesced monitor updates** — `map[string]CheckResult` keyed by `MonitorID`, latest wins. Grouped `UPDATE uptime_monitors SET status=?, last_check=?, latency=?, failure_count=?, last_status_change=COALESCE(?, last_status_change) WHERE id=?` — one per distinct monitor. `next_check_at` untouched (scheduler owns it). -3. **Coalesced host updates** — `map[string]HostCheckResult` for `uptime_hosts` (`status`, `last_check`, `latency`, `failure_count`, `last_status_change`). - -All inside **one** `db.Transaction(...)` per flush ⇒ ~2–4 write statements / 500 ms for the whole subsystem, vs today's 2 writes *per check*. - -**These DB writes are a persistence MIRROR, not the source of truth.** Authoritative `status` / `failure_count` / debounce state lives in the pool's `monState` / `hostState` maps (§3.2.1). The ingester keeps the DB roughly current so the summary endpoint/UI have fresh data and the *next* process's `SeedState` has a good baseline. A dropped write costs at most one stale row until the next flushed check for that monitor — it **cannot** suppress or delay a transition (B3), because runtime detection never reads these columns. - -**Drop-on-full metric:** `droppedCount` at `GET /api/v1/uptime/health`. Logged `Warn` (rate-limited) like `StatsIngester`. - -**Shutdown semantics:** see §3.1.4 — the pool closing `results` (after its worker `WaitGroup` drains) is what ends `Run`, after one final `flush()`. A hard crash loses at most `uptimeFlushInterval` (500 ms) of un-flushed heartbeats — acceptable for monitoring data; the debounce maps are unaffected by the loss. - -#### 3.3.3 Debounce + transition detection are authoritative in memory (B3) - -**The failure-debounce counter must never depend on a droppable async DB round-trip.** Under sustained ingester saturation — the exact overload this feature targets — a design that recomputed `FailureCount` from the last-persisted row would drop successive failing-check results, never persist the increment, keep reading a stale-low count, never reach `maxRetries`, and **never fire the down alert**. So the counter is owned in memory: - -1. The pool's `monState` map (`{status, failureCount, lastStatusChange, lastNotifiedDown}`) is **seeded once from the DB at pool start** (`SeedState`, §3.2.1) and is the debounce source of truth for the whole process lifetime. -2. On every check result the **worker**, holding `monMu`: - - reads `monState[monitorID]`; - - applies the existing debounce (`uptime_service.go` `checkMonitor` logic, ~lines 928–952): success ⇒ `status="up"`, `failureCount=0`; failure ⇒ `failureCount++`, `status="down"` once `failureCount >= job.Monitor.MaxRetries` (MaxRetries is a *static* config field — safe to read from the scheduler snapshot); - - computes `StatusChanged = old.status != new.status && old.status != "pending"`; - - writes the updated entry back (including `lastNotifiedDown` if it dispatches below); releases `monMu`. -3. If `StatusChanged`, the worker **synchronously** (before emitting the result) invokes the existing notification path: - - `down` ⇒ `notifier.queueDownNotification(monitor, msg, durationStr)` (30 s batch window unchanged — coalesces multi-service outages); - - `up` ⇒ `notifier.sendRecoveryNotification(monitor, durationStr)`. -4. The `CheckResult` carries the already-resolved `NewMonitorStatus` / `FailureCount` / `StatusChanged` / `StatusChangedAt`; the ingester copies them (mirror only). - -Result: alerts fire on the worker goroutine that observed the transition, with **zero** buffering latency, and **a dropped `CheckResult` cannot delay or suppress a subsequent transition** — the next check reads the still-correct in-memory `monState`. - -**Manual `POST /:id/check`** uses the same `monMu` + `monState`. A manual check racing a scheduled check of the same monitor no longer under-counts the failure streak (the old "both read N, both write N+1" hazard): the lock serializes the two RMWs. A double-*notify* in that narrow window is still possible and still deduped by `NotificationService` + `lastNotifiedDown` (5-min host-down guard / 30 s monitor-down batch) — documented, not fixed (unchanged). - -**Restart reseed staleness (ties to §3.1.4 / §3.9):** after a hard crash mid-saturation, `SeedState` reads whatever the ingester last flushed — `failureCount` may be stale-low by a few. A monitor one failed check from `down` then needs 1–2 extra cycles post-restart to re-accumulate and fire. Bounded (≤ 2 intervals ≈ 60 s), self-correcting, alert still fires — just slightly later. Acceptable. +#### 3.2.2 Rename existing stages + close the two pin gaps (Commit 1) + +- `Dockerfile:302` — `FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine AS caddy-builder` → `FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine@sha256: AS caddy-inline` (N4 — digest-pin the base). +- `Dockerfile:577` — same treatment for `crowdsec-builder` → `crowdsec-inline`. +- Add near `:64`, with `# renovate: datasource=go` annotations (B4): + ```dockerfile + # renovate: datasource=go depName=github.com/zhangjiayin/caddy-geoip2 + ARG CADDY_GEOIP2_VERSION= + # renovate: datasource=go depName=github.com/mholt/caddy-ratelimit + ARG CADDY_RATELIMIT_VERSION= + ``` + and change `Dockerfile:391-392` to `--with github.com/zhangjiayin/caddy-geoip2@v${CADDY_GEOIP2_VERSION}` / `--with github.com/mholt/caddy-ratelimit@v${CADDY_RATELIMIT_VERSION}` (declare both ARGs inside `caddy-inline` alongside the other `ARG CADDY_*` at `:305-310`). +- **Delete** the dead `crowdsec-fallback` stage (`:713-748`) and the now-dead `CROWDSEC_RELEASE_SHA256` ARG (`:22`, `:586`) — N1. + +Apart from the base-image digest and the two plugin `@version` suffixes, **no logic inside the two stages changes**. All `go get` patches, module-cache source patches, and embeds-version assertions are retained verbatim — they are the security recipe and the toolchain image is *the* place they now run. + +#### 3.2.3 New `toolchain-prebuilt` and `toolchain-runtime` stages + +Insert after `crowdsec-inline` (where `crowdsec-fallback` used to be, now deleted): + +```dockerfile +# ---- Prebuilt toolchain (default source for caddy-builder / crowdsec-builder) ---- +# Digest-pinned. Contains /usr/bin/caddy and /crowdsec-out/{crowdsec,cscli,config} +# at the SAME paths the inline stages produce, so the COPY --from lines in the +# final stage need no change. +FROM ${CHARON_TOOLCHAIN_IMAGE}@${CHARON_TOOLCHAIN_DIGEST} AS toolchain-prebuilt + +# ---- Toolchain image assembly target (built by toolchain-image.yml) ---- +# NOT part of the app build graph (nothing FROMs it there). `docker buildx build +# --target toolchain-runtime` produces the publishable multi-arch image. +FROM ${ALPINE_IMAGE} AS toolchain-runtime +COPY --from=caddy-inline /usr/bin/caddy /usr/bin/caddy +COPY --from=crowdsec-inline /crowdsec-out/crowdsec /crowdsec-out/crowdsec +COPY --from=crowdsec-inline /crowdsec-out/cscli /crowdsec-out/cscli +COPY --from=crowdsec-inline /crowdsec-out/config /crowdsec-out/config +# Provenance label so `docker inspect` on the toolchain image shows the key. +LABEL io.charon.toolchain.key="${CHARON_TOOLCHAIN_TAG}" + +# ---- Effective builder stages: alias to prebuilt image OR inline compile ---- +FROM ${CADDY_BUILDER_SRC} AS caddy-builder +FROM ${CROWDSEC_BUILDER_SRC} AS crowdsec-builder +``` -**Test (C5):** saturate the ingester (`results` full, every send dropping); feed a monitor `maxRetries` consecutive `down` results through the worker; assert the `down` transition **is** detected and `queueDownNotification` **is** called despite every `CheckResult` being dropped. +`FROM ${ARG} AS name` where the ARG resolves to a **prior stage name** is valid BuildKit; unreferenced stages (`caddy-inline` etc. when `…_SRC=toolchain-prebuilt`) are pruned from the graph and never built. When `…_SRC=caddy-inline`, `toolchain-prebuilt` is still declared but unreferenced → also pruned, so a fork build never needs to pull the image. ---- +**`crowdsec-fallback` (`:713-748`):** deleted in Commit 1 — it is dead code (verified, §2.1 correction). Nothing referenced it before this change. If the reviewer wants it retained as an escape hatch, it stays out of the toolchain key and out of the graph regardless. -### 3.4 Component D — Retention pruner +#### 3.2.4 Final-stage `COPY --from` lines — UNCHANGED, plus a cheap embed assertion (N5) -#### 3.4.1 `UptimePruner` +`Dockerfile:807`, `:814`, `:815`, `:817` keep referencing `caddy-builder` / `crowdsec-builder` and the same source paths. This is the whole point of putting the binaries at identical paths in `toolchain-runtime`. -```go -const ( - prunerInterval = 1 * time.Hour - pruneChunkSize = 5000 - pruneChunkPause = 50 * time.Millisecond // steady-state: yield the single connection between chunks - firstPassChunkPause = 250 * time.Millisecond // first (cold, huge) pass: yield longer — see §3.4.2 / N1 - walCheckpointRowThreshold = 50_000 // TRUNCATE checkpoint after a big prune -) +**N5 — add a post-`COPY` assertion in the final stage.** Today the "did the binary embed the fixed cel-go / grpc-go" checks (`Dockerfile:564`, `:569`) run *inside* `caddy-inline` — so on the prebuilt path they only ever executed when the toolchain image was built, and a wrong/rolled-back `CHARON_TOOLCHAIN_DIGEST` (or a hand-edited pin pointing at an old image) would sail through the app build silently. Add a small `RUN` right after `COPY --from=caddy-builder … /usr/bin/caddy` (and the crowdsec copies): -type UptimePruner struct { - db *gorm.DB - cfg *uptimeConfig // reads uptime.heartbeat_retention_days (hot) - now func() time.Time - firstPassDone atomic.Bool // widens the chunk pause until the first clean pass completes -} - -func (p *UptimePruner) Run(ctx context.Context) // go p.Run(ctx); first pass ~30s after boot, then hourly -func (p *UptimePruner) pruneOnce(ctx) (deleted int64, err error) +```dockerfile +RUN set -e; \ + caddy list-modules 2>/dev/null | grep -q 'http.handlers.rate_limit' || { echo "toolchain image missing expected caddy plugins"; exit 1; }; \ + go_ver_check() { command -v go >/dev/null && go version -m "$1" || true; }; \ + /usr/local/bin/cscli version >/dev/null || { echo "cscli from toolchain image not runnable"; exit 1; } ``` -**`pruneOnce`:** - -``` -cutoff := now().Add(-retentionDays * 24h) -pause := pruneChunkPause; if !p.firstPassDone.Load() { pause = firstPassChunkPause } -total := 0 -for { - if ctx.Err() != nil { return total, ctx.Err() } - res := db.Exec(` - DELETE FROM uptime_heartbeats - WHERE id IN ( - SELECT id FROM uptime_heartbeats - WHERE created_at < ? - ORDER BY id - LIMIT ? - )`, cutoff, pruneChunkSize) - total += res.RowsAffected - if res.Error != nil { return total, res.Error } - if res.RowsAffected < pruneChunkSize { break } // caught up - time.Sleep(pause) // release connection to API/ingester -} -if total >= walCheckpointRowThreshold { - db.Exec(`PRAGMA wal_checkpoint(TRUNCATE)`) // reclaim WAL growth from a large prune -} +The final stage has no Go toolchain, so a full `go version -m` embed check is not possible there — instead assert (a) the Caddy binary loads and lists the expected custom plugins (`rate_limit`, `crowdsec`, `geoip2`, `coraza`), (b) `cscli version` runs and prints the expected `v${CROWDSEC_VERSION}`. A wrong-arch or stale-recipe image fails these immediately. The authoritative embeds-version assertions remain in `caddy-inline` and run in `toolchain-image.yml`. Additionally, a CI step in `docker-build.yml` (it already has a "Caddy/CrowdSec CVE verification" step post-build, `merge-and-publish`) runs `docker run --rm go version -m /usr/bin/caddy | grep …` against the *final* image for the full check — extend that existing step to also assert the toolchain `LABEL io.charon.toolchain.key` matches `scripts/toolchain-key.sh`. + +### 3.3 Renovate / pin-tracking + +- Every `# renovate:` annotation on the version ARGs stays. Renovate keeps bumping `CADDY_VERSION` etc. as today; the two new plugin ARGs (B4) and the digest-pinned `golang` base (N4) get annotations too. +- A Renovate bump to any of those ARGs now *also* needs a toolchain rebuild. The **freshness guard** (§3.4.2) turns that into a hard PR failure with a one-line fix (`workflow_dispatch` the toolchain workflow, or wait for the bot), so a Renovate PR that bumps `CADDY_VERSION` cannot merge with a stale toolchain. +- **N7 (corrected):** the `CHARON_TOOLCHAIN_IMAGE` / `_TAG` / `_DIGEST` three-ARG split with a **content-hash tag** (`caddy-crowdsec-`) is *not* something Renovate's `datasource=docker` manager tracks out of the box — it has no semver/digest series to follow on that tag. It simply won't fire, which is harmless: the daily rebuild + digest-bump bot (§3.4.3) is the sole authority on that pin. Do **not** add a Renovate entry implying it works; add a comment in `renovate.json` stating the toolchain digest is bot-owned. + +### 3.4 New workflow: `.github/workflows/toolchain-image.yml` + +Builds & publishes `ghcr.io/wikid82/charon-toolchain`. + +#### 3.4.1 Triggers, permissions, concurrency + +```yaml +name: Toolchain Image — Build & Publish +on: + schedule: + - cron: '0 6 * * *' # DAILY 06:00 UTC — committed scope (B2). --no-cache --pull. + workflow_dispatch: + inputs: + force_rebuild: { type: boolean, default: true, description: "Build with --no-cache --pull" } + pull_request: + paths: + - 'Dockerfile' # coarse; the key script decides if it truly changed + - '.github/workflows/toolchain-image.yml' + - 'scripts/toolchain-key.sh' + - 'scripts/verify-toolchain-pin.sh' + - 'scripts/lib/dockerfile-stage.sh' + - '.trivyignore' + # The Tuesday `security-weekly-rebuild.yml` also `workflow_call`s this workflow for the + # heavier "full Trivy report + SARIF + JSON artifact" pass; the daily `schedule` above + # is the freshness driver. Two entry points, one build definition. + workflow_call: + inputs: + force_rebuild: { type: boolean, default: true } + publish: { type: boolean, default: true } # PR path builds but does not push :latest +concurrency: + group: toolchain-image-${{ github.ref }} + cancel-in-progress: false # never cancel a publish mid-push +permissions: + contents: read + packages: write # push to GHCR + security-events: write # Trivy SARIF + pull-requests: write # bot digest-bump PR (schedule/dispatch/workflow_call only) ``` -- Subquery form (not `DELETE ... LIMIT`) — required for the `modernc.org/sqlite` driver. -- `ORDER BY id` makes each chunk delete the oldest rows first and keeps the plan index-friendly (`id` is the PK). -- **Per-chunk latency, honest range:** on a warm table a 5 000-row chunk delete is ~10–30 ms. On a **cold, huge table (first pass)** each chunk can be **100–500 ms** — with `SetMaxOpenConns(1)` + `busy_timeout=5000` the ingester flush and API writes block for that window. So the first pass uses `firstPassChunkPause = 250 ms` (5× the steady-state pause) to keep the single connection available between chunks; steady-state hourly passes (tiny) use `50 ms`. Worst added API/ingester write latency during the first pass ≈ one chunk ≈ up to ~500 ms, intermittently, for the pass's duration. -- **`PRAGMA optimize`** runs on a 24 h sub-cadence (every 24th successful pass). **`VACUUM` is explicitly deferred** — it locks the whole DB and rewrites the file; WAL checkpoint reclaims most space for a hard-delete workload. +**Daily cadence is committed scope, not optional (B2).** See §3.8 for the baseline analysis that requires it. The `schedule` trigger runs `--no-cache --pull` every day at 06:00 UTC; on a day with no digest change it is a ~30-minute no-op (acceptable — one runner, off-peak). `security-weekly-rebuild.yml` keeps its Tuesday slot for the fuller scan/report but is no longer the *only* forced-rebuild driver. -#### 3.4.2 First run on a large existing table — honest at the 500-monitor target +- **`pull_request` from a fork:** GitHub grants only `contents: read`, no `packages: write`. The job's publish/push steps are guarded `if: github.event.pull_request.head.repo.full_name == github.repository`. On a fork PR the workflow still *builds* `--target toolchain-runtime` (validates the recipe compiles) but does not push and does not open a bot PR. The fork's *app* build meanwhile uses the inline fallback (§3.7), so a fork PR is fully testable without the image. +- **`timeout-minutes: 45`** (cold amd64+arm64 cross-compile of both stages ≈ 25–30 min + Trivy). -**Row-count math (corrected).** The "13 M rows" figure is the *current 100-monitor* case (100 × 1440 checks/day × 90 days). At this spec's **500-monitor target with the 90-day default retention** the table holds **≈ 65 M rows in steady state** (500 × 1440 × 90), or ≈ 130 M at a 30 s interval floor. An instance that has run for *years* past 90 days without pruning can hold several hundred million. +#### 3.4.2 Tag key derivation — `scripts/toolchain-key.sh` -**What prune-first actually buys.** It bounds the **worst** case: on a multi-hundred-million-row instance, trimming to the ~65 M steady state *before* building the index avoids a `CREATE INDEX` over the full table (10+ minutes). It does **not** make the first-boot index build "seconds" — on a healthy 500-monitor instance the build still runs over ~65 M rows and is a **bounded multi-minute operation that contends for the single write connection** for its duration (readers still serve from WAL; writers — ingester flushes, API writes — see elevated latency and may drop-on-full while it runs). This is an honest, bounded first-boot-only cost, not a stall that is designed away. +Deterministic, content-addressed. Output: `caddy-crowdsec-<16 hex>`. -**Why it is still acceptable (mitigations that hold):** -- It runs in a **background goroutine**, never on a request path or a blocking migration step. The server is up and serving throughout. -- The runtime **summary endpoint stays available** (correct results, slower, 30 s-cached) the whole time — no route downtime, no 503. -- It is **idempotent (`CREATE INDEX IF NOT EXISTS`) and retried at the end of every clean, caught-up prune pass** — a failed or `ctx`-interrupted attempt self-heals on the next pass; there is no "stuck unbuilt until restart" hole. -- Heartbeat writes that drop-on-full during the build self-heal on the next check (monitoring data). -- Operators who cannot tolerate the window: run `charon migrate` in a maintenance window (eager index build there, with an explicit warning log — §3.5.6 / S7), or temporarily **lower `uptime.heartbeat_retention_days` before first boot** to shrink both the first prune and the index build. +Inputs to the SHA-256: -**The `uptime.heartbeat_retention_days` default stays 90** (user's explicit decision — not changed here). +1. The **exact text** of the `caddy-inline` stage (`Dockerfile` from `FROM … AS caddy-inline` to the blank line before the next `FROM`), extracted by the **shared** `extract_stage` routine in `scripts/lib/dockerfile-stage.sh` (N9 — one copy, `source`d by both `toolchain-key.sh` and `verify-toolchain-pin.sh`). +2. The **exact text** of the `crowdsec-inline` stage (same routine). +3. The resolved default values of every ARG in the §2.2 table — **including the two new `CADDY_GEOIP2_VERSION` / `CADDY_RATELIMIT_VERSION` plugin pins (B4)** — parsed from the `ARG NAME=default` lines, so a bump to `CADDY_VERSION` (or a plugin) changes the key even though the stage body only interpolates `${…}`. +4. The `tonistiigi/xx` pin line (`:73`) **and the digest-pinned `golang:${GO_VERSION}-alpine@sha256:…` base lines of both inline stages (N4)**. +5. `sha256sum .trivyignore`. +6. A `SCHEMA_VERSION` constant in the script (bump to force a global rebuild if the recipe-extraction logic itself changes). -**Ordering & retry.** `pruneOnce` returns `(deleted int64, err error)`. At the **end of every hourly pass** where `pruneOnce` returned `err == nil` and the chunk loop reached its "caught up" break (not a `ctx` abort), `Run` sets `firstPassDone` and issues `CREATE INDEX IF NOT EXISTS idx_heartbeat_monitor_created ON uptime_heartbeats (monitor_id, created_at)`. On a healthy instance this lands on the first pass; on a huge instance the first pass trims first, then the (still multi-minute but bounded) build runs; a transient failure retries next hour. No `sync.Once`. Risk restated in §6 (R3/R4/R7). - ---- - -### 3.5 Component E — Batch summary endpoint (kills the N+1) - -#### 3.5.1 Route - -``` -GET /api/v1/uptime/monitors/summary?beats=<1..60> +```bash +#!/usr/bin/env bash +# scripts/lib/dockerfile-stage.sh — SHARED (N9). sourced by both scripts. +extract_stage() { # $1 = stage name, $2 = Dockerfile path + awk -v s="$1" ' + $0 ~ ("AS "s"$") {c=1} + c {print} + c && /^$/ && NR>1 {exit} + END { if (!c) { print "extract_stage: no stage \"" s "\"" > "/dev/stderr"; exit 3 } }' "$2" +} ``` -Registered in `routes.go` next to the existing uptime routes; same auth (`management` group / JWT). `beats` optional, **default 30**, capped at 60. The Uptime page list view requests the default 30; an expanded/detail view may request up to 60. - -#### 3.5.2 Response schema (snake_case, explicit `json` tags) - -```jsonc -[ - { - "id": "0f8c...-uuid", - "name": "API Server", - "type": "http", - "url": "https://api.example.com", - "enabled": true, - "status": "up", // resolved monitor status - "latency": 45, // ms, last check - "last_check": "2026-08-27T12:00:00Z", // nullable - "interval": 30, - "proxy_host_id": 12, // nullable, for UI grouping - "remote_server_id": null, - "uptime_24h": 99.86, // % up over last 24h, computed from heartbeats (nullable if no data). Always present in the response. - "recent_beats": [ // chronological ASC, length <= beats param (default 30, cap 60) - { "status": "up", "latency": 44, "created_at": "2026-08-27T11:31:00Z" }, - { "status": "up", "latency": 46, "created_at": "2026-08-27T11:31:30Z" }, - { "status": "down", "latency": 0, "created_at": "2026-08-27T11:32:00Z" } - ] - } -] +```bash +#!/usr/bin/env bash +# scripts/toolchain-key.sh — prints the deterministic toolchain image tag. +set -euo pipefail +SCHEMA_VERSION=2 # rev-2: added plugin pins + digest-pinned golang base to the key +df="${1:-Dockerfile}" +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=scripts/lib/dockerfile-stage.sh +source "$here/lib/dockerfile-stage.sh" + +caddy_stage="$(extract_stage caddy-inline "$df")" +crowdsec_stage="$(extract_stage crowdsec-inline "$df")" +# sanity: each stage must be non-trivial and actually build something +for s in "$caddy_stage" "$crowdsec_stage"; do + [[ "$(wc -l <<<"$s")" -ge 20 ]] && grep -q 'go build\|xx-go build' <<<"$s" \ + || { echo "toolchain-key: stage extraction looks wrong" >&2; exit 3; } +done +{ + echo "schema=$SCHEMA_VERSION" + printf '%s\n' "$caddy_stage" "$crowdsec_stage" + grep -E '^ARG (GO_VERSION|ALPINE_IMAGE|CROWDSEC_VERSION|EXPR_LANG_VERSION|XNET_VERSION|XCRYPTO_VERSION|KLAUSPOST_COMPRESS_VERSION|GRPC_VERSION|CADDY_VERSION|CADDY_CANDIDATE_VERSION|CADDY_USE_CANDIDATE|CADDY_PATCH_SCENARIO|CADDY_SECURITY_VERSION|CORAZA_CADDY_VERSION|CADDY_GEOIP2_VERSION|CADDY_RATELIMIT_VERSION)=' "$df" + grep -E 'tonistiigi/xx:|^FROM .*golang:.*-alpine@sha256:' "$df" + sha256sum .trivyignore | cut -d' ' -f1 +} | sha256sum | cut -c1-16 | sed 's/^/caddy-crowdsec-/' ``` -Go types (in `uptime_summary_service.go`): - -```go -type MonitorSummary struct { - ID string `json:"id"` - Name string `json:"name"` - Type string `json:"type"` - URL string `json:"url"` - Enabled bool `json:"enabled"` - Status string `json:"status"` - Latency int64 `json:"latency"` - LastCheck *time.Time `json:"last_check"` - Interval int `json:"interval"` - ProxyHostID *uint `json:"proxy_host_id"` - RemoteServerID *uint `json:"remote_server_id"` - Uptime24h *float64 `json:"uptime_24h"` - RecentBeats []BeatDTO `json:"recent_beats"` -} - -type BeatDTO struct { - Status string `json:"status"` - Latency int64 `json:"latency"` - CreatedAt time.Time `json:"created_at"` -} +Freshness guard — `scripts/verify-toolchain-pin.sh` (runs in `quality-checks.yml` on every PR, fast, no Docker build). **B7 — failure-closed on same-repo PRs:** + +```bash +#!/usr/bin/env bash +set -euo pipefail +KEY="$(scripts/toolchain-key.sh)" +PINNED_TAG="$(grep -E '^ARG CHARON_TOOLCHAIN_TAG=' Dockerfile | cut -d= -f2)" +PINNED_DIGEST="$(grep -E '^ARG CHARON_TOOLCHAIN_DIGEST=' Dockerfile | cut -d= -f2)" + +# Is this a trusted, same-repo run (has/should-have a registry-read token)? +# - push / same-repo pull_request / workflow_dispatch / schedule -> SAME_REPO=1 +# - pull_request from a fork -> SAME_REPO=0 +SAME_REPO=1 +if [[ "${GITHUB_EVENT_NAME:-}" == "pull_request" \ + && "${GITHUB_EVENT_PULL_REQUEST_HEAD_REPO_FULL_NAME:-}" != "${GITHUB_REPOSITORY:-}" ]]; then + SAME_REPO=0 +fi + +if [[ "$KEY" != "$PINNED_TAG" ]]; then + echo "::error::Toolchain recipe/pins changed (recomputed $KEY, Dockerfile pins $PINNED_TAG)." + echo "::error::Run the 'Toolchain Image' workflow (workflow_dispatch) or wait for the bot PR, then bump ARG CHARON_TOOLCHAIN_TAG/DIGEST." + exit 1 +fi + +if [[ "$SAME_REPO" == "1" ]]; then + # HARD requirement: the tool AND the token must be present, and the pinned digest + # MUST resolve and MUST equal what GHCR serves for :$KEY. No silent skip. + command -v regctl >/dev/null || { echo "::error::regctl missing on a same-repo run — cannot verify digest"; exit 1; } + : "${GHCR_READ_TOKEN:?::error::GHCR_READ_TOKEN unset on a same-repo run — cannot verify digest}" + REMOTE_DIGEST="$(regctl image digest "ghcr.io/wikid82/charon-toolchain:$KEY")" \ + || { echo "::error:::$KEY does not resolve in GHCR — toolchain image was never published for this pin"; exit 1; } + if [[ "$REMOTE_DIGEST" != "$PINNED_DIGEST" ]]; then + echo "::error::Dockerfile pins $PINNED_DIGEST but GHCR :$KEY = $REMOTE_DIGEST (hand-edited or stale)." + exit 1 + fi + echo "Toolchain pin verified (same-repo): $KEY @ $PINNED_DIGEST" +else + # Fork PR: no packages:read, cannot reach GHCR. Degrade to tag-only equality + # (already checked above). The real digest check runs when a maintainer + # re-dispatches the same-repo event (see security-pr.yml workflow_run gate). + echo "::warning::Fork PR — digest existence not verified (no registry access). Tag matches recomputed key." +fi ``` -#### 3.5.3 Query strategy — one windowed query, not N - -`UptimeSummaryService.GetSummary(ctx, beats int) ([]MonitorSummary, error)`: - -1. **Cache check** — `summaryCache`-style struct (`sync.Mutex` + value + `expiresAt`), `ttl = 30 * time.Second`, keyed by `beats`. Copied from `stats_service.go:36–56`. Hit ⇒ return. -2. **Monitor metadata** — `SELECT ... FROM uptime_monitors ORDER BY name ASC` (≤ 500 rows, one query). -3. **Recent beats** — one windowed query: +`GHCR_READ_TOKEN` is `${{ secrets.GITHUB_TOKEN }}` (has `packages: read` for a repo-internal package once N8's package-linking is done); `regctl` is installed by the job (`ghcr.io/regclient/regctl` container or `iarekylew00t/regctl-installer`). - ```sql - SELECT monitor_id, status, latency, created_at - FROM ( - SELECT monitor_id, status, latency, created_at, - ROW_NUMBER() OVER (PARTITION BY monitor_id ORDER BY created_at DESC) AS rn - FROM uptime_heartbeats - WHERE created_at >= ? -- now - 24h (bounds the scan; also feeds uptime_24h) - ) - WHERE rn <= ? -- beats - ORDER BY monitor_id, created_at ASC; - ``` +- On a **same-repo PR that legitimately bumps a pin**: `toolchain-image.yml` (path trigger) builds & pushes `:`, and its `sync-pin-on-pr` job commits the `CHARON_TOOLCHAIN_TAG`/`DIGEST` bump onto the PR head branch, so the guard goes green within the same PR. +- On a **fork PR that bumps a pin**: guard fails on the tag mismatch with instructions to have a maintainer dispatch the workflow — acceptable, rare, safe. The fork's app build meanwhile uses the inline fallback (§3.7). - Backed by new index `idx_heartbeat_monitor_created (monitor_id, created_at)` (§3.5.6) — correct but slower without it. -4. **24h uptime** — one grouped query over the same window: +#### 3.4.3 Jobs - ```sql - SELECT monitor_id, - SUM(CASE WHEN status = 'up' THEN 1 ELSE 0 END) * 100.0 / COUNT(*) AS pct - FROM uptime_heartbeats - WHERE created_at >= ? -- now - 24h - GROUP BY monitor_id; - ``` -5. **Assemble** in Go (map join on `monitor_id`), set cache, return. - -Total: **3 SQL queries** regardless of monitor count (was N+1 HTTP round-trips + N queries). Steady-state p95 target < 300 ms at 500 monitors / 24 h of heartbeats (~720 rows/monitor in-window at 30 s → ~360 k row scan, index-covered). - -**Automated perf gate (S5).** C7 adds `TestUptimeSummary_PerfBudget` (`-short`-skippable): seeds 500 monitors each with 24 h of 60 s-interval heartbeats (~360 k rows), builds `idx_heartbeat_monitor_created`, then asserts `GetSummary(ctx, 30)` (cache cleared) completes **under 2 s wall-clock** — a deliberately loose CI-stable ceiling. The < 300 ms p95 is the real target, tracked via the QA run's timing output but not the hard CI gate (runner variance). §7 acceptance criterion #4a. - -#### 3.5.4 Detail history endpoint — kept, paginated, capped - -`GET /api/v1/uptime/monitors/:id/history` stays for the expanded/detail view only. Changes to `UptimeHandler.GetHistory` + `UptimeService.GetMonitorHistory`: - -- `limit`: default **60**, **hard cap 500** (currently uncapped `strconv.Atoi`). Values ≤ 0 → default. -- New optional `before` query param (RFC3339): returns heartbeats with `created_at < before`, for "load older" paging. Query: `WHERE monitor_id = ? AND created_at < ? ORDER BY created_at DESC LIMIT ?`. -- Response unchanged (`[]UptimeHeartbeat`). - -#### 3.5.5 `GET /api/v1/uptime/health` - -New `UptimeHandler.Health` (mirrors `StatsHandler.GetStatsHealth`): - -```jsonc -GET /api/v1/uptime/health → 200 -{ - "heartbeats_dropped": 0, // UptimeIngester.DroppedCount() - "checks_enqueue_dropped": 0, // UptimeWorkerPool.EnqueueDropped() - "queue_depth": 3, // UptimeWorkerPool.QueueDepth() - "worker_pool_size": 30 -} +``` +build-toolchain: + - checkout + - KEY=$(scripts/toolchain-key.sh); echo to $GITHUB_OUTPUT + - Set up QEMU? NO. Set up Buildx. + - login GHCR (skip on fork) + - docker buildx build + --target toolchain-runtime + --platform linux/amd64,linux/arm64 + $( [[ force_rebuild || schedule ]] && echo --no-cache --pull ) + --cache-from type=gha,scope=toolchain + --cache-to type=gha,mode=max,scope=toolchain + -t ghcr.io/wikid82/charon-toolchain:${KEY} + $( same-repo && echo -t ghcr.io/wikid82/charon-toolchain:latest ) + -t ghcr.io/wikid82/charon-toolchain:$(date +%Y%m%d) + $( same-repo && echo --push || echo --output=type=cacheonly ) + --iidfile /tmp/toolchain-iid.txt + . + - DIGEST=$(regctl image digest ghcr.io/wikid82/charon-toolchain:${KEY}) + - outputs: key, digest + +trivy-scan: + needs: build-toolchain + - trivy image --severity CRITICAL,HIGH --exit-code 1 --ignorefile .trivyignore \ + ghcr.io/wikid82/charon-toolchain@${{ needs.build-toolchain.outputs.digest }} + - trivy image --format sarif ... → upload-sarif (category: toolchain-image:trivy) + - continue-on-error on the gate step is FALSE on schedule/dispatch (must be clean), + TRUE on PR (report-only; the app-image Trivy gates still run downstream) + +sync-pin-on-pr: # only when event == pull_request && same-repo && pins moved + needs: [build-toolchain] + - sed -i "s|^ARG CHARON_TOOLCHAIN_TAG=.*|ARG CHARON_TOOLCHAIN_TAG=${KEY}|" Dockerfile + - sed -i "s|^ARG CHARON_TOOLCHAIN_DIGEST=.*|ARG CHARON_TOOLCHAIN_DIGEST=${DIGEST}|" Dockerfile + - git commit -m "chore(docker): sync toolchain image pin to ${KEY}" && git push (to PR head branch) + +open-bump-pr: # event == schedule | workflow_dispatch | workflow_call ; NEVER on pull_request + needs: [build-toolchain, trivy-scan] + if: digest changed vs Dockerfile pin + - sed -i the two ARG lines (CHARON_TOOLCHAIN_TAG, CHARON_TOOLCHAIN_DIGEST) + - docker build --check -f Dockerfile . + - peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1 + base: development + branch: bot/bump-toolchain-image # updated in place if already open + title: "feat(security): refresh bundled proxy toolchain image" + labels: dependencies, automated, docker, security + body: old→new digest, Trivy CRITICAL/HIGH summary, verification checklist + - on failure: actions/github-script → open issue "🚨 Toolchain image rebuild failed" ``` -#### 3.5.6 Index change — deferred creation (prune-first ordering) - -The summary + history + prune access patterns want a composite `idx_heartbeat_monitor_created (monitor_id, created_at)`. The existing `idx_heartbeat_lookup (monitor_id, status, created_at)` on `models.UptimeHeartbeat` is **kept unchanged** (used elsewhere). - -**The new index is NOT declared via a struct tag** and is NOT built by `AutoMigrate`. On a long-lived instance `uptime_heartbeats` can be millions of rows; building the index through the single connection at startup would block all writes for tens of seconds to minutes. Instead the index is created **after the retention pruner has trimmed the table**, so the build runs against an already-small dataset: +The bot-PR title/body must **not** name the specific CVE/dependency (CLAUDE.md `(security)` vagueness rule): "refresh bundled proxy toolchain image so the shipped Caddy/CrowdSec binaries pick up upstream fixes." -1. `models.UptimeHeartbeat` tags are left as they are today (no `idx_heartbeat_monitor_created` entry). -2. `UptimePruner` owns the index creation with a **retry-until-success** loop — no `sync.Once`. At the **end of every hourly pass** where `pruneOnce` returned `err == nil` and the chunk loop reached its "caught up" break (the first such pass runs ~30 s after boot; see §3.4.2), the pruner issues: - ```sql - CREATE INDEX IF NOT EXISTS idx_heartbeat_monitor_created - ON uptime_heartbeats (monitor_id, created_at); - ``` - `CREATE INDEX IF NOT EXISTS` is idempotent: on a healthy instance it lands on the first pass and every later pass is a ~free no-op; if the first pass errored or was `ctx`-interrupted, the next hourly pass retries. There is no path where a transient early failure leaves the index unbuilt until a process restart. -3. **`charon migrate` CLI** (`backend/cmd/api/main.go`, the `case "migrate"` block with its own `db.AutoMigrate(...)` list): after `AutoMigrate`, log a warning and run the same `CREATE INDEX IF NOT EXISTS` **unconditionally**: - ```go - logger.Log().Warn("building index idx_heartbeat_monitor_created on uptime_heartbeats; " + - "on a large database this can take several minutes and holds a write lock for the duration") - db.Exec("CREATE INDEX IF NOT EXISTS idx_heartbeat_monitor_created ON uptime_heartbeats (monitor_id, created_at)") - ``` - `charon migrate` is **not** prune-first — it runs the build against the full table. That is acceptable *only* because the operator invoked it deliberately in a maintenance window; the warning makes the cost visible, and the Phase 5 deploy note (S7) tells operators to lower `uptime.heartbeat_retention_days` first if the table is huge and they want the build to finish quickly. -4. **Before the index exists** (server start → pruner's first successful pass), `UptimeSummaryService.GetSummary` **stays available with correct results** — the `ROW_NUMBER()` query falls back to `idx_heartbeat_lookup` (`(monitor_id, …)` prefix) or a `created_at >= now-24h`-bounded scan, cushioned by the 30 s TTL cache (≈ 2 executions/min). Slower, not broken; **never** 503-gated. `UptimeSummaryService` logs once when it first observes the index present (cheap `PRAGMA index_list` on a cache miss). +**Trivy gate semantics (revised):** the `--exit-code 1` CRITICAL/HIGH step is **blocking** on `schedule` / `workflow_dispatch` / `workflow_call` (a known CRITICAL in the bundled binaries turns the daily run red → failure issue). On `pull_request` it is **report-only** (`continue-on-error: true`) because the app-image Trivy gates in `docker-build.yml` / `security-pr.yml` still run downstream and a contributor PR must not be blocked by a pre-existing bundled-binary finding they did not introduce. -**Tradeoff (honest, per S2).** On a healthy 500-monitor instance the first-boot index build still runs over **~65 M rows** and is a **bounded multi-minute operation that contends for the single write connection** (readers unaffected — WAL). Prune-first only removes the *pathological* case (hundreds of millions of rows on years-stale instances). The reasons this is acceptable are in §3.4.2: background goroutine, no route downtime, retried-until-success, operator escape hatches. It is **not** claimed to be a "no-stall" design. See revised R3/R4/R7 in §6. +#### 3.4.4 Repurpose `security-weekly-rebuild.yml` (N6) -**Commit placement:** the index creation lives in **Commit 6 (retention pruner)**, not Commit 2. Commit 2 only adds `NextCheckAt` on `uptime_monitors` (a ≤ 500-row table — trivially fast via the existing struct-tag/AutoMigrate path). The `migrate`-CLI `CREATE INDEX IF NOT EXISTS` line also lands in Commit 6. +Replace its `Build Docker image (NO CACHE)` step — which builds a `charon:security-scan-YYYYMMDD` app image **that nothing consumes** — with `uses: ./.github/workflows/toolchain-image.yml` (`workflow_call`, `force_rebuild: true`). Keep its Trivy CRITICAL/HIGH table + SARIF upload + JSON artifact + failure `::warning::` steps, re-pointed at the toolchain digest. -#### 3.5.7 Frontend changes +- Its `permissions:` block (`security-weekly-rebuild.yml:21`, currently `contents: read`, and job-level `:36-39` `contents/packages/security-events`) **must add `pull-requests: write`** — a `workflow_call`ed workflow cannot request perms the caller did not grant, so the caller must grant everything `open-bump-pr` needs (`contents: write`, `pull-requests: write`, `packages: write`, `security-events: write`). +- Keep `TRIVY_SARIF_CATEGORY` stable to avoid duplicate code-scanning tracks; rename the value to `…:trivy-toolchain`. +- **Cadence:** the Tuesday slot stays for the fuller report; the **daily** `schedule` in `toolchain-image.yml` (§3.4.1) is the freshness driver. What the forced rebuild actually catches is stated precisely in §3.8 — **not** "upstream `go get` MVS drift" (that claim was wrong, see §3.8 / B3). -`frontend/src/api/uptime.ts`: +### 3.5 Multi-arch handling (hard constraint) -```ts -export interface BeatDTO { status: string; latency: number; created_at: string; } -export interface MonitorSummary { - id: string; name: string; type: string; url: string; enabled: boolean; - status: string; latency: number; last_check: string | null; interval: number; - proxy_host_id?: number | null; remote_server_id?: number | null; - uptime_24h: number | null; recent_beats: BeatDTO[]; -} -export const getMonitorsSummary = async (beats = 30): Promise => { - const res = await client.get(`/uptime/monitors/summary?beats=${beats}`); - return res.data; -}; -// getMonitorHistory gains an optional `before` cursor param. -export const getMonitorHistory = async (id: string, limit = 60, before?: string) => { ... }; -``` +**Decision: publish a genuine multi-arch manifest list, built without QEMU via `$BUILDPLATFORM` cross-compilation.** -`frontend/src/hooks/useUptimeSummary.ts` (new): +Justification: +- `caddy-inline` is already `FROM --platform=$BUILDPLATFORM golang:…` + `GOOS=$TARGETOS GOARCH=$TARGETARCH go build` (CGO off). `docker buildx build --platform linux/amd64,linux/arm64` runs this stage once per target platform, all on the amd64 host; each pass emits the correct-arch `caddy`. No emulation. +- `crowdsec-inline` is `FROM --platform=$BUILDPLATFORM golang:…` + `COPY --from=xx / /` + `xx-apk add … musl` + `CGO_ENABLED=1 xx-go build`. `tonistiigi/xx` provides the cross linker/sysroot; this is exactly how CrowdSec cross-compiles today for the arm64 leg of `docker-build.yml`. No emulation. +- `toolchain-runtime` is `FROM ${ALPINE_IMAGE}` + `COPY` only — no `RUN`, so nothing arch-specific executes; BuildKit assembles one layer per platform from the matching `caddy-inline`/`crowdsec-inline` outputs. +- Result: `ghcr.io/wikid82/charon-toolchain:` is a manifest list with `linux/amd64` and `linux/arm64` children. In the app build, `FROM …@sha256: AS toolchain-prebuilt` **without** `--platform` → BuildKit auto-selects the child matching the app build's `$TARGETPLATFORM`. So `docker-build.yml`'s `build-amd64` pulls the amd64 child, `build-arm64` (QEMU) pulls the arm64 child, and each does a plain `COPY --from` instead of running the builders. +- **N2 — accurate framing:** the Caddy/CrowdSec compile was *never* QEMU-emulated on the arm64 leg — both builder stages are `FROM --platform=$BUILDPLATFORM` and always cross-compiled natively on the amd64 host. QEMU on `build-arm64` only ever executed the *final* arm64 stage's `RUN` lines (apk installs, setcap, GeoIP fetch, verification). The real win here is **no compile at all** on any app build (cold or warm, amd64 or arm64) — not "arm64 stops emulating a compile". The arm64 leg still runs its final-stage `RUN` lines under QEMU exactly as before. +- The pinned `CHARON_TOOLCHAIN_DIGEST` is the **manifest-list digest** (arch-independent), so one pin covers both arches. -```ts -export const useUptimeSummary = () => - useQuery({ queryKey: ['uptimeSummary'], queryFn: () => getMonitorsSummary(30), refetchInterval: 30000 }); -``` +Runner cost: the toolchain workflow does ~30 min of cross-compile once a day on one `ubuntu-latest` (mostly cache-hit no-ops between pin bumps), versus today's cold ~14-min compile on effectively every app build across `docker-build` (×2 arch), `nightly-build` (daily), `security-pr`, `supply-chain-pr`, `e2e-tests-split`, and 4 integration workflows. -`frontend/src/pages/Uptime.tsx`: +### 3.6 `--no-cache-filter` retarget (Commit 1) then removal (Commit 4, R6) — exact edits -- `Uptime` component calls `useUptimeSummary()` **once**. Grouping (`proxyHostMonitors` / `remoteServerMonitors` / `otherMonitors`) and alpha sort operate on `MonitorSummary[]`. -- `MonitorCard` prop type changes `UptimeMonitor` → `MonitorSummary`; it **no longer calls `useQuery(['uptimeHistory', ...])`**. `history` becomes `monitor.recent_beats`; `latestBeat`, `effectiveStatus`, the heartbeat bar, latency, and last-check all read from props. `hasHistory = recent_beats.length > 0`. -- Heartbeat bar: the fixed "last 60" grid becomes `recent_beats.length` wide (default 30), padded with empty slots up to a `BEAT_BAR_SLOTS` constant (set to 30 to match the new default). The `title`/tooltip copy ("Last 60 checks") updates to reflect the slot count. An expanded/detail view can request 60 and render the wider bar. -- `checkMutation` success handler invalidates `['uptimeSummary']` (was `['monitors']` + `['uptimeHistory', id]`). -- `deleteMutation` / `toggleMutation` / `syncMutation` invalidate `['uptimeSummary']`. -- Create/Edit modals: `interval` `` `min="30"` (was `10`); on blur/submit clamp `< 30 → 30`; helper text "Minimum 30 seconds". Submit sends `interval` unchanged otherwise. -- Detail/expanded view (existing "Configure" path or a future drill-in) is the only remaining caller of `getMonitorHistory`, now with `limit`/`before` paging. -- **Manual "check now" / "sync" queue-full feedback (N5):** `POST /uptime/monitors/:id/check` returns `503 {"error":"check queue is full, try again"}` when the pool is saturated; `POST /system/uptime/check` returns `{"enqueued": N, "dropped": M}`. `checkMutation` / `syncMutation` surface a `503` (or any `dropped > 0`) as a toast — "Check queue full, try again in a moment" — instead of a silent success. `api/uptime.ts` types: `checkMonitor` may reject with a 503; `syncMonitors` response gains `enqueued` / `dropped`. -- `UptimeWidget.tsx` may switch `getMonitors` → `getMonitorsSummary` for consistency (**optional**, low-risk; keep `getMonitors` for now if it complicates the commit). +**Two-step, per B5.** Commit 1 changes the *value* at every site from `caddy-builder,crowdsec-builder` to `caddy-inline,crowdsec-inline` (so the recurrence guard keeps invalidating the actual `RUN` layers through the rename). Commit 4 — only after `verify-toolchain-pin` is a live required check — deletes them entirely. The table below is the Commit 4 removal list; Commit 1 touches the same sites with a value change. -`getMonitors` (plain list) is **retained** — still used by `UptimeWidget`, the proxy-host form, and tests. +| File | Edit (Commit 4 = delete; Commit 1 = retarget value first) | +|---|---| +| `docker-build.yml:463-464` | delete the two `--no-cache-filter …` array lines in the `build-amd64` `BUILD_CMD` | +| `docker-build.yml:549-550` | delete the two `--no-cache-filter …` lines in `build-arm64` `BUILD_CMD` | +| `docker-build.yml:392` | rewrite comment: drop "no-cache-filter passed as native buildx flags"; note toolchain image is digest-pinned so layer cache is authoritative | +| `security-pr.yml:157-164` | remove the `with: no-cache-filters:` block + its 6-line justification comment from the `build-charon-image` step | +| `supply-chain-pr.yml:252-261` | same removal | +| `e2e-tests-split.yml:224` | delete `no-cache-filters: caddy-builder,crowdsec-builder` from the `docker/build-push-action` `with:` | +| `nightly-build.yml:243` | delete `no-cache-filters: caddy-builder,crowdsec-builder` | +| `.github/actions/build-charon-image/action.yml` | remove the `no-cache-filters` input (`:11-33` decl) and the `no-cache-filters: ${{ inputs.no-cache-filters }}` passthrough (`:52`); rewrite the `description:` to state the toolchain image is prebuilt+digest-pinned and every stage is layer-cached | +| `crowdsec-integration.yml`, `waf-integration.yml`, `rate-limit-integration.yml`, `cerberus-integration.yml` | no change needed (none passes the input) — but verify after the input is deleted that the composite still resolves (it will; input had a default) | ---- +After removal, add to each build step (where not already present) `--build-arg CHARON_TOOLCHAIN_DIGEST` is **not** needed — the Dockerfile default is authoritative. CI passes nothing extra on the happy path. -### 3.6 Component F — Config surface +### 3.7 Fork PR / bootstrap / offline fallback (R4, hard constraint) -#### 3.6.1 New `models.Setting` rows (seeded in `routes.go`, `FirstOrCreate` like `feature.uptime.enabled`) +Three cases, one mechanism (`CADDY_BUILDER_SRC` / `CROWDSEC_BUILDER_SRC` build-args, §3.2.1): -| Key | Type | Default | Bounds | Hot-reload? | -|---|---|---|---|---| -| `uptime.default_interval_seconds` | `int` | `60` | 30 – 86400 | **Yes** — scheduler reads via `uptimeConfig` snapshot (60 s TTL) when clamping legacy/zero intervals and hydrating new monitors. | -| `uptime.worker_pool_size` | `int` | `30` | 1 – 200 | **No** — pool sized at construction; change requires restart. `GET /uptime/health` surfaces the active value so operators can confirm. | -| `uptime.heartbeat_retention_days` | `int` | `90` | 1 – 3650 | **Yes** — `UptimePruner` reads the snapshot at the start of each hourly pass. | - -`Category = "uptime"` on all three. - -#### 3.6.2 `uptimeConfig` — hot-reloading snapshot - -```go -type uptimeConfig struct { - db *gorm.DB - mu sync.RWMutex - val cachedUptimeCfg - exp time.Time - now func() time.Time // injectable clock — test seam (N8) - ttl time.Duration // default 60s -} -// snapshot() refreshes from the Settings table if c.now() > exp, else returns cached. -func (c *uptimeConfig) DefaultIntervalSeconds() int // reads snapshot() -func (c *uptimeConfig) RetentionDays() int // reads snapshot() -func (c *uptimeConfig) forceRefresh() // test-only: expire the cache now - -func clampInterval(seconds int, cfg *uptimeConfig) int { - if seconds <= 0 { seconds = cfg.DefaultIntervalSeconds() } - if seconds < 30 { seconds = 30 } - return seconds -} -``` +| Case | Detection | Behavior | +|---|---|---| +| **Fork PR** (no `packages: write`, cannot pull an internal image) | job-level expression `github.event.pull_request.head.repo.full_name != github.repository` sets `TOOLCHAIN_SRC=inline` | Every app-image build step passes `--build-arg CADDY_BUILDER_SRC=${{ env.CADDY_SRC }} --build-arg CROWDSEC_BUILDER_SRC=${{ env.CROWDSEC_SRC }}` where the two env vars are `caddy-inline`/`crowdsec-inline` on a fork and `toolchain-prebuilt`/`toolchain-prebuilt` otherwise. Full from-source compile (~14 min). Layer cache (`type=gha`) still applies to the fork's own repeated runs. **Because this path exists, the job `timeout-minutes` for every fork-reachable build job stays ≥ 20 (see §3.9 / B6) — it is NOT cut to 15.** | +| **Bootstrap** (toolchain image does not yet exist) | first `toolchain-image.yml` run publishes it; until then `CHARON_TOOLCHAIN_DIGEST` is a placeholder | Commit 1 publishes the image manually (`workflow_dispatch`) and links/marks the GHCR package internal (N8) **before** Commit 2 flips the Dockerfile default. Freshness guard lands in Commit 3; the `--no-cache-filter` sites are only removed in Commit 4, after the guard is live. | +| **Local `docker build`** (dev, offline, or not logged into GHCR) | developer choice | `docker build .` uses the pinned image (one ~30 MB pull, then cached). Offline / air-gapped: `make build-offline` → `docker build --build-arg CADDY_BUILDER_SRC=caddy-inline --build-arg CROWDSEC_BUILDER_SRC=crowdsec-inline .`. | -Shared by `UptimeScheduler`, `UptimePruner`, and `UptimeService` (for monitor-creation default resolution — §3.6.3). Read-only; writes go through the normal Settings endpoint. `now` + `forceRefresh` exist so hot-reload tests can force a TTL expiry without sleeping. +**Security non-regression:** the release/CVE-gate paths — `docker-build.yml` (amd64+arm64), `nightly-build.yml`, `security-pr.yml`, `supply-chain-pr.yml` — always use the default (`toolchain-prebuilt`, digest-pinned, **daily-`--no-cache --pull`-rebuilt-and-scanned**). Fork PRs use `caddy-inline`, which is byte-for-byte the same recipe (same `go get pkg@fixed` lines, same embeds-version assertions) — a fork build is *not weaker*, just slower and unpinned. A fork PR cannot merge without a maintainer re-running the trusted same-repo path (`security-pr.yml` already gates this via its `workflow_run` trust-boundary check at `:146`), at which point the real prebuilt+scanned image is exercised. -**Field-name note:** `UptimeService` already has an unrelated `config UptimeConfig` field (timeout/threshold struct). The injected `*uptimeConfig` is stored as `s.uptimeCfg` to avoid the collision. +### 3.8 Security-guarantee analysis (CVE-2026-84304-class recurrence) -#### 3.6.3 Server-side interval-floor validation +#### 3.8.1 The true current baseline (B2 — corrected) -- **`SettingsHandler.UpdateSetting`** — add a `uptime.*` branch (precedent: the `backup.*` / `security.admin_whitelist` branches at `settings_handler.go:143`): parse int, enforce the bounds in the table above, `400` with `{"error": "...", "error_code": "invalid_uptime_setting"}` on violation. -- **`UptimeHandler.Create`** — `CreateMonitorRequest.Interval`: if `> 0 && < 30` → `400 {"error": "interval must be at least 30 seconds"}`. If `0` → passed through; `CreateMonitor` resolves it to `cfg.DefaultIntervalSeconds()` at **write time** (via `clampInterval(interval, s.uptimeCfg)`) so the stored value is always concrete and visible in the UI. -- **`UptimeService.UpdateMonitor`** — the `interval` whitelist branch (`uptime_service.go:1295`) gains the same floor check via `clampInterval`; reject `> 0 && < 30` with a typed `ErrIntervalTooLow`; the handler maps it to `400`. -- **ALL monitor-creation paths route through the same write-time resolution (S3).** `CreateMonitor`, **and the auto-create sites** `SyncMonitors` (`uptime_service.go` ~223 and ~320), `SyncAndCheckForHost` (~1402), and the new `SyncAndCheckForRemoteServer` — currently every one of these hardcodes `Interval: 60` on the struct literal. Replace each `Interval: 60` with `Interval: clampInterval(0, s.uptimeCfg)` (i.e. resolve to `uptime.default_interval_seconds`), so proxy-host / remote-server monitors honour the admin global default instead of being pinned to 60 s. `clampInterval(0, …)` also floors correctly if an admin sets the default below 30. -- **`CreateMonitor` signature** unchanged (`name, url, type, interval, maxRetries`) — only the internal default/floor resolution changes. -- **Test (C5):** set `uptime.default_interval_seconds = 45`, force a `uptimeConfig` refresh, run `SyncAndCheckForHost` / `SyncAndCheckForRemoteServer`, assert the created monitor's `Interval == 45`. +Rev 1 understated this. Today, `--no-cache-filter caddy-builder,crowdsec-builder` forces a from-scratch rebuild of the two builder stages: -#### 3.6.4 Frontend +- on **`nightly-build.yml`** — `schedule: '0 9 * * *'`, i.e. **daily**, and it builds the *shipped* `nightly` multi-arch image (`nightly-build.yml:229-243`); +- on **every** `docker-build.yml` release build (push to `main`/`development`/`nightly`, every version tag); +- on **every** `security-pr.yml` / `supply-chain-pr.yml` CVE-gate run (per PR); +- on **every** `e2e-tests-split.yml` image build (per PR / per run). -**Per-monitor interval field** — covered in §3.5.7 (Create/Edit modal `min="30"` + clamp + helper text). +So the effective current cadence at which the bundled Caddy/CrowdSec binaries are recompiled from source (re-running every `go get pkg@fixed`, re-resolving `go mod tidy`, re-pulling base images via the accompanying `--pull`) is **at least daily, and in practice several times a day on active days**. The weekly `security-scan-YYYYMMDD` image is a *scan* artifact, not the only rebuild. -**Admin "Uptime" settings card — IN SCOPE for this PR (Commit 8).** Added to `frontend/src/pages/SystemSettings.tsx`, which already renders the `feature.uptime.enabled` toggle and already wires `getSettings` / `updateSetting` from `frontend/src/api/settings.ts` and the `Card` primitives (`components/ui/Card`). No new page or route. +#### 3.8.2 What actually changes, and why the new cadence is acceptable -- **Placement:** a new `` ("Uptime Monitoring", below the existing feature-flags card). Rendered only when `feature.uptime.enabled` is on (reuse the `featureFlags` query already in the file). -- **Fields** (three number inputs, seeded from the `settings` map returned by `getSettings`): +After this change the forced-rebuild driver is the **daily** `schedule` on `toolchain-image.yml` (§3.4.1) plus per-PR rebuilds whenever a tracked pin moves. Refresh latency for the *shipped* image becomes: `daily toolchain rebuild` → `bot PR` → `human merge of bot PR` → next app build picks up the new digest. - | Field | Setting key | Input bounds (client, must match §3.6.1 server bounds) | Helper text | - |---|---|---|---| - | Default check interval (seconds) | `uptime.default_interval_seconds` | `min=30 max=86400 step=1` | "New monitors inherit this. Applies within ~60 s, no restart." | - | Worker pool size | `uptime.worker_pool_size` | `min=1 max=200 step=1` | "Concurrent checks. **Requires a restart to take effect.** Current active value shown on the Uptime page health indicator." | - | Heartbeat retention (days) | `uptime.heartbeat_retention_days` | `min=1 max=3650 step=1` | "Older heartbeats are permanently deleted. Applies within ~1 h, no restart." | +| Property | Today | After | +|---|---|---| +| Bundled-binary recompile cadence (no pin moved) | daily (nightly) + per active PR | **daily** (toolchain `schedule`) | +| Latency from a new toolchain digest to it being in the shipped image | 0 (next nightly/release builds it directly) | **daily rebuild + bot-PR merge latency** (target: merge within 1 business day; the bot PR is `feat(security)`-labelled and shows in the same queue as a Renovate security bump) | +| Human step in the loop | none | **yes — a maintainer merges `bot/bump-toolchain-image`** | -- **Validation:** client-side bounds check on blur/submit mirroring the server (`< min` / `> max` / non-integer ⇒ inline error, save button disabled). The server is still authoritative — a rejected `POST /api/v1/settings` (`400 { error_code: "invalid_uptime_setting" }`) surfaces as a toast. -- **Save:** one `useMutation` calling `updateSetting(key, String(value), 'uptime', 'int')` per changed field (same pattern as the existing `saveSettingsMutation` in the file), then `invalidateQueries(['settings'])`. Only changed fields are written. The three writes are **independent** — each key is a standalone tuning knob with no cross-coupling — so partial success (e.g. field 2 rejected by the server while fields 1 and 3 persist) is acceptable; the card re-reads `getSettings` after the mutation settles and re-renders from the actual persisted state, so the UI always reflects what is stored rather than what was attempted. -- **i18n:** add `systemSettings.uptime.*` keys (card title, three labels, three helper texts, validation messages) to the locale files. -- **No dedicated typed endpoint** — the generic `POST /api/v1/settings` with the `uptime.*` validation branch (§3.6.3) is sufficient; `backup.*`-style endpoint carve-out is not needed because these keys have no cron/side-effect coupling. +The added human-merge step is the real trade. It is acceptable because: (a) the daily rebuild + Trivy gate still *detects* a problem on the same ~24 h cadence as today — only *shipping* the fix now waits on a PR merge; (b) the bot PR is small (two ARG lines), CI-verified, and lands in the security review queue the team already watches for Renovate; (c) an urgent case is a one-click `workflow_dispatch` + expedited merge (~30 min end to end, §6); (d) the alternative — auto-committing digest bumps to `development` with no review — is worse for a security-sensitive artifact. **The daily cadence (not weekly) is therefore committed scope**, precisely so the *detection* cadence matches today's; only the merge step is new. ---- +#### 3.8.3 What the forced `--no-cache --pull` rebuild does and does NOT catch (B3 — corrected) -### 3.7 Error handling & edge cases +Rev 1 claimed the weekly rebuild's "fresh `go mod tidy` MVS → new binary" catches upstream fixes to **unpinned transitive** deps. **That claim is withdrawn — it is false:** -| Scenario | Behavior | -|---|---| -| Worker queue full (thundering herd / pool starvation) | `TryEnqueue` → false; scheduler leaves the monitor/host due, retries next 5 s tick; `checks_enqueue_dropped` increments; `WARN` (rate-limited). No goroutine leak, no lost monitor. | -| Ingester channel full | `emit` drops the result; `heartbeats_dropped` increments; the **DB row** is briefly stale until the next flushed check. Detection is **unaffected** — the authoritative `monState` was already updated synchronously by the worker (§3.3.3), and the notification (if any) already fired. | -| Failing monitor + sustained ingester saturation (B3) | Every `CheckResult` drops, so nothing persists — but the worker still increments `monState[id].failureCount` under `monMu` on each check, so the `down` transition **is** detected at `failureCount >= maxRetries` and the alert fires. The DB catches up whenever a flush next succeeds. | -| Process restart mid-cycle | `SeedState` reseeds `monState`/`hostState` from the DB mirror (≤ last flush); scheduler cold-start reads `next_check_at`, jitter-backfills past-due over 60 s. `failureCount` may be stale-low by a few → a near-transition monitor fires ≤ 1–2 cycles later. No stampede, no missed alert. | -| Monitor disabled while a check is in flight | Worker checks `job.Monitor.Enabled` and **emits nothing** if false; scheduler drops it from `monSchedule` on the next `rescan()` (≤ 30 s). | -| Monitor deleted while queued | Worker runs, ingester `UPDATE ... WHERE id = ?` affects 0 rows (no-op); a dangling heartbeat row is pruned by retention; `DeleteMonitor` bulk-deletes heartbeats so the window is tiny. `monState` entry is GC'd on the next `rescan()`. Accepted. | -| Host recovers | Host-check worker sets `hostState[hid]="up"`; scheduler stops skipping that host's TCP monitors; each resumes on its next due tick and re-derives status from its first real result. | -| `interval` below 30 via direct API | `400` (Create/Update). Legacy DB rows with `interval < 30` → `clampInterval` floors to 30 at schedule time; not rewritten unless the monitor is edited. | -| `interval = 0` (legacy rows / auto-created monitors) | `clampInterval` → `uptime.default_interval_seconds`. Auto-create paths now pass `0` deliberately (S3). | -| SQLite `database is locked` during ingester flush | Flush wrapped in `db.Transaction`; on lock error, log + keep the batch for the next flush (bounded retry, mirrors `createMonitorWithRetry`). Drop the batch after 3 failed flushes to bound memory. Detection unaffected (mirror only). | -| Pruner delete contends with API | `pruneChunkPause` between chunks releases the connection. **Steady-state**: 50 ms pause, ~10–30 ms/chunk. **First cold pass on a huge table**: 250 ms pause, up to ~500 ms/chunk — added API/ingester write latency is intermittent, up to ~one chunk, for the pass's duration (§3.4.2 / N1). | -| Summary query before `idx_heartbeat_monitor_created` exists | Runs unindexed (`idx_heartbeat_lookup` prefix / 24 h-bounded scan), correct results, 30 s-cached, **never 503**. Clears once the pruner builds the index (§3.5.6). | -| Window function unsupported (old SQLite) | Not possible with bundled `modernc.org/sqlite`. Guarded by a unit test; fallback per-monitor `LIMIT` loop documented, not implemented. | -| `feature.uptime.enabled = false` | Scheduler tick no-ops (flag cached 60 s); pool idle; ingester idle; **pruner still runs** (retention applies while checking is paused). Summary serves last-known data. | -| Manual `POST /uptime/monitors/:id/check` when pool saturated | `Enqueue` blocks ≤ 2 s then `503 {"error":"check queue is full, try again"}` — surfaced as a toast (N5). | -| Manual `POST /system/uptime/check` when pool saturated | `CheckAll()` returns `{"enqueued": N, "dropped": M}` (never a silent all-drop); frontend toasts if `dropped > 0` (N5). | -| Orthrus monitor, subsystem down | Unchanged (`"Orthrus subsystem unavailable"` → `down` heartbeat). | -| Summary endpoint before any heartbeats exist | `recent_beats: []`, `uptime_24h: null`, `status` from `uptime_monitors.status` (`"pending"`). | -| **Backup / restore** (S6) | See §3.9. Restore-then-restart = ordinary crash-recovery cold start. Live restore without restart = self-heals within one `rescan()` (≤ 30 s) + 1–2 check cycles per monitor; `UptimeScheduler.Rehydrate()` (called from the restore reconcile step) makes it immediate. | - -### 3.8 Data flow - -#### 3.8.1 One monitor check, end to end +- `go mod tidy` / MVS is **deterministic**. It selects the *minimum* version satisfying the constraints in `go.mod`/`go.sum`. An upstream project publishing a patched `v1.2.4` does **not** cause MVS to move off `v1.2.3` unless something in the require graph raises the lower bound. "Latest patch" is not an MVS input. +- `docker buildx build --no-cache` invalidates *layer* cache. It does **not** clear the `RUN --mount=type=cache,target=/go/pkg/mod` BuildKit cache mount — the Go module cache persists across `--no-cache` builds. (`--pull` only refreshes `FROM` images.) -``` -scheduler tick (5s) - └─ monDue = {m1 (interval 30, next_check_at 12:00:00)} , now=12:00:03 - ├─ loadJobSnapshots([m1]) ── 1 batched SELECT (static fields; dynamic cols ignored for debounce) - ├─ m1.Type=="tcp" && host known-down? ── pool.HostState(hid) → no → proceed - ├─ pool.TryEnqueue(UptimeJob{Kind: Monitor, Monitor: m1}) ── ok - ├─ monSchedule[m1] = 12:00:33 ; writeback[m1] = 12:00:33 - └─ flushWriteback() ── 1 grouped UPDATE uptime_monitors.next_check_at -worker (1 of 30) - ├─ raw := runCheck(job, sharedClient) ── ValidateExternalURL (L1 DNS) → client.Do (L2 safeDialer, keep-alive) - │ └─ latency=44ms, success=true - ├─ monMu.Lock() - │ ├─ e := monState["m1"] ── AUTHORITATIVE: {status:"down", failureCount:2, ...} - │ ├─ success ⇒ new = {status:"up", failureCount:0, lastStatusChange:now} - │ ├─ StatusChanged = ("down" != "up" && "down" != "pending") = true - │ └─ monState["m1"] = new - ├─ monMu.Unlock() - ├─ notifier.sendRecoveryNotification(m1, "3m 12s") ── SYNC, fires now (before emit) - └─ pool.emit(CheckResult{m1, HeartbeatStatus:"up", Latency:44, Message:"HTTP 200", - NewMonitorStatus:"up", FailureCount:0, StatusChanged:true, StatusChangedAt:now}) -ingester flush (≤500ms later) — MIRROR write, not source of truth - └─ Transaction: - ├─ CreateInBatches([]UptimeHeartbeat{ {m1,"up",44,"HTTP 200"} }) - └─ UPDATE uptime_monitors SET status='up', last_check=…, latency=44, failure_count=0, - last_status_change=… WHERE id='m1' - (if this flush is DROPPED: monState already says "up"; DB catches up on the next successful flush; - the recovery alert already fired — nothing is lost that matters) -UI (next 30s refetch) - └─ GET /uptime/monitors/summary ── cache hit or 3 queries ── card shows UP, 44ms, sparkline -``` +**What the daily `--no-cache --pull` toolchain rebuild genuinely catches:** -#### 3.8.2 Host goes down — short-circuit fan-out +| Vector | Caught? | Mechanism | +|---|---|---| +| Upstream fix to a **pinned** dep (any §2.2 ARG, incl. the two new plugin pins, or a literal `go get x@vN` in the stage body, or the stage text itself) | ✅ per-PR | Renovate/manual bump → `toolchain-key.sh` changes → `verify-toolchain-pin` **fails the PR** until the toolchain is rebuilt and the digest synced | +| **Base-image** drift — new `golang:1.27.1-alpine` / `alpine@sha256:…` / plugin-source-image CVEs | ✅ daily | `--pull` re-resolves the `FROM` digests; with N4's digest-pinned golang base, a Renovate digest bump also trips the key | +| **Alpine package** drift in `toolchain-runtime` / final stage (`apk upgrade`) | ✅ daily (toolchain) + per-release (app `--pull`, kept) | fresh `apk` index on `--no-cache` | +| Trivy signature DB gaining a new match against an **already-shipped** bundled version | ✅ daily | Trivy runs against the toolchain digest every day; new CRITICAL/HIGH → red run + failure issue | +| Upstream security fix to a genuinely **unpinned transitive** Go dep, where nothing raises the MVS lower bound | ❌ — **same gap as today** | only closed by a human adding an explicit `go get dep@fixed` pin (the existing pattern — the stage already has ~40 such pins). Renovate's Go-module manager + the `caddy-major-monitor.yml` / dependency-review tooling surface these; this spec does not change that surface either way. | -``` -scheduler host pass (5s tick) - └─ hostDue = {h7} → pool.TryEnqueue(UptimeJob{Kind: Host, Host: h7}) ; hostSchedule[h7] += minChildInterval -worker - ├─ raw := runHostCheck(job, hostDialer) ── single 3s dial to a child port → fail - ├─ hostMu.Lock() - │ ├─ hostState["h7"] = {status:"down"|"pending"→…, failureCount++} - │ └─ failureCount >= 2 ⇒ transition up→down - ├─ hostMu.Unlock() - ├─ for each tcp child mN of h7 where monState[mN].status != "down": - │ ├─ synthesize CheckResult{HeartbeatStatus:"down", Latency:0, Message:"Host unreachable", Synthetic:true} - │ ├─ monMu: run it through the SAME debounce → monState[mN].status="down", StatusChanged per-child - │ └─ pool.emit(that CheckResult) - ├─ notifier.queueDownNotification(...) ×1 ── SYNC; 30s batch window coalesces → one "N services down on h7" alert - └─ pool.emit(HostCheckResult{h7, Status:"down", ...}) -scheduler (subsequent monitor passes, while h7 down) - └─ for tcp child mN of h7: pool.HostState("h7").Status == "down" - └─ SKIP enqueue; advance monSchedule[mN] += interval (no new heartbeats — nothing changed) -ingester flush - └─ writes the synthetic child `down` heartbeats + coalesced uptime_monitors / uptime_hosts column updates (dumb copy) -h7 recovers → host-check worker sets hostState["h7"].Status="up" → scheduler resumes enqueuing mN on next due tick -``` +**Net:** the recurrence guarantee for *pinned* deps is **strengthened** (a stale pin now hard-fails a PR instead of relying on a cache-key accident). The *unpinned-transitive* gap is **unchanged** — it exists identically today and is out of scope here; the plan explicitly does not claim to close it. -#### 3.8.3 Shutdown (teardown chain, §3.1.4) +**Strengthening vs today:** the daily toolchain Trivy gate is **blocking** on `schedule`/`dispatch`/`workflow_call` (`exit-code 1`) — today's `security-weekly-rebuild.yml` has `continue-on-error: true` on its first Trivy step, so a known CRITICAL currently only produces a `::warning::`. After this change it produces a red run + a tracked GitHub issue, daily. -``` -ctx.Done() - ├─ scheduler: final flushWriteback(); return ── no more enqueues - ├─ syncLoop: return - ├─ pool.Run: stop reading `jobs` → workerWG.Wait() (each in-flight check ≤ hardCap 20s) → close(results) - ├─ ingester: for r := range results { ... } drains until `results` closed → one final flush() → return - └─ pruner: ctx.Err() between chunks → abort current pass → return - (process shutdown grace must be ≥ hardCap + ~2s — verify server.Run shutdown timeout, §3.1.4) -``` +**Optional further hardening (follow-up, not committed):** a twice-daily `schedule` guarded by "rebuild only if no published tag for the current key OR last publish > 12 h" — halves detection latency for modest runner cost. ---- +### 3.9 Timeout right-sizing (R1 side-effect) -### 3.9 Backup / restore interaction (S6) +| Job | File:line | Now | After | Rationale | +|---|---|---|---|---| +| `build-amd64` | `docker-build.yml:403`, `:441` | 15 / 15 | **20 / 20** | No compile on hot path; 20 gives headroom for a cold GHA cache miss on the *fast* stages + cache export + push. `docker-build.yml` never runs the inline fallback (release path, same-repo only), so 20 is safe. | +| `build-arm64` | `docker-build.yml:487`, `:527` | 25 / 25 | **keep 25** | QEMU still runs the final arm64 stage's `RUN` lines + `COPY` from the arm64 toolchain child; 25 stays comfortable. | +| `merge-and-publish` | `docker-build.yml:582` | 10 | keep 10 | unaffected | +| `security-pr` build | `security-pr.yml:32` | 20 | **keep 20** | **B6:** this job IS fork-reachable and runs the inline compile (~14 min) on a fork PR → 14 + checkout + Trivy + overhead would blow a 15-min cap. Keep 20. Update the `:32` comment to: "20m — warm same-repo build ~6–8 m; fork PRs compile the toolchain inline (~14 m + scan), which sets the floor." | +| `supply-chain-pr` build | `supply-chain-pr.yml:34` | 20 | **keep 20** | same reasoning; update comment `:34` identically. | +| `cerberus/crowdsec/waf/rate-limit-integration` | each `:29` | 20 | **keep 20** | fork-reachable + integration test work on top; 20 still right. Update the "first run … full cold build" comments to: "fork PRs build the toolchain inline; same-repo runs `COPY` it from the pinned image". | +| `e2e-tests-split.yml` build job | `:252` etc. | 60 | keep 60 | already generous; fork inline compile fits easily. | +| `toolchain-image.yml` | new | — | **45** | cold amd64+arm64 cross-compile of both stages + Trivy | +| `security-weekly-rebuild.yml` | `:35` | 60 | keep 60 (now mostly the `workflow_call` to toolchain-image) | -**Goroutine start ordering.** The uptime background components are launched inside `routes.Register(ctx, …)`, which runs **after** `database.Connect` and after any pending-restore boot-swap performed during `main.go` / database init. So on a **restore-then-restart** (the pending-restore path, and the recommended flow for `RehydrateLiveDatabase`), the scheduler/pool/pruner cold-start against the already-restored DB — identical to ordinary crash-recovery cold start, **no special handling required**. `SeedState` reseeds `monState`/`hostState` from the restored rows; the scheduler hydrates `next_check_at` (jitter-backfilling past-due entries — an old backup just means "everything is due", bounded by the 60 s spread + per-tick cap + queue cap, i.e. the R5 stampede mitigation already covers it). +**B6 reconciliation, explicit:** §3.7 establishes that `security-pr.yml`, `supply-chain-pr.yml`, and the four `*-integration.yml` jobs run the ~14-minute inline compile on fork PRs. Therefore **no job reachable by a fork inline build has its timeout cut**. Only `build-amd64` (release path, same-repo-only, never inline) is raised 15→20. If the team later wants tighter same-repo feedback, the reduction can be made conditional: `timeout-minutes: ${{ github.event.pull_request.head.repo.full_name == github.repository && 15 || 20 }}` — noted as an option, not adopted now (keeps the YAML simpler and 20 min idle-capacity cost is negligible). -**Live restore without a restart** (`RehydrateLiveDatabase` swaps table *contents* under the running `*gorm.DB` handle). Immediately after, the in-memory `monSchedule` / `hostSchedule` / `monState` / `hostState` reflect *pre-restore* data: -- Monitors deleted by the restore keep getting enqueued until the next `rescan()` (≤ 30 s), where the mirror `UPDATE ... WHERE id=?` affects 0 rows — harmless. -- Monitors added by the restore are picked up by the same `rescan()` (≤ 30 s) and seeded into `monState`. -- Existing monitors whose restored `status`/`failure_count` differ from the in-memory entry re-converge within 1–2 real check cycles (bounded, no corruption, alert timing shifts by ≤ ~2 intervals). +**Stale-comment fixes:** `docker-build.yml:381` ("amd64's fast native build") — reword: the Caddy/CrowdSec compile now lives in the prebuilt toolchain image, and the arm64 builder stages were always cross-compiled (never QEMU) regardless. Fix the dangling `docs/plans/current_spec.md §1.1` cross-reference in the same comment block (it points at the retired uptime spec) to cite this document. Grep `.github/**` for `no-cache-filter` / `xcaddy` / `10-14m` / `12-14 min` / `cold build` / `full cold build` and reconcile every comment (Commit 6). -To make a live restore **immediate** rather than eventually-consistent, `RestoreBackupSafe`'s reconcile step (which already reloads Caddy config) also calls `UptimeScheduler.Rehydrate()` — which re-runs cold-start hydration under `s.mu` and calls `pool.ReseedState()`. This is a small, localized hook (the cold-start seeding code is already factored out for `Run`). Spec'd in **Commit 5** with a test (`backup_service` reconcile invokes `Rehydrate`; post-`Rehydrate` schedule/state match the restored DB). +### 3.10 Error handling / edge cases -**Large first prune after restoring an old backup.** If the restored DB is weeks/months stale, the pruner's next hourly pass is a large one — handled by the same chunked, wider-first-pass path as §3.4.2 (the `firstPassDone` flag resets on `Rehydrate()` so the wider pause reapplies). No separate handling. +| Scenario | Handling | +|---|---| +| Toolchain image pull fails mid app-build (GHCR outage) | app build fails fast with BuildKit's `failed to resolve source` — no silent fallback to a stale local layer. CI: `nick-fields/retry` already wraps `build-amd64`/`build-arm64` (3× / 10s). Document: maintainers can re-run or pass the inline build-args. | +| `regctl` not on runner | every workflow that runs `verify-toolchain-pin.sh` installs `regctl` first (`iarekylew00t/regctl-installer` or `docker run ghcr.io/regclient/regctl`). **B7:** on a same-repo run the script `exit 1`s if `regctl` or `GHCR_READ_TOKEN` is missing — it does **not** silently skip. Only a fork PR (`SAME_REPO=0`) degrades to tag-only comparison, with a `::warning::`. | +| Two toolchain builds race (dispatch + path-trigger on same commit) | `concurrency: toolchain-image-${{ github.ref }}`, `cancel-in-progress: false` → serialized; second is a cache hit / no-op. | +| Bot PR already open | `peter-evans/create-pull-request` updates the existing `bot/bump-toolchain-image` branch in place (same as GeoLite2 bot). | +| `toolchain-key.sh` awk stage-extraction breaks if a future edit removes the blank line between stages | script asserts each `extract_stage` returned ≥ 20 lines and contains `go build`; exits non-zero with a clear message otherwise. Unit-tested (§7). | +| Digest pinned but tag `:latest` moved (someone pushed manually) | guard compares against `:${KEY}` (content tag), never `:latest`; manual `:latest` pushes are cosmetic. | +| `CADDY_USE_CANDIDATE=1` experiment build | changes `toolchain-key.sh` output (ARG is in the hashed set) → distinct tag → distinct image; experiment is isolated, never collides with the mainline pin. | +| Renovate bumps `ghcr.io/wikid82/charon-toolchain` digest directly | harmless; guard still requires `TAG == key`, so a digest-only Renovate bump without a matching key change fails the guard and is closed in favor of the bot PR. Document in `renovate.json` a `packageRules` comment. | +| arm64 toolchain child missing (build published amd64-only by mistake) | app `build-arm64` fails at `FROM …@` with "no match for platform" — loud. `toolchain-image.yml` asserts `regctl manifest get` lists both platforms before pushing `:latest`. | --- ## 4. Implementation Plan -### Phase 1 — E2E specs (`test.fixme`) +The phases map 1:1 onto the Commit Slicing Strategy (§12); this is the same plan viewed as work packages. -`tests/monitoring/uptime-monitoring-scale.spec.ts` (new), `page.route`-mocked, all `test.fixme` initially: +### Phase 1 — "Spec behavior": key/guard scripts + toolchain workflow + stage split (Commit 1) -1. **Per-monitor interval honored** — create a monitor with `interval: 30`; mock `POST /uptime/monitors` echoing `interval: 30`; assert the create form rejects `interval: 10` client-side (min 30, helper text visible) and that the payload sent has `interval: 30`. -2. **Uptime page loads fast with many monitors** — mock `GET /uptime/monitors/summary` with a **100-monitor** fixture (each with `recent_beats` of 30, the new default); assert exactly **one** request to `**/uptime/monitors/summary`, **zero** requests to `**/uptime/monitors/*/history`, all 100 cards render with status badge + sparkline, and the page is interactive under a generous budget. -3. **Retention prunes old heartbeats** — this is backend-observable only; E2E asserts the **admin-facing signal**: `GET /uptime/health` mock returns `heartbeats_dropped: 0` and the settings round-trip for `uptime.heartbeat_retention_days` (set to 30, reload, value persists). The actual delete is covered by a Go test (Phase 2/6). -4. **Summary endpoint drives card state** — mock a monitor whose `status: "down"` with a trailing `down` beat; assert the card shows DOWN without any history call. +- `scripts/lib/dockerfile-stage.sh` (shared `extract_stage`), `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, with **bats unit tests** under `scripts/tests/` (house style: `scripts/*.sh` + a `bats` runner; add `shellcheck` + `bats` to the fast-lint set). +- No E2E/Playwright surface — this is CI/build infra. The executable "spec behavior" is: `toolchain-key.sh` is stable across a no-op Dockerfile reformat and changes when a tracked ARG / plugin pin / golang-base digest / stage line / `.trivyignore` changes; `verify-toolchain-pin.sh` is failure-closed on same-repo runs (B7). +- `.github/workflows/toolchain-image.yml` (daily `schedule` + `workflow_dispatch` + `pull_request` paths + `workflow_call`; build/publish + trivy-scan only). +- `Dockerfile`: rename stages, delete dead `crowdsec-fallback`, pin the two plugins + the golang base digest, add `toolchain-runtime` + temp aliases; **retarget the no-cache filters to `caddy-inline`/`crowdsec-inline`**. +- Manual `workflow_dispatch` first publish → capture `:` + manifest-list digest; set GHCR package **Internal** (N8). -### Phase 2 — Backend +### Phase 2 — Consume the pinned image; wire the fallback selector (Commit 2) -Ordered per the Commit Slicing Strategy (§Commit Slicing). Each backend commit is TDD (`backend-dev`): red test → implementation → green, `go test ./...` for the touched packages, `go build ./...`, `staticcheck`/`make lint-fast`. **New errors are wrapped `fmt.Errorf("context: %w", err)` per CLAUDE.md (N7)** — call it out in each commit's review. +- `Dockerfile`: `CHARON_TOOLCHAIN_*` ARGs + `toolchain-prebuilt` + `FROM ${…_SRC} AS caddy-builder`. +- Fork-detection build-args on every app-image build step + **new `builder-src` input on the `build-charon-image` composite** (§3.7, N10). +- `Makefile` `build-offline`. -- **Foundation:** `NextCheckAt` field (on `uptime_monitors` only) + `uptimeConfig` snapshot (with `now`/`forceRefresh` test seam, N8) + 3 Setting seeds + interval-floor validation (`SettingsHandler.UpdateSetting`, `UptimeHandler.Create`, `UptimeService.UpdateMonitor`, `CreateMonitor` write-time resolution). **No** heartbeat-table index change here (deferred to the Pruner step / §3.5.6). No runtime behavior change to checking yet. -- **Ingester:** `UptimeIngester` + `CheckResult` + `HostCheckResult` + tests (mirror `stats_ingester_test.go`: drop-on-full, batch by count, batch by timer, coalesced monitor **and** host updates, type-switch routing, `results`-closed terminates `Run` + final flush, `ctx.Done()` alone does **not** terminate). **Also constructed (not `Run`) in `routes.go` this commit (S1).** -- **Worker pool + shared client:** `network.WithKeepAlive` option (idleTimeout 30 s) + `safeclient` tests (keep-alive reuse; connection older than `idleTimeout` not reused; link-local/metadata still blocked); `UptimeWorkerPool` with `Kind`-discriminated jobs, `monState` + `hostState` maps + `SeedState`/`ReseedState`/`EnsureMonitorState`, synchronous debounce + transition + notification + host-down child fan-out, `workerWG`-based shutdown that closes `results`; `uptime_check.go` (`runCheck` + `runHostCheck`, pure, no DB, no state); de-block `checkHost` (single dial, no sleep-retry). Tests: enqueue/drop, per-check deadline, SSRF parity, `SeedState` from DB, `monMu` serializes concurrent RMW, host-check job path. **Also constructed (not `Run`) in `routes.go` this commit (S1).** -- **Scheduler + remote-server sync hook + restore rehydrate:** `UptimeScheduler` (two schedule maps, host + monitor hydration, jittered backfill, host-down short-circuit consult, batched write-back, `rescan()`, `Rehydrate()`); start **all** `Run` loops in `routes.go` (`ingester.Run`, `pool.Run`, `scheduler.Run`, `syncLoop.Run`) replacing the ticker go-func; delete the old block; collapse `checkMonitor`/`checkHost` onto `runCheck`/`runHostCheck` + the pool; drop `CheckAll()` from `runInitialUptimeBootstrap`; `CheckAll()` returns `(enqueued, dropped int)`; `SyncAndCheckForRemoteServer` / `SyncMonitorForRemoteServer` + `RemoteServerHandler` `UptimeService` dependency + create/update/delete hooks; replace hardcoded `Interval: 60` in all auto-create paths with `clampInterval(0, cfg)` (S3); `RestoreBackupSafe` reconcile calls `scheduler.Rehydrate()` (S6). **Verify `server.Run` shutdown grace ≥ `hardCap` + ~2 s (S4).** Tests: monitor + **host** due-selection; interval clamp (30 floor, 0→default, auto-create honours `default_interval_seconds` — S3); backfill spread; **teardown chain — in-flight check's heartbeat still written on immediate `ctx` cancel (S4)**; **B3 — saturated ingester, every `CheckResult` dropped, `down` transition still detected + notified**; **B2 — host→down fan-out + scheduler skip + recovery**; write-back grouping; remote-server hook create/update/delete; `Rehydrate()` re-syncs after a simulated live restore (S6). -- **Pruner + deferred index:** `UptimePruner` + chunked delete (subquery form) + `pruneChunkPause` + WAL checkpoint threshold + `PRAGMA optimize` cadence + `CREATE INDEX IF NOT EXISTS idx_heartbeat_monitor_created` retried at the end of every clean, caught-up pass until it lands (no `sync.Once`); `charon migrate` CLI gains the eager unconditional `CREATE INDEX IF NOT EXISTS`. Tests: deletes only rows older than cutoff; chunk loop terminates; respects hot config change; `ctx` abort mid-loop does not attempt the index; a clean caught-up pass creates it; a second pass with the index present is a no-op (no error); a pass that errors then a later clean pass still creates it. -- **Summary endpoint:** `UptimeSummaryService` + 30 s TTL cache + 3-query strategy + `GET /uptime/monitors/summary` + `GET /uptime/health`; cap + `before` cursor on `GetHistory`. Tests: one windowed query returns ≤ `beats` per monitor chronological ASC; cache hit avoids re-query; `uptime_24h` math; `beats` clamp; history `limit` cap 500; `before` paging. +### Phase 3 — Guardrails: freshness + app-side assertions (Commit 3) -### Phase 3 — Frontend (`frontend-dev`) +- `verify-toolchain-pin` → required check in `quality-checks.yml` (with `regctl` + `GHCR_READ_TOKEN`). +- `sync-pin-on-pr` + `open-bump-pr` jobs in `toolchain-image.yml`. +- N5 final-stage `RUN` assertions; extend `docker-build.yml`'s post-build verification to check the toolchain `LABEL` key. -- `api/uptime.ts`: `MonitorSummary`, `BeatDTO`, `getMonitorsSummary(beats = 30)`, `before` param on `getMonitorHistory`. -- `hooks/useUptimeSummary.ts` (`getMonitorsSummary(30)`). -- `pages/Uptime.tsx`: single summary query; `MonitorCard` reads props (remove per-card `useQuery`); invalidations retargeted to `['uptimeSummary']`; heartbeat bar `BEAT_BAR_SLOTS = 30`; interval field `min=30` + clamp + helper; manual check/sync queue-full → toast (N5). -- `pages/SystemSettings.tsx`: new admin "Uptime Monitoring" card — three `uptime.*` number fields with client-side bounds validation, restart-required note on `worker_pool_size`, save via `updateSetting(..., 'uptime', 'int')`, gated on `feature.uptime.enabled`. -- Vitest: `Uptime.test.tsx` / `Uptime.spec.tsx` / `Uptime.tcp-ux.test.tsx` updated — assert no per-card history fetch, cards render from summary fixture; `api/__tests__/uptime.test.ts` — new client fn; form validation test for the 30 s floor; **`SystemSettings` test — Uptime card render / bounds rejection / save / feature-flag gating**. -- `npm run type-check`, `npm run test`, `npm run build`. +### Phase 4 — Remove the forced rebuilds; reroute the security scan (Commits 4–5) -### Phase 4 — Integration & E2E +- Delete every `--no-cache-filter` / `no-cache-filters` + the composite `no-cache-filters` input (Commit 4) — only after Phase 3's guard is live. +- Repurpose `security-weekly-rebuild.yml` → `workflow_call` into `toolchain-image.yml`; blocking Trivy on `schedule`/`dispatch`/`workflow_call`; caller `permissions:` gains `contents: write` + `pull-requests: write` (N6) (Commit 5). -- Flip Phase 1 `test.fixme` → `test`; adjust mock payload shapes to the final schema. -- Run the touched specs: `npx playwright test tests/monitoring/uptime-monitoring-scale.spec.ts tests/monitoring/uptime-monitoring.spec.ts --project=firefox` from repo root. -- `tests/a11y/uptime.a11y.spec.ts` — re-run under firefox; fix any regressions from the card markup change. -- Manual smoke against a local backend with ~100 seeded monitors (optional but recommended): confirm `/uptime/monitors/summary` p95 and one-request behavior in the network panel. +### Phase 5 — Timeouts, comment sweep, docs (Commit 6) -### Phase 5 — Documentation & deployment - -- `docs/features/uptime-monitoring.md` — rewrite "How It Works / Check Cycle" for per-monitor intervals; new "Scaling & performance" section (worker pool, ingester, retention, the three `uptime.*` settings + bounds + hot-reload table, admin settings card); note the accepted double-DNS-lookup. -- `ARCHITECTURE.md` — see §5. -- No infra/CI changes. No new env vars. -- **Deploy note (upgrade on a large existing DB — be honest, per S2/S7):** - - First boot: the pruner trims `uptime_heartbeats` to the retention window (chunked, wider first-pass pause), then — at the end of that and every later clean pass, retried until it lands — builds `idx_heartbeat_monitor_created`. Both run in a **background goroutine**; the server is up and the summary endpoint serves correct (if slower) results throughout — **no route downtime**. - - **On a 500-monitor instance the retained table is ~65 M rows even after trimming**, so the first index build is a **bounded multi-minute operation** that adds write-lock contention on the single connection for its duration (reads via WAL are unaffected; some heartbeat writes may drop-on-full and self-heal). This is expected, not a bug. - - Operators who want to avoid that window entirely: **lower `uptime.heartbeat_retention_days` before first boot**, or run `charon migrate` in a maintenance window — the CLI builds the index eagerly (not prune-first) and now logs a `WARN` that the build "can take several minutes and holds a write lock". +- Timeout edits (§3.9 — only `build-amd64` 15→20; CVE-gate jobs stay 20 per B6), stale-comment reconciliation. +- `ARCHITECTURE.md`, `SECURITY.md`/`docs/security.md`, new `docs/ci/toolchain-image.md`, `CONTRIBUTING.md`, `renovate.json` (§9). --- -## 5. ARCHITECTURE.md updates required - -Add a **"Uptime Subsystem"** subsection (sibling of "Stats Subsystem", ~line 356), documenting: - -- **`UptimeScheduler`** (`internal/services/uptime_scheduler.go`): single goroutine, ~5 s tick. Maintains **two** in-memory schedule maps — monitors (keyed on `uptime_monitors.next_check_at`, persisted via batched write-back) and hosts (due = min child interval, in-memory only). Per tick: a host-connectivity pass then a monitor pass; the monitor pass consults the pool's `hostState` map to **skip** TCP monitors of a known-down host. Advances due-times by the per-monitor `Interval` (30 s floor; `uptime.default_interval_seconds` for zero/legacy/auto-created). Jittered cold-start backfill (60 s) prevents a restart stampede. `Rehydrate()` re-syncs after a live DB restore. Replaces the former global 1-minute `CheckAll()` + `checkAllHosts()` ticker. -- **`UptimeWorkerPool`** (`internal/services/uptime_worker_pool.go`): fixed-size (`uptime.worker_pool_size`, default 30) pool over a bounded (512) `Kind`-discriminated job channel (monitor check / host check); drop-on-full → `checks_enqueue_dropped`. Owns the **authoritative in-memory debounce state**: `monState` (per-monitor `{status, failureCount, lastStatusChange, lastNotifiedDown}`) and `hostState` (per-host), both seeded from the DB at start. The **worker** — not the ingester — read-modify-writes this state synchronously, computes transitions, dispatches notifications, and (on a host→down) fans out synthetic `down` child results for the host's TCP monitors. One shared SSRF-safe keep-alive `*http.Client` (`network.NewSafeHTTPClient(..., network.WithKeepAlive(100, 4, 30s))`) — same `safeDialer` / redirect / localhost+RFC1918 policy as the retired per-check client. Host TCP pre-check is a single non-blocking dial (was `2s × MaxRetries` sleep-retry). Shutdown: `workerWG.Wait()` on in-flight checks, then closes the `results` channel. -- **`UptimeIngester`** (`internal/services/uptime_ingester.go`): mirrors `StatsIngester`'s batching but is a **pure persistence mirror** — it does **no** transition detection and **no** fan-out; it copies pre-computed columns. Receives `CheckResult | HostCheckResult` on a channel the **pool owns and closes**; `Run` ends only when that channel is closed (guaranteeing no in-flight result is lost at shutdown), then does a final flush. Batch-inserts `uptime_heartbeats` and coalesces `uptime_monitors` / `uptime_hosts` column updates every 500 ms or 100 results in one transaction. Drop-on-full → `heartbeats_dropped` at `GET /api/v1/uptime/health`; a dropped write **cannot** suppress an alert because detection never reads these columns at runtime. -- **`UptimePruner`** (`internal/services/uptime_pruner.go`): hourly, chunked `DELETE` (5 000 rows/chunk via `WHERE id IN (SELECT ... LIMIT n)`; 50 ms inter-chunk pause steady-state, 250 ms on the first cold pass) of `uptime_heartbeats` older than `uptime.heartbeat_retention_days` (default 90). `PRAGMA wal_checkpoint(TRUNCATE)` after a large prune; `PRAGMA optimize` daily. No downsampling; `VACUUM` deliberately not used. Also **owns lazy creation of `idx_heartbeat_monitor_created`** — not a struct-tag/AutoMigrate index; `CREATE INDEX IF NOT EXISTS` is issued at the end of every clean, caught-up pass (retried until it lands). Prune-first bounds the pathological (hundreds-of-millions-row) case; on a healthy 500-monitor instance the first build still runs over ~65 M rows and is a bounded multi-minute, write-contending operation (background; no route downtime). `charon migrate` builds it eagerly, with a `WARN` log. -- **Targeted monitor sync on host mutation** — proxy-host create/update/delete already drive `UptimeService.SyncAndCheckForHost` / `SyncMonitorForHost` / inline monitor cleanup; **remote-server create/update/delete now do the same** via `SyncAndCheckForRemoteServer` / `SyncMonitorForRemoteServer` / inline cleanup (`RemoteServerHandler` gains a nil-guarded `UptimeService` dependency). The 5-minute `UptimeSyncLoop` is the backstop. Auto-created monitors inherit `uptime.default_interval_seconds` (not a hardcoded 60). -- **`uptimeConfig`** (`internal/services/uptime_config.go`): a hot-reloading (60 s TTL) snapshot of the three `uptime.*` Settings, shared by the scheduler, pruner, and `UptimeService`. Read-only; writes go through `POST /api/v1/settings`. -- **`UptimeSummaryService`** (`internal/services/uptime_summary_service.go`): serves `GET /api/v1/uptime/monitors/summary` from **three** queries (monitor metadata; one `ROW_NUMBER()` windowed recent-beats query, default 30 beats / cap 60; one grouped 24 h-uptime query) with a 30 s TTL cache — same pattern as `StatsService`. Correct (slower) even before `idx_heartbeat_monitor_created` exists; never 503-gated. Replaces the per-card N+1 history fetch. -- **New settings** (`models.Setting`, `Category="uptime"`): `uptime.default_interval_seconds` (60; 30–86400; hot-reload), `uptime.worker_pool_size` (30; 1–200; **restart** to apply), `uptime.heartbeat_retention_days` (90; 1–3650; hot-reload). Editable via `POST /api/v1/settings` and the SystemSettings "Uptime Monitoring" card. +## 5. Acceptance Criteria (Definition of Done) + +1. **No app-image build compiles Caddy/CrowdSec on the happy path.** A CI `build-amd64` run with a warm cache shows no `xcaddy`/`go build … caddy`/`xx-go build … crowdsec` step; total job < 8 min. Verified from the run log in the PR. +2. **`build-amd64` / integration jobs no longer time out** across 3 consecutive CI runs on the PR (main gate that this feature exists to fix). +3. **`verify-toolchain-pin` is a required check** and: (a) passes on `main` HEAD, (b) **fails** on a deliberate commit that bumps `CADDY_VERSION` (or a plugin pin) without rebuilding, (c) **fails** on a deliberate commit that hand-edits `CHARON_TOOLCHAIN_DIGEST` to a wrong-but-valid digest on a same-repo run (proves B7 failure-closed — not a tag-only check), (d) passes again after `sync-pin-on-pr` runs. Demonstrated with temporary commits that are then reverted. +4. **Multi-arch intact:** `docker buildx imagetools inspect ghcr.io/wikid82/charon-toolchain:` lists `linux/amd64` + `linux/arm64`; the merged app image manifest still lists both; `docker run --rm --platform linux/arm64 /usr/bin/caddy version` and `… cscli version` succeed (in `docker-build.yml`'s existing verification step). Confirm the arm64 leg still runs only its final-stage `RUN` under QEMU (N2 — it never compiled the builders). +5. **Fallback works:** a CI leg builds the app image with `--build-arg CADDY_BUILDER_SRC=caddy-inline --build-arg CROWDSEC_BUILDER_SRC=crowdsec-inline` and passes the in-`caddy-inline` embeds-version assertions; a simulated fork run stays under the 20-min job cap (B6). +6. **Security guarantee mechanized:** the daily `schedule` on `toolchain-image.yml` runs `--no-cache --pull` + a **blocking** Trivy CRITICAL/HIGH gate; a wired failure issue; a new digest opens `bot/bump-toolchain-image`. `security-weekly-rebuild.yml` routes through the same `workflow_call`. §3.8.3's "does NOT catch unpinned-transitive MVS drift" statement is reflected verbatim in `SECURITY.md` (no overclaim). Dry-run via `workflow_dispatch` on the PR branch. +7. **No `--no-cache-filter` / `no-cache-filters` string remains** under `.github/workflows` or `.github/actions` (grep clean; `docs/` history excepted). The composite action no longer exposes a `no-cache-filters` input. +8. **All existing CI green:** `docker-build.yml`, `nightly-build.yml` (dispatch), `security-pr.yml`, `supply-chain-pr.yml`, `e2e-tests-split.yml`, 4× integration workflows pass on the PR. +9. **Backend/frontend untouched:** `cd backend && go build ./... && go test ./...` and `cd frontend && npm run build && npm run type-check` unaffected (no diff there). GORM security scan **N/A** (no `backend/internal/models/**` change). +10. **`ARCHITECTURE.md` updated** (§9) and `docs/` build/CI docs updated; `docs-writer` pass done. +11. **Trivy on the final app image** (existing `merge-and-publish` step) shows **no new** CRITICAL/HIGH versus the pre-change baseline — the bundled binaries are the same recipe. +12. Lefthook / staticcheck / `make lint-fast` clean; shell scripts pass `shellcheck` (add to `lefthook` if not already). -Update the **API Endpoints** area (or add an "Uptime" table) with: +--- -| Method | Path | Description | -|---|---|---| -| `GET` | `/api/v1/uptime/monitors/summary` | Per-monitor status + latency + last check + `recent_beats` sparkline series + 24 h uptime, one response, 30 s cached | -| `GET` | `/api/v1/uptime/monitors/:id/history` | Detailed heartbeat history for one monitor — `limit` (≤ 500), `before` cursor | -| `GET` | `/api/v1/uptime/health` | Ingester `heartbeats_dropped`, pool `checks_enqueue_dropped`, `queue_depth`, `worker_pool_size` | +## 6. Risks & Rollback -Update **§4 Database (SQLite + GORM)** "Pragma Settings" / concurrency note to mention that the uptime write path, like stats, is funnelled through a buffered ingester rather than writing on the request/check goroutine — reinforcing why `SetMaxOpenConns(1)` remains viable at 500 monitors. Add that authoritative uptime **debounce state lives in memory** (pool `monState`/`hostState` maps); the DB columns are a persistence mirror. +| Risk | Likelihood | Impact | Mitigation | Rollback | +|---|---|---|---|---| +| Toolchain image stale vs an urgent 0-day; bot-PR-merge latency too slow | Low | High | daily rebuild + Trivy **detects** on ~24 h cadence (unchanged from today); urgent path = `workflow_dispatch` + expedited merge (~30 min); follow-up twice-daily toggle (§3.8.3) | n/a — detection cadence matches today; only the merge step is new (§3.8.2) | +| `FROM ${ARG} AS name` selector unsupported on a pinned BuildKit | Low | Med | verified against BuildKit ≥ 0.11 (repo uses current `buildx`); `docker build --check` + `--print` in Commit 1/2 gates | drop the selector; make `caddy-inline`/`crowdsec-inline` the direct stage names and gate the prebuilt image behind an explicit per-workflow `--build-arg` | +| GHCR outage blocks all builds (new hard dependency) | Low | High | `nick-fields/retry` wraps the release builds; documented inline fallback; Docker Hub mirror is a follow-up | flip the selector build-args to `caddy-inline` fleet-wide via a one-line workflow edit | +| `toolchain-key.sh` false-negative (misses a security-relevant change) | Med | High | key hashes full stage **text** + all consumed ARGs (incl. the 2 plugin pins) + golang-base digest + `.trivyignore` + `SCHEMA_VERSION`; bats tests; the **daily** `--no-cache --pull` rebuild is the base-image/`apk` backstop even if the key never moves (it is NOT a backstop for unpinned-transitive MVS — §3.8.3) | bump `SCHEMA_VERSION` → global rebuild + re-pin | +| Bot PR churn (digest bump when nothing meaningful changed) | Low | Low | `open-bump-pr` fires only when the **manifest-list digest** actually changes; a no-op day (same base digests, same `apk` index) reproduces the same digest → no PR | close PR; tune to "digest changed AND (Trivy delta OR >7 d since last bump)" | +| Fork PRs slower (full inline compile) | High (every fork PR) | Low | expected; fork CI already runs long; CVE-gate job timeouts stay at 20 (B6); documented in `CONTRIBUTING.md` | none needed | +| Timeout bump to `build-amd64` masks a real slowdown | Low | Low | AC #1 asserts < 8 min actual; a run > 12 min is investigated | revert timeout to 15 | +| Human forgets to merge the bot PR for days | Med | Med | bot PR carries the `security` label → shows in the same queue as Renovate security bumps; `repo-health.yml`/stale-bot surfaces it; runbook says target ≤ 1 business day | expedite; or `workflow_dispatch` + merge | -Update the **Security section's SSRF-client note** (the "Keep-alives disabled" default in `network.NewSafeHTTPClient`): add a line that the uptime worker pool constructs a **pooled variant** via `network.WithKeepAlive(100, 4, 30s)` — `safeDialer` still re-validates every new connection; `idleTimeout` is 30 s to bound Layer-2 staleness on reused connections. +**Whole-PR rollback:** revert the single merged commit. The `charon-toolchain` package stays in GHCR (harmless, unreferenced; `container-prune.yml` ages it out). `security-weekly-rebuild.yml` returns to building the throwaway scan image; the Dockerfile returns to inline `caddy-builder`/`crowdsec-builder` + `--no-cache-filter` — **security posture identical to today**. No data migration, no runtime change; app image content byte-identical (same recipe). -Add `uptime_heartbeats` retention to the **"Migrations" / data-lifecycle** notes (hard-delete, configurable window `uptime.heartbeat_retention_days` default 90, background pruner). Note that `idx_heartbeat_monitor_created` is created **lazily by the pruner** (`CREATE INDEX IF NOT EXISTS` at the end of every clean, caught-up pass, retried until it lands), not by AutoMigrate — so upgrades never see a *migration-time* stall, though on a large instance the first background build is still a bounded multi-minute write-contending operation (`charon migrate` builds it eagerly, with a warning log, for out-of-band migration). +**Contingency:** if the selector-stage approach hits a BuildKit bug in one workflow only, that workflow can pin `--build-arg CADDY_BUILDER_SRC=caddy-inline` as a temporary per-workflow escape hatch while keeping the prebuilt default everywhere else — no revert of the whole feature. --- -## 6. Risks & mitigations +## 7. Testing strategy (validate without a 14-min wait) -| # | Risk | Likelihood / impact | Mitigation | -|---|---|---|---| -| R1 | **Single-connection contention persists even with batching.** The ingester still writes through the one SQLite connection; a slow flush blocks API reads. | Med / Med | Ingester writes are now ~2–4 statements / 500 ms (vs ~17 writes/s today) — a large net reduction. Flush is one transaction. Pruner yields between chunks. If contention still shows in QA, raise `uptimeFlushInterval` and/or lower `uptimeBatchSize` is counter-productive — instead increase batch size so flushes are rarer. Monitored via `/uptime/health` + existing `/api/v1/health/db`. | -| R2 | **Down alert suppressed under ingester saturation** — a failure-debounce counter that depended on the droppable async DB write would never reach `maxRetries` while results drop, so the transition is never detected. This is the exact overload the feature targets. | — / High if mishandled | **Designed out (B3):** the debounce counter is authoritative in the pool's in-memory `monState` map (seeded from DB once at start), read-modify-written **synchronously by the worker** under `monMu` on every result. The ingester's `failure_count`/`status` write is a persistence *mirror* only. A dropped `CheckResult` cannot delay or suppress a transition. Transition detection + `queueDownNotification`/`sendRecoveryNotification` also run synchronously on the worker before enqueue. Covered by the C5 "drop-does-not-suppress-alert" test (§3.3.3) and the notification-without-ingester test. Residual: a hard crash mid-saturation can reseed `failureCount` stale-low → alert fires ≤ 1–2 cycles later (bounded, §3.3.3). | -| R3 | **First retention prune on a huge `uptime_heartbeats` table.** At the 500-monitor target the steady-state table is ~65 M rows (90 d); a years-stale instance can be several hundred million. | High / Med | Chunked delete (5 000/chunk); **first cold pass** uses a 250 ms inter-chunk pause (5× steady state) so the single connection stays available; each cold-table chunk can be 100–500 ms (§3.4.2 / N1). Runs in a background goroutine ~30 s after boot; server serves throughout. WAL checkpoint reclaims file growth. No migration-time bulk operation. The first pass trims *before* the index build (§3.5.6, R4). | -| R4 | **`idx_heartbeat_monitor_created` first-boot build.** Even after a clean prune the build runs over ~65 M rows on a 500-monitor instance — a **bounded multi-minute operation that contends for the single write connection** (readers unaffected — WAL). | Med / Med (honest, per S2 — *not* "designed away") | The index is **not** built by AutoMigrate. `UptimePruner` issues `CREATE INDEX IF NOT EXISTS` at the end of every clean, caught-up pass — **retried hourly, idempotent**, so a failed/interrupted attempt self-heals without a restart. Prune-first only removes the *pathological* (hundreds-of-millions) case, not the multi-minute build itself. Acceptable because: background goroutine (no request-path / migration stall); summary endpoint stays available (correct, 30 s-cached, never 503); heartbeat drops during the build self-heal. Operator escape hatches: `charon migrate` in a maintenance window (eager, with a warning log — S7), or lower `uptime.heartbeat_retention_days` before first boot. The 90-day default is unchanged (user decision). | -| R4a | **Host-check scheduling is a new subsystem (B1).** `UptimeScheduler` now also hydrates + schedules `UptimeHost` rows and the `hostState` map is shared between the host-check worker (writer) and the scheduler (reader). | Low / Med | Hosts are few (one per distinct upstream) so the host schedule + cold-start wave are tiny. `hostState` is a plain `sync.RWMutex`-guarded map; the scheduler only ever RLocks it for the short-circuit check. Host due-times are in-memory only (no schema change). Covered by scheduler host-due-selection tests + a host→down fan-out test (C5). | -| R5 | **Scheduler cold-start stampede** — a restart makes all 500 monitors due at once → 500 enqueues in one tick → queue saturation + latency spike. | Med / Med | Jittered backfill: past-due monitors get `next_check_at = now + rand(0, min(interval, 60s))`, spreading enqueues to ~`monitors/60` per second. Per-tick enqueue cap (200). Queue cap 512 absorbs the rest; overflow retried next tick. | -| R6 | **Worker-pool starvation when many hosts are down** — failing checks hold workers for the full connect timeout, throughput collapses, *all* latencies inflate (the exact symptom we're fixing). | Med / High | Short connect timeout (3 s) + per-check hard deadline (≤ 20 s, usually = interval). Host-down short-circuit skips TCP monitors of known-down hosts entirely. Drop-on-full degrades gracefully (delayed checks + metric) rather than unbounded goroutine growth. Documented sizing formula; 500-monitor/mostly-down deployments raise `uptime.worker_pool_size` to 60–90. `/uptime/health` `queue_depth` + `checks_enqueue_dropped` make starvation observable. | -| R7 | **`ROW_NUMBER()` window query cost** at 500 monitors × 24 h of beats — worst during the first-boot window before `idx_heartbeat_monitor_created` exists (which, per R4, can now be several minutes on a large instance). | Low / Med | Bounded by `created_at >= now-24h`; covered by the index once built; 30 s TTL cache ⇒ ≤ 2 executions/min regardless of viewers. **The route stays available with correct results the whole time (never 503-gated)** — deliberate, bounded. Steady-state p95 target < 300 ms, gated by the C7 `TestUptimeSummary_PerfBudget` timed test (< 2 s CI-stable ceiling, index present; §3.5.3 / S5). Fallback (per-monitor `LIMIT` loop) documented, not implemented. | -| R8 | **Summary payload size.** | Low / Low | **`beats` default is 30** (per user decision), cap 60. 500 monitors × 30 beats ≈ ~700–900 KB uncompressed → **~50–75 KB gzipped** (gzip already on for the API). 30 s TTL cache. The list view uses the default 30; only an expanded/detail view requests 60. | -| R9 | **Lost `next_check_at` write-back on crash** → duplicate check for some monitors after restart. | Low / Low | In-memory `monSchedule` is the runtime source of truth; write-back is best-effort batched every tick. Worst case: one extra check per affected monitor, absorbed by cold-start jitter. | -| R10 | **Behavior change: detection timing shifts** — per-monitor interval instead of fixed 60 s; auto-created monitors now inherit `uptime.default_interval_seconds` instead of a hardcoded 60 (S3); host "down" now after 2× the host cadence. | High / Low | Intended (goal #2). Documented in `docs/features/uptime-monitoring.md`. `uptime.default_interval_seconds` defaults to 60, so an untouched deployment sees no cadence change; legacy monitor rows keep their stored interval until edited. | -| R11 | **Keep-alive pooled connection skips Layer-2 re-resolution for its lifetime.** | Low / Low | No actual SSRF — an *established* TCP connection can't be re-bound to a new IP; `safeDialer` still validates every **new** connection. `idleTimeout` cut to **30 s** (from 90 s) to bound the staleness window; `safeclient_test.go` asserts a connection older than `idleTimeout` is not reused, and that link-local/metadata stay blocked with keep-alive on. | -| R12 | **Two orchestration passes on the same files** (per CLAUDE.md incident note). | — / Med | Single PR, sequential commits; `qa-security` runs last, after all commits land, never in parallel with implementation on `uptime_*` files. | -| R13 | **Result loss at shutdown** — an in-flight worker emitting after the ingester returned would lose that result's persistence (and, post-B3, the DB would never learn a transition the `monState` already applied → next restart reseeds stale). | — / Med | Explicit teardown chain (§3.1.4): scheduler stops enqueuing → pool `workerWG.Wait()`s in-flight checks then **closes `results`** → ingester `range`s `results` until closed, then final flush. The ingester does **not** exit on `ctx.Done()` alone. C5 test asserts an in-flight check's heartbeat is still written when `ctx` is cancelled immediately. Grace-period requirement (≥ `hardCap` + ~2 s) flagged as a C5 implementation check against `server.Run`'s shutdown timeout. | -| R14 | **Post-restore stale in-memory state** — after a live `RehydrateLiveDatabase` the scheduler/pool maps reflect pre-restore data. | Low / Low | Restore-then-restart (pending-restore path + the recommended `RehydrateLiveDatabase` flow) = ordinary cold start, no special handling (§3.9 — goroutines start after DB init). Live-restore-without-restart self-heals within one `rescan()` (≤ 30 s) + 1–2 check cycles; `UptimeScheduler.Rehydrate()` called from the restore reconcile step makes it immediate. No data corruption in any case. | +| What | How | Where | +|---|---|---| +| `toolchain-key.sh` determinism | shell/bats test: run twice → identical; reformat whitespace outside the stages → identical; change a `go get` line inside `caddy-inline` → differs; bump `CADDY_VERSION` default → differs; touch `.trivyignore` → differs | `scripts/tests/toolchain-key.bats`, runs in `quality-checks.yml` (< 5 s) | +| `verify-toolchain-pin.sh` | bats matrix: matching pin → exit 0; mismatched tag → exit 1 (actionable message); **same-repo run + missing `regctl`/token → exit 1** (B7 failure-closed, mocked); **same-repo run + GHCR digest ≠ pinned → exit 1**; fork run (`SAME_REPO=0`) + no registry access → exit 0 with `::warning::` | `scripts/tests/verify-toolchain-pin.bats` | +| Selector stage resolves both ways | `docker build --check` + `docker buildx build --target caddy-builder --print` (BuildKit dry-run, no compile) for both `CADDY_BUILDER_SRC` values | new `toolchain-image.yml` PR-path job, seconds | +| Cache behavior (the actual fix) | CI observation: run `build-amd64` twice on the PR; second run's log shows `CACHED` for every stage and **no** `xcaddy` / `xx-go build` compile lines; assert job wall-time < 8 min via a step that checks `$SECONDS` | PR CI, no local 14-min wait | +| Guard-live check (B5) | in Commit 1's gate: `docker buildx build --no-cache-filter caddy-inline` re-runs the `xcaddy build` step (not `CACHED`); with the old `--no-cache-filter caddy-builder` value on the renamed graph it would show `CACHED` | Commit 1 CI leg | +| Fallback correctness | one CI leg builds the app image with the inline build-args; the in-`caddy-inline` `go version -m /usr/bin/caddy | grep 'cel-go … v0.29'` / `grpc … v${GRPC_VERSION}` assertions (`Dockerfile:564`, `:569`) are the test — they already fail the build if the binary is wrong; plus the new N5 final-stage assertion | `toolchain-image.yml` PR-path matrix leg | +| Wrong-digest detection (N5) | build the app image against a deliberately old `CHARON_TOOLCHAIN_DIGEST` → N5 final-stage `RUN` fails ("missing expected caddy plugins" / bad `cscli version`) | Commit 3 CI leg | +| Multi-arch child selection | `docker buildx imagetools inspect` two-platform assertion in `toolchain-image.yml` (before pushing `:latest`); `docker run --platform linux/arm64 … caddy version` in `docker-build.yml`'s existing post-build verification | existing + new assertion | +| Daily rebuild + bot | `workflow_dispatch` `toolchain-image.yml` from the PR branch with `force_rebuild: true`; confirm it publishes, scans, and (if digest changes) opens a draft `bot/bump-toolchain-image` PR; blocking Trivy gate on the dispatch path | manual, once, during PR review | +| No regression in app-image Trivy | compare `merge-and-publish` Trivy JSON artifact on the PR vs a recent `main` run — diff must be empty for CRITICAL/HIGH | PR CI artifact | +| E2E | existing `e2e-tests-split.yml` runs unchanged against the built image; targeted local run per CLAUDE.md DoD only if a spec is touched (none is) | CI | + +**Local dev validation (fast):** `scripts/toolchain-key.sh` + `bats scripts/tests/` (seconds); `docker buildx build --target caddy-builder --print` (no compile); pulling the published toolchain image and running `docker build .` end-to-end is a ~30 MB pull + fast stages only (~4–6 min), well under the old 14-min floor. --- -## 7. Acceptance Criteria - -Feature is done when **all** of the following hold on the PR: - -### Functional - -1. Creating a monitor with `interval: 45` results in that monitor being checked ~every 45 s (observable via heartbeat `created_at` spacing); `interval: 10` is rejected `400` by the API and blocked client-side with helper text. -2. A monitor with `interval: 0` or a legacy `interval < 30` is checked at `uptime.default_interval_seconds` / 30 s respectively (clamp works). -2a. **Auto-created monitors honour the admin default (S3).** With `uptime.default_interval_seconds = 45`, a monitor created by `SyncAndCheckForHost` / `SyncAndCheckForRemoteServer` / `SyncMonitors` has `Interval == 45` (not the old hardcoded 60) — verified by a Go test. -3. Changing `uptime.default_interval_seconds` via `POST /api/v1/settings` (or the admin Uptime settings card) takes effect within ≤ 60 s without a restart; an out-of-bounds value is rejected `400`. -3a. The **admin "Uptime Monitoring" settings card** (`SystemSettings.tsx`) renders the three `uptime.*` fields seeded from `GET /api/v1/settings`, enforces the §3.6.1 bounds client-side (out-of-bounds ⇒ inline error + disabled save), persists in-bounds values via `POST /api/v1/settings` with `category=uptime`, labels `worker_pool_size` as restart-required, and is hidden when `feature.uptime.enabled` is off. -4. `GET /api/v1/uptime/monitors/summary` returns one array with `status`, `latency`, `last_check`, `interval`, `uptime_24h` (always present), and up to `beats` chronological `recent_beats` per monitor (**default 30**, cap 60); the Uptime page issues **exactly one** request to it and **zero** to `/uptime/monitors/*/history` on initial load and on refetch. -4a. **(S5) Automated perf gate:** `TestUptimeSummary_PerfBudget` (`-short`-skippable) seeds 500 monitors × 24 h of heartbeats, builds `idx_heartbeat_monitor_created`, and asserts `GetSummary(ctx, 30)` (cache cleared) completes **< 2 s** wall-clock. (The < 300 ms p95 target is tracked from the QA timing output, not hard-gated.) -5. `GET /api/v1/uptime/monitors/:id/history?limit=99999` returns at most 500 rows; `before=` returns only older rows. -6. `GET /api/v1/uptime/health` returns `heartbeats_dropped`, `checks_enqueue_dropped`, `queue_depth`, `worker_pool_size`. -7. Heartbeats older than `uptime.heartbeat_retention_days` are deleted within one hour of the pruner running; rows within the window are untouched; the delete is chunked (test — loop terminates, chunk size respected, first cold pass uses the wider pause). `idx_heartbeat_monitor_created` is created at the end of a clean, caught-up pruner pass (not attempted on a `ctx`-aborted pass); a pass that errored then a later clean pass still creates it; a pass with the index already present is a no-op. -8. **(B3)** The failure-debounce counter and up→down / down→up transition detection are computed against the pool's in-memory `monState` map, **not** a persisted row. Verified: (a) a monitor's `down` transition fires the notification **without** the ingester running / flushing; (b) with the ingester saturated and **every** `CheckResult` dropped, feeding `maxRetries` consecutive `down` results still detects the transition and calls `queueDownNotification`. -8a. **(B1/B2)** Host connectivity is scheduled by `UptimeScheduler`'s host pass (host due-selection test); on a host `up→down` transition the **worker** (not the ingester) writes `hostState`, fans out synthetic `down` child heartbeats for the host's TCP monitors, and fires one consolidated notification; while the host is down the scheduler skips enqueueing those TCP monitors; on recovery they resume. Verified by a host-down/recovery integration test. -9. **(S4)** Graceful shutdown teardown chain: `ctx` cancel → scheduler stops enqueuing → pool `workerWG.Wait()`s in-flight checks and closes `results` → ingester drains the closed channel and does a final flush. Test: a check that is in-flight when `ctx` is cancelled still has its heartbeat row written (no result loss); all goroutines exit (`goleak` or a `ctx`-cancel wait). -9a. **(S6)** After a live DB restore, calling `UptimeScheduler.Rehydrate()` re-syncs the schedule + state maps to the restored data (test); without it, state self-heals within one `rescan()` + 1–2 check cycles (documented, §3.9). -10. The shared HTTP client preserves SSRF protections: checks to `127.0.0.1` and `10.x` succeed (as today), checks to `169.254.169.254` / link-local fail, redirects are not followed — with keep-alive enabled; a pooled connection older than `idleTimeout` (30 s) is not reused. - -### Non-functional / DoD (per CLAUDE.md "Task Completion Protocol" — referenced, not reproduced) - -10a. Remote-server create/update/delete drive the targeted uptime-monitor sync (`SyncAndCheckForRemoteServer` / `SyncMonitorForRemoteServer` / inline delete cleanup) — verified by `remote_server_handler_test.go` and `uptime_service_*_test.go`. -11. **Targeted Playwright** (`tests/monitoring/uptime-monitoring-scale.spec.ts` + `tests/monitoring/uptime-monitoring.spec.ts` + `tests/a11y/uptime.a11y.spec.ts`, `--project=firefox`) pass locally; full/cross-browser deferred to CI. -12. **GORM security scan** (`./scripts/scan-gorm-security.sh --check`) — zero CRITICAL/HIGH (triggered: `models/uptime.go` + new raw-ish queries). -13. **Backend coverage ≥ 85%** (`scripts/go-test-coverage.sh`) — new services each have their own `_test.go`; **Frontend coverage ≥ 85%** (`scripts/frontend-test-coverage.sh`). -14. **Local patch coverage preflight** (`bash scripts/local-patch-report.sh`) — artifacts generated, patch coverage green. -15. **CodeQL Go + JS + Trivy** — zero high/critical (this adds new code paths/endpoints ⇒ run locally per DoD). -16. `lefthook run pre-commit` clean; `make lint-fast` / staticcheck clean (no `--no-verify`). -17. `cd backend && go build ./...` and `cd frontend && npm run build` succeed; `cd frontend && npm run type-check` clean. -18. All existing/adjacent tests updated and green: `uptime_service_*_test.go`, `uptime_handler_test.go`, `remote_server_handler_test.go`, `routes_uptime_bootstrap_test.go`, `Uptime.test.tsx`, `Uptime.spec.tsx`, `Uptime.tcp-ux.test.tsx`, `SystemSettings` test, `api/__tests__/uptime.test.ts`. -19. `ARCHITECTURE.md` (§5) and `docs/features/uptime-monitoring.md` updated. -20. `supervisor` review passed against the plan; `qa-security` audit (`docs/reports/qa_report.md`) has no blocking findings. - ---- +## 8. Component complexity estimate -## Commit Slicing Strategy +| Component | Complexity | Notes | +|---|---|---| +| Dockerfile stage rename + delete dead `crowdsec-fallback` + pin 2 plugins + digest-pin golang base + selector + `toolchain-runtime` + N5 assertion | **M** | mostly mechanical; selector pattern needs `--check` validation; plugin/base pins need one-time version resolution; COPY paths chosen to keep final stage untouched | +| `toolchain-image.yml` | **L** | multi-arch build, GHCR push, Trivy+SARIF, `sync-pin-on-pr`, `open-bump-pr`, fork guards, `workflow_call`, daily `schedule` | +| `scripts/lib/dockerfile-stage.sh` + `toolchain-key.sh` + `verify-toolchain-pin.sh` (failure-closed) + bats | **M** | shared awk extraction; robust asserts; B7 same-repo/fork branching; token+regctl plumbing in CI | +| Retarget then strip `--no-cache-filter` across 6 workflows + composite action | **S–M** | Commit 1 retarget (value change) + Commit 4 removal + composite input deletion (public interface change) + comment rewrites | +| Repurpose `security-weekly-rebuild.yml` | **M** | swap build step for `workflow_call`; caller `permissions:` must grant `contents: write` + `pull-requests: write` (N6); keep Trivy plumbing; blocking gate | +| Fork-detection build-args in every build step **+ new `builder-src` input on the `build-charon-image` composite (public interface change, 4 integration callers)** | **M** | ~8 build steps across 6 workflows + composite input + per-caller `head.repo.full_name` expression (N10 — was S–M, raised to M) | +| Timeout + comment reconciliation | **S** | grep-driven sweep; only `build-amd64` actually changes value | +| `ARCHITECTURE.md` + docs + `docs/ci/toolchain-image.md` runbook | **S–M** | §9 list + new runbook incl. one-time package-visibility step | -**Decision:** One feature, **one PR** on `feat/uptime-monitoring-scale`, merged only when the whole Definition of Done passes. The work is decomposed into **9 ordered commits**. Each commit builds and passes its own gate; the PR as a whole passes the full DoD (§7.11–7.20 — CLAUDE.md "Task Completion Protocol", not reproduced here). No feature-splitting across PRs. +--- -Dependency order: `C1` (specs, independent) → `C2` (foundation, no behavior change) → `C3` ingester **(constructed in `routes.go`, `Run` not started)** → `C4` pool+client **(constructed in `routes.go`, `Run` not started; `monState`/`hostState` types)** → `C5` scheduler + start all `Run` loops + remote-server hook + restore-rehydrate (needs C3+C4) → `C6` pruner + deferred `idx_heartbeat_monitor_created` (needs C2) → `C7` summary endpoint + `/uptime/health` (needs C2, C4 for the pool/ingester refs, C6 for the index) → `C8` frontend (needs C7) → `C9` hardening (needs all). +## 9. `ARCHITECTURE.md` / documentation update list -**Why C3/C4 construct-but-don't-`Run` (S1):** C7's `/uptime/health` handler needs the pool + ingester *references*. If those were constructed in C5, `git revert C5` after C7 landed would leave C7's handler holding nil deps and the tree would not build. Constructing them in C3/C4 (inert until C5 starts their `Run` loops) makes **`git revert C5` a genuine "restore the old ticker with C3/C4/C6/C7 dormant"** — the pool/ingester sit idle, `QueueDepth()`/`DroppedCount()` return zero, `/uptime/health` still serves valid JSON, and the legacy `checkMonitor` path is what runs. +| File | Section | Change | +|---|---|---| +| `ARCHITECTURE.md` | §"Deployment Architecture / Multi-Stage Dockerfile" (`:1082`) | replace the illustrative snippet's build-from-source framing; add a "Prebuilt toolchain image" subsection: what `charon-toolchain` contains, that Caddy/CrowdSec are compiled there (not in the app build), digest-pinned in `Dockerfile`, rebuilt **daily** `--no-cache --pull`, freshness-guarded, with the fork/offline inline fallback | +| `ARCHITECTURE.md` | §"Infrastructure" table (`:158`) | add row: **Bundled proxy toolchain** — `ghcr.io/wikid82/charon-toolchain` — multi-arch prebuilt Caddy + CrowdSec, daily-rebuilt + Trivy-gated | +| `ARCHITECTURE.md` | §"Directory Structure" (`:286`) | note `.github/workflows/toolchain-image.yml`, `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh`; note removal of the `crowdsec-fallback` Dockerfile stage | +| `ARCHITECTURE.md` | §"Security Architecture / Layer 2: CrowdSec Integration" (`:780`) and the defense-in-depth intro (`:750`) | note the CrowdSec agent + bouncer-enabled Caddy are supply-chain-hardened via the scanned, digest-pinned toolchain image; recurrence guarantee = **daily** `--no-cache --pull` toolchain rebuild + blocking Trivy gate + bot PR (+ per-PR `verify-toolchain-pin` for pinned-dep bumps). Be precise per §3.8.3: it does not close the unpinned-transitive-MVS gap (unchanged from today) | +| `ARCHITECTURE.md` | §"Development Workflow / Local Development Setup" (`:1204`) | add the offline build note (`--build-arg …_SRC=…-inline`) and `make build-offline` | +| `CONTRIBUTING.md` | build section | fork PRs compile the toolchain from source (slower CI); maintainers re-dispatch for the prebuilt path | +| `docs/features.md` | — | no user-facing capability change → **no edit** (per CLAUDE.md keep brief) | +| `docs/security.md` / `SECURITY.md` | supply-chain / build integrity paragraph | describe the toolchain image, its **daily** `--no-cache --pull` rebuild + blocking Trivy gate, the digest pin, and the `verify-toolchain-pin` freshness guard as the mechanism that keeps bundled binaries patched; state the §3.8.3 scope precisely (pinned-dep + base-image drift covered; unpinned-transitive MVS gap unchanged from today) — do not overclaim | +| new `docs/ci/toolchain-image.md` | — | operator/maintainer runbook: how the key works, how to force a rebuild, how to respond to the bot PR / failure issue, how to roll back | +| `Makefile` | — | `build-offline` target | +| `renovate.json` | — | comment on the `charon-toolchain` datasource entry: digest bumps are owned by the bot workflow, not Renovate | --- -### Commit 1 — E2E specs for new behavior (`test.fixme`) +## 10. API / schema impact -- **Scope:** Failing-by-design E2E coverage of the three headline behaviors, mock-response style. -- **Files:** - - `tests/monitoring/uptime-monitoring-scale.spec.ts` (new) — all `test.fixme`. - - `tests/fixtures/uptime.ts` (new) — `makeSummaryFixture(n)`, `makeBeatSeries(n)` helpers. -- **Depends on:** nothing. -- **Validation gate:** - - `npx playwright test tests/monitoring/uptime-monitoring-scale.spec.ts --project=firefox` → all `fixme` (skipped), 0 failures. - - `cd frontend && npm run type-check` (spec + fixtures type-check). -- **Notes:** Establishes the acceptance shape (§4 Phase 1). No app code touched. +**None.** No REST endpoint, no GORM model, no migration, no `internal/**` code, no frontend, no DB. This is entirely CI/build-graph and repo tooling. `routes.go` AutoMigrate untouched. -### Commit 2 — Foundation: `NextCheckAt`, indexes, config keys, interval-floor validation - -- **Scope:** Schema + config + validation only. **No change to checking behavior** (old ticker still runs). -- **Files:** - - `backend/internal/models/uptime.go` — add `NextCheckAt time.Time` to `UptimeMonitor` (+ `gorm:"index"`). **Do NOT touch `UptimeHeartbeat` tags** — the `idx_heartbeat_monitor_created` composite is created lazily by the pruner in C6 (§3.5.6), not via a struct tag. - - `backend/internal/services/uptime_config.go` (new) — `uptimeConfig` snapshot + `clampInterval`. - - `backend/internal/api/routes/routes.go` — `FirstOrCreate` seeds for `uptime.default_interval_seconds` (60), `uptime.worker_pool_size` (30), `uptime.heartbeat_retention_days` (90), `Category:"uptime"`. (`&models.UptimeMonitor{}` already in `AutoMigrate` — the `next_check_at` column + its index are added automatically; ≤ 500-row table, sub-millisecond.) - - `backend/internal/api/handlers/settings_handler.go` — `uptime.*` validation branch in `UpdateSetting` (bounds per §3.6.1). - - `backend/internal/api/handlers/uptime_handler.go` — `Create` rejects `0 < interval < 30` → `400`. - - `backend/internal/services/uptime_service.go` — `CreateMonitor` resolves `interval<=0 → cfg.DefaultIntervalSeconds()`, floors `<30 → 30` (or errors — see §3.6.3); `UpdateMonitor` `interval` branch adds floor check returning `ErrIntervalTooLow`; construct/inject `uptimeConfig`. - - Tests: `uptime_config_test.go` (new), `settings_handler_test.go` (+cases), `uptime_handler_test.go` (+cases), `uptime_service_test.go` (+`CreateMonitor`/`UpdateMonitor` floor cases). -- **Depends on:** C1 (order only). -- **Validation gate:** - - `cd backend && go test ./internal/models/... ./internal/services/... ./internal/api/handlers/... ./internal/api/routes/...` - - `cd backend && go build ./...`; `make lint-fast`. - - `./scripts/scan-gorm-security.sh --check` (touches `models/uptime.go`). -- **Rollback:** Pure additive; revert commit. `NextCheckAt` column left in place is harmless (unused). - -### Commit 3 — `UptimeIngester` (dumb persistence mirror) + `CheckResult` / `HostCheckResult` +--- -- **Scope:** New ingester component + result types. **Constructed in `routes.go`, `Run` not started** (S1) — nothing sends to it yet, so it is inert. -- **Files:** - - `backend/internal/services/uptime_ingester.go` (new) — `CheckResult`, `HostCheckResult`, `UptimeIngester` (`results <-chan any`, `noteDropped`, `DroppedCount`, `Run(ctx)` that returns on `results` **closed** — not `ctx.Done()` — with a final flush, `Stop` test helper). Pure column-copy flush: heartbeat batch insert + coalesced `uptime_monitors` / `uptime_hosts` updates in one transaction; **no** transition logic, **no** fan-out. - - `backend/internal/api/routes/routes.go` — create the `results` channel; `ingester := services.NewUptimeIngester(db, results)`; store the ref. **Do not** `go ingester.Run(ctx)` yet. - - `backend/internal/services/uptime_ingester_test.go` (new) — drop-on-full (`noteDropped` increments), batch-by-count, batch-by-timer, type-switch routing of `CheckResult` vs `HostCheckResult`, `Run` terminates only when `results` is closed + does a final flush, `ctx.Done()` alone does **not** terminate `Run`, transaction lock-error bounded-retry. -- **Depends on:** C2. -- **Validation gate:** - - `cd backend && go test ./internal/services/... -run Uptime`; `cd backend && go test ./internal/api/routes/...` - - `go build ./...`; `make lint-fast`; `./scripts/scan-gorm-security.sh --check`. -- **Rollback:** Constructed-but-idle. Revert is isolated (also removes the `routes.go` construction line). +## 11. Out-of-scope / follow-ups -### Commit 4 — Bounded worker pool (state maps, host jobs) + keep-alive SSRF client; de-block host pre-check +- Mirror `charon-toolchain` to Docker Hub for GHCR-outage resilience. +- Twice-daily toolchain freshness trigger (§3.8.3 hardening toggle). +- Conditional `timeout-minutes` expression on `security-pr` / `supply-chain-pr` to give same-repo runs a tighter 15-min budget while forks keep 20 (§3.9 B6) — deferred to keep the YAML simple. +- Fold `gosu-builder` / `backend-builder` into the toolchain image too (they are already fast; low value). +- Cosign-sign the toolchain image and verify the signature in the app build `FROM` (needs BuildKit attestation verification; separate spec). +- **N11 (confirmation, no action):** `orthrus-build.yml` builds `./agent/Dockerfile` — a **different** image (the Orthrus agent), with its own `cache-from/to type=gha` and no `caddy-builder`/`crowdsec-builder` stages. Verified out of scope; this spec makes no change to it. -- **Scope:** `UptimeWorkerPool` with the authoritative `monState`/`hostState` maps, `Kind`-discriminated jobs, `network.WithKeepAlive`, pure `runCheck`/`runHostCheck` extraction, single-dial host pre-check. **Constructed in `routes.go`, `Run` not started** (S1). Legacy `checkMonitor`/`checkHost` still active this commit — `runCheck`/`runHostCheck` are parallel pure functions used only by the pool until C5 collapses the old paths (transient duplication, called out per N3). -- **Files:** - - `backend/internal/network/safeclient.go` — add `WithKeepAlive(maxIdle, perHost int, idleTimeout time.Duration)` option + conditional `Transport` fields (default byte-for-byte unchanged). - - `backend/internal/network/safeclient_test.go` — keep-alive on: connection reuse within `idleTimeout` (httptest, assert connection count); **connection older than `idleTimeout` (30 s) is NOT reused**; link-local / cloud-metadata still blocked; redirects still not followed. - - `backend/internal/services/uptime_check.go` (new) — `runCheck(ctx, job, client) rawResult` + `runHostCheck(ctx, job, dialer) rawResult`: pure probes (HTTP/TCP/orthrus; single host dial), **no DB, no state-map access, no notifications**. - - `backend/internal/services/uptime_worker_pool.go` (new) — `UptimeJobKind`/`UptimeJob`, `monStateEntry`/`hostStateEntry`, `UptimeWorkerPool` (`SeedState`, `ReseedState`, `EnsureMonitorState`, `Run`, `TryEnqueue`, `Enqueue`, `QueueDepth`, `EnqueueDropped`, `HostState`), shared keep-alive client + `hostDialer` construction, `handle()` dispatch: debounce RMW under `monMu`/`hostMu`, synchronous transition + notification, host→down synthetic child fan-out, `workerWG` shutdown that closes `results`. - - `backend/internal/api/routes/routes.go` — `pool := services.NewUptimeWorkerPool(db, results, ingester, cfg, notifier, poolSize)`; store the ref. **Do not** `go pool.Run(ctx)` yet. - - `backend/internal/services/uptime_service.go` — `checkHost` inner `for retry` sleep-loop removed → single dial (keeps the cross-cycle `FailureThreshold` debounce). - - Tests: `uptime_worker_pool_test.go` (new — enqueue/`TryEnqueue`-drop, `Enqueue` 2 s timeout, per-check deadline, `SeedState` populates from DB, `monMu` serializes two concurrent RMWs for the same monitor giving the correct streak, `JobHostCheck` path, `workerWG` drains + closes `results` on `ctx` cancel), `uptime_check_test.go` (new — pure probe outcomes, SSRF parity: `127.0.0.1`/`10.x` allowed, link-local blocked), `uptime_service_test.go` (host pre-check no longer sleeps — assert wall-clock), `uptime_service_race_test.go` (adjusted). -- **Depends on:** C3 (`CheckResult`/`HostCheckResult`, the `results` channel). -- **Validation gate:** - - `cd backend && go test ./internal/network/... ./internal/services/... ./internal/api/routes/...` - - `go build ./...`; `make lint-fast`. - - CodeQL Go local (`lefthook run pre-commit`) — new network option is SSRF-adjacent. -- **Rollback:** `WithKeepAlive` additive; pool constructed-but-idle. Revert isolated (also removes the `routes.go` construction line). +--- -### Commit 5 — Scheduler goes live; teardown chain; remote-server hook; restore rehydrate +## 12. Commit Slicing Strategy -- **Scope:** The behavior switch. `UptimeScheduler` (monitor + **host** schedules) + `UptimeSyncLoop` are created and **all `Run` loops are started** (ingester, pool, scheduler, sync loop); the old ticker go-func is deleted; `checkMonitor`/`checkHost` collapse onto `runCheck`/`runHostCheck` + the pool + ingester. Plus: remote-server sync hooks, auto-create default-interval fix (S3), restore rehydrate (S6), and the shutdown grace check (S4). -- **Files:** - - `backend/internal/services/uptime_scheduler.go` (new) — `monSchedule` + `hostSchedule` maps, `hydrate()` (monitor + host cold-start, jittered backfill), per-tick host pass + monitor pass with the `pool.HostState` short-circuit, batched `next_check_at` write-back, feature-flag gate, `rescan()` (new/disabled monitors + `hostMinInt` recompute + `pool.EnsureMonitorState`), `Rehydrate()` (re-`hydrate()` + `pool.ReseedState()`), `ctx` shutdown = "stop enqueuing". - - `backend/internal/services/uptime_service.go` — `CheckAll()` re-implemented to enqueue every enabled host + monitor into the pool and return `(enqueued, dropped int)` (N5); `checkMonitor`/`CheckMonitor` and `checkHost` route through the pool (delete direct `s.DB.Create(&heartbeat)` / `s.DB.Save(&monitor)` from the check path); replace **every** hardcoded `Interval: 60` in `SyncMonitors` (~223, ~320), `SyncAndCheckForHost` (~1402) with `clampInterval(0, s.uptimeCfg)` (S3); new `SyncAndCheckForRemoteServer(remoteServerID uint)` / `SyncMonitorForRemoteServer(remoteServerID uint) error` (also `clampInterval(0, …)`), `hostMutexes` key `remote-`, Orthrus-unbound-UUID → silent no-op. - - `backend/internal/api/handlers/remote_server_handler.go` — `RemoteServerHandler` + `NewRemoteServerHandler` gain a nil-guarded `uptimeService *services.UptimeService`. `Create` → `go SyncAndCheckForRemoteServer`; `Update` → `go SyncMonitorForRemoteServer` (log on error); `Delete` → inline `WHERE remote_server_id = ?` → `DeleteMonitor` before `h.service.Delete(...)` (mirrors `proxy_host_handler.go:755-761`). - - `backend/internal/services/backup_service.go` — `RestoreBackupSafe` reconcile step calls `scheduler.Rehydrate()` after the DB is restored (S6). Requires a scheduler ref reachable from the restore path (inject or a small setter, mirroring how the Caddy manager ref is threaded). - - `backend/internal/api/routes/routes.go` — delete the ticker go-func; construct `UptimeScheduler` + `UptimeSyncLoop`; `go X.Run(ctx)` for **ingester, pool, scheduler, sync loop** (pruner started in C6); `runInitialUptimeBootstrap` loses `CheckAll()`; `POST /system/uptime/check` returns `{enqueued,dropped}`, `POST /uptime/monitors/:id/check` uses `pool.Enqueue` (503 on full); pass `uptimeService` into `NewRemoteServerHandler(...)` (~897); wire the scheduler ref to the restore path. **Verify `server.Run` / `http.Server.Shutdown` grace ≥ `hardCap` (20 s) + ~2 s (S4)** — raise it or lower `hardCap` if short; note the finding in the PR. - - `backend/internal/api/routes/routes_uptime_bootstrap_test.go` — drop `CheckAll` from the `uptimeBootstrapService` interface + tests. - - `backend/internal/api/handlers/uptime_handler.go` — `CheckMonitor` uses `pool.Enqueue` (503); `Sync`/system-check handlers surface `{enqueued,dropped}`. - - Tests: `uptime_scheduler_test.go` (new — monitor **and host** due-selection; interval clamp incl. auto-create honours `default_interval_seconds` — S3; backfill spread; write-back grouping; new/disabled reconcile; `Rehydrate()` re-syncs after a simulated live restore — S6; `ctx` cancel stops enqueuing), `uptime_worker_pool_test.go` / a new `uptime_pipeline_test.go` (**S4** in-flight-check heartbeat still written on immediate `ctx` cancel; **B3** saturated ingester + every `CheckResult` dropped → `down` still detected + `queueDownNotification` called; **B2** host→down fan-out + scheduler skip + recovery), `uptime_service_*_test.go` updated for the pool-routed write path + remote-server sync cases, `uptime_handler_test.go`, `remote_server_handler_test.go` (new constructor arg + hook invocation), `backup_service` test (reconcile invokes `Rehydrate`), `routes_test.go` if it asserts the ticker. -- **Depends on:** C3, C4. -- **Validation gate:** - - `cd backend && go test ./internal/services/... ./internal/api/handlers/... ./internal/api/routes/...` - - `go build ./...`; `make lint-fast`; `./scripts/scan-gorm-security.sh --check`. - - CodeQL Go local — new execution path + fan-out. - - Manual: `go run ./cmd/api` with a few seeded monitors — per-monitor cadence, clean `ctx` shutdown (goroutines exit, no panic), and a killed target flips to `down` and alerts. -- **Rollback:** Highest-risk commit, but isolated: `git revert ` restores the legacy ticker and leaves C3/C4 (pool/ingester constructed but idle), C6, C7 all dormant and building — because their construction lives in C3/C4, not here (S1). +**Decision:** one feature = **one PR** targeting `development`, sliced into 6 ordered logical commits. Each commit builds and passes its own gate; the PR merges only when the full DoD (§5) passes. Not split across multiple PRs. -### Commit 6 — Retention pruner + deferred `idx_heartbeat_monitor_created` +The `weekly-nightly-promotion.yml` "merge commit only" rule is **not** engaged — this PR follows the normal `development` flow and touches no promotion machinery. -- **Scope:** Hourly chunked retention delete, **plus** deferred `idx_heartbeat_monitor_created` creation retried at the end of every clean, caught-up pass until it lands (prune-before-index ordering, §3.5.6 / §3.4.2). -- **Files:** - - `backend/internal/services/uptime_pruner.go` (new) — hourly loop; `pruneOnce(ctx) (deleted int64, err error)` chunked subquery `DELETE` (`WHERE id IN (SELECT id ... LIMIT 5000)`), `pruneChunkPause`, WAL checkpoint threshold, `PRAGMA optimize` cadence, `ctx` abort. After each pass where `pruneOnce` returned `err == nil` and reached its "caught up" break, `Run` issues `CREATE INDEX IF NOT EXISTS idx_heartbeat_monitor_created ON uptime_heartbeats (monitor_id, created_at)` (no `sync.Once` — idempotent, re-attempted hourly until it succeeds). - - `backend/cmd/api/main.go` — in the `case "migrate":` block, after `db.AutoMigrate(...)`: `logger.Log().Warn("building index idx_heartbeat_monitor_created on uptime_heartbeats; on a large database this can take several minutes and holds a write lock for the duration")` then an unconditional `db.Exec("CREATE INDEX IF NOT EXISTS idx_heartbeat_monitor_created ON uptime_heartbeats (monitor_id, created_at)")` (operator-initiated maintenance window; idempotent, harmless on fresh DBs) — S7. - - `backend/internal/api/routes/routes.go` — construct `UptimePruner` + `go pruner.Run(ctx)`; `firstPassDone` widens the inter-chunk pause until the first clean pass (N1). - - `backend/internal/services/uptime_pruner_test.go` (new) — deletes only `< cutoff`; chunk loop terminates at `RowsAffected < chunk`; honors hot config change; `ctx` mid-loop abort **does not** attempt the index; a clean caught-up pass **does** create the index (assert via `PRAGMA index_list(uptime_heartbeats)`); a subsequent pass with the index already present is a no-op (no error); a first pass that returns an error followed by a later clean pass still creates the index. -- **Depends on:** C2 (`uptime.heartbeat_retention_days`, `uptimeConfig`). -- **Validation gate:** - - `cd backend && go test ./internal/services/... -run Pruner`; `cd backend && go test ./cmd/api/... -run Migrate` (if a migrate-CLI test exists; else `go build ./cmd/api`). - - `go build ./...`; `make lint-fast`; `./scripts/scan-gorm-security.sh --check` (raw `DELETE` / `CREATE INDEX` `Exec`). -- **Rollback:** Independent goroutine; revert removes the pruner and the deferred index creation with no schema impact (the index, if already built on a running instance, is harmless to leave — or drop it manually). +**B5 — the CVE-recurrence guard is never inert.** The old plan had a window (Commit 1→3) where the stages were renamed to `caddy-inline`/`crowdsec-inline` but the workflows still said `--no-cache-filter caddy-builder` — a filter on an *alias* node does not invalidate the `RUN` layers that moved into `caddy-inline`, so the guard was silently dead. Fixed below: **Commit 1 retargets every `--no-cache-filter` / `no-cache-filters` from `caddy-builder,crowdsec-builder` to `caddy-inline,crowdsec-inline` in the same commit as the rename**, and the freshness guard (`verify-toolchain-pin`, Commit 3) is in place **before** those filters are removed (Commit 4). Every commit's gate below explicitly checks that *some* live mechanism forces a from-source recompile when a pin/recipe changes. -### Commit 7 — Batch summary endpoint + `/uptime/health` + history pagination +### Commit 1 — `feat(security): add toolchain-image workflow, key tooling, split builder stages` +- **Scope:** the toolchain build/publish workflow + key/guard scripts; Dockerfile stage split; **retarget the no-cache filters to the RUN-bearing stage names**; first manual publish; make the new GHCR package internal. - **Files:** - - `backend/internal/services/uptime_summary_service.go` (new) — `MonitorSummary`, `BeatDTO`, `UptimeSummaryService`, 30 s TTL cache (ported `summaryCache` shape), 3-query strategy. - - `backend/internal/api/handlers/uptime_handler.go` — `Summary(c)` (`beats` clamp 1..60), `Health(c)`; `GetHistory` — `limit` cap 500, `before` RFC3339 cursor. - - `backend/internal/services/uptime_service.go` — `GetMonitorHistory(id, limit, before)` signature + cap. - - `backend/internal/api/routes/routes.go` — `management.GET("/uptime/monitors/summary", uptimeHandler.Summary)`, `management.GET("/uptime/health", uptimeHandler.Health)`. Wire `UptimeSummaryService` + the pool/ingester refs (constructed in C3/C4) into the handler. - - Tests: `uptime_summary_service_test.go` (new — windowed query returns ≤ `beats` chronological ASC, cache hit skips query, `uptime_24h` math, empty-history case, **correct with and without `idx_heartbeat_monitor_created` present**, never 503-gated on index absence; **`TestUptimeSummary_PerfBudget`** — `-short`-skippable, 500-monitor + 24 h-heartbeat seed, index built, `GetSummary(ctx, 30)` < 2 s wall-clock — S5), `uptime_handler_test.go` (+`Summary` `beats` default 30 / clamp 60, `Health`, history cap 500, `before` paging), `routes_test.go` (+**N4**: assert both `GET /uptime/monitors/summary` and `GET /uptime/monitors/:id/history` resolve to their handlers — mixed static/param route on the same segment). -- **Depends on:** C2 (config keys), C4 (pool/ingester refs for `Health` — constructed there per S1, so C7 builds even if C5 is reverted), C6 (the `idx_heartbeat_monitor_created` index — summary is correct without it but meets the perf gate only with it). + - `scripts/lib/dockerfile-stage.sh`, `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh` (new) + - `scripts/tests/toolchain-key.bats` (+ wire into `quality-checks.yml` as a **non-blocking** job for now) + - `.github/workflows/toolchain-image.yml` (new — `schedule` daily + `workflow_dispatch` + `pull_request` paths + `workflow_call`; **build/publish + trivy-scan jobs only**; `sync-pin-on-pr` / `open-bump-pr` land in Commit 3) + - `Dockerfile` — rename `caddy-builder→caddy-inline`, `crowdsec-builder→crowdsec-inline`; **delete the dead `crowdsec-fallback` stage** (`:713-748`) and its now-dead `CROWDSEC_RELEASE_SHA256` ARG (N1); **pin the two xcaddy plugins** `CADDY_GEOIP2_VERSION` / `CADDY_RATELIMIT_VERSION` (B4); **digest-pin the `golang:${GO_VERSION}-alpine` base** of both inline stages (N4); add `toolchain-runtime` assembly stage; add temporary aliases `FROM caddy-inline AS caddy-builder` / `FROM crowdsec-inline AS crowdsec-builder` so the app build is unchanged this commit. + - **All six no-cache-filter sites + composite action** (§3.6) — change the value `caddy-builder,crowdsec-builder` → `caddy-inline,crowdsec-inline` (do **not** remove yet). +- **Dependencies:** none. +- **Bootstrap / N8:** after CI publishes the first image via `workflow_dispatch`, in GHCR set the `charon-toolchain` package visibility to **Internal** (or link it to the repo and grant the repo `packages: read`) so cross-workflow `FROM ghcr.io/…/charon-toolchain@digest` works with the default `GITHUB_TOKEN`. Document this one-time manual step in `docs/ci/toolchain-image.md` (Commit 6) and in the PR description. - **Validation gate:** - - `cd backend && go test ./internal/services/... ./internal/api/handlers/... ./internal/api/routes/...` - - `go build ./...`; `make lint-fast`; `./scripts/scan-gorm-security.sh --check`. - - CodeQL Go local (`lefthook run pre-commit`) — new endpoints. -- **Rollback:** Additive endpoints + one changed service signature; revert also reverts the `GetMonitorHistory` signature (update call sites). Isolated from execution model — and from C5, since the `Health` deps come from C4. + 1. `bats scripts/tests/` green; `shellcheck scripts/*.sh scripts/lib/*.sh` clean. + 2. `scripts/toolchain-key.sh` is stable across a whitespace-only reformat outside the two stages, and **changes** when (a) a `go get` line inside `caddy-inline` is edited, (b) `CADDY_VERSION` / `CADDY_GEOIP2_VERSION` default is bumped, (c) the golang base digest changes, (d) `.trivyignore` changes. + 3. `workflow_dispatch` toolchain-image.yml on the branch → publishes `ghcr.io/wikid82/charon-toolchain:caddy-crowdsec-`; `docker buildx imagetools inspect` shows **both** `linux/amd64` and `linux/arm64`. Record `:` + manifest-list digest for Commit 2. + 4. **Guard-live check:** on a scratch build, `docker buildx build --no-cache-filter caddy-inline …` shows the `xcaddy build` step running (not `CACHED`); with the old `--no-cache-filter caddy-builder` value it would show `CACHED` — confirm the retarget is what keeps the guard effective. -### Commit 8 — Frontend: summary-driven Uptime page + interval floor + admin Uptime settings card +### Commit 2 — `feat(security): build app image from the pinned toolchain image` -- **Scope:** (1) Uptime page reads the batch summary endpoint (kills N+1); (2) per-monitor interval field with 30 s floor; (3) new admin "Uptime Monitoring" settings card on `SystemSettings.tsx` for the three `uptime.*` global keys (§3.6.4). +- **Scope:** default path consumes the prebuilt image by digest; inline stages become the selectable fallback; fork detection wired. - **Files:** - - `frontend/src/api/uptime.ts` — `BeatDTO`, `MonitorSummary`, `getMonitorsSummary(beats = 30)`, `before` param on `getMonitorHistory`; `syncMonitors` response type gains `enqueued?` / `dropped?`; document that `checkMonitor` may reject with `503` (N5). - - `frontend/src/hooks/useUptimeSummary.ts` (new) — `getMonitorsSummary(30)`, `refetchInterval: 30000`, key `['uptimeSummary']`. - - `frontend/src/pages/Uptime.tsx` — single `useUptimeSummary()`; `MonitorCard` reads `monitor.recent_beats` from props, **remove** per-card `useQuery(['uptimeHistory'])`; retarget all `invalidateQueries` to `['uptimeSummary']`; heartbeat bar becomes `BEAT_BAR_SLOTS = 30` wide with updated tooltip copy; interval `` + clamp + helper text in Create + Edit modals; `checkMutation` / `syncMutation` `onError` (or `dropped > 0`) → toast "Check queue full, try again in a moment" instead of a silent success (N5). - - `frontend/src/pages/SystemSettings.tsx` — new `` "Uptime Monitoring" (three number inputs: `uptime.default_interval_seconds`, `uptime.worker_pool_size`, `uptime.heartbeat_retention_days`), client-side bounds validation matching §3.6.1, helper text noting `worker_pool_size` needs a restart while the other two hot-reload ~60 s / ~1 h, `useMutation` → `updateSetting(key, String(v), 'uptime', 'int')` per changed field → `invalidateQueries(['settings'])`. Gated on `feature.uptime.enabled` (reuse the existing `featureFlags` query in the file). - - `frontend/src/components/UptimeWidget.tsx` — optional switch to `getMonitorsSummary`; otherwise unchanged. - - Tests: - - `frontend/src/pages/__tests__/Uptime.test.tsx`, `Uptime.spec.tsx`, `Uptime.tcp-ux.test.tsx` — updated to summary fixture; assert exactly one `getMonitorsSummary` call and **zero** history fetches on load/refetch; interval-floor form validation (reject 10, accept 30); manual check → `503` (or `dropped > 0`) surfaces a toast (N5). - - `frontend/src/api/__tests__/uptime.test.ts` — `getMonitorsSummary` (URL, `beats=30` default, `beats` passthrough); `getMonitorHistory` `before` param. - - `frontend/src/pages/__tests__/SystemSettings.test.tsx` (or the existing SystemSettings test file) — **new: Uptime settings card** — renders the three fields seeded from `getSettings`, rejects out-of-bounds input (e.g. interval 10, pool size 0, retention 4000) with the save button disabled, saves in-bounds values via `updateSetting` with `category='uptime'`, card hidden when `feature.uptime.enabled` is off. - - `frontend/src/components/__tests__/ProxyHostForm-uptime.test.tsx` — check interval field if present. - - i18n: add `uptime.checkIntervalHelper` (min 30 s) and `systemSettings.uptime.*` keys (card title, three labels, three helper texts, validation messages) to the locale files. -- **Depends on:** C7. -- **Coverage implication:** the settings card + its validation add ~1 component's worth of new frontend LOC — its dedicated test above keeps the 85 % frontend patch-coverage gate satisfied; do not merge the card without the card test. + - `Dockerfile` — add `CHARON_TOOLCHAIN_IMAGE/TAG/DIGEST` ARGs (values from Commit 1's publish), `CADDY_BUILDER_SRC`/`CROWDSEC_BUILDER_SRC` selector ARGs, `toolchain-prebuilt` stage; replace the temp aliases with `FROM ${CADDY_BUILDER_SRC} AS caddy-builder` / `FROM ${CROWDSEC_BUILDER_SRC} AS crowdsec-builder`. + - **Every app-image build step** in `docker-build.yml`, `nightly-build.yml`, `security-pr.yml`, `supply-chain-pr.yml`, `e2e-tests-split.yml`, and the **`build-charon-image` composite action** (new `builder-src` input, default `toolchain-prebuilt`, with the `head.repo.full_name` expression in each caller) — pass `--build-arg CADDY_BUILDER_SRC=… --build-arg CROWDSEC_BUILDER_SRC=…` (`toolchain-prebuilt` same-repo, `caddy-inline`/`crowdsec-inline` on forks). + - `Makefile` — `build-offline` target. +- **Dependencies:** Commit 1. +- **Note on the guard in this window:** default builds no longer run `caddy-inline` at all, so the retargeted `--no-cache-filter caddy-inline` is a no-op there — **intended**: the only path that still compiles is the fork/inline path, and the filter remains live *there*. The pin↔digest binding on the default path is enforced by Commit 3's freshness guard, added before any filter is removed (Commit 4). +- **Validation gate:** `docker build --check`; `docker build .` (default) → pulls the image, **no `xcaddy`/`xx-go build` in the log**, image boots, final-stage N5 assertions pass, `caddy version` + `cscli version` OK; `make build-offline` (inline) → compiles and passes the in-`caddy-inline` embeds-version assertions **and** still honours `--no-cache-filter caddy-inline`; `docker buildx build --target caddy-builder --print` resolves for both selector values; simulated fork run (push from a fork or manual expression override) uses the inline path and stays under the 20-min job cap (B6). + +### Commit 3 — `feat(security): enforce toolchain pin freshness + app-side embed assertions` + +- **Scope:** `verify-toolchain-pin` becomes a **required** check; `sync-pin-on-pr` + `open-bump-pr` jobs added to `toolchain-image.yml`; N5 final-stage assertions; extend `docker-build.yml`'s existing post-build CVE-verification step to also check the toolchain `LABEL` key. +- **Files:** `.github/workflows/quality-checks.yml` (required `verify-toolchain-pin` job, with `regctl` install + `GHCR_READ_TOKEN`), `.github/workflows/toolchain-image.yml` (add `sync-pin-on-pr`, `open-bump-pr`), `Dockerfile` (N5 `RUN` assertion after the `COPY --from` lines), `docker-build.yml` (extend verification step), `renovate.json` (comment: toolchain digest is bot-owned, N7). +- **Dependencies:** Commits 1–2 (a real pin must exist to guard). - **Validation gate:** - - `cd frontend && npm run test -- uptime Uptime SystemSettings` (targeted) then full `npm run test`. - - `cd frontend && npm run type-check`; `cd frontend && npm run build`. -- **Rollback:** Frontend-only; revert restores per-card history and removes the settings card. Backend summary endpoint + `uptime.*` keys stay (keys remain editable via `POST /api/v1/settings`). - -### Commit 9 — Hardening: flip E2E live, docs, ARCHITECTURE + 1. Temp commit bumping `CADDY_VERSION` (no rebuild) → `verify-toolchain-pin` **fails** with the actionable message; the `toolchain-image.yml` path trigger rebuilds and `sync-pin-on-pr` pushes the `TAG`/`DIGEST` bump onto the branch → check green → revert temp commit. + 2. Temp commit hand-editing `CHARON_TOOLCHAIN_DIGEST` to a valid-but-wrong digest → `verify-toolchain-pin` **fails** on the same-repo digest-mismatch branch (proves B7 failure-closed: it is not a tag-only check). + 3. `open-bump-pr` runs only on `schedule`/`workflow_dispatch`/`workflow_call`, never `pull_request` (assert via a dry `workflow_dispatch`). + 4. Build an app image against a deliberately wrong (old) toolchain digest → the N5 final-stage assertion fails the build (proves a bad pin is caught even if `verify-toolchain-pin` were bypassed). -- **Files:** - - `tests/monitoring/uptime-monitoring-scale.spec.ts` — `test.fixme` → `test`; finalize mock payloads to the shipped schema. - - `tests/monitoring/uptime-monitoring.spec.ts` / `tests/a11y/uptime.a11y.spec.ts` — adjust for the new card data source if needed. - - `docs/features/uptime-monitoring.md` — per-monitor intervals, scaling section, `uptime.*` settings table + bounds + hot-reload, accepted double-DNS note. - - `ARCHITECTURE.md` — "Uptime Subsystem" subsection, endpoint table rows, DB concurrency note (per §5). - - `docs/features.md` — one-line touch if it summarizes uptime. -- **Depends on:** C1–C8. -- **Validation gate (also the PR-level DoD gate):** - - `npx playwright test tests/monitoring/uptime-monitoring-scale.spec.ts tests/monitoring/uptime-monitoring.spec.ts tests/a11y/uptime.a11y.spec.ts --project=firefox` — all green. - - `bash scripts/local-patch-report.sh` — artifacts + patch coverage green. - - `scripts/go-test-coverage.sh` ≥ 85%; `scripts/frontend-test-coverage.sh` ≥ 85%. - - CodeQL Go + JS + Trivy — zero high/critical. - - `lefthook run pre-commit` clean; `cd backend && go build ./...`; `cd frontend && npm run build && npm run type-check`. - - `./scripts/scan-gorm-security.sh --check` — zero CRITICAL/HIGH. +### Commit 4 — `perf(ci): drop the forced from-source rebuilds; rely on the pinned image + guard` ---- +- **Scope:** remove every `--no-cache-filter` / `no-cache-filters` and the composite `no-cache-filters` input — now safe because (a) the default path never compiles, (b) `verify-toolchain-pin` enforces pin↔digest freshness per PR, (c) the daily toolchain rebuild + Trivy gate covers base-image drift, (d) the N5 assertion catches a wrong digest. +- **Files:** `docker-build.yml` (`:463-464`, `:549-550`), `security-pr.yml` (`:157-164` block), `supply-chain-pr.yml` (`:252-261` block), `e2e-tests-split.yml` (`:224`), `nightly-build.yml` (`:243`), `.github/actions/build-charon-image/action.yml` (delete the `no-cache-filters` input decl `:11-33` + passthrough `:52`; rewrite `description`). +- **Dependencies:** Commit 3 (guard must be live *before* the filters go). +- **Validation gate:** `grep -rn "no-cache-filter" .github/workflows .github/actions` → empty (comments/docs excluded); `docker-build.yml build-amd64` run twice on the branch → second run every stage `CACHED`, wall-time **< 8 min**, zero compile lines; all 8 build-consuming workflows green; a bump-a-pin temp commit still fails `verify-toolchain-pin` (guard still live via the freshness mechanism, not the deleted filter). -### Rollback & contingency (PR-level) +### Commit 5 — `feat(security): route the security rebuild through the toolchain image` -- **Pre-merge:** the risky behavior switch is isolated to **C5** (starting the `Run` loops + collapsing `checkMonitor`). `git revert ` restores the legacy 60 s ticker; C3/C4 leave the pool/ingester **constructed but idle** (so C7's `/uptime/health` still builds and returns zeros), and C6/C7/C8 stay dormant. This is a *real* isolated revert precisely because pool/ingester construction was pushed down to C3/C4 (S1) — nothing after C5 hard-depends on the `Run` loops being started. -- **Post-merge, field regression:** the fastest kill-switch is `feature.uptime.enabled = false` (Setting) — stops the scheduler, pool, and ingester (pruner keeps running, which is desirable). Then a targeted revert PR of the whole feature if needed. -- **Pruner misbehaving:** set `uptime.heartbeat_retention_days` to a very large value (e.g. 3650) to effectively pause deletion without a deploy. -- **Pool starvation in the field:** raise `uptime.worker_pool_size` and restart; `/uptime/health` confirms the new size. -- **Data safety:** no destructive migration. `NextCheckAt` and `idx_heartbeat_monitor_created` are additive. Heartbeat pruning is the only delete and is bounded by a configurable, admin-visible window with a large default (90 d). +- **Scope:** `security-weekly-rebuild.yml` `workflow_call`s `toolchain-image.yml` instead of building a throwaway app image; blocking Trivy on `schedule`/`dispatch`/`workflow_call`; caller grants all perms the bot job needs (N6). +- **Files:** `.github/workflows/security-weekly-rebuild.yml` (swap build step; `permissions:` add `contents: write` + `pull-requests: write` at job level; keep Trivy table/SARIF/JSON/`::warning::`; rename `TRIVY_SARIF_CATEGORY` value to `…:trivy-toolchain`). +- **Dependencies:** Commits 1, 3. +- **Validation gate:** `workflow_dispatch` on the branch → toolchain rebuilds `--no-cache --pull`; Trivy runs; SARIF uploads under the stable category; **no** bot PR when the digest is unchanged; temporarily drop a known-ignored item from `.trivyignore` → `schedule`-path Trivy step is **red** and the failure issue is created → restore `.trivyignore`. Confirm the daily `schedule` on `toolchain-image.yml` (added Commit 1) now also produces a bot PR path via `open-bump-pr` (Commit 3) when the digest moves. ---- +### Commit 6 — `docs(ci): document the toolchain image; right-size timeouts; sweep stale comments` -## 8. Resolved decisions +- **Scope:** timeout edits (§3.9), stale-comment sweep, all `ARCHITECTURE.md` / docs updates (§9). +- **Files:** `docker-build.yml` (`build-amd64` timeout `:403`/`:441` 15→20; comment `:381` + dangling `§1.1` cross-ref), `security-pr.yml` (`:32` comment only — timeout **stays 20**, B6), `supply-chain-pr.yml` (`:34` comment only — stays 20), `*-integration.yml` (`:29` comments), `ARCHITECTURE.md` (§9 rows), `SECURITY.md` / `docs/security.md`, `docs/ci/toolchain-image.md` (new runbook — incl. the N8 one-time package-visibility step), `CONTRIBUTING.md`, `renovate.json` comment, `Makefile` (if not in C2). +- **Dependencies:** Commits 1–5. +- **Validation gate:** `grep -rn "xcaddy\|no-cache-filter\|cold build\|full cold build\|10-14m\|12-14 min" .github/` reconciled; markdown lint; `docs-writer` review; full CI green; DoD §5 all boxes checked. -All six open questions were answered by the user on 2026-08-27 and are folded into the spec above. Recorded here for traceability: +### PR-level rollback / contingency -1. **`worker_pool_size` hot-reload — restart-only.** Pool is sized at construction; `GET /api/v1/uptime/health` surfaces the active value. §3.6.1 keeps "No" for hot-reload. No pool live-resizing. -2. **Admin UI for the 3 `uptime.*` settings — IN THIS PR (Commit 8).** A dedicated "Uptime Monitoring" card on `frontend/src/pages/SystemSettings.tsx` with client-side bounds validation matching §3.6.1, a restart-required note on `worker_pool_size`, wired through `POST /api/v1/settings` (`category=uptime`). See §3.6.4, Commit 8, §7.3a, §7.13. -3. **`uptime_24h` on the summary response — KEPT.** The 3-query strategy stands; the field is always present (nullable when no data). §3.5.2 / §3.5.3. -4. **Remote-server targeted sync — HOOK ADDED (Commit 5).** `SyncAndCheckForRemoteServer` / `SyncMonitorForRemoteServer` + inline delete cleanup on `RemoteServerHandler` create/update/delete, mirroring the proxy-host hooks. The 5-minute `UptimeSyncLoop` remains the backstop. §3.1.3, Commit 5, §7.10a. -5. **`beats` default — 30** (cap unchanged at 60). List view uses 30; an expanded/detail view may request 60. ~50–75 KB gzipped at 500 monitors. Updated in §3.5.1, §3.5.2, §3.5.7, §3.8, §7.4, R8. -6. **Index build ordering — PRUNE FIRST, THEN INDEX; retry-until-success.** `idx_heartbeat_monitor_created` is no longer an AutoMigrate/struct-tag index; `UptimePruner` issues `CREATE INDEX IF NOT EXISTS` at the end of **every** clean, caught-up prune pass (no `sync.Once` — retried hourly until it lands). Prune-first bounds the *pathological* (hundreds-of-millions-row) case; on a healthy 500-monitor instance the first build still runs over ~65 M rows and is a bounded multi-minute, write-contending background operation (no route downtime, never 503; §3.4.2 / R4 state this honestly, per supervisor S2). `charon migrate` builds it eagerly with a `WARN` log. Index-creation work moved from Commit 2 to Commit 6. §3.5.6, §3.4.2, revised R3/R4/R7, Commit 6. +- **Rollback:** revert the single merged commit. The `charon-toolchain` package stays in GHCR unreferenced (`container-prune.yml` ages it out). `security-weekly-rebuild.yml` reverts to its prior behavior. The Dockerfile reverts to inline `caddy-builder`/`crowdsec-builder` with `--no-cache-filter` — **identical security posture to today**. Zero runtime/app-image content change (same recipe), so nothing to migrate or re-release. +- **Contingency (partial):** + - Freshness guard misbehaves post-merge → make `verify-toolchain-pin` non-required (repo setting); the daily `--no-cache --pull` toolchain rebuild + Trivy gate + N5 assertion still protect the guarantee. + - Selector stage breaks one workflow → set that workflow's `--build-arg CADDY_BUILDER_SRC=caddy-inline` as a temporary escape hatch (its `--no-cache-filter caddy-inline` was removed in Commit 4 but can be re-added to that one workflow) — no full revert. + - GHCR unavailable for a release → the release build fails fast; run it again, or fleet-flip the selector build-args to `caddy-inline` via a one-line workflow edit. +- **Forward-fix preferred over revert** for anything touching the security guarantee (CLAUDE.md: long-term fix over quick patch). --- -### Supervisor review (2026-08-27) — REVISE → resolved +## 13. Handoff -`docs/reports/supervisor_review.md` returned REVISE on the first draft. All Blocking + Should-fix + Nice-to-have items are folded in: - -| Item | Resolution | Where | -|---|---|---| -| **B1** host-check scheduling | `UptimeScheduler` gains a `hostSchedule` map + cold-start host hydration + a per-tick host pass enqueuing `UptimeJob{Kind: JobHostCheck}`; host due-times in-memory only. | §3.0, §3.1.2, §3.2.3, R4a | -| **B2** host-down short-circuit owner | The **worker** (not the ingester) owns host transition detection + synthetic child `down` fan-out + the consolidated notification; a shared `pool.hostState` map is read (RLock) by the scheduler for the skip decision. Ingester stays a dumb writer. `UptimeJob.Kind` added; `HostCheckResult` type added. | §3.0, §3.2.1, §3.2.3, §3.3.1/3.3.2, §3.8.2 | -| **B3** debounce vs droppable write | Authoritative `pool.monState` map (seeded from DB once), read-modify-written **synchronously by the worker** under `monMu`; the ingester's `status`/`failure_count` write is a persistence mirror. A dropped `CheckResult` cannot suppress a transition. New test: saturated ingester + all drops → `down` still detected + notified. | §3.0, §3.2.1, §3.3.3, §3.8.1, R2, §7.8 | -| **S1** C5 revertibility | Pool + ingester **constructed** in C3/C4 (`Run` started in C5). `git revert C5` genuinely restores the old ticker with C3/C4/C6/C7 dormant + building. | dependency-order note, Commits 3/4/5/7, Rollback | -| **S2** deferred-index premise | Row-count math corrected (~65 M steady-state at 500 monitors, not 13 M); first-boot index build honestly described as a bounded multi-minute write-contending op; 90-day default unchanged (user decision). | §3.4.2, §3.5.6, R3/R4/R7, Phase 5 | -| **S3** auto-created interval | `SyncMonitors` / `SyncAndCheckForHost` / `SyncAndCheckForRemoteServer` create monitors with `clampInterval(0, cfg)` → honour `uptime.default_interval_seconds`. Test added. | §3.1.3, §3.6.3, §7.2a, R10, Commit 5 | -| **S4** shutdown handshake | Explicit ordered teardown chain enforced by channel ownership (scheduler stops → pool `workerWG.Wait()` + closes `results` → ingester drains-until-closed + final flush). Grace-period check + no-result-loss test. | §3.1.4, §3.8.3, R13, §7.9, Commit 5 | -| **S5** p95 gate | `TestUptimeSummary_PerfBudget` (`-short`-skippable): 500-monitor + 24 h seed, index built, `GetSummary(30)` < 2 s wall-clock. | §3.5.3, §7.4a, R7, Commit 7 | -| **S6** backup/restore | §3.9: restore-then-restart = ordinary cold start (goroutines start after DB init); live restore self-heals within one `rescan()` + 1–2 cycles; `UptimeScheduler.Rehydrate()` called from `RestoreBackupSafe` reconcile makes it immediate. | §3.7 row, §3.9, R14, §7.9a, Commit 5 | -| **S7** `charon migrate` warning | Warning log before the eager `CREATE INDEX` + Phase 5 deploy-note sentence. | §3.5.6 item 3, Phase 5, Commit 6 | -| **N1** pruner chunk latency | Honest 100–500 ms/chunk on a cold huge table; `firstPassChunkPause = 250 ms` for the first pass. | §3.4.1, §3.7, R3, Commit 6 | -| **N2** keep-alive idleTimeout | 90 s → **30 s**; `safeclient_test.go` asserts a connection older than `idleTimeout` is not reused; 500-distinct-hosts churn noted. | §3.2.2, R11, Commit 4 | -| **N3** transient check-logic dup C4→C5 | Called out in the C4 scope note. | Commit 4 | -| **N4** mixed static/param route smoke test | `routes_test.go` asserts `/uptime/monitors/summary` and `/uptime/monitors/:id/history` both resolve. | Commit 7 | -| **N5** manual check drops silently | `POST /:id/check` → 503; `POST /system/uptime/check` → `{enqueued,dropped}`; frontend toasts. | §3.5.7, §3.7, Commit 5, Commit 8 | -| **N6** ARCHITECTURE.md omissions | `uptimeConfig` + the 3 `uptime.*` keys + the pooled-SSRF-client note added to the §5 update list. | §5 | -| **N7** error wrapping | `fmt.Errorf("context: %w", err)` called out in the Phase 2 per-commit requirement. | §4 Phase 2 | -| **N8** `uptimeConfig` test seam | `now func() time.Time` + `forceRefresh()`. | §3.6.2, Commit 2 | +On approval: route to **supervisor** for plan review; iterate here until approved; then present to the user for explicit go-ahead before implementation. Implementation is CI/build-only → delegate commit-by-commit primarily to **devops** (with **docs-writer** for Commit 6), each commit gated as above, then **supervisor** re-review, then **qa-security** last against `SECURITY.md` + DoD. From 17cec8008d806b9e2c34e29843cd628987a4dbf3 Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Mon, 7 Sep 2026 15:53:31 -0400 Subject: [PATCH 02/19] ci: add toolchain image workflow, key tooling, and split builder stages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extract the custom Caddy + CrowdSec build recipe into a separately-versioned prebuilt image so ordinary app builds stop recompiling xcaddy / CrowdSec on the hot path (spec docs/plans/current_spec.md §12 Commit 1). - scripts/lib/dockerfile-stage.sh — shared `extract_stage` (N9), sourced by both key scripts. - scripts/toolchain-key.sh — deterministic content-addressed tag (caddy-crowdsec-) over the two inline stage bodies + consumed version ARGs (incl. the two now-pinned xcaddy plugins) + the xx / digest-pinned golang base lines + .trivyignore + a SCHEMA_VERSION. - scripts/verify-toolchain-pin.sh — failure-closed freshness guard (B7): on a trusted same-repo run a missing regctl / missing GHCR_READ_TOKEN / unresolvable :$KEY / digest mismatch each exit 1; only a fork degrades to tag-only with a ::warning::. - scripts/tests/*.bats — determinism + sensitivity + failure-closed matrix (§7); wired into quality-checks.yml as a non-blocking job for now. - .github/workflows/toolchain-image.yml — daily schedule + workflow_dispatch + pull_request(paths) + workflow_call; multi-arch $BUILDPLATFORM cross-compile (no QEMU) of --target toolchain-runtime, GHCR publish on same-repo, Trivy CRITICAL/HIGH gate (blocking off-PR, report-only on PR) + SARIF. The sync-pin-on-pr / open-bump-pr jobs land in Commit 3. - Dockerfile: * rename caddy-builder -> caddy-inline, crowdsec-builder -> crowdsec-inline; * digest-pin the golang:${GO_VERSION}-alpine base of both (N4) and feed the digest line to the key; * pin the two loose xcaddy plugins — CADDY_GEOIP2_VERSION (full pseudo-version, caddy-geoip2 has no semver tags) and CADDY_RATELIMIT_VERSION=0.1.0 (B4) — and feed both to the key; * delete the dead crowdsec-fallback stage and the now-unused CROWDSEC_RELEASE_SHA256 ARG (N1); * add the toolchain-runtime assembly stage (binaries at identical paths) with an io.charon.toolchain.key provenance LABEL; * add temporary aliases `FROM caddy-inline AS caddy-builder` / `FROM crowdsec-inline AS crowdsec-builder` so the app build is byte-identical this commit (selector + prebuilt-image consumption land in Commit 2). - Retarget every --no-cache-filter / no-cache-filters value from caddy-builder,crowdsec-builder to caddy-inline,crowdsec-inline across docker-build.yml, security-pr.yml, supply-chain-pr.yml, e2e-tests-split.yml, nightly-build.yml, and the build-charon-image composite doc comment — so the CVE-recurrence guard keeps invalidating the real RUN layers through the rename (B5). Full removal is Commit 4, after the freshness guard is live (Commit 3). Plugin/base versions are best-effort here; the exact baked CADDY_GEOIP2_VERSION is confirmed from the first toolchain-image CI run's xcaddy log and re-pinned. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .github/actions/build-charon-image/action.yml | 32 ++- .github/workflows/docker-build.yml | 8 +- .github/workflows/e2e-tests-split.yml | 2 +- .github/workflows/nightly-build.yml | 2 +- .github/workflows/quality-checks.yml | 25 ++ .github/workflows/security-pr.yml | 2 +- .github/workflows/supply-chain-pr.yml | 2 +- .github/workflows/toolchain-image.yml | 249 ++++++++++++++++++ Dockerfile | 115 ++++---- scripts/lib/dockerfile-stage.sh | 74 ++++++ scripts/tests/helpers/toolchain_fixture.bash | 157 +++++++++++ scripts/tests/toolchain-key.bats | 98 +++++++ scripts/tests/verify-toolchain-pin.bats | 95 +++++++ scripts/toolchain-key.sh | 81 ++++++ scripts/verify-toolchain-pin.sh | 113 ++++++++ 15 files changed, 987 insertions(+), 68 deletions(-) create mode 100644 .github/workflows/toolchain-image.yml create mode 100755 scripts/lib/dockerfile-stage.sh create mode 100644 scripts/tests/helpers/toolchain_fixture.bash create mode 100644 scripts/tests/toolchain-key.bats create mode 100644 scripts/tests/verify-toolchain-pin.bats create mode 100755 scripts/toolchain-key.sh create mode 100755 scripts/verify-toolchain-pin.sh diff --git a/.github/actions/build-charon-image/action.yml b/.github/actions/build-charon-image/action.yml index 7678c7fd7..c669ddc1f 100644 --- a/.github/actions/build-charon-image/action.yml +++ b/.github/actions/build-charon-image/action.yml @@ -15,21 +15,25 @@ inputs: no-cache-filters: description: >- Comma-separated Dockerfile stages to force-rebuild (never restore from the - layer cache). Empty by default: every stage, including the expensive - caddy-builder xcaddy step, is GHA layer-cached — that is where the build - time is recovered. Suitable for the integration-test callers - (waf/crowdsec/rate-limit/cerberus), which exercise runtime behaviour and - do not care about dependency freshness. + layer cache). Empty by default: every stage is GHA layer-cached — that is + where the build time is recovered. Suitable for the integration-test + callers (waf/crowdsec/rate-limit/cerberus), which exercise runtime + behaviour and do not care about dependency freshness. - CVE-scan-gate callers (security-pr.yml, supply-chain-pr.yml) MUST override - this with `caddy-builder,crowdsec-builder`. The caddy-builder and - crowdsec-builder stages patch pinned transitive dependencies in-place - (`go get pkg@fixed` in their Stage 2 blocks). A global build-arg bump does - not reliably invalidate the GHA layer-cache key for a stage that only - *consumes* that arg (the same edge case that produced CVE-2026-45135 and - the 2026-09-04 grpc-go v1.83.0 recurrence), so a restored stale layer keeps - shipping the superseded, still-vulnerable version. The release image - (docker-build.yml) already force-rebuilds caddy-builder for this reason. + CVE-scan-gate callers (security-pr.yml, supply-chain-pr.yml) currently + override this with `caddy-inline,crowdsec-inline` (the RUN-bearing + from-source stages, renamed from caddy-builder/crowdsec-builder). Those + stages patch pinned transitive dependencies in-place (`go get pkg@fixed`), + and a global build-arg bump does not reliably invalidate the GHA + layer-cache key for a stage that only *consumes* that arg (the same edge + case that produced CVE-2026-45135 and the 2026-09-04 grpc-go v1.83.0 + recurrence). + + NOTE: on the default app-build path those stages are no longer compiled — + their output is COPY --from'd out of the digest-pinned, daily-rebuilt-and- + scanned toolchain image, and pin freshness is enforced per-PR by + scripts/verify-toolchain-pin.sh. This input is removed entirely in a later + commit once that guard is a required check. required: false default: '' runs: diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index c2db0c013..afe3a9ce8 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -460,8 +460,8 @@ jobs: "${TAG_ARGS[@]}" --cache-from type=gha,scope=docker-build-amd64 --cache-to type=gha,mode=max,scope=docker-build-amd64 - --no-cache-filter caddy-builder - --no-cache-filter crowdsec-builder + --no-cache-filter caddy-inline + --no-cache-filter crowdsec-inline --pull --build-arg "VERSION=${{ needs.setup.outputs.version }}" --build-arg "BUILD_DATE=${{ needs.setup.outputs.created }}" @@ -546,8 +546,8 @@ jobs: "${TAG_ARGS[@]}" --cache-from type=gha,scope=docker-build-arm64 --cache-to type=gha,mode=max,scope=docker-build-arm64 - --no-cache-filter caddy-builder - --no-cache-filter crowdsec-builder + --no-cache-filter caddy-inline + --no-cache-filter crowdsec-inline --pull --build-arg "VERSION=${{ needs.setup.outputs.version }}" --build-arg "BUILD_DATE=${{ needs.setup.outputs.created }}" diff --git a/.github/workflows/e2e-tests-split.yml b/.github/workflows/e2e-tests-split.yml index fe0ded30d..1de9995ee 100644 --- a/.github/workflows/e2e-tests-split.yml +++ b/.github/workflows/e2e-tests-split.yml @@ -221,7 +221,7 @@ jobs: tags: ${{ steps.resolve-image.outputs.image_tag }} cache-from: type=gha cache-to: type=gha,mode=max - no-cache-filters: caddy-builder,crowdsec-builder + no-cache-filters: caddy-inline,crowdsec-inline - name: Save Docker image if: steps.resolve-image.outputs.image_source == 'build' diff --git a/.github/workflows/nightly-build.yml b/.github/workflows/nightly-build.yml index a835c23c6..2e7a60066 100644 --- a/.github/workflows/nightly-build.yml +++ b/.github/workflows/nightly-build.yml @@ -240,7 +240,7 @@ jobs: ALPINE_IMAGE=${{ steps.alpine.outputs.image }} cache-from: type=gha cache-to: type=gha,mode=max - no-cache-filters: caddy-builder,crowdsec-builder + no-cache-filters: caddy-inline,crowdsec-inline provenance: true sbom: true diff --git a/.github/workflows/quality-checks.yml b/.github/workflows/quality-checks.yml index f6a459c2e..6ffda7f22 100644 --- a/.github/workflows/quality-checks.yml +++ b/.github/workflows/quality-checks.yml @@ -73,6 +73,31 @@ jobs: run: | bash scripts/ci/check-codecov-trigger-parity.sh + toolchain-key-tests: + name: Toolchain key / freshness-guard scripts (bats) + runs-on: ubuntu-latest + # Non-blocking for now (spec §12 Commit 1). The blocking `verify-toolchain-pin` + # required check is added in Commit 3. + continue-on-error: true + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Install bats + shellcheck + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y bats shellcheck + + - name: shellcheck (shell severity=error) + run: | + shellcheck --severity=error \ + scripts/toolchain-key.sh \ + scripts/verify-toolchain-pin.sh \ + scripts/lib/dockerfile-stage.sh + + - name: bats + run: bats scripts/tests/toolchain-key.bats scripts/tests/verify-toolchain-pin.bats + backend-quality: name: Backend (Go) runs-on: ubuntu-latest diff --git a/.github/workflows/security-pr.yml b/.github/workflows/security-pr.yml index fd457a96b..3ea30d38b 100644 --- a/.github/workflows/security-pr.yml +++ b/.github/workflows/security-pr.yml @@ -161,7 +161,7 @@ jobs: # `go get pkg@fixed` patch lives INSIDE the cached-and-skipped stage). # Matches nightly-build.yml and e2e-tests-split.yml. CVE-scan gate: # correctness beats the few minutes of rebuild time. - no-cache-filters: caddy-builder,crowdsec-builder + no-cache-filters: caddy-inline,crowdsec-inline - name: Check for PR image artifact id: check-artifact diff --git a/.github/workflows/supply-chain-pr.yml b/.github/workflows/supply-chain-pr.yml index 5ee3bfece..5de24c5ab 100644 --- a/.github/workflows/supply-chain-pr.yml +++ b/.github/workflows/supply-chain-pr.yml @@ -258,7 +258,7 @@ jobs: # INSIDE the cached-and-skipped stage). Matches nightly-build.yml and # e2e-tests-split.yml. This is a CVE-scan gate; correctness beats the # few minutes of xcaddy/crowdsec rebuild time. - no-cache-filters: caddy-builder,crowdsec-builder + no-cache-filters: caddy-inline,crowdsec-inline - name: Expose local image name if: github.event_name != 'workflow_run' diff --git a/.github/workflows/toolchain-image.yml b/.github/workflows/toolchain-image.yml new file mode 100644 index 000000000..4d0e8f6aa --- /dev/null +++ b/.github/workflows/toolchain-image.yml @@ -0,0 +1,249 @@ +# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json +--- +name: Toolchain Image — Build & Publish + +# Builds and publishes ghcr.io/wikid82/charon-toolchain — a digest-pinned, +# multi-arch prebuilt image carrying the custom Caddy + CrowdSec binaries that +# the app Dockerfile's caddy-inline / crowdsec-inline stages produce. The app +# build COPY --from's this image instead of recompiling xcaddy / CrowdSec on +# every run (spec docs/plans/current_spec.md §3.1). +# +# Entry points (spec §3.4.1): +# schedule (daily 06:00 UTC) -> --no-cache --pull freshness rebuild +# workflow_dispatch -> manual rebuild (default --no-cache --pull) +# pull_request (paths) -> validate the recipe still compiles; same-repo +# PRs also publish :; forks build cacheonly +# workflow_call -> heavier scan/report pass from +# security-weekly-rebuild.yml +# +# The sync-pin-on-pr / open-bump-pr jobs (which move the Dockerfile digest pin) +# are added in a later commit, once the freshness guard is a required check. + +on: + schedule: + - cron: '0 6 * * *' + workflow_dispatch: + inputs: + force_rebuild: + description: 'Build with --no-cache --pull' + type: boolean + default: true + pull_request: + paths: + - 'Dockerfile' + - '.github/workflows/toolchain-image.yml' + - 'scripts/toolchain-key.sh' + - 'scripts/verify-toolchain-pin.sh' + - 'scripts/lib/dockerfile-stage.sh' + - '.trivyignore' + workflow_call: + inputs: + force_rebuild: + type: boolean + default: true + publish: + type: boolean + default: true + +concurrency: + group: toolchain-image-${{ github.ref }} + cancel-in-progress: false + +permissions: + contents: read + packages: write + security-events: write + pull-requests: write + +env: + TOOLCHAIN_IMAGE: ghcr.io/wikid82/charon-toolchain + +jobs: + build-toolchain: + name: Build & publish toolchain image + runs-on: ubuntu-latest + timeout-minutes: 45 + outputs: + key: ${{ steps.key.outputs.key }} + digest: ${{ steps.digest.outputs.digest }} + same_repo: ${{ steps.trust.outputs.same_repo }} + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Classify trust (same-repo vs fork) + id: trust + env: + EVENT_NAME: ${{ github.event_name }} + # Each job that needs trust classification maps head.repo.full_name + # into env explicitly (supervisor caution #1 / B7). + PR_HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }} + REPO: ${{ github.repository }} + run: | + set -euo pipefail + same_repo=true + if [[ "$EVENT_NAME" == "pull_request" && "$PR_HEAD_REPO" != "$REPO" ]]; then + same_repo=false + fi + echo "same_repo=$same_repo" >> "$GITHUB_OUTPUT" + echo "Trust: same_repo=$same_repo (event=$EVENT_NAME head=${PR_HEAD_REPO:-n/a})" + + - name: Compute toolchain key + id: key + run: | + set -euo pipefail + KEY="$(bash scripts/toolchain-key.sh)" + echo "key=$KEY" >> "$GITHUB_OUTPUT" + echo "Toolchain key: $KEY" + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0 + + - name: Log in to GitHub Container Registry + if: steps.trust.outputs.same_repo == 'true' + uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Decide cache flags + id: flags + env: + EVENT_NAME: ${{ github.event_name }} + FORCE_REBUILD: ${{ inputs.force_rebuild }} + run: | + set -euo pipefail + no_cache="" + if [[ "$EVENT_NAME" == "schedule" || "$FORCE_REBUILD" == "true" ]]; then + no_cache="--no-cache --pull" + fi + echo "no_cache=$no_cache" >> "$GITHUB_OUTPUT" + echo "cache flags: '${no_cache:-}'" + + - name: Build toolchain image (multi-arch $BUILDPLATFORM cross-compile, no QEMU) + id: build + env: + SAME_REPO: ${{ steps.trust.outputs.same_repo }} + KEY: ${{ steps.key.outputs.key }} + NO_CACHE: ${{ steps.flags.outputs.no_cache }} + run: | + set -euo pipefail + DATE_TAG="$(date -u +%Y%m%d)" + TAGS=(-t "${TOOLCHAIN_IMAGE}:${KEY}" -t "${TOOLCHAIN_IMAGE}:${DATE_TAG}") + OUT="--output=type=cacheonly" + if [[ "$SAME_REPO" == "true" ]]; then + TAGS+=(-t "${TOOLCHAIN_IMAGE}:latest") + OUT="--push" + fi + + # shellcheck disable=SC2086 # NO_CACHE is an intentional word-split flag list + docker buildx build \ + --target toolchain-runtime \ + --platform linux/amd64,linux/arm64 \ + $NO_CACHE \ + --cache-from type=gha,scope=toolchain \ + --cache-to type=gha,mode=max,scope=toolchain \ + --build-arg "CHARON_TOOLCHAIN_TAG=${KEY}" \ + "${TAGS[@]}" \ + $OUT \ + . + + - name: Resolve published manifest-list digest + id: digest + if: steps.trust.outputs.same_repo == 'true' + env: + KEY: ${{ steps.key.outputs.key }} + run: | + set -euo pipefail + DIGEST="$(docker buildx imagetools inspect "${TOOLCHAIN_IMAGE}:${KEY}" \ + --format '{{json .Manifest}}' | jq -r '.digest')" + if [[ -z "$DIGEST" || "$DIGEST" == "null" ]]; then + echo "::error::Could not resolve manifest-list digest for ${TOOLCHAIN_IMAGE}:${KEY}" + exit 1 + fi + echo "digest=$DIGEST" >> "$GITHUB_OUTPUT" + echo "Published ${TOOLCHAIN_IMAGE}:${KEY} @ ${DIGEST}" + + - name: Assert multi-arch manifest (linux/amd64 + linux/arm64) + if: steps.trust.outputs.same_repo == 'true' + env: + KEY: ${{ steps.key.outputs.key }} + run: | + set -euo pipefail + PLATFORMS="$(docker buildx imagetools inspect "${TOOLCHAIN_IMAGE}:${KEY}" --raw \ + | jq -r '.manifests[]?.platform | select(.) | "\(.os)/\(.architecture)"' | sort -u)" + echo "Published platforms:" + echo "$PLATFORMS" + echo "$PLATFORMS" | grep -qx 'linux/amd64' || { echo "::error::missing linux/amd64 child"; exit 1; } + echo "$PLATFORMS" | grep -qx 'linux/arm64' || { echo "::error::missing linux/arm64 child"; exit 1; } + + - name: Summary + if: always() + run: | + { + echo "## Toolchain image" + echo "- key: \`${{ steps.key.outputs.key }}\`" + echo "- digest: \`${{ steps.digest.outputs.digest || 'not published (fork PR / cacheonly build)' }}\`" + echo "- trust: same_repo=${{ steps.trust.outputs.same_repo }}" + } >> "$GITHUB_STEP_SUMMARY" + + trivy-scan: + name: Trivy scan (toolchain image) + needs: build-toolchain + runs-on: ubuntu-latest + timeout-minutes: 20 + permissions: + contents: read + security-events: write + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Determine scan reference + id: ref + env: + SAME_REPO: ${{ needs.build-toolchain.outputs.same_repo }} + DIGEST: ${{ needs.build-toolchain.outputs.digest }} + run: | + set -euo pipefail + if [[ "$SAME_REPO" == "true" && -n "$DIGEST" ]]; then + echo "scan=true" >> "$GITHUB_OUTPUT" + echo "image_ref=${TOOLCHAIN_IMAGE}@${DIGEST}" >> "$GITHUB_OUTPUT" + echo "Scanning ${TOOLCHAIN_IMAGE}@${DIGEST}" + else + echo "scan=false" >> "$GITHUB_OUTPUT" + echo "Fork PR / image not published — no remote image to scan." + fi + + - name: Trivy vulnerability scan (CRITICAL/HIGH gate) + if: steps.ref.outputs.scan == 'true' + uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 + with: + image-ref: ${{ steps.ref.outputs.image_ref }} + format: 'table' + severity: 'CRITICAL,HIGH' + exit-code: '1' + trivyignores: '.trivyignore' + version: 'v0.74.0' + # Report-only on PRs (the app-image Trivy gates still run downstream); + # blocking on schedule / workflow_dispatch / workflow_call (spec §3.4.3). + continue-on-error: ${{ github.event_name == 'pull_request' }} + + - name: Trivy vulnerability scan (SARIF) + if: steps.ref.outputs.scan == 'true' + uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 + with: + image-ref: ${{ steps.ref.outputs.image_ref }} + format: 'sarif' + output: 'trivy-toolchain.sarif' + severity: 'CRITICAL,HIGH,MEDIUM' + trivyignores: '.trivyignore' + version: 'v0.74.0' + + - name: Upload Trivy SARIF + if: steps.ref.outputs.scan == 'true' + uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9 + with: + sarif_file: 'trivy-toolchain.sarif' + category: '.github/workflows/toolchain-image.yml:trivy-toolchain' diff --git a/Dockerfile b/Dockerfile index add68da4f..b6f961170 100644 --- a/Dockerfile +++ b/Dockerfile @@ -8,6 +8,19 @@ ARG VCS_REF # Set BUILD_DEBUG=1 to build with debug symbols (required for Delve debugging) ARG BUILD_DEBUG=0 +# ---- Prebuilt Caddy + CrowdSec toolchain image ---- +# Built by .github/workflows/toolchain-image.yml from the caddy-inline / +# crowdsec-inline stages below (--target toolchain-runtime). Bumped by that +# workflow's bot PR when a security-relevant input moves OR the DAILY +# `--no-cache --pull` rebuild produces a new digest. The freshness-guard CI +# check (scripts/verify-toolchain-pin.sh) fails any PR where TAG/DIGEST is +# stale for the current pins. +ARG CHARON_TOOLCHAIN_IMAGE=ghcr.io/wikid82/charon-toolchain +# NOT Renovate-tracked (a content-hash tag has no series to follow, N7) — the +# toolchain-image.yml bot owns these two lines. +ARG CHARON_TOOLCHAIN_TAG=caddy-crowdsec-1efe7f19fa52a512 +ARG CHARON_TOOLCHAIN_DIGEST=sha256:0000000000000000000000000000000000000000000000000000000000000000 + # ---- Pinned Toolchain Versions ---- # renovate: datasource=docker depName=golang versioning=docker ARG GO_VERSION=1.27.1 @@ -18,8 +31,6 @@ ARG ALPINE_IMAGE=alpine:3.24.1@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db # ---- Shared CrowdSec Version ---- # renovate: datasource=github-releases depName=crowdsecurity/crowdsec ARG CROWDSEC_VERSION=1.8.1 -# CrowdSec fallback tarball checksum (v${CROWDSEC_VERSION}) -ARG CROWDSEC_RELEASE_SHA256=deae1f43ddf1118339dc4f774289d745c957802423d0310ad1d2990067d05ea8 # ---- Shared Go Security Patches ---- # renovate: datasource=github-tags depName=expr-lang/expr extractVersion=^v(?.+)$ @@ -62,6 +73,15 @@ ARG CADDY_PATCH_SCENARIO=B ARG CADDY_SECURITY_VERSION=1.1.64 # renovate: datasource=go depName=github.com/corazawaf/coraza-caddy/v2 ARG CORAZA_CADDY_VERSION=2.6.0 +# xcaddy plugins that previously resolved "latest" at build time (B4). Pinned so +# a toolchain-key.sh input moves when the plugin does. caddy-geoip2 publishes NO +# semver tags, so its pin is the full pseudo-version (leading v included) and the +# `--with` line interpolates it directly (no added `v`); the renovate marker is +# kept for discoverability but does not track a pseudo-version (same caveat as N7). +# renovate: datasource=go depName=github.com/zhangjiayin/caddy-geoip2 +ARG CADDY_GEOIP2_VERSION=v0.0.0-20260623062220-3675c6e7e63d +# renovate: datasource=go depName=github.com/mholt/caddy-ratelimit +ARG CADDY_RATELIMIT_VERSION=0.1.0 ## When an official caddy image tag isn't available on the host, use a ## plain Alpine base image and overwrite its caddy binary with our ## xcaddy-built binary in the later COPY step. This avoids relying on @@ -296,10 +316,22 @@ RUN --mount=type=cache,target=/root/.cache/go-build \ -o charon ./cmd/api; \ fi -# ---- Caddy Builder ---- +# ---- Caddy Builder (inline / from-source) ---- # Build Caddy from source to ensure we use the latest Go version and dependencies # This fixes vulnerabilities found in the pre-built Caddy images (e.g. CVE-2025-59530, stdlib issues) -FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine AS caddy-builder +# +# This stage is the single source of truth for the Caddy build recipe. On the +# default app-build path its output is NOT recompiled — it is COPY --from'd out +# of the digest-pinned toolchain image (see the toolchain-prebuilt / caddy-builder +# selector stages further down). It is compiled here only by +# .github/workflows/toolchain-image.yml (--target toolchain-runtime) and on the +# fork / bootstrap / offline fallback path. +# +# N4: the golang:${GO_VERSION}-alpine tag is a moving reference; digest-pin it so +# a silent upstream base rebuild is caught by toolchain-key.sh. The pinned digest +# is refreshed by the daily toolchain rebuild's `--pull` + Renovate. +# renovate: datasource=docker depName=golang +FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine@sha256:cf6fca6641884b8433441b2b0652976f975e1d0fdd26d177eaaf8596087f3125 AS caddy-inline ARG TARGETOS ARG TARGETARCH ARG CADDY_VERSION @@ -308,6 +340,8 @@ ARG CADDY_USE_CANDIDATE ARG CADDY_PATCH_SCENARIO ARG CADDY_SECURITY_VERSION ARG CORAZA_CADDY_VERSION +ARG CADDY_GEOIP2_VERSION +ARG CADDY_RATELIMIT_VERSION # renovate: datasource=go depName=github.com/caddyserver/xcaddy ARG XCADDY_VERSION=0.4.7 ARG EXPR_LANG_VERSION @@ -388,8 +422,8 @@ RUN --mount=type=cache,target=/root/.cache/go-build \ --with github.com/greenpau/caddy-security@v${CADDY_SECURITY_VERSION} \ --with github.com/corazawaf/coraza-caddy/v2@v${CORAZA_CADDY_VERSION} \ --with github.com/hslatman/caddy-crowdsec-bouncer@v0.12.1 \ - --with github.com/zhangjiayin/caddy-geoip2 \ - --with github.com/mholt/caddy-ratelimit \ + --with github.com/zhangjiayin/caddy-geoip2@${CADDY_GEOIP2_VERSION} \ + --with github.com/mholt/caddy-ratelimit@v${CADDY_RATELIMIT_VERSION} \ --output /tmp/caddy-initial; \ # Find the build directory created by xcaddy BUILDDIR=$(ls -td /tmp/buildenv_* 2>/dev/null | head -1); \ @@ -571,10 +605,15 @@ RUN --mount=type=cache,target=/root/.cache/go-build \ # Clean up temporary build directories rm -rf /tmp/buildenv_* /tmp/caddy-initial' -# ---- CrowdSec Builder ---- +# ---- CrowdSec Builder (inline / from-source) ---- # Build CrowdSec from source to ensure we use Go 1.26.3+ and avoid stdlib vulnerabilities # (CVE-2025-58183, CVE-2025-58186, CVE-2025-58187, CVE-2025-61729) -FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine AS crowdsec-builder +# +# Like caddy-inline, this is the single source of truth for the CrowdSec build +# recipe. Compiled by toolchain-image.yml and the fork/offline fallback only; the +# default app build COPY --from's its output out of the pinned toolchain image. +# renovate: datasource=docker depName=golang +FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine@sha256:cf6fca6641884b8433441b2b0652976f975e1d0fdd26d177eaaf8596087f3125 AS crowdsec-inline COPY --from=xx / / WORKDIR /tmp/crowdsec @@ -583,7 +622,6 @@ ARG TARGETPLATFORM ARG TARGETOS ARG TARGETARCH ARG CROWDSEC_VERSION -ARG CROWDSEC_RELEASE_SHA256 ARG EXPR_LANG_VERSION ARG XNET_VERSION ARG XCRYPTO_VERSION @@ -709,43 +747,28 @@ RUN --mount=type=cache,target=/root/.cache/go-build \ RUN mkdir -p /crowdsec-out/config && \ cp -r config/* /crowdsec-out/config/ || true -# ---- CrowdSec Fallback (for architectures where build fails) ---- -FROM ${ALPINE_IMAGE} AS crowdsec-fallback - -SHELL ["/bin/ash", "-o", "pipefail", "-c"] - -WORKDIR /tmp/crowdsec - -ARG TARGETARCH -ARG CROWDSEC_VERSION -ARG CROWDSEC_RELEASE_SHA256 - -# hadolint ignore=DL3018 -RUN apk add --no-cache curl ca-certificates - -# Download static binaries as fallback (only available for amd64) -# For other architectures, create empty placeholder files so COPY doesn't fail -# hadolint ignore=DL3059,SC2015 -RUN set -eux; \ - mkdir -p /crowdsec-out/bin /crowdsec-out/config; \ - if [ "$TARGETARCH" = "amd64" ]; then \ - echo "Downloading CrowdSec binaries for amd64 (fallback)..."; \ - curl -fSL --retry 3 --retry-delay 5 --retry-all-errors \ - "https://github.com/crowdsecurity/crowdsec/releases/download/v${CROWDSEC_VERSION}/crowdsec-release.tgz" \ - -o /tmp/crowdsec.tar.gz && \ - echo "${CROWDSEC_RELEASE_SHA256} /tmp/crowdsec.tar.gz" | sha256sum -c - && \ - tar -xzf /tmp/crowdsec.tar.gz -C /tmp && \ - cp "/tmp/crowdsec-v${CROWDSEC_VERSION}/cmd/crowdsec-cli/cscli" /crowdsec-out/bin/ && \ - cp "/tmp/crowdsec-v${CROWDSEC_VERSION}/cmd/crowdsec/crowdsec" /crowdsec-out/bin/ && \ - chmod +x /crowdsec-out/bin/* && \ - if [ -d "/tmp/crowdsec-v${CROWDSEC_VERSION}/config" ]; then \ - cp -r "/tmp/crowdsec-v${CROWDSEC_VERSION}/config/"* /crowdsec-out/config/; \ - fi && \ - echo "CrowdSec fallback binaries installed successfully"; \ - else \ - echo "CrowdSec binaries not available for $TARGETARCH - skipping"; \ - touch /crowdsec-out/bin/.placeholder /crowdsec-out/config/.placeholder; \ - fi +# ---- Toolchain image assembly target (built by toolchain-image.yml) ---- +# NOT part of the app build graph — nothing FROMs it here. `docker buildx build +# --target toolchain-runtime` produces the publishable multi-arch image that the +# default app build then COPY --from's. The binaries land at the SAME paths the +# inline stages produce, so the final-stage COPY --from lines need no change. +FROM ${ALPINE_IMAGE} AS toolchain-runtime +ARG CHARON_TOOLCHAIN_TAG +COPY --from=caddy-inline /usr/bin/caddy /usr/bin/caddy +COPY --from=crowdsec-inline /crowdsec-out/crowdsec /crowdsec-out/crowdsec +COPY --from=crowdsec-inline /crowdsec-out/cscli /crowdsec-out/cscli +COPY --from=crowdsec-inline /crowdsec-out/config /crowdsec-out/config +# Provenance: `docker inspect` on the toolchain image shows the content key. +LABEL io.charon.toolchain.key="${CHARON_TOOLCHAIN_TAG}" + +# ---- Effective builder stages ---- +# Commit 1: temporary aliases so the app build is byte-identical while the +# selector / prebuilt-image consumption lands in Commit 2. The retargeted +# `--no-cache-filter caddy-inline,crowdsec-inline` in CI keeps invalidating the +# real RUN layers (which now live in caddy-inline / crowdsec-inline) through the +# rename — the CVE-recurrence guard is never inert (B5). +FROM caddy-inline AS caddy-builder +FROM crowdsec-inline AS crowdsec-builder # ---- Final Runtime with Caddy ---- FROM ${ALPINE_IMAGE} diff --git a/scripts/lib/dockerfile-stage.sh b/scripts/lib/dockerfile-stage.sh new file mode 100755 index 000000000..c40b76733 --- /dev/null +++ b/scripts/lib/dockerfile-stage.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# scripts/lib/dockerfile-stage.sh +# +# SHARED helper (spec §3.4.2 / N9). Sourced by BOTH scripts/toolchain-key.sh and +# scripts/verify-toolchain-pin.sh so the "extract one Dockerfile stage" logic +# exists in exactly one place. +# +# extract_stage +# Prints the body of the named build stage: its `FROM ... AS ` header +# through its last real instruction line. Comment / blank lines that sit +# between the stage's last instruction and the next `FROM` (i.e. the *next* +# stage's preamble) are NOT included, so re-wording a stage's header comment +# does not perturb the *previous* stage's extracted text — and therefore does +# not perturb the toolchain key for unrelated edits. Comments that appear +# mid-stage (followed by more instructions before the next `FROM`) ARE kept. +# Exits 3 if the stage is not found. +# +# This file is meant to be `source`d, not executed. + +# shellcheck shell=bash + +extract_stage() { + local stage="$1" file="$2" + + if [[ -z "$stage" || -z "$file" ]]; then + echo "extract_stage: usage: extract_stage " >&2 + return 2 + fi + if [[ ! -f "$file" ]]; then + echo "extract_stage: no such file: $file" >&2 + return 2 + fi + + awk -v s="$stage" ' + # Stage name declared by a FROM line ("" if it has no `AS`). Handles + # `FROM --platform=$BUILDPLATFORM img:tag@sha256:... AS name`. + function stagename(line, n) { + if (match(line, /[ \t][Aa][Ss][ \t]+[A-Za-z0-9._-]+[ \t]*$/)) { + n = substr(line, RSTART) + sub(/^[ \t]+[Aa][Ss][ \t]+/, "", n) + sub(/[ \t]+$/, "", n) + return n + } + return "" + } + function flush_pending( i) { + for (i = 1; i <= np; i++) print pending[i] + np = 0 + } + /^FROM[ \t]/ { + nm = stagename($0) + if (capturing && nm != s) { exit } # next stage reached; drop pending buffer + if (nm == s) { found = 1; capturing = 1; np = 0; print $0; next } + } + capturing { + # Hold blank / comment-only lines: they might be the next stage preamble. + if ($0 ~ /^[ \t]*$/ || $0 ~ /^[ \t]*#/) { + pending[++np] = $0 + } else { + flush_pending() + print $0 + } + next + } + END { + if (!found) { + print "extract_stage: no stage \"" s "\"" > "/dev/stderr" + exit 3 + } + # Trailing pending lines (comments/blanks before the next FROM or EOF) + # are intentionally dropped. + } + ' "$file" +} diff --git a/scripts/tests/helpers/toolchain_fixture.bash b/scripts/tests/helpers/toolchain_fixture.bash new file mode 100644 index 000000000..d13d3a4b9 --- /dev/null +++ b/scripts/tests/helpers/toolchain_fixture.bash @@ -0,0 +1,157 @@ +#!/usr/bin/env bash +# Shared bats helper: builds an isolated fake repo containing the real +# scripts/lib/dockerfile-stage.sh + scripts/toolchain-key.sh + +# scripts/verify-toolchain-pin.sh alongside a synthetic Dockerfile / .trivyignore +# fixture that is structurally valid for toolchain-key.sh's sanity checks +# (>= 20 body lines per inline stage, each containing a `go build` / `xx-go build`). +# +# Usage from a .bats file: +# load helpers/toolchain_fixture +# setup() { tf_setup; } +# teardown() { tf_teardown; } +# +# Exposes: $TF_ROOT (fake repo root), $TF_DF ($TF_ROOT/Dockerfile), +# $TF_BIN (a PATH-prepended dir for command stubs). + +# shellcheck shell=bash + +tf_setup() { + TF_REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../.." && pwd)" + TF_ROOT="$(mktemp -d)" + TF_BIN="$TF_ROOT/.bin" + mkdir -p "$TF_ROOT/scripts/lib" "$TF_BIN" + + cp "$TF_REPO_ROOT/scripts/lib/dockerfile-stage.sh" "$TF_ROOT/scripts/lib/" + cp "$TF_REPO_ROOT/scripts/toolchain-key.sh" "$TF_ROOT/scripts/" + cp "$TF_REPO_ROOT/scripts/verify-toolchain-pin.sh" "$TF_ROOT/scripts/" + chmod +x "$TF_ROOT/scripts/"*.sh + + TF_DF="$TF_ROOT/Dockerfile" + tf_write_dockerfile + printf '.cache/\nsome-cve-id\n' > "$TF_ROOT/.trivyignore" + + export PATH="$TF_BIN:$PATH" +} + +tf_teardown() { + [[ -n "${TF_ROOT:-}" && -d "$TF_ROOT" ]] && rm -rf "$TF_ROOT" +} + +# Write the fixture Dockerfile. Any already-exported TF_* knobs below let +# individual tests perturb one input at a time. +tf_write_dockerfile() { + local caddy_version="${TF_CADDY_VERSION:-2.11.4}" + local geoip2_version="${TF_GEOIP2_VERSION:-v0.0.0-20260623062220-3675c6e7e63d}" + local caddy_get_line="${TF_CADDY_GET_LINE:- _retry go get golang.org/x/net@v0.58.0; \\}" + local golang_digest="${TF_GOLANG_DIGEST:-sha256:cf6fca6641884b8433441b2b0652976f975e1d0fdd26d177eaaf8596087f3125}" + + cat > "$TF_DF" < "$TF_BIN/regctl" < tag" { + run key_of + [ "$status" -eq 0 ] + [[ "$output" =~ ^caddy-crowdsec-[0-9a-f]{16}$ ]] +} + +@test "deterministic: two runs on the same Dockerfile agree" { + a="$(key_of)" + b="$(key_of)" + [ "$a" = "$b" ] +} + +@test "stable across a whitespace reformat OUTSIDE the inline stages" { + before="$(key_of)" + # Add blank lines / trailing space to the global ARG block and the final stage, + # none of which is part of caddy-inline / crowdsec-inline. + printf '\n\n# a new trailing comment\n' >> "$TF_DF" + sed -i '1a # extra header comment' "$TF_DF" + after="$(key_of)" + [ "$before" = "$after" ] +} + +@test "changes when a go get line INSIDE caddy-inline changes" { + before="$(key_of)" + export TF_CADDY_GET_LINE=' _retry go get golang.org/x/net@v9.9.9; \' + tf_write_dockerfile + after="$(key_of)" + [ "$before" != "$after" ] +} + +@test "changes when the CADDY_VERSION default is bumped" { + before="$(key_of)" + export TF_CADDY_VERSION=2.11.5 + tf_write_dockerfile + after="$(key_of)" + [ "$before" != "$after" ] +} + +@test "changes when the CADDY_GEOIP2_VERSION plugin pin is bumped (B4)" { + before="$(key_of)" + export TF_GEOIP2_VERSION='v0.0.0-20270101000000-abcdefabcdef' + tf_write_dockerfile + after="$(key_of)" + [ "$before" != "$after" ] +} + +@test "changes when the digest-pinned golang base moves (N4)" { + before="$(key_of)" + export TF_GOLANG_DIGEST='sha256:1111111111111111111111111111111111111111111111111111111111111111' + tf_write_dockerfile + after="$(key_of)" + [ "$before" != "$after" ] +} + +@test "changes when .trivyignore changes" { + before="$(key_of)" + printf 'another-cve\n' >> "$TF_ROOT/.trivyignore" + after="$(key_of)" + [ "$before" != "$after" ] +} + +@test "fails loudly when a required inline stage is missing" { + sed -i 's/ AS caddy-inline/ AS caddy-scratch/' "$TF_DF" + run key_of + [ "$status" -ne 0 ] + [[ "$output" == *"caddy-inline"* ]] +} + +@test "fails loudly when stage extraction is truncated to a stub" { + # Blank the caddy-inline body down to just the FROM line. + awk ' + /AS caddy-inline$/ { print; skip=1; next } + skip && /^FROM / { skip=0 } + skip { next } + { print } + ' "$TF_DF" > "$TF_DF.new" && mv "$TF_DF.new" "$TF_DF" + run key_of + [ "$status" -ne 0 ] + [[ "$output" == *"extraction looks wrong"* || "$output" == *"caddy-inline"* ]] +} diff --git a/scripts/tests/verify-toolchain-pin.bats b/scripts/tests/verify-toolchain-pin.bats new file mode 100644 index 000000000..8cee19b85 --- /dev/null +++ b/scripts/tests/verify-toolchain-pin.bats @@ -0,0 +1,95 @@ +#!/usr/bin/env bats +# +# scripts/verify-toolchain-pin.sh — failure-closed freshness guard (spec §7, B7). +# +# matching pin (fork) -> exit 0 (::warning::) +# mismatched tag -> exit 1 (actionable) +# same-repo + regctl absent -> exit 1 +# same-repo + GHCR_READ_TOKEN unset -> exit 1 +# same-repo + GHCR digest != pinned digest -> exit 1 +# same-repo + GHCR digest == pinned digest -> exit 0 + +load helpers/toolchain_fixture + +setup() { + tf_setup + # Pin the Dockerfile tag to the real recomputed key so tag-equality passes + # unless a test deliberately breaks it. + KEY="$(bash "$TF_ROOT/scripts/toolchain-key.sh" "$TF_DF")" + GOOD_DIGEST="sha256:abc123abc123abc123abc123abc123abc123abc123abc123abc123abc123abcd0" + sed -i "s|^ARG CHARON_TOOLCHAIN_TAG=.*|ARG CHARON_TOOLCHAIN_TAG=${KEY}|" "$TF_DF" + sed -i "s|^ARG CHARON_TOOLCHAIN_DIGEST=.*|ARG CHARON_TOOLCHAIN_DIGEST=${GOOD_DIGEST}|" "$TF_DF" +} +teardown() { tf_teardown; } + +verify() { bash "$TF_ROOT/scripts/verify-toolchain-pin.sh" "$TF_DF"; } + +@test "fork PR with a matching tag: exit 0 + warning, no digest check" { + export GITHUB_EVENT_NAME=pull_request + export GITHUB_REPOSITORY=wikid82/Charon + export GITHUB_EVENT_PULL_REQUEST_HEAD_REPO_FULL_NAME=contributor/Charon + run verify + [ "$status" -eq 0 ] + [[ "$output" == *"::warning::Fork PR"* ]] +} + +@test "mismatched tag: exit 1 with actionable message (any trust level)" { + sed -i "s|^ARG CHARON_TOOLCHAIN_TAG=.*|ARG CHARON_TOOLCHAIN_TAG=caddy-crowdsec-deadbeefdeadbeef|" "$TF_DF" + export GITHUB_EVENT_NAME=pull_request + export GITHUB_REPOSITORY=wikid82/Charon + export GITHUB_EVENT_PULL_REQUEST_HEAD_REPO_FULL_NAME=contributor/Charon + run verify + [ "$status" -eq 1 ] + [[ "$output" == *"recipe/pins changed"* ]] + [[ "$output" == *"Toolchain Image"* ]] +} + +@test "same-repo push + regctl absent: exit 1 (failure-closed)" { + export GITHUB_EVENT_NAME=push + export GITHUB_REPOSITORY=wikid82/Charon + export GHCR_READ_TOKEN=tok + run env PATH="$(tf_min_path)" bash "$TF_ROOT/scripts/verify-toolchain-pin.sh" "$TF_DF" + [ "$status" -eq 1 ] + [[ "$output" == *"regctl missing"* ]] +} + +@test "same-repo push + GHCR_READ_TOKEN unset: exit 1 (failure-closed)" { + tf_stub_regctl "$GOOD_DIGEST" + export GITHUB_EVENT_NAME=push + export GITHUB_REPOSITORY=wikid82/Charon + unset GHCR_READ_TOKEN + run verify + [ "$status" -eq 1 ] + [[ "$output" == *"GHCR_READ_TOKEN unset"* ]] +} + +@test "same-repo PR + GHCR digest disagrees with the pinned digest: exit 1" { + tf_stub_regctl "sha256:0000000000000000000000000000000000000000000000000000000000000bad" + export GITHUB_EVENT_NAME=pull_request + export GITHUB_REPOSITORY=wikid82/Charon + export GITHUB_EVENT_PULL_REQUEST_HEAD_REPO_FULL_NAME=wikid82/Charon + export GHCR_READ_TOKEN=tok + run verify + [ "$status" -eq 1 ] + [[ "$output" == *"hand-edited or stale"* ]] +} + +@test "same-repo PR + GHCR digest matches the pinned digest: exit 0" { + tf_stub_regctl "$GOOD_DIGEST" + export GITHUB_EVENT_NAME=pull_request + export GITHUB_REPOSITORY=wikid82/Charon + export GITHUB_EVENT_PULL_REQUEST_HEAD_REPO_FULL_NAME=wikid82/Charon + export GHCR_READ_TOKEN=tok + run verify + [ "$status" -eq 0 ] + [[ "$output" == *"verified (same-repo)"* ]] +} + +@test "workflow_dispatch is treated as trusted same-repo (failure-closed)" { + export GITHUB_EVENT_NAME=workflow_dispatch + export GITHUB_REPOSITORY=wikid82/Charon + export GHCR_READ_TOKEN=tok + run env PATH="$(tf_min_path)" bash "$TF_ROOT/scripts/verify-toolchain-pin.sh" "$TF_DF" + [ "$status" -eq 1 ] + [[ "$output" == *"regctl missing"* ]] +} diff --git a/scripts/toolchain-key.sh b/scripts/toolchain-key.sh new file mode 100755 index 000000000..e91f3aa00 --- /dev/null +++ b/scripts/toolchain-key.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# scripts/toolchain-key.sh +# +# Prints the deterministic, content-addressed tag for the prebuilt Caddy + +# CrowdSec toolchain image (spec §3.4.2). +# +# Output: caddy-crowdsec-<16 hex> +# +# The tag is a SHA-256 over every security-relevant toolchain input: +# 1. the exact text of the `caddy-inline` Dockerfile stage +# 2. the exact text of the `crowdsec-inline` Dockerfile stage +# 3. the resolved default values of every version ARG the two stages consume +# (including the two now-pinned xcaddy plugins, B4) +# 4. the `tonistiigi/xx` pin line and the digest-pinned `golang:*-alpine` +# builder-base lines of both inline stages (N4) +# 5. sha256 of .trivyignore +# 6. a SCHEMA_VERSION constant (bump to force a global rebuild if this +# extraction logic itself changes) +# +# Because the stage *bodies* only interpolate ${ARG}, a bump to e.g. CADDY_VERSION +# would not change (1)/(2); (3) is what makes such a bump change the key. +# +# Usage: scripts/toolchain-key.sh [path/to/Dockerfile] + +set -euo pipefail + +# rev-2: added the two xcaddy plugin pins + the digest-pinned golang base lines +# to the hashed input set. +SCHEMA_VERSION=2 + +df="${1:-Dockerfile}" +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# shellcheck source=scripts/lib/dockerfile-stage.sh +source "$here/lib/dockerfile-stage.sh" + +if [[ ! -f "$df" ]]; then + echo "toolchain-key: Dockerfile not found: $df" >&2 + exit 2 +fi + +df_dir="$(cd "$(dirname "$df")" && pwd)" +if [[ -f "$df_dir/.trivyignore" ]]; then + trivyignore="$df_dir/.trivyignore" +elif [[ -f .trivyignore ]]; then + trivyignore=".trivyignore" +else + echo "toolchain-key: .trivyignore not found (looked in $df_dir and CWD)" >&2 + exit 2 +fi + +caddy_stage="$(extract_stage caddy-inline "$df")" +crowdsec_stage="$(extract_stage crowdsec-inline "$df")" + +# Sanity: each stage must be non-trivial and actually build a binary. Guards +# against a future edit that removes a stage or breaks extraction (spec §3.10). +for pair in "caddy-inline:$caddy_stage" "crowdsec-inline:$crowdsec_stage"; do + name="${pair%%:*}" + body="${pair#*:}" + if [[ "$(printf '%s\n' "$body" | wc -l)" -lt 20 ]] \ + || ! printf '%s\n' "$body" | grep -Eq 'go build|xx-go build'; then + echo "toolchain-key: stage '$name' extraction looks wrong (too short or no build step)" >&2 + exit 3 + fi +done + +# ARG names whose default values feed the key. Keep in sync with spec §2.2 / §3.4.2. +arg_re='^ARG (GO_VERSION|ALPINE_IMAGE|CROWDSEC_VERSION|EXPR_LANG_VERSION|XNET_VERSION|XCRYPTO_VERSION|KLAUSPOST_COMPRESS_VERSION|GRPC_VERSION|CADDY_VERSION|CADDY_CANDIDATE_VERSION|CADDY_USE_CANDIDATE|CADDY_PATCH_SCENARIO|CADDY_SECURITY_VERSION|CORAZA_CADDY_VERSION|CADDY_GEOIP2_VERSION|CADDY_RATELIMIT_VERSION)=' + +key="$( + { + echo "schema=$SCHEMA_VERSION" + printf '%s\n' "$caddy_stage" + printf '%s\n' "$crowdsec_stage" + grep -E "$arg_re" "$df" + grep -E 'tonistiigi/xx:|^FROM .*golang:.*-alpine@sha256:' "$df" + sha256sum "$trivyignore" | cut -d' ' -f1 + } | sha256sum | cut -c1-16 +)" + +printf 'caddy-crowdsec-%s\n' "$key" diff --git a/scripts/verify-toolchain-pin.sh b/scripts/verify-toolchain-pin.sh new file mode 100755 index 000000000..7a425e316 --- /dev/null +++ b/scripts/verify-toolchain-pin.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# scripts/verify-toolchain-pin.sh +# +# Freshness guard for the prebuilt toolchain image pin (spec §3.4.2, B7). +# Fast, no Docker build. Run on every PR (wired into quality-checks.yml in +# spec Commit 3). +# +# It asserts: +# 1. The recomputed content key (scripts/toolchain-key.sh) equals the +# ARG CHARON_TOOLCHAIN_TAG pinned in the Dockerfile. A mismatch means a +# tracked pin / recipe line moved without the toolchain image being +# rebuilt and re-pinned -> exit 1 with an actionable message. +# 2. On a TRUSTED same-repo run (SAME_REPO=1) it is FAILURE-CLOSED: +# - `regctl` MUST be installed (else exit 1) +# - GHCR_READ_TOKEN MUST be set (else exit 1) +# - `:$KEY` MUST resolve in GHCR (else exit 1) +# - the resolved digest MUST equal CHARON_TOOLCHAIN_DIGEST (else exit 1) +# There is NO silent skip on the trusted path. +# 3. Only a FORK run (SAME_REPO=0), which has no registry access, degrades to +# tag-only equality with a `::warning::`. +# +# SAME_REPO detection (spec §3.4.2): +# push / same-repo pull_request / workflow_dispatch / schedule -> SAME_REPO=1 +# pull_request whose head repo != GITHUB_REPOSITORY -> SAME_REPO=0 +# +# Env: +# GITHUB_EVENT_NAME +# GITHUB_EVENT_PULL_REQUEST_HEAD_REPO_FULL_NAME (each calling workflow must +# map github.event.pull_request +# .head.repo.full_name here) +# GITHUB_REPOSITORY +# GHCR_READ_TOKEN (= secrets.GITHUB_TOKEN with packages:read) +# TOOLCHAIN_IMAGE (optional override; default from Dockerfile / hard default) + +set -euo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +repo_root="$(cd "$here/.." && pwd)" +df="${1:-$repo_root/Dockerfile}" + +if [[ ! -f "$df" ]]; then + echo "::error::verify-toolchain-pin: Dockerfile not found: $df" + exit 2 +fi + +arg_value() { # $1 = ARG name + grep -E "^ARG $1=" "$df" | head -n1 | cut -d= -f2- +} + +KEY="$("$here/toolchain-key.sh" "$df")" +PINNED_TAG="$(arg_value CHARON_TOOLCHAIN_TAG)" +PINNED_DIGEST="$(arg_value CHARON_TOOLCHAIN_DIGEST)" +TOOLCHAIN_IMAGE="${TOOLCHAIN_IMAGE:-$(arg_value CHARON_TOOLCHAIN_IMAGE)}" +TOOLCHAIN_IMAGE="${TOOLCHAIN_IMAGE:-ghcr.io/wikid82/charon-toolchain}" + +if [[ -z "$PINNED_TAG" ]]; then + echo "::error::verify-toolchain-pin: ARG CHARON_TOOLCHAIN_TAG not found in $df" + exit 1 +fi + +# --- Trust classification ----------------------------------------------------- +SAME_REPO=1 +if [[ "${GITHUB_EVENT_NAME:-}" == "pull_request" \ + && "${GITHUB_EVENT_PULL_REQUEST_HEAD_REPO_FULL_NAME:-}" != "${GITHUB_REPOSITORY:-}" ]]; then + SAME_REPO=0 +fi + +# --- (1) Tag / key equality (always) ---------------------------------------- +if [[ "$KEY" != "$PINNED_TAG" ]]; then + echo "::error::Toolchain recipe/pins changed (recomputed $KEY, Dockerfile pins $PINNED_TAG)." + echo "::error::Run the 'Toolchain Image' workflow (workflow_dispatch) or wait for the bot PR," + echo "::error::then bump ARG CHARON_TOOLCHAIN_TAG / CHARON_TOOLCHAIN_DIGEST in the Dockerfile." + exit 1 +fi + +# --- (2) / (3) Digest verification ----------------------------------------- +if [[ "$SAME_REPO" == "1" ]]; then + if ! command -v regctl >/dev/null 2>&1; then + echo "::error::regctl missing on a same-repo run — cannot verify the toolchain digest." + exit 1 + fi + if [[ -z "${GHCR_READ_TOKEN:-}" ]]; then + echo "::error::GHCR_READ_TOKEN unset on a same-repo run — cannot verify the toolchain digest." + exit 1 + fi + if [[ -z "$PINNED_DIGEST" ]]; then + echo "::error::ARG CHARON_TOOLCHAIN_DIGEST not found in $df — cannot verify on a same-repo run." + exit 1 + fi + + # Authenticate regctl to GHCR (best-effort; the digest call below is the real gate). + regctl registry login ghcr.io \ + --user "${GITHUB_ACTOR:-x-access-token}" \ + --pass-stdin <<<"$GHCR_READ_TOKEN" >/dev/null 2>&1 || true + + if ! REMOTE_DIGEST="$(regctl image digest "${TOOLCHAIN_IMAGE}:${KEY}" 2>/dev/null)"; then + echo "::error::${TOOLCHAIN_IMAGE}:${KEY} does not resolve in GHCR — the toolchain image" + echo "::error::was never published for this pin. Dispatch the 'Toolchain Image' workflow." + exit 1 + fi + + if [[ "$REMOTE_DIGEST" != "$PINNED_DIGEST" ]]; then + echo "::error::Dockerfile pins CHARON_TOOLCHAIN_DIGEST=$PINNED_DIGEST but" + echo "::error::${TOOLCHAIN_IMAGE}:${KEY} currently resolves to $REMOTE_DIGEST (hand-edited or stale)." + exit 1 + fi + + echo "Toolchain pin verified (same-repo): $KEY @ $PINNED_DIGEST" +else + echo "::warning::Fork PR — toolchain digest existence not verified (no registry access)." + echo "::warning::Tag matches the recomputed key ($KEY). A maintainer re-running the trusted" + echo "::warning::same-repo path performs the full digest check." +fi From c0ee1195eb433a9ecf2f5f7964da89e4ec977565 Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Mon, 7 Sep 2026 16:12:10 -0400 Subject: [PATCH 03/19] refactor(docker): build the app image from the prebuilt toolchain image MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The default app-build path now COPY --from's the digest-pinned prebuilt toolchain image instead of compiling xcaddy / CrowdSec; the inline stages become the selectable fork / bootstrap / offline fallback (spec §12 Commit 2). - Dockerfile: * CHARON_TOOLCHAIN_DIGEST set to the first published manifest-list (OCI index) digest sha256:56ea3568… for ghcr.io/wikid82/charon-toolchain :caddy-crowdsec-1efe7f19fa52a512 (multi-arch: linux/amd64 + linux/arm64 confirmed). Baked plugin versions confirmed from the toolchain build's xcaddy log: caddy-geoip2 v0.0.0-20260623062220-3675c6e7e63d, caddy-ratelimit v0.1.0 — unchanged from Commit 1's pins. * add CADDY_BUILDER_SRC / CROWDSEC_BUILDER_SRC selector ARGs (default toolchain-prebuilt). * add the toolchain-prebuilt stage (FROM ${CHARON_TOOLCHAIN_IMAGE}@${CHARON_TOOLCHAIN_DIGEST}) and replace the temp aliases with FROM ${CADDY_BUILDER_SRC} AS caddy-builder / FROM ${CROWDSEC_BUILDER_SRC} AS crowdsec-builder. The unreferenced alternative is pruned — a fork build never pulls the image. * add org.opencontainers.image.source to the toolchain-runtime LABEL so the GHCR package links to the repo. - .github/actions/build-charon-image: new `builder-src` (prebuilt|inline) and `ghcr-token` inputs; a srcsel resolver step maps them to the two build-args; a GHCR login step (skipped for inline) so BuildKit can pull the private image. - Every app-image build path passes the selector + a fork-detection expression (`head.repo.full_name != '' && != github.repository` -> inline): * docker-build.yml build-amd64 / build-arm64 (raw buildx, via job env); * nightly-build.yml (hardcoded prebuilt — same-repo only); * e2e-tests-split.yml build job (+ GHCR login + packages: read); * security-pr.yml / supply-chain-pr.yml / the 4 *-integration.yml workflows (via the composite `builder-src` + `ghcr-token`, + packages: read). The retargeted `--no-cache-filter caddy-inline,crowdsec-inline` stays for now: a no-op on the prebuilt path (stages pruned), live on the inline fork path (B5). Full removal is Commit 4, after the freshness guard is required (Commit 3). - Makefile: `build-offline` target (--build-arg *_BUILDER_SRC=*-inline). Local gate: `docker buildx build --target caddy-builder` on the default path pulls the toolchain image (40 MB) with NO xcaddy / go build step; the pulled caddy lists http.handlers.{rate_limit,crowdsec,geoip2} + geoip2 and reports v2.11.4. `docker build --check` clean; toolchain-key unchanged. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .github/actions/build-charon-image/action.yml | 45 +++++++++++++++++++ .github/workflows/cerberus-integration.yml | 5 +++ .github/workflows/crowdsec-integration.yml | 4 ++ .github/workflows/docker-build.yml | 10 +++++ .github/workflows/e2e-tests-split.yml | 15 +++++++ .github/workflows/nightly-build.yml | 2 + .github/workflows/rate-limit-integration.yml | 5 +++ .github/workflows/security-pr.yml | 13 ++++-- .github/workflows/supply-chain-pr.yml | 17 ++++--- .github/workflows/waf-integration.yml | 5 +++ Dockerfile | 45 +++++++++++++------ Makefile | 12 ++++- 12 files changed, 153 insertions(+), 25 deletions(-) diff --git a/.github/actions/build-charon-image/action.yml b/.github/actions/build-charon-image/action.yml index c669ddc1f..30703a23b 100644 --- a/.github/actions/build-charon-image/action.yml +++ b/.github/actions/build-charon-image/action.yml @@ -12,6 +12,24 @@ inputs: description: Value passed as the CI build-arg. required: false default: 'false' + builder-src: + description: >- + Which source the Caddy / CrowdSec binaries come from: + prebuilt (default) — COPY --from the digest-pinned toolchain image + (ghcr.io/wikid82/charon-toolchain), no compile. + inline — compile caddy-inline / crowdsec-inline from source + (fork PR / bootstrap / offline; ~14 min). + required: false + default: prebuilt + ghcr-token: + description: >- + Token with packages:read, used to log in to ghcr.io so BuildKit can pull + the (private) prebuilt toolchain image. Callers pass secrets.GITHUB_TOKEN. + Composite actions cannot read the secrets context themselves, so this must + be threaded in from the calling workflow. Ignored when builder-src=inline; + a fork PR that cannot supply a working token should pass builder-src=inline. + required: false + default: '' no-cache-filters: description: >- Comma-separated Dockerfile stages to force-rebuild (never restore from the @@ -39,8 +57,33 @@ inputs: runs: using: composite steps: + - name: Resolve builder source stages + id: srcsel + shell: bash + env: + BUILDER_SRC: ${{ inputs.builder-src }} + run: | + set -euo pipefail + case "${BUILDER_SRC:-prebuilt}" in + inline) + echo "caddy=caddy-inline" >> "$GITHUB_OUTPUT" + echo "crowdsec=crowdsec-inline" >> "$GITHUB_OUTPUT" ;; + prebuilt) + echo "caddy=toolchain-prebuilt" >> "$GITHUB_OUTPUT" + echo "crowdsec=toolchain-prebuilt" >> "$GITHUB_OUTPUT" ;; + *) + echo "::error::build-charon-image: invalid builder-src '${BUILDER_SRC}' (want prebuilt|inline)" + exit 1 ;; + esac - name: Set up Docker Buildx uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0 + - name: Log in to GHCR (pull prebuilt toolchain image) + if: inputs.builder-src != 'inline' && inputs.ghcr-token != '' + uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ inputs.ghcr-token }} - name: Build image uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7 with: @@ -51,6 +94,8 @@ runs: tags: ${{ inputs.tag }} build-args: | CI=${{ inputs.ci }} + CADDY_BUILDER_SRC=${{ steps.srcsel.outputs.caddy }} + CROWDSEC_BUILDER_SRC=${{ steps.srcsel.outputs.crowdsec }} cache-from: type=gha,scope=charon-integration-image cache-to: type=gha,mode=max,scope=charon-integration-image no-cache-filters: ${{ inputs.no-cache-filters }} diff --git a/.github/workflows/cerberus-integration.yml b/.github/workflows/cerberus-integration.yml index bf47e2b9f..143adb3bc 100644 --- a/.github/workflows/cerberus-integration.yml +++ b/.github/workflows/cerberus-integration.yml @@ -21,6 +21,7 @@ concurrency: permissions: contents: read + packages: read jobs: cerberus-integration: @@ -32,6 +33,10 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Build Docker image (Local) uses: ./.github/actions/build-charon-image + with: + # Fork PRs cannot pull the private toolchain image -> compile inline. + builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} + ghcr-token: ${{ secrets.GITHUB_TOKEN }} - name: Run Cerberus integration tests id: cerberus-test diff --git a/.github/workflows/crowdsec-integration.yml b/.github/workflows/crowdsec-integration.yml index c7d399962..9d2280bde 100644 --- a/.github/workflows/crowdsec-integration.yml +++ b/.github/workflows/crowdsec-integration.yml @@ -21,6 +21,7 @@ concurrency: permissions: contents: read + packages: read jobs: crowdsec-integration: @@ -34,6 +35,9 @@ jobs: uses: ./.github/actions/build-charon-image with: ci: 'true' + # Fork PRs cannot pull the private toolchain image -> compile inline. + builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} + ghcr-token: ${{ secrets.GITHUB_TOKEN }} - name: Run CrowdSec integration tests id: crowdsec-test diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index afe3a9ce8..3df9733a0 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -399,6 +399,10 @@ jobs: if: needs.setup.result == 'success' && needs.setup.outputs.skip_build != 'true' env: HAS_DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN != '' }} + # Release path is same-repo only and never runs the inline compile; a fork + # PR (empty or foreign head repo) would still fall back to caddy-inline. + CADDY_BUILDER_SRC: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'caddy-inline' || 'toolchain-prebuilt' }} + CROWDSEC_BUILDER_SRC: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'crowdsec-inline' || 'toolchain-prebuilt' }} runs-on: ubuntu-latest timeout-minutes: 15 permissions: @@ -467,6 +471,8 @@ jobs: --build-arg "BUILD_DATE=${{ needs.setup.outputs.created }}" --build-arg "VCS_REF=${{ env.TRIGGER_HEAD_SHA }}" --build-arg "ALPINE_IMAGE=${{ needs.setup.outputs.alpine_image }}" + --build-arg "CADDY_BUILDER_SRC=${{ env.CADDY_BUILDER_SRC }}" + --build-arg "CROWDSEC_BUILDER_SRC=${{ env.CROWDSEC_BUILDER_SRC }}" --iidfile /tmp/image-digest-amd64.txt . ) @@ -483,6 +489,8 @@ jobs: if: needs.setup.result == 'success' && needs.setup.outputs.skip_build != 'true' env: HAS_DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN != '' }} + CADDY_BUILDER_SRC: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'caddy-inline' || 'toolchain-prebuilt' }} + CROWDSEC_BUILDER_SRC: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'crowdsec-inline' || 'toolchain-prebuilt' }} runs-on: ubuntu-latest timeout-minutes: 25 permissions: @@ -553,6 +561,8 @@ jobs: --build-arg "BUILD_DATE=${{ needs.setup.outputs.created }}" --build-arg "VCS_REF=${{ env.TRIGGER_HEAD_SHA }}" --build-arg "ALPINE_IMAGE=${{ needs.setup.outputs.alpine_image }}" + --build-arg "CADDY_BUILDER_SRC=${{ env.CADDY_BUILDER_SRC }}" + --build-arg "CROWDSEC_BUILDER_SRC=${{ env.CROWDSEC_BUILDER_SRC }}" --iidfile /tmp/image-digest-arm64.txt . ) diff --git a/.github/workflows/e2e-tests-split.yml b/.github/workflows/e2e-tests-split.yml index 1de9995ee..711026ce5 100644 --- a/.github/workflows/e2e-tests-split.yml +++ b/.github/workflows/e2e-tests-split.yml @@ -123,6 +123,9 @@ jobs: build: name: Prepare Application Image runs-on: ubuntu-latest + permissions: + contents: read + packages: read outputs: image_source: ${{ steps.resolve-image.outputs.image_source }} image_ref: ${{ steps.resolve-image.outputs.image_ref }} @@ -209,6 +212,14 @@ jobs: if: steps.resolve-image.outputs.image_source == 'build' uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 + - name: Log in to GHCR (pull prebuilt toolchain image) + if: steps.resolve-image.outputs.image_source == 'build' && github.event.pull_request.head.repo.full_name == github.repository + uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - name: Build Docker image id: build-image if: steps.resolve-image.outputs.image_source == 'build' @@ -221,6 +232,10 @@ jobs: tags: ${{ steps.resolve-image.outputs.image_tag }} cache-from: type=gha cache-to: type=gha,mode=max + # Fork PRs cannot pull the private toolchain image -> compile inline. + build-args: | + CADDY_BUILDER_SRC=${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'caddy-inline' || 'toolchain-prebuilt' }} + CROWDSEC_BUILDER_SRC=${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'crowdsec-inline' || 'toolchain-prebuilt' }} no-cache-filters: caddy-inline,crowdsec-inline - name: Save Docker image diff --git a/.github/workflows/nightly-build.yml b/.github/workflows/nightly-build.yml index 2e7a60066..717d84ae0 100644 --- a/.github/workflows/nightly-build.yml +++ b/.github/workflows/nightly-build.yml @@ -238,6 +238,8 @@ jobs: VCS_REF=${{ github.sha }} BUILD_DATE=${{ github.event.repository.pushed_at }} ALPINE_IMAGE=${{ steps.alpine.outputs.image }} + CADDY_BUILDER_SRC=toolchain-prebuilt + CROWDSEC_BUILDER_SRC=toolchain-prebuilt cache-from: type=gha cache-to: type=gha,mode=max no-cache-filters: caddy-inline,crowdsec-inline diff --git a/.github/workflows/rate-limit-integration.yml b/.github/workflows/rate-limit-integration.yml index faa49223b..a1d88899c 100644 --- a/.github/workflows/rate-limit-integration.yml +++ b/.github/workflows/rate-limit-integration.yml @@ -21,6 +21,7 @@ concurrency: permissions: contents: read + packages: read jobs: rate-limit-integration: @@ -32,6 +33,10 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Build Docker image (Local) uses: ./.github/actions/build-charon-image + with: + # Fork PRs cannot pull the private toolchain image -> compile inline. + builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} + ghcr-token: ${{ secrets.GITHUB_TOKEN }} - name: Run rate limit integration tests id: ratelimit-test diff --git a/.github/workflows/security-pr.yml b/.github/workflows/security-pr.yml index 3ea30d38b..faca444f6 100644 --- a/.github/workflows/security-pr.yml +++ b/.github/workflows/security-pr.yml @@ -24,6 +24,7 @@ concurrency: permissions: contents: read + packages: read jobs: security-scan: @@ -156,11 +157,15 @@ jobs: if: github.event_name == 'push' || github.event_name == 'pull_request' uses: ./.github/actions/build-charon-image with: - # Force a clean rebuild of the two from-source stages so a stale GHA - # layer cache cannot retain a superseded pinned dependency (the + # Default path COPY --from's the digest-pinned toolchain image; fork + # PRs (no private-package pull) compile caddy-inline/crowdsec-inline. + builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} + ghcr-token: ${{ secrets.GITHUB_TOKEN }} + # Still force-rebuild the from-source stages on the inline path so a + # stale GHA layer cannot retain a superseded pinned dependency (the # `go get pkg@fixed` patch lives INSIDE the cached-and-skipped stage). - # Matches nightly-build.yml and e2e-tests-split.yml. CVE-scan gate: - # correctness beats the few minutes of rebuild time. + # Removed entirely in a later commit once verify-toolchain-pin is a + # required check and the default path no longer compiles at all. no-cache-filters: caddy-inline,crowdsec-inline - name: Check for PR image artifact diff --git a/.github/workflows/supply-chain-pr.yml b/.github/workflows/supply-chain-pr.yml index 5de24c5ab..088a5dc3d 100644 --- a/.github/workflows/supply-chain-pr.yml +++ b/.github/workflows/supply-chain-pr.yml @@ -23,6 +23,7 @@ concurrency: permissions: contents: read + packages: read pull-requests: write security-events: write actions: read @@ -251,13 +252,15 @@ jobs: if: github.event_name != 'workflow_run' uses: ./.github/actions/build-charon-image with: - # Force a clean rebuild of the two from-source stages so a stale GHA - # layer cache cannot retain a superseded pinned dependency (e.g. a - # caddy-builder layer built before a grpc-go / x-crypto pin bump still - # embeds the old, vulnerable version — the `go get pkg@fixed` lives - # INSIDE the cached-and-skipped stage). Matches nightly-build.yml and - # e2e-tests-split.yml. This is a CVE-scan gate; correctness beats the - # few minutes of xcaddy/crowdsec rebuild time. + # Default path COPY --from's the digest-pinned toolchain image; fork + # PRs (no private-package pull) compile caddy-inline/crowdsec-inline. + builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} + ghcr-token: ${{ secrets.GITHUB_TOKEN }} + # Still force-rebuild the from-source stages on the inline path so a + # stale GHA layer cannot retain a superseded pinned dependency (the + # `go get pkg@fixed` patch lives INSIDE the cached-and-skipped stage). + # Removed entirely in a later commit once verify-toolchain-pin is a + # required check and the default path no longer compiles at all. no-cache-filters: caddy-inline,crowdsec-inline - name: Expose local image name diff --git a/.github/workflows/waf-integration.yml b/.github/workflows/waf-integration.yml index 231d92e33..c3dbccc9a 100644 --- a/.github/workflows/waf-integration.yml +++ b/.github/workflows/waf-integration.yml @@ -21,6 +21,7 @@ concurrency: permissions: contents: read + packages: read jobs: waf-integration: @@ -32,6 +33,10 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - name: Build Docker image (Local) uses: ./.github/actions/build-charon-image + with: + # Fork PRs cannot pull the private toolchain image -> compile inline. + builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} + ghcr-token: ${{ secrets.GITHUB_TOKEN }} - name: Run WAF integration tests id: waf-test diff --git a/Dockerfile b/Dockerfile index b6f961170..d1fd68716 100644 --- a/Dockerfile +++ b/Dockerfile @@ -17,9 +17,17 @@ ARG BUILD_DEBUG=0 # stale for the current pins. ARG CHARON_TOOLCHAIN_IMAGE=ghcr.io/wikid82/charon-toolchain # NOT Renovate-tracked (a content-hash tag has no series to follow, N7) — the -# toolchain-image.yml bot owns these two lines. +# toolchain-image.yml bot owns these two lines. DIGEST is the arch-independent +# manifest-list (OCI index) digest, so one pin covers linux/amd64 + linux/arm64. ARG CHARON_TOOLCHAIN_TAG=caddy-crowdsec-1efe7f19fa52a512 -ARG CHARON_TOOLCHAIN_DIGEST=sha256:0000000000000000000000000000000000000000000000000000000000000000 +ARG CHARON_TOOLCHAIN_DIGEST=sha256:6575f4c6a9f76074870c64df9dd4c9ebee812342f37f52ae5ef8f511ba9f8f00 + +# Stage selector — default consumes the prebuilt toolchain image (no compile). +# Fork PRs / bootstrap / offline builds pass +# --build-arg CADDY_BUILDER_SRC=caddy-inline --build-arg CROWDSEC_BUILDER_SRC=crowdsec-inline +# (e.g. `make build-offline`) to compile from source instead. +ARG CADDY_BUILDER_SRC=toolchain-prebuilt +ARG CROWDSEC_BUILDER_SRC=toolchain-prebuilt # ---- Pinned Toolchain Versions ---- # renovate: datasource=docker depName=golang versioning=docker @@ -758,17 +766,28 @@ COPY --from=caddy-inline /usr/bin/caddy /usr/bin/caddy COPY --from=crowdsec-inline /crowdsec-out/crowdsec /crowdsec-out/crowdsec COPY --from=crowdsec-inline /crowdsec-out/cscli /crowdsec-out/cscli COPY --from=crowdsec-inline /crowdsec-out/config /crowdsec-out/config -# Provenance: `docker inspect` on the toolchain image shows the content key. -LABEL io.charon.toolchain.key="${CHARON_TOOLCHAIN_TAG}" - -# ---- Effective builder stages ---- -# Commit 1: temporary aliases so the app build is byte-identical while the -# selector / prebuilt-image consumption lands in Commit 2. The retargeted -# `--no-cache-filter caddy-inline,crowdsec-inline` in CI keeps invalidating the -# real RUN layers (which now live in caddy-inline / crowdsec-inline) through the -# rename — the CVE-recurrence guard is never inert (B5). -FROM caddy-inline AS caddy-builder -FROM crowdsec-inline AS crowdsec-builder +# Provenance: `docker inspect` on the toolchain image shows the content key; +# image.source links the GHCR package to the repo so same-repo CI can pull it. +LABEL io.charon.toolchain.key="${CHARON_TOOLCHAIN_TAG}" \ + org.opencontainers.image.source="https://github.com/Wikid82/charon" + +# ---- Prebuilt toolchain (default source for caddy-builder / crowdsec-builder) ---- +# Digest-pinned OCI index; BuildKit auto-selects the child matching $TARGETPLATFORM. +# Contains /usr/bin/caddy and /crowdsec-out/{crowdsec,cscli,config} at the SAME +# paths the inline stages produce, so the final-stage COPY --from lines are +# unchanged. Pruned from the graph (never pulled) when *_BUILDER_SRC=*-inline. +FROM ${CHARON_TOOLCHAIN_IMAGE}@${CHARON_TOOLCHAIN_DIGEST} AS toolchain-prebuilt + +# ---- Effective builder stages: prebuilt image OR inline compile ---- +# `FROM ${ARG} AS name` where the ARG resolves to a prior stage name is valid +# BuildKit; the unreferenced alternative is pruned and never built/pulled. +# On the default (prebuilt) path caddy-inline / crowdsec-inline are not in the +# graph, so the retargeted `--no-cache-filter caddy-inline,crowdsec-inline` in CI +# is a no-op there and live only on the fork/offline inline path (B5). The +# pin<->digest binding on the default path is enforced by +# scripts/verify-toolchain-pin.sh (wired as a required check in Commit 3). +FROM ${CADDY_BUILDER_SRC} AS caddy-builder +FROM ${CROWDSEC_BUILDER_SRC} AS crowdsec-builder # ---- Final Runtime with Caddy ---- FROM ${ALPINE_IMAGE} diff --git a/Makefile b/Makefile index 31fb347d3..ab443c043 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help install test build run clean docker-build docker-run release go-check gopls-logs lint-fast lint-staticcheck-only security-local +.PHONY: help install test build run clean docker-build docker-run build-offline release go-check gopls-logs lint-fast lint-staticcheck-only security-local # Default target help: @@ -105,6 +105,16 @@ docker-build-versioned: -t charon:latest \ . +# Build the image WITHOUT pulling the prebuilt toolchain image — compiles the +# custom Caddy + CrowdSec binaries from source (caddy-inline / crowdsec-inline). +# Use offline / air-gapped, or when not logged in to GHCR. Slow (~14 min extra). +build-offline: + docker build \ + --build-arg CADDY_BUILDER_SRC=caddy-inline \ + --build-arg CROWDSEC_BUILDER_SRC=crowdsec-inline \ + -t charon:offline \ + . + # Run Docker containers (production) docker-run: docker compose -f .docker/compose/docker-compose.yml up -d From 53134273f4c6c8ad694071a170d02bb494f4d3c2 Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Mon, 7 Sep 2026 16:45:19 -0400 Subject: [PATCH 04/19] feat(security): harden build-pipeline integrity verification MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec §12 Commit 3 (+ Rev 2.1 determinism amendment, see docs/plans/current_spec.md §3.4.3 — flagged for supervisor re-review). - quality-checks.yml: * `toolchain-key-tests` (bats) is now BLOCKING (dropped continue-on-error). * new REQUIRED `verify-toolchain-pin` job — installs regctl (iarekylew00t/regctl-installer, SHA-pinned) and runs scripts/verify-toolchain-pin.sh with GHCR_READ_TOKEN=secrets.GITHUB_TOKEN and head.repo.full_name mapped into env (B7). Failure-closed on same-repo runs. - toolchain-image.yml: * DETERMINISTIC build (fixes a self-perpetuating sync-pin bot-commit loop): `--provenance=false --sbom=false`, a fixed `SOURCE_DATE_EPOCH=1700000000`, and `--output type=image,"name=…",push=true,rewrite-timestamp=true`. Two independent builds now produce an identical manifest-list digest. The toolchain image is an internal build input; the app image's provenance/SBOM (docker-build.yml) is separate and unaffected. * SKIP-IF-ALREADY-PUBLISHED: on any non-forced event, if `:$KEY` resolves in GHCR the job reuses that digest and skips the build/push entirely. Only schedule / workflow_dispatch force_rebuild / workflow_call rebuild+repush. * `sync-pin-on-pr` — idempotent + self-trigger-safe: guarded `github.actor != 'github-actions[bot]'`, and no-ops unless `git diff --quiet Dockerfile` shows a real change. Pushes the two-line TAG/DIGEST bump onto the PR head branch (::notice:: to re-run the check). * `open-bump-pr` — off the PR path only (schedule / workflow_dispatch / workflow_call); opens `bot/bump-toolchain-image` against base `development` (peter-evans/create-pull-request, SHA-pinned) titled `feat(security): refresh bundled proxy toolchain image`; on failure opens a tracked issue. - Dockerfile N5 — after the final-stage COPY --from=caddy-builder / crowdsec-builder: * assert the toolchain caddy binary exposes http.handlers.{rate_limit,crowdsec, geoip2,waf} (the 4 recipe plugins; the WAF module is `http.handlers.waf`); * assert cscli runs and emits its recognisable version block (CrowdSec 1.8.x prints an empty `version:` field regardless of the -X ldflag). A wrong-arch / rolled-back digest fails the app build here, not only at toolchain-build time. - docker-build.yml merge-and-publish: new "Verify pinned toolchain image matches the recipe (N5)" step — pulls the digest-pinned toolchain image and asserts its io.charon.toolchain.key LABEL == scripts/toolchain-key.sh. - renovate.json: packageRules entry — the charon-toolchain digest is bot-owned, not Renovate-tracked (N7), disabled there. - docs/plans/current_spec.md §3.4.3: Rev 2.1 amendment documenting the three determinism / loop-prevention fixes above. Local: full `docker build .` (default prebuilt path) passes both N5 assertions; two independent deterministic multi-arch builds produce identical manifest-list digest sha256:6575f4c6a9f76074870c64df9dd4c9ebee812342f37f52ae5ef8f511ba9f8f00 (pinned in Commit 2); `docker build --check` clean; toolchain key unchanged (caddy-crowdsec-1efe7f19fa52a512). Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .github/renovate.json | 7 + .github/workflows/docker-build.yml | 17 ++ .github/workflows/quality-checks.yml | 27 ++- .github/workflows/toolchain-image.yml | 244 ++++++++++++++++++++++++-- Dockerfile | 28 +++ docs/plans/current_spec.md | 46 +++-- 6 files changed, 339 insertions(+), 30 deletions(-) diff --git a/.github/renovate.json b/.github/renovate.json index 12df0cdf9..862b79f4e 100644 --- a/.github/renovate.json +++ b/.github/renovate.json @@ -494,6 +494,13 @@ ], "semanticCommitType": "chore" }, + { + "description": "The ghcr.io/wikid82/charon-toolchain digest pin (Dockerfile ARG CHARON_TOOLCHAIN_TAG / CHARON_TOOLCHAIN_DIGEST) is owned by .github/workflows/toolchain-image.yml's bot PR, NOT Renovate: its content-hash tag (caddy-crowdsec-) has no version series to follow (spec N7). Disabled here so a stray digest bump cannot land without the matching key change the freshness guard requires.", + "matchPackageNames": [ + "ghcr.io/wikid82/charon-toolchain" + ], + "enabled": false + }, { "description": "Group GitHub Actions non-major updates into one PR", "matchManagers": [ diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index 3df9733a0..2035a9243 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -785,6 +785,23 @@ jobs: path: /tmp/charon-pr-image.tar retention-days: 1 # Only needed for workflow duration + - name: Verify pinned toolchain image matches the recipe (N5) + run: | + set -euo pipefail + KEY="$(bash scripts/toolchain-key.sh)" + PIN_IMG="$(grep -E '^ARG CHARON_TOOLCHAIN_IMAGE=' Dockerfile | cut -d= -f2)" + PIN_DIGEST="$(grep -E '^ARG CHARON_TOOLCHAIN_DIGEST=' Dockerfile | cut -d= -f2)" + echo "recomputed recipe key : $KEY" + echo "pinned image : ${PIN_IMG}@${PIN_DIGEST}" + docker pull "${PIN_IMG}@${PIN_DIGEST}" + LABEL_KEY="$(docker inspect --format '{{ index .Config.Labels "io.charon.toolchain.key" }}' "${PIN_IMG}@${PIN_DIGEST}")" + echo "toolchain image label : ${LABEL_KEY:-}" + if [ "$LABEL_KEY" != "$KEY" ]; then + echo "::error::Pinned toolchain image label ('${LABEL_KEY}') != recomputed recipe key ('${KEY}') — the Dockerfile digest pin is stale or hand-edited." + exit 1 + fi + echo "✅ Pinned toolchain image matches the current recipe." + - name: Verify Caddy Security Patches (CVE-2025-68156) timeout-minutes: 2 continue-on-error: true diff --git a/.github/workflows/quality-checks.yml b/.github/workflows/quality-checks.yml index 6ffda7f22..a4dffff15 100644 --- a/.github/workflows/quality-checks.yml +++ b/.github/workflows/quality-checks.yml @@ -76,9 +76,6 @@ jobs: toolchain-key-tests: name: Toolchain key / freshness-guard scripts (bats) runs-on: ubuntu-latest - # Non-blocking for now (spec §12 Commit 1). The blocking `verify-toolchain-pin` - # required check is added in Commit 3. - continue-on-error: true steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 @@ -98,6 +95,30 @@ jobs: - name: bats run: bats scripts/tests/toolchain-key.bats scripts/tests/verify-toolchain-pin.bats + verify-toolchain-pin: + name: Toolchain pin freshness (verify-toolchain-pin) + runs-on: ubuntu-latest + # Failure-closed on trusted same-repo runs (spec §3.4.2 / B7). A stale + # Dockerfile toolchain TAG/DIGEST for the current pins fails this required + # check with an actionable message. + permissions: + contents: read + packages: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Install regctl + uses: iarekylew00t/regctl-installer@c2202c17a65fe59371c71ecc169c9e58c3710a15 # v4.0.16 + + - name: Verify the pinned toolchain image matches the recipe + env: + GHCR_READ_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GITHUB_EVENT_NAME: ${{ github.event_name }} + # Map head.repo.full_name into env explicitly (B7 / supervisor caution #1). + GITHUB_EVENT_PULL_REQUEST_HEAD_REPO_FULL_NAME: ${{ github.event.pull_request.head.repo.full_name }} + GITHUB_REPOSITORY: ${{ github.repository }} + run: bash scripts/verify-toolchain-pin.sh + backend-quality: name: Backend (Go) runs-on: ubuntu-latest diff --git a/.github/workflows/toolchain-image.yml b/.github/workflows/toolchain-image.yml index 4d0e8f6aa..e8c66233d 100644 --- a/.github/workflows/toolchain-image.yml +++ b/.github/workflows/toolchain-image.yml @@ -57,6 +57,10 @@ permissions: env: TOOLCHAIN_IMAGE: ghcr.io/wikid82/charon-toolchain + # Fixed epoch for `rewrite-timestamp`: fully decouples the manifest-list digest + # from git history / build time, so an unchanged recipe (same toolchain key) + # always reproduces the SAME digest. Only bump if a global rebuild is wanted. + SOURCE_DATE_EPOCH: '1700000000' jobs: build-toolchain: @@ -107,46 +111,86 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - - name: Decide cache flags - id: flags + - name: Decide build plan + id: plan env: EVENT_NAME: ${{ github.event_name }} FORCE_REBUILD: ${{ inputs.force_rebuild }} + SAME_REPO: ${{ steps.trust.outputs.same_repo }} + KEY: ${{ steps.key.outputs.key }} run: | set -euo pipefail - no_cache="" + + # Forced rebuild: schedule, or an explicit force_rebuild=true input + # (workflow_dispatch / workflow_call). PR / plain paths never force. + forced=false if [[ "$EVENT_NAME" == "schedule" || "$FORCE_REBUILD" == "true" ]]; then - no_cache="--no-cache --pull" + forced=true + fi + no_cache="" + [[ "$forced" == "true" ]] && no_cache="--no-cache --pull" + + # Skip-if-already-published: on a non-forced run, if :$KEY already + # resolves in GHCR there is nothing to do — reuse the existing digest. + # This is a deterministic image (see the build step), so :$KEY for a + # given recipe is a single stable digest; rebuilding would be a no-op. + should_build=true + existing_digest="" + if [[ "$forced" != "true" && "$SAME_REPO" == "true" ]]; then + if existing_digest="$(docker buildx imagetools inspect "${TOOLCHAIN_IMAGE}:${KEY}" \ + --format '{{json .Manifest}}' 2>/dev/null | jq -r '.digest')" \ + && [[ -n "$existing_digest" && "$existing_digest" != "null" ]]; then + should_build=false + echo "::notice::${TOOLCHAIN_IMAGE}:${KEY} already published (${existing_digest}) — skipping rebuild." + else + existing_digest="" + fi fi - echo "no_cache=$no_cache" >> "$GITHUB_OUTPUT" - echo "cache flags: '${no_cache:-}'" - - name: Build toolchain image (multi-arch $BUILDPLATFORM cross-compile, no QEMU) + { + echo "forced=$forced" + echo "no_cache=$no_cache" + echo "should_build=$should_build" + echo "existing_digest=$existing_digest" + } >> "$GITHUB_OUTPUT" + echo "plan: forced=$forced should_build=$should_build cache='${no_cache:-}'" + + - name: Build toolchain image (deterministic multi-arch, no QEMU) id: build + if: steps.plan.outputs.should_build == 'true' env: SAME_REPO: ${{ steps.trust.outputs.same_repo }} KEY: ${{ steps.key.outputs.key }} - NO_CACHE: ${{ steps.flags.outputs.no_cache }} + NO_CACHE: ${{ steps.plan.outputs.no_cache }} run: | set -euo pipefail DATE_TAG="$(date -u +%Y%m%d)" - TAGS=(-t "${TOOLCHAIN_IMAGE}:${KEY}" -t "${TOOLCHAIN_IMAGE}:${DATE_TAG}") - OUT="--output=type=cacheonly" + NAMES="${TOOLCHAIN_IMAGE}:${KEY},${TOOLCHAIN_IMAGE}:${DATE_TAG}" if [[ "$SAME_REPO" == "true" ]]; then - TAGS+=(-t "${TOOLCHAIN_IMAGE}:latest") - OUT="--push" + NAMES="${NAMES},${TOOLCHAIN_IMAGE}:latest" + OUT="type=image,\"name=${NAMES}\",push=true,rewrite-timestamp=true" + else + OUT="type=cacheonly" fi + # Deterministic: no provenance / SBOM attestation manifests (their + # content embeds per-run timestamps + builder identity, which would + # change the OCI-index digest on every build for identical layers). + # This image is an internal build INPUT, not a released artifact — the + # app image's own provenance/SBOM is generated separately and is + # unaffected. rewrite-timestamp + SOURCE_DATE_EPOCH stabilise layer + # mtimes. Result: identical key => identical manifest-list digest. # shellcheck disable=SC2086 # NO_CACHE is an intentional word-split flag list docker buildx build \ --target toolchain-runtime \ --platform linux/amd64,linux/arm64 \ $NO_CACHE \ + --provenance=false \ + --sbom=false \ --cache-from type=gha,scope=toolchain \ --cache-to type=gha,mode=max,scope=toolchain \ --build-arg "CHARON_TOOLCHAIN_TAG=${KEY}" \ - "${TAGS[@]}" \ - $OUT \ + --output "${OUT}" \ . - name: Resolve published manifest-list digest @@ -154,16 +198,21 @@ jobs: if: steps.trust.outputs.same_repo == 'true' env: KEY: ${{ steps.key.outputs.key }} + EXISTING: ${{ steps.plan.outputs.existing_digest }} run: | set -euo pipefail - DIGEST="$(docker buildx imagetools inspect "${TOOLCHAIN_IMAGE}:${KEY}" \ - --format '{{json .Manifest}}' | jq -r '.digest')" + if [[ -n "$EXISTING" ]]; then + DIGEST="$EXISTING" + else + DIGEST="$(docker buildx imagetools inspect "${TOOLCHAIN_IMAGE}:${KEY}" \ + --format '{{json .Manifest}}' | jq -r '.digest')" + fi if [[ -z "$DIGEST" || "$DIGEST" == "null" ]]; then echo "::error::Could not resolve manifest-list digest for ${TOOLCHAIN_IMAGE}:${KEY}" exit 1 fi echo "digest=$DIGEST" >> "$GITHUB_OUTPUT" - echo "Published ${TOOLCHAIN_IMAGE}:${KEY} @ ${DIGEST}" + echo "Toolchain ${TOOLCHAIN_IMAGE}:${KEY} @ ${DIGEST}" - name: Assert multi-arch manifest (linux/amd64 + linux/arm64) if: steps.trust.outputs.same_repo == 'true' @@ -247,3 +296,164 @@ jobs: with: sarif_file: 'trivy-toolchain.sarif' category: '.github/workflows/toolchain-image.yml:trivy-toolchain' + + # On a same-repo PR that legitimately moved a tracked pin, the recomputed key + # no longer matches the Dockerfile ARG. This job pushes the two-line + # TAG/DIGEST bump onto the PR head branch so verify-toolchain-pin goes green. + sync-pin-on-pr: + name: Sync toolchain pin onto the PR branch + needs: build-toolchain + # Idempotent + self-trigger-safe: never act on the bot's own commits, only + # commit when the Dockerfile pin genuinely changes. With the deterministic + # build above (same key => same digest) this essentially never fires unless a + # tracked pin actually moved on the PR. + if: >- + github.event_name == 'pull_request' && + github.event.pull_request.head.repo.full_name == github.repository && + github.actor != 'github-actions[bot]' && + needs.build-toolchain.outputs.digest != '' + runs-on: ubuntu-latest + permissions: + contents: write + steps: + - name: Checkout PR head + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + ref: ${{ github.event.pull_request.head.ref }} + token: ${{ secrets.GITHUB_TOKEN }} + + - name: Sync ARG CHARON_TOOLCHAIN_TAG / DIGEST if stale + env: + KEY: ${{ needs.build-toolchain.outputs.key }} + DIGEST: ${{ needs.build-toolchain.outputs.digest }} + HEAD_REF: ${{ github.event.pull_request.head.ref }} + run: | + set -euo pipefail + cur_tag="$(grep -E '^ARG CHARON_TOOLCHAIN_TAG=' Dockerfile | cut -d= -f2)" + cur_digest="$(grep -E '^ARG CHARON_TOOLCHAIN_DIGEST=' Dockerfile | cut -d= -f2)" + if [[ "$cur_tag" == "$KEY" && "$cur_digest" == "$DIGEST" ]]; then + echo "Toolchain pin already fresh ($KEY @ $DIGEST) — nothing to sync." + exit 0 + fi + sed -i "s|^ARG CHARON_TOOLCHAIN_TAG=.*|ARG CHARON_TOOLCHAIN_TAG=${KEY}|" Dockerfile + sed -i "s|^ARG CHARON_TOOLCHAIN_DIGEST=.*|ARG CHARON_TOOLCHAIN_DIGEST=${DIGEST}|" Dockerfile + if git diff --quiet -- Dockerfile; then + echo "sed produced no net change — nothing to commit." + exit 0 + fi + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add Dockerfile + git commit -m "chore(docker): sync toolchain image pin to ${KEY}" + git push origin "HEAD:${HEAD_REF}" + echo "::notice::Toolchain pin synced to ${KEY} @ ${DIGEST}. Re-run the 'Toolchain pin freshness' check (GITHUB_TOKEN pushes do not auto-retrigger PR checks)." + + # Off the PR path only: if the rebuilt digest differs from the Dockerfile pin, + # open a bot PR bumping it (modelled on update-geolite2.yml). Keeps base + # `development` per spec §3.3/§3.4. + open-bump-pr: + name: Open toolchain digest-bump PR + needs: [build-toolchain, trivy-scan] + if: >- + needs.build-toolchain.outputs.same_repo == 'true' && + needs.build-toolchain.outputs.digest != '' && + (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') + runs-on: ubuntu-latest + permissions: + contents: write + pull-requests: write + issues: write + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Decide whether the pin moved + id: moved + env: + KEY: ${{ needs.build-toolchain.outputs.key }} + DIGEST: ${{ needs.build-toolchain.outputs.digest }} + run: | + set -euo pipefail + cur_tag="$(grep -E '^ARG CHARON_TOOLCHAIN_TAG=' Dockerfile | cut -d= -f2)" + cur_digest="$(grep -E '^ARG CHARON_TOOLCHAIN_DIGEST=' Dockerfile | cut -d= -f2)" + echo "old_tag=$cur_tag" >> "$GITHUB_OUTPUT" + echo "old_digest=$cur_digest" >> "$GITHUB_OUTPUT" + if [[ "$cur_tag" == "$KEY" && "$cur_digest" == "$DIGEST" ]]; then + echo "moved=false" >> "$GITHUB_OUTPUT" + echo "Toolchain pin unchanged ($KEY @ $DIGEST) — no bot PR." + else + echo "moved=true" >> "$GITHUB_OUTPUT" + fi + + - name: Update the ARG lines + if: steps.moved.outputs.moved == 'true' + env: + KEY: ${{ needs.build-toolchain.outputs.key }} + DIGEST: ${{ needs.build-toolchain.outputs.digest }} + run: | + set -euo pipefail + sed -i "s|^ARG CHARON_TOOLCHAIN_TAG=.*|ARG CHARON_TOOLCHAIN_TAG=${KEY}|" Dockerfile + sed -i "s|^ARG CHARON_TOOLCHAIN_DIGEST=.*|ARG CHARON_TOOLCHAIN_DIGEST=${DIGEST}|" Dockerfile + DOCKER_BUILDKIT=1 docker build --check -f Dockerfile . + + - name: Create Pull Request + if: steps.moved.outputs.moved == 'true' + uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1 + with: + base: development + branch: bot/bump-toolchain-image + delete-branch: true + title: "feat(security): refresh bundled proxy toolchain image" + labels: | + dependencies + automated + docker + security + commit-message: | + feat(security): refresh bundled proxy toolchain image + + Rebuilds the prebuilt Caddy/CrowdSec toolchain image so the shipped + binaries pick up upstream fixes, and bumps the digest pin in the + Dockerfile. + body: | + 🤖 **Automated bundled-toolchain refresh** + + The prebuilt Caddy/CrowdSec toolchain image was rebuilt (daily + `--no-cache --pull`, or a tracked-pin change) and produced a new + digest. + + | | Tag | Digest | + |---|---|---| + | Old | `${{ steps.moved.outputs.old_tag }}` | `${{ steps.moved.outputs.old_digest }}` | + | New | `${{ needs.build-toolchain.outputs.key }}` | `${{ needs.build-toolchain.outputs.digest }}` | + + The Trivy CRITICAL/HIGH gate on the new image passed on the rebuild + run. Merging updates the shipped Caddy/CrowdSec binaries. + + ### Verification + - [ ] CI green (`verify-toolchain-pin`, Docker Build, integration) + - [ ] Trivy diff on the final app image shows no new CRITICAL/HIGH + + Auto-generated by `.github/workflows/toolchain-image.yml`. + + - name: Report failure via GitHub issue + if: failure() + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 + with: + script: | + const runUrl = `https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`; + await github.rest.issues.create({ + owner: context.repo.owner, + repo: context.repo.repo, + title: '🚨 Toolchain image rebuild / digest-bump failed', + body: [ + 'The `.github/workflows/toolchain-image.yml` rebuild or its digest-bump PR step failed.', + '', + `- Run: ${runUrl}`, + `- Trigger: ${context.eventName}`, + '', + 'The shipped Caddy/CrowdSec binaries may be missing an upstream fix until this is resolved.', + 'Re-run via `workflow_dispatch` once fixed.', + ].join('\n'), + labels: ['bug', 'automated', 'ci-cd', 'docker', 'security'], + }); diff --git a/Dockerfile b/Dockerfile index d1fd68716..89f0d9469 100644 --- a/Dockerfile +++ b/Dockerfile @@ -851,10 +851,38 @@ COPY --from=caddy-builder /usr/bin/caddy /usr/bin/caddy # Allow non-root to bind privileged ports (80/443) securely RUN setcap 'cap_net_bind_service=+ep' /usr/bin/caddy +# N5 — app-side sanity check on the toolchain-provided Caddy binary. On the +# prebuilt path the "does it embed the fixed cel-go / grpc-go" assertions ran +# only when the toolchain image was built, so a wrong / rolled-back +# CHARON_TOOLCHAIN_DIGEST (or a hand-edited pin) would sail through silently. +# The final stage has no Go toolchain, so instead assert the binary loads and +# exposes the four custom plugins the recipe adds — a wrong-arch or stale-recipe +# image fails here immediately. The authoritative embed-version checks remain in +# caddy-inline (run by toolchain-image.yml) and docker-build.yml's post-build step. +RUN set -e; \ + mods="$(/usr/bin/caddy list-modules 2>/dev/null)"; \ + for m in http.handlers.rate_limit http.handlers.crowdsec http.handlers.geoip2 http.handlers.waf; do \ + printf '%s\n' "$mods" | grep -qx "$m" \ + || { echo "ERROR: toolchain caddy binary missing expected module: $m"; printf '%s\n' "$mods"; exit 1; }; \ + done; \ + echo "Verified toolchain caddy binary exposes rate_limit / crowdsec / geoip2 / waf(coraza)" + # Copy CrowdSec binaries from the crowdsec-builder stage (built with Go 1.26.3+) # This ensures we don't have stdlib vulnerabilities from older Go versions COPY --from=crowdsec-builder /crowdsec-out/crowdsec /usr/local/bin/crowdsec COPY --from=crowdsec-builder /crowdsec-out/cscli /usr/local/bin/cscli + +# N5 — app-side sanity check on the toolchain-provided cscli binary: it must run +# and emit its recognisable version block. (CrowdSec 1.8.x prints an empty +# `version:` field here regardless of the -X ldflag, so match a stable field +# instead.) A wrong-arch / stale-recipe image fails this immediately. +RUN set -e; \ + /usr/local/bin/cscli version >/tmp/cscli-v.txt 2>&1 \ + || { echo "ERROR: toolchain cscli is not runnable"; cat /tmp/cscli-v.txt; exit 1; }; \ + grep -q 'Constraint_api' /tmp/cscli-v.txt \ + || { echo "ERROR: toolchain cscli version output not recognised"; cat /tmp/cscli-v.txt; exit 1; }; \ + rm -f /tmp/cscli-v.txt; \ + echo "Verified toolchain cscli runs (GoVersion: $(/usr/local/bin/cscli version 2>&1 | sed -n 's/^GoVersion: //p'))" # Copy CrowdSec configuration files to .dist directory (will be used at runtime) COPY --from=crowdsec-builder /crowdsec-out/config /etc/crowdsec.dist # Verify config files were copied successfully diff --git a/docs/plans/current_spec.md b/docs/plans/current_spec.md index 17fe72b44..987445077 100644 --- a/docs/plans/current_spec.md +++ b/docs/plans/current_spec.md @@ -502,26 +502,50 @@ fi #### 3.4.3 Jobs +> **Amendment (Rev 2.1, post-approval — flagged for supervisor re-review).** +> BuildKit's default provenance / SBOM attestation manifests embed per-run +> timestamps + builder identity, so the OCI-index (manifest-list) digest of an +> otherwise byte-identical build changes on every run. Combined with +> `sync-pin-on-pr` + the path-filtered `pull_request` trigger this produced a +> self-perpetuating bot-commit loop on the feature PR. Fixes, all in this PR: +> +> 1. **Deterministic build.** `--provenance=false --sbom=false`, a **fixed** +> `SOURCE_DATE_EPOCH` (`1700000000`), and +> `--output type=image,"name=…:KEY,…:DATE,…:latest",push=true,rewrite-timestamp=true`. +> The toolchain image is an internal build *input*; the app image's own +> provenance/SBOM (in `docker-build.yml`) is separate and unaffected. Result: +> identical toolchain key ⇒ identical manifest-list digest (verified by two +> independent builds producing the same digest). +> 2. **Skip-if-already-published.** On any non-forced event (`pull_request`, +> plain path trigger) the job first `imagetools inspect`s `:${KEY}`; if it +> resolves, it SKIPS the build/push entirely and emits that existing digest. +> Only `schedule` / `workflow_dispatch force_rebuild=true` / `workflow_call` +> actually rebuild + repush. This also removes the ~30-min rebuild from +> unrelated Dockerfile PRs. +> 3. **`sync-pin-on-pr` is idempotent + self-trigger-safe:** guarded +> `github.actor != 'github-actions[bot]'` and no-ops unless +> `git diff --quiet Dockerfile` shows a real change after the sed. + ``` build-toolchain: - checkout - KEY=$(scripts/toolchain-key.sh); echo to $GITHUB_OUTPUT - Set up QEMU? NO. Set up Buildx. - login GHCR (skip on fork) - - docker buildx build + - PLAN: forced = (schedule || force_rebuild); if !forced && same-repo && + `imagetools inspect :${KEY}` resolves -> should_build=false, reuse that digest + - if should_build: SOURCE_DATE_EPOCH=1700000000 docker buildx build --target toolchain-runtime --platform linux/amd64,linux/arm64 - $( [[ force_rebuild || schedule ]] && echo --no-cache --pull ) + $( forced && echo --no-cache --pull ) + --provenance=false --sbom=false --cache-from type=gha,scope=toolchain --cache-to type=gha,mode=max,scope=toolchain - -t ghcr.io/wikid82/charon-toolchain:${KEY} - $( same-repo && echo -t ghcr.io/wikid82/charon-toolchain:latest ) - -t ghcr.io/wikid82/charon-toolchain:$(date +%Y%m%d) - $( same-repo && echo --push || echo --output=type=cacheonly ) - --iidfile /tmp/toolchain-iid.txt + --output type=image,"name=…:KEY,…:$(date +%Y%m%d)$( same-repo && echo ,…:latest )",push=$( same-repo && echo true || echo false via type=cacheonly ),rewrite-timestamp=true . - - DIGEST=$(regctl image digest ghcr.io/wikid82/charon-toolchain:${KEY}) - - outputs: key, digest + - DIGEST = existing_digest (if skipped) else + $(docker buildx imagetools inspect …:${KEY} --format '{{json .Manifest}}' | jq -r .digest) + - outputs: key, digest, same_repo trivy-scan: needs: build-toolchain @@ -531,11 +555,13 @@ trivy-scan: - continue-on-error on the gate step is FALSE on schedule/dispatch (must be clean), TRUE on PR (report-only; the app-image Trivy gates still run downstream) -sync-pin-on-pr: # only when event == pull_request && same-repo && pins moved +sync-pin-on-pr: # pull_request && same-repo && actor != github-actions[bot] needs: [build-toolchain] - sed -i "s|^ARG CHARON_TOOLCHAIN_TAG=.*|ARG CHARON_TOOLCHAIN_TAG=${KEY}|" Dockerfile - sed -i "s|^ARG CHARON_TOOLCHAIN_DIGEST=.*|ARG CHARON_TOOLCHAIN_DIGEST=${DIGEST}|" Dockerfile + - if `git diff --quiet Dockerfile`: exit 0 (no commit — idempotent) - git commit -m "chore(docker): sync toolchain image pin to ${KEY}" && git push (to PR head branch) + - ::notice:: re-run the freshness check (GITHUB_TOKEN pushes don't re-trigger PR checks) open-bump-pr: # event == schedule | workflow_dispatch | workflow_call ; NEVER on pull_request needs: [build-toolchain, trivy-scan] From ba0d9df6a97628081243fc5ab882290125cf473c Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Mon, 7 Sep 2026 20:33:53 -0400 Subject: [PATCH 05/19] perf(ci): stop forcing from-source rebuilds of the bundled toolchain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec §12 Commit 4. Now safe: (a) the default app-build path never compiles the inline stages, (b) verify-toolchain-pin is a live required check enforcing pin<->digest freshness per PR (Commit 3), (c) the daily toolchain rebuild + blocking Trivy gate covers base-image drift, (d) the N5 assertions catch a wrong digest in the app build. - docker-build.yml: remove the four `--no-cache-filter caddy-inline` / `crowdsec-inline` lines from build-amd64 / build-arm64; reword the retry-block comment. - security-pr.yml / supply-chain-pr.yml: drop the `no-cache-filters:` override (and its justification comment) from the build-charon-image `with:` block. - e2e-tests-split.yml / nightly-build.yml: drop the `no-cache-filters:` input. - build-charon-image composite: delete the `no-cache-filters` input declaration and its `docker/build-push-action` passthrough; rewrite `description:` to state the binaries come from the digest-pinned toolchain image and every stage is layer-cached. `grep -rn no-cache-filter .github/` is clean. actionlint + lefthook green. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .github/actions/build-charon-image/action.yml | 35 ++++--------------- .github/workflows/docker-build.yml | 11 +++--- .github/workflows/e2e-tests-split.yml | 1 - .github/workflows/nightly-build.yml | 1 - .github/workflows/security-pr.yml | 6 ---- .github/workflows/supply-chain-pr.yml | 6 ---- 6 files changed, 12 insertions(+), 48 deletions(-) diff --git a/.github/actions/build-charon-image/action.yml b/.github/actions/build-charon-image/action.yml index 30703a23b..ca988c88a 100644 --- a/.github/actions/build-charon-image/action.yml +++ b/.github/actions/build-charon-image/action.yml @@ -1,8 +1,12 @@ name: Build Charon image description: >- - Build the Charon Docker image locally for integration / security-scan jobs, - with GitHub Actions layer caching (type=gha) so the multi-stage build is not - rebuilt cold on every run. Loads the image into the local Docker daemon. + Build the Charon Docker image locally for integration / security-scan jobs. + The custom Caddy + CrowdSec binaries come from the digest-pinned, daily- + rebuilt-and-Trivy-scanned prebuilt toolchain image + (ghcr.io/wikid82/charon-toolchain) via COPY --from — no xcaddy / CrowdSec + compile on this path. Every remaining stage is GitHub Actions layer-cached + (type=gha). Pin freshness is enforced per-PR by + scripts/verify-toolchain-pin.sh. Loads the image into the local Docker daemon. inputs: tag: description: Image tag to load locally. @@ -30,30 +34,6 @@ inputs: a fork PR that cannot supply a working token should pass builder-src=inline. required: false default: '' - no-cache-filters: - description: >- - Comma-separated Dockerfile stages to force-rebuild (never restore from the - layer cache). Empty by default: every stage is GHA layer-cached — that is - where the build time is recovered. Suitable for the integration-test - callers (waf/crowdsec/rate-limit/cerberus), which exercise runtime - behaviour and do not care about dependency freshness. - - CVE-scan-gate callers (security-pr.yml, supply-chain-pr.yml) currently - override this with `caddy-inline,crowdsec-inline` (the RUN-bearing - from-source stages, renamed from caddy-builder/crowdsec-builder). Those - stages patch pinned transitive dependencies in-place (`go get pkg@fixed`), - and a global build-arg bump does not reliably invalidate the GHA - layer-cache key for a stage that only *consumes* that arg (the same edge - case that produced CVE-2026-45135 and the 2026-09-04 grpc-go v1.83.0 - recurrence). - - NOTE: on the default app-build path those stages are no longer compiled — - their output is COPY --from'd out of the digest-pinned, daily-rebuilt-and- - scanned toolchain image, and pin freshness is enforced per-PR by - scripts/verify-toolchain-pin.sh. This input is removed entirely in a later - commit once that guard is a required check. - required: false - default: '' runs: using: composite steps: @@ -98,4 +78,3 @@ runs: CROWDSEC_BUILDER_SRC=${{ steps.srcsel.outputs.crowdsec }} cache-from: type=gha,scope=charon-integration-image cache-to: type=gha,mode=max,scope=charon-integration-image - no-cache-filters: ${{ inputs.no-cache-filters }} diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index 2035a9243..f2ba56655 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -389,8 +389,11 @@ jobs: # today, which requires the raw `docker buildx build` CLI rather than the JS action. # Retry is therefore implemented the same way as today: raw buildx CLI wrapped in # nick-fields/retry at the step level (same action pin, same 3-attempts/10s-wait - # config), with cache-from/cache-to/no-cache-filter passed as native buildx flags - # (functionally identical to docker/build-push-action's equivalent inputs). + # config), with cache-from/cache-to passed as native buildx flags (functionally + # identical to docker/build-push-action's equivalent inputs). The Caddy/CrowdSec + # compile lives in the digest-pinned, daily-rebuilt-and-scanned toolchain image + # now, so normal type=gha layer caching is authoritative for every stage; pin + # freshness is enforced per-PR by scripts/verify-toolchain-pin.sh. build-amd64: needs: setup # An explicit job-level `if:` REPLACES (does not append to) the implicit @@ -464,8 +467,6 @@ jobs: "${TAG_ARGS[@]}" --cache-from type=gha,scope=docker-build-amd64 --cache-to type=gha,mode=max,scope=docker-build-amd64 - --no-cache-filter caddy-inline - --no-cache-filter crowdsec-inline --pull --build-arg "VERSION=${{ needs.setup.outputs.version }}" --build-arg "BUILD_DATE=${{ needs.setup.outputs.created }}" @@ -554,8 +555,6 @@ jobs: "${TAG_ARGS[@]}" --cache-from type=gha,scope=docker-build-arm64 --cache-to type=gha,mode=max,scope=docker-build-arm64 - --no-cache-filter caddy-inline - --no-cache-filter crowdsec-inline --pull --build-arg "VERSION=${{ needs.setup.outputs.version }}" --build-arg "BUILD_DATE=${{ needs.setup.outputs.created }}" diff --git a/.github/workflows/e2e-tests-split.yml b/.github/workflows/e2e-tests-split.yml index 711026ce5..669d0a855 100644 --- a/.github/workflows/e2e-tests-split.yml +++ b/.github/workflows/e2e-tests-split.yml @@ -236,7 +236,6 @@ jobs: build-args: | CADDY_BUILDER_SRC=${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'caddy-inline' || 'toolchain-prebuilt' }} CROWDSEC_BUILDER_SRC=${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'crowdsec-inline' || 'toolchain-prebuilt' }} - no-cache-filters: caddy-inline,crowdsec-inline - name: Save Docker image if: steps.resolve-image.outputs.image_source == 'build' diff --git a/.github/workflows/nightly-build.yml b/.github/workflows/nightly-build.yml index 717d84ae0..c73401e80 100644 --- a/.github/workflows/nightly-build.yml +++ b/.github/workflows/nightly-build.yml @@ -242,7 +242,6 @@ jobs: CROWDSEC_BUILDER_SRC=toolchain-prebuilt cache-from: type=gha cache-to: type=gha,mode=max - no-cache-filters: caddy-inline,crowdsec-inline provenance: true sbom: true diff --git a/.github/workflows/security-pr.yml b/.github/workflows/security-pr.yml index faca444f6..6157c1c42 100644 --- a/.github/workflows/security-pr.yml +++ b/.github/workflows/security-pr.yml @@ -161,12 +161,6 @@ jobs: # PRs (no private-package pull) compile caddy-inline/crowdsec-inline. builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} ghcr-token: ${{ secrets.GITHUB_TOKEN }} - # Still force-rebuild the from-source stages on the inline path so a - # stale GHA layer cannot retain a superseded pinned dependency (the - # `go get pkg@fixed` patch lives INSIDE the cached-and-skipped stage). - # Removed entirely in a later commit once verify-toolchain-pin is a - # required check and the default path no longer compiles at all. - no-cache-filters: caddy-inline,crowdsec-inline - name: Check for PR image artifact id: check-artifact diff --git a/.github/workflows/supply-chain-pr.yml b/.github/workflows/supply-chain-pr.yml index 088a5dc3d..50d5a90f1 100644 --- a/.github/workflows/supply-chain-pr.yml +++ b/.github/workflows/supply-chain-pr.yml @@ -256,12 +256,6 @@ jobs: # PRs (no private-package pull) compile caddy-inline/crowdsec-inline. builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} ghcr-token: ${{ secrets.GITHUB_TOKEN }} - # Still force-rebuild the from-source stages on the inline path so a - # stale GHA layer cannot retain a superseded pinned dependency (the - # `go get pkg@fixed` patch lives INSIDE the cached-and-skipped stage). - # Removed entirely in a later commit once verify-toolchain-pin is a - # required check and the default path no longer compiles at all. - no-cache-filters: caddy-inline,crowdsec-inline - name: Expose local image name if: github.event_name != 'workflow_run' From c1ee1074ecc07f35f0b2b4ae61edf33cc3bc6f9a Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Mon, 7 Sep 2026 21:01:19 -0400 Subject: [PATCH 06/19] ci: route the weekly security rebuild through the toolchain image MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec §12 Commit 5 / §3.4.4. security-weekly-rebuild.yml no longer builds a throwaway `charon:security-scan-YYYYMMDD` app image that nothing consumes. Instead: - `toolchain-rebuild` job — `uses: ./.github/workflows/toolchain-image.yml` (workflow_call, force_rebuild: true). This drives the same deterministic `--no-cache --pull` rebuild + publish + BLOCKING Trivy CRITICAL/HIGH gate + `:trivy-toolchain` SARIF + (if the digest moved) `bot/bump-toolchain-image` PR as the daily schedule. N6: the job grants `contents: write`, `pull-requests: write`, `issues: write` (plus packages / security-events) so the reusable workflow's open-bump-pr job has the perms it needs — a workflow_call'ed workflow cannot request perms the caller did not grant. - `extended-report` job — pulls the rebuilt toolchain digest and produces the fuller weekly artefacts: a MEDIUM/LOW Trivy JSON artifact (90-day retention) and an installed-package table in the step summary. No second SARIF upload (the reusable workflow already owns the `:trivy-toolchain` category). - toolchain-image.yml gains `workflow_call` `outputs.key` / `outputs.digest` so the caller can reference the rebuilt digest. actionlint + lefthook green. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .github/workflows/security-weekly-rebuild.yml | 187 ++++++------------ .github/workflows/toolchain-image.yml | 7 + 2 files changed, 62 insertions(+), 132 deletions(-) diff --git a/.github/workflows/security-weekly-rebuild.yml b/.github/workflows/security-weekly-rebuild.yml index 04cc5cf2b..54a21c40d 100644 --- a/.github/workflows/security-weekly-rebuild.yml +++ b/.github/workflows/security-weekly-rebuild.yml @@ -3,6 +3,13 @@ name: Weekly Security Rebuild # Note: This workflow filename has remained consistent. The related docker-publish.yml # was replaced by docker-build.yml in commit f640524b (Dec 21, 2025). # GitHub Advanced Security may show warnings about the old filename until its tracking updates. +# +# Since the prebuilt-toolchain-image change this workflow no longer builds a +# throwaway `charon:security-scan-YYYYMMDD` app image that nothing consumes. It +# drives the SAME deterministic `--no-cache --pull` rebuild + blocking Trivy +# CRITICAL/HIGH gate + digest-bump bot PR as the daily schedule, by +# `workflow_call`ing toolchain-image.yml, then adds the fuller MEDIUM/LOW JSON +# report artifact + installed-package table the weekly slot has always produced. on: schedule: @@ -22,180 +29,96 @@ concurrency: permissions: contents: read -env: - REGISTRY: ghcr.io - IMAGE_NAME: ${{ github.repository_owner }}/charon - # Canonical SARIF category owner for weekly rebuild Trivy uploads. - # Keep this stable to prevent duplicate/stale code scanning tracks. - TRIVY_SARIF_CATEGORY: .github/workflows/security-weekly-rebuild.yml:trivy-weekly - jobs: - security-rebuild: - name: Security Rebuild & Scan - runs-on: ubuntu-latest - timeout-minutes: 60 + # Deterministic --no-cache --pull rebuild + publish + blocking Trivy + # CRITICAL/HIGH gate + :trivy-toolchain SARIF + (if the digest moved) + # bot/bump-toolchain-image PR. N6: a workflow_call'ed workflow cannot request + # perms the caller did not grant, so grant everything toolchain-image.yml's + # open-bump-pr job needs. + toolchain-rebuild: + name: Toolchain rebuild + scan + uses: ./.github/workflows/toolchain-image.yml permissions: - contents: read + contents: write packages: write security-events: write - + pull-requests: write + issues: write + with: + # The weekly slot always forces a clean deterministic rebuild. + force_rebuild: true + + extended-report: + name: Extended Trivy report (MEDIUM/LOW + package table) + needs: toolchain-rebuild + runs-on: ubuntu-latest + timeout-minutes: 20 + permissions: + contents: read + packages: read + env: + TOOLCHAIN_REF: ghcr.io/${{ github.repository_owner }}/charon-toolchain@${{ needs.toolchain-rebuild.outputs.digest }} steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: - # Explicitly fetch the current HEAD of the ref at run time, not the - # SHA that was frozen when this scheduled job was queued. Without this, - # a queued job can run days later with stale code. ref: ${{ github.ref_name }} - - name: Normalize image name - run: | - echo "IMAGE_NAME=$(echo "${{ env.IMAGE_NAME }}" | tr '[:upper:]' '[:lower:]')" >> "$GITHUB_ENV" - - - name: Set up QEMU - uses: docker/setup-qemu-action@1f40c72289eff860ee54a304f1438e3cff362e0a # v4.3.0 - - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0 - - - name: Resolve Debian base image digest - id: base-image + - name: Normalize image ref run: | - docker pull debian:trixie-slim - DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' debian:trixie-slim) - echo "digest=$DIGEST" >> "$GITHUB_OUTPUT" - echo "Base image digest: $DIGEST" + echo "TOOLCHAIN_REF=$(echo "$TOOLCHAIN_REF" | tr '[:upper:]' '[:lower:]')" >> "$GITHUB_ENV" - name: Log in to Container Registry uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 with: - registry: ${{ env.REGISTRY }} + registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - - name: Extract metadata - id: meta - uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0 - with: - images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} - tags: | - type=raw,value=security-scan-{{date 'YYYYMMDD'}} - - - name: Build Docker image (NO CACHE) - id: build - uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7 - with: - context: . - platforms: linux/amd64 - push: true - tags: ${{ steps.meta.outputs.tags }} - labels: ${{ steps.meta.outputs.labels }} - no-cache: ${{ github.event_name == 'schedule' || inputs.force_rebuild }} - pull: true # Always pull fresh base images to get latest security patches - build-args: | - VERSION=security-scan - BUILD_DATE=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.created'] }} - VCS_REF=${{ github.sha }} - BASE_IMAGE=${{ steps.base-image.outputs.digest }} - - - name: Run Trivy vulnerability scanner (CRITICAL+HIGH) - uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 - with: - image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }} - format: 'table' - severity: 'CRITICAL,HIGH' - exit-code: '1' # Fail workflow if vulnerabilities found - version: 'v0.74.0' - continue-on-error: true - - - name: Run Trivy vulnerability scanner (SARIF) - id: trivy-sarif - uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 - with: - image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }} - format: 'sarif' - output: 'trivy-weekly-results.sarif' - severity: 'CRITICAL,HIGH,MEDIUM' - version: 'v0.74.0' - trivyignores: '.trivyignore' - - - name: Upload Trivy results to GitHub Security - id: upload-trivy-weekly - uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9 - with: - sarif_file: 'trivy-weekly-results.sarif' - category: ${{ env.TRIVY_SARIF_CATEGORY }} - - - name: Verify SARIF was uploaded - if: always() - run: | - set -euo pipefail - if [ ! -f trivy-weekly-results.sarif ]; then - echo "::error::SARIF file trivy-weekly-results.sarif was not produced. Check Trivy scan step." - exit 1 - fi - - if [ "${{ steps.upload-trivy-weekly.outcome }}" != "success" ]; then - echo "::error::SARIF upload step did not succeed (outcome: ${{ steps.upload-trivy-weekly.outcome }})" - exit 1 - fi - - RESULT_COUNT=$(jq '[.runs[].results // [] | .[]] | length' trivy-weekly-results.sarif 2>/dev/null || echo "parse_error") - if [ "$RESULT_COUNT" = "parse_error" ]; then - echo "::error::SARIF file is not valid JSON or missing expected structure." - exit 1 - fi - - echo "SARIF upload verified. Finding count: ${RESULT_COUNT}" - echo "SARIF_RESULT_COUNT=${RESULT_COUNT}" >> "$GITHUB_STEP_SUMMARY" - - - name: Run Trivy vulnerability scanner (JSON for artifact) + - name: Run Trivy (JSON, all severities) for the artifact uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 with: - image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }} + image-ref: ${{ env.TOOLCHAIN_REF }} format: 'json' - output: 'trivy-weekly-results.json' + output: 'trivy-toolchain-weekly.json' severity: 'CRITICAL,HIGH,MEDIUM,LOW' version: 'v0.74.0' - name: Upload Trivy JSON results uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: - name: trivy-weekly-scan-${{ github.run_number }} - path: trivy-weekly-results.json + name: trivy-toolchain-weekly-${{ github.run_number }} + path: trivy-toolchain-weekly.json retention-days: 90 - - name: Check Debian package versions + - name: Installed package versions (key security packages) run: | { - echo "## 📦 Installed Package Versions" + echo "## 📦 Bundled toolchain — key package versions" echo "" - echo "Checking key security packages:" echo '```' - docker run --rm --entrypoint "" "${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}" \ - sh -c "dpkg -l | grep -E 'libc-ares|curl|libcurl|openssl|libssl' || echo 'No matching packages found'" + docker run --rm --entrypoint "" "${TOOLCHAIN_REF}" \ + sh -c "apk info -v 2>/dev/null | grep -E 'zlib|openssl|libcrypto|libssl|musl|xz' || echo 'no matching apk packages (scratch/distroless layers)'" \ + || echo "toolchain image has no shell — inspect via the JSON artifact" echo '```' } >> "$GITHUB_STEP_SUMMARY" - - name: Create security scan summary + - name: Weekly rebuild summary if: always() run: | { - echo "## 🔒 Weekly Security Rebuild Complete" + echo "## 🔒 Weekly Toolchain Rebuild Complete" echo "" - echo "- **Build Date:** $(date -u +"%Y-%m-%d %H:%M:%S UTC")" - echo "- **Image:** ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}" - echo "- **SARIF Category:** ${{ env.TRIVY_SARIF_CATEGORY }}" - echo "- **Cache Used:** No (forced fresh build)" - echo "- **Trivy Scan:** Completed (see Security tab for details)" + echo "- **Date:** $(date -u +"%Y-%m-%d %H:%M:%S UTC")" + echo "- **Toolchain key:** \`${{ needs.toolchain-rebuild.outputs.key }}\`" + echo "- **Digest:** \`${{ needs.toolchain-rebuild.outputs.digest }}\`" + echo "- **Blocking CRITICAL/HIGH gate:** ran in the reusable toolchain-image workflow" + echo "- **SARIF:** uploaded under category .github/workflows/toolchain-image.yml:trivy-toolchain" echo "" - echo "### Next Steps:" - echo "1. Review Security tab for new vulnerabilities" - echo "2. Check Trivy JSON artifact for detailed package info" - echo "3. If critical CVEs found, trigger production rebuild" + echo "If the digest moved, bot/bump-toolchain-image has been opened." } >> "$GITHUB_STEP_SUMMARY" - - name: Notify on security issues (optional) + - name: Notify on failure if: failure() run: | - echo "::warning::Weekly security scan found HIGH or CRITICAL vulnerabilities. Review the Security tab." + echo "::warning::Weekly toolchain rebuild / extended scan failed. Review the Security tab and the run log." diff --git a/.github/workflows/toolchain-image.yml b/.github/workflows/toolchain-image.yml index e8c66233d..b24938d58 100644 --- a/.github/workflows/toolchain-image.yml +++ b/.github/workflows/toolchain-image.yml @@ -44,6 +44,13 @@ on: publish: type: boolean default: true + outputs: + key: + description: The content-addressed toolchain tag (caddy-crowdsec-). + value: ${{ jobs.build-toolchain.outputs.key }} + digest: + description: The published manifest-list (OCI index) digest. + value: ${{ jobs.build-toolchain.outputs.digest }} concurrency: group: toolchain-image-${{ github.ref }} From 01f5ba405c9bd5323ecf21da7c312a36faa8d123 Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Mon, 7 Sep 2026 21:05:38 -0400 Subject: [PATCH 07/19] ci: document the toolchain image, right-size build timeouts, sweep stale comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec §12 Commit 6 / §3.9 / §9. Timeouts (§3.9): - docker-build.yml build-amd64: 15 -> 20 (job + nested nick-fields/retry). No compile on the hot path; 20 gives headroom for a cold GHA cache miss on the fast stages + cache export + push. build-arm64 stays 25; merge-and-publish stays 10; e2e build stays 60. - security-pr / supply-chain-pr / the 4 *-integration jobs KEEP timeout-minutes: 20 (B6 — fork-reachable, they run the ~14-min inline compile). Comments only. Stale-comment sweep: - docker-build.yml: reworded the build-split comment; dropped the dangling `docs/plans/current_spec.md §1.1` cross-ref (pointed at the retired uptime spec) and the QEMU-emulated-compile framing (both builder stages were always $BUILDPLATFORM cross-compiled). - e2e-tests-split.yml: fixed the malformed `# v4uses: docker/setup-buildx-action@…` comment mash on the Set up Docker Buildx step. - Integration + CVE-gate timeout comments reworded to describe the prebuilt vs fork-inline paths. Docs (§9): - ARCHITECTURE.md: Infrastructure table row; new "Prebuilt toolchain image" subsection under Deployment Architecture / Multi-Stage Dockerfile; Directory Structure note (new scripts/workflow, crowdsec-fallback removed, stage renames); Security Architecture / Layer 2 supply-chain-hardening note with the precise §3.8.3 scope; Local Development Setup build note + make build-offline. - SECURITY.md: new "Build Integrity — Bundled Caddy / CrowdSec Toolchain" section (daily deterministic rebuild + blocking Trivy + failure-closed verify-toolchain-pin; unpinned-transitive-MVS gap stated verbatim, no overclaim). - CONTRIBUTING.md: fork PRs compile the toolchain from source; make build-offline. - new docs/ci/toolchain-image.md: operator/maintainer runbook (how the key works, determinism, triggers, force-rebuild, respond to the failure issue / verify-toolchain-pin failure, rollback, one-time bootstrap notes). Ignore-file check (CLAUDE.md): `Dockerfile: COPY scripts/` does copy the new shell scripts into the runtime image (a few KB), consistent with the dozens of scripts/*.sh already shipped — no .dockerignore change. .gitignore: source files that must be committed, no glob match. .codecov.yml: shell/bats/YAML carry no Go/TS coverage — no change. Recorded in docs/ci/toolchain-image.md and the PR. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .github/workflows/cerberus-integration.yml | 2 +- .github/workflows/crowdsec-integration.yml | 2 +- .github/workflows/docker-build.yml | 15 ++-- .github/workflows/e2e-tests-split.yml | 2 +- .github/workflows/rate-limit-integration.yml | 2 +- .github/workflows/security-pr.yml | 2 +- .github/workflows/supply-chain-pr.yml | 2 +- .github/workflows/waf-integration.yml | 2 +- ARCHITECTURE.md | 77 ++++++++++++++++++ CONTRIBUTING.md | 16 ++++ SECURITY.md | 34 ++++++++ docs/ci/toolchain-image.md | 86 ++++++++++++++++++++ 12 files changed, 229 insertions(+), 13 deletions(-) create mode 100644 docs/ci/toolchain-image.md diff --git a/.github/workflows/cerberus-integration.yml b/.github/workflows/cerberus-integration.yml index 143adb3bc..5bcb3998b 100644 --- a/.github/workflows/cerberus-integration.yml +++ b/.github/workflows/cerberus-integration.yml @@ -27,7 +27,7 @@ jobs: cerberus-integration: name: Cerberus Security Stack Integration runs-on: ubuntu-latest - timeout-minutes: 20 + timeout-minutes: 20 # 20m: same-repo runs COPY Caddy/CrowdSec from the pinned toolchain image (~2-4m build); fork PRs compile them inline (~14m) + test work, which sets the floor (B6). steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 diff --git a/.github/workflows/crowdsec-integration.yml b/.github/workflows/crowdsec-integration.yml index 9d2280bde..d7945b103 100644 --- a/.github/workflows/crowdsec-integration.yml +++ b/.github/workflows/crowdsec-integration.yml @@ -27,7 +27,7 @@ jobs: crowdsec-integration: name: CrowdSec Bouncer Integration runs-on: ubuntu-latest - timeout-minutes: 20 # 20m: warm GHA cache builds in ~8-12m; first run on a fresh branch / after cache eviction is still a full cold build. + timeout-minutes: 20 # 20m: same-repo runs COPY Caddy/CrowdSec from the pinned toolchain image (~2-4m build); fork PRs compile them inline (~14m) + test work, which sets the floor (B6). steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index f2ba56655..8915a5b6f 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -377,10 +377,13 @@ jobs: # `build-amd64` / `build-arm64` build+push a single platform each to a throwaway, # run-scoped tag (never exposed to end users), then `merge-and-publish` composes both - # digests into one multi-platform manifest list. This isolates arm64's slow - # QEMU-emulated cross-compile from amd64's fast native build so each gets its own - # right-sized timeout instead of sharing one 20-minute budget (root cause of the - # arm64 timeout flake this restructuring fixes — see docs/plans/current_spec.md §1.1). + # digests into one multi-platform manifest list, so each per-arch leg gets its own + # right-sized timeout instead of sharing one budget. The custom Caddy/CrowdSec + # compile no longer runs on either leg — both COPY --from the digest-pinned + # toolchain image (see docs/plans/current_spec.md §3.1); the builder stages were + # always $BUILDPLATFORM cross-compiled (never QEMU-emulated) regardless. QEMU on + # build-arm64 still runs only the final arm64 stage's RUN lines (apk, setcap, + # GeoIP fetch, verification). # # NOTE on retry mechanism: docker/build-push-action cannot be nested inside # nick-fields/retry (retry's `command:` only accepts a shell string, not a `uses:` @@ -407,7 +410,7 @@ jobs: CADDY_BUILDER_SRC: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'caddy-inline' || 'toolchain-prebuilt' }} CROWDSEC_BUILDER_SRC: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'crowdsec-inline' || 'toolchain-prebuilt' }} runs-on: ubuntu-latest - timeout-minutes: 15 + timeout-minutes: 20 permissions: contents: read packages: write @@ -445,7 +448,7 @@ jobs: id: build uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0 with: - timeout_minutes: 15 + timeout_minutes: 20 max_attempts: 3 retry_wait_seconds: 10 retry_on: error diff --git a/.github/workflows/e2e-tests-split.yml b/.github/workflows/e2e-tests-split.yml index 669d0a855..5812b6a39 100644 --- a/.github/workflows/e2e-tests-split.yml +++ b/.github/workflows/e2e-tests-split.yml @@ -210,7 +210,7 @@ jobs: - name: Set up Docker Buildx if: steps.resolve-image.outputs.image_source == 'build' - uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 + uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0 - name: Log in to GHCR (pull prebuilt toolchain image) if: steps.resolve-image.outputs.image_source == 'build' && github.event.pull_request.head.repo.full_name == github.repository diff --git a/.github/workflows/rate-limit-integration.yml b/.github/workflows/rate-limit-integration.yml index a1d88899c..1bad9e9f1 100644 --- a/.github/workflows/rate-limit-integration.yml +++ b/.github/workflows/rate-limit-integration.yml @@ -27,7 +27,7 @@ jobs: rate-limit-integration: name: Rate Limiting Integration runs-on: ubuntu-latest - timeout-minutes: 20 # 20m: warm GHA cache builds in ~8-12m; first run on a fresh branch / after cache eviction is still a full cold build. + timeout-minutes: 20 # 20m: same-repo runs COPY Caddy/CrowdSec from the pinned toolchain image (~2-4m build); fork PRs compile them inline (~14m) + test work, which sets the floor (B6). steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 diff --git a/.github/workflows/security-pr.yml b/.github/workflows/security-pr.yml index 6157c1c42..be1da3d68 100644 --- a/.github/workflows/security-pr.yml +++ b/.github/workflows/security-pr.yml @@ -30,7 +30,7 @@ jobs: security-scan: name: Trivy Binary Scan runs-on: ubuntu-latest - timeout-minutes: 20 # 20m: cold GHA cache (first run / post-eviction) is a full ~10-14m image build; warm-cache builds are far quicker. + timeout-minutes: 20 # 20m: same-repo warm build ~6-8m (toolchain image is COPY --from, not compiled); fork PRs compile the toolchain inline (~14m) + scan, which sets the floor (B6). # Run for manual dispatch, direct PR/push, or successful upstream workflow_run if: >- github.event_name == 'workflow_dispatch' || diff --git a/.github/workflows/supply-chain-pr.yml b/.github/workflows/supply-chain-pr.yml index 50d5a90f1..34927169b 100644 --- a/.github/workflows/supply-chain-pr.yml +++ b/.github/workflows/supply-chain-pr.yml @@ -32,7 +32,7 @@ jobs: verify-supply-chain: name: Verify Supply Chain runs-on: ubuntu-latest - timeout-minutes: 20 # 20m: cold GHA cache (first run / post-eviction) is a full ~10-14m image build; warm-cache builds are far quicker. + timeout-minutes: 20 # 20m: same-repo warm build ~6-8m (toolchain image is COPY --from, not compiled); fork PRs compile the toolchain inline (~14m) + scan, which sets the floor (B6). # Run for: manual dispatch, or successful workflow_run triggered by push/PR if: > github.event_name == 'workflow_dispatch' || diff --git a/.github/workflows/waf-integration.yml b/.github/workflows/waf-integration.yml index c3dbccc9a..b455efc5f 100644 --- a/.github/workflows/waf-integration.yml +++ b/.github/workflows/waf-integration.yml @@ -27,7 +27,7 @@ jobs: waf-integration: name: Coraza WAF Integration runs-on: ubuntu-latest - timeout-minutes: 20 # 20m: warm GHA cache builds in ~8-12m; first run on a fresh branch / after cache eviction is still a full cold build. + timeout-minutes: 20 # 20m: same-repo runs COPY Caddy/CrowdSec from the pinned toolchain image (~2-4m build); fork PRs compile them inline (~14m) + test work, which sets the floor (B6). steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 6b1102aa5..134cbc655 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -163,6 +163,7 @@ graph TB | **Base Image** | Debian Trixie Slim | Latest | Security-hardened base | | **CI/CD** | GitHub Actions | N/A | Automated testing and deployment | | **Registry** | Docker Hub + GHCR | N/A | Image distribution | +| **Bundled proxy toolchain** | `ghcr.io/wikid82/charon-toolchain` | digest-pinned | Multi-arch prebuilt custom Caddy + CrowdSec binaries; deterministic daily `--no-cache --pull` rebuild + blocking Trivy gate; digest pinned in `Dockerfile` and freshness-guarded per-PR | | **Security Scanning** | Trivy + Grype + Semgrep | Latest | Vulnerability detection | | **SBOM Generation** | Syft | Latest | Software Bill of Materials | | **Signature Verification** | Cosign | Latest | Supply chain integrity | @@ -305,8 +306,17 @@ graph TB - **`.docker/`**: All Docker-related files (prevents root clutter) - **`docs/implementation/`**: Archived implementation documentation - **`docs/plans/`**: Active planning documents (`current_spec.md`) +- **`docs/ci/`**: CI/build operator runbooks (`toolchain-image.md`) - **`test-results/`**: Test artifacts (gitignored) +**Bundled-toolchain tooling** (see Deployment Architecture → Prebuilt toolchain image): +`.github/workflows/toolchain-image.yml`, `scripts/toolchain-key.sh`, +`scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh`. The +`Dockerfile` `crowdsec-fallback` stage was removed (dead code); the +`caddy-builder` / `crowdsec-builder` stages were renamed `caddy-inline` / +`crowdsec-inline` and are now compiled only by the toolchain workflow and the +fork/offline fallback. + --- ## Core Components @@ -794,6 +804,19 @@ graph LR - **Local Only:** No external API calls - **API Mode:** Sync with CrowdSec cloud for global intelligence +**Supply-chain hardening:** the CrowdSec agent + `cscli` and the bouncer-enabled +Caddy binary are built from source with pinned, in-place-patched transitive +dependencies and shipped via the scanned, digest-pinned toolchain image +(`ghcr.io/wikid82/charon-toolchain`). The CVE-2026-84304-class recurrence +guarantee ("upstream ships a fix, no repo pin changes") is mechanised by: the +**daily** deterministic `--no-cache --pull` toolchain rebuild + **blocking** +Trivy gate + digest-bump bot PR, plus the per-PR `verify-toolchain-pin` check for +tracked-pin bumps. This covers pinned-dependency and base-image drift; it does +**not** close the pre-existing gap for a security fix to a genuinely *unpinned* +transitive Go dependency where nothing raises the MVS lower bound — that is +unchanged from before and is closed only by adding an explicit `go get dep@fixed` +pin (the stage already carries ~40 such pins). + ### Layer 3: Access Control Lists (ACL) **Purpose:** IP-based access control @@ -1121,6 +1144,47 @@ ENTRYPOINT ["/docker-entrypoint.sh"] CMD ["/app/charon"] ``` +> The real `Dockerfile` is a much larger multi-stage build. The illustrative +> snippet above is schematic only. + +#### Prebuilt toolchain image + +Charon ships a **custom Caddy v2 binary** (built with `xcaddy` + in-place +transitive-dependency security patches) and a **custom CrowdSec agent** built +from source. Compiling both takes ~14 minutes, so it is **not** done on every app +image build. Instead: + +- The recipe lives in the main `Dockerfile` as the `caddy-inline` / + `crowdsec-inline` stages (single source of truth). +- `.github/workflows/toolchain-image.yml` builds `--target toolchain-runtime` + into `ghcr.io/wikid82/charon-toolchain` — a **multi-arch** (`linux/amd64` + + `linux/arm64`) manifest list, cross-compiled without QEMU. The build is + **deterministic** (`--provenance=false --sbom=false`, fixed + `SOURCE_DATE_EPOCH`, `rewrite-timestamp`): an unchanged recipe reproduces an + identical manifest-list digest. +- Triggers: a **daily** `--no-cache --pull` schedule, `workflow_dispatch`, a + `pull_request` touching a tracked input, and a `workflow_call` from + `security-weekly-rebuild.yml`. Each publish runs a Trivy CRITICAL/HIGH gate + (blocking off the PR path). +- The app `Dockerfile` pins `ARG CHARON_TOOLCHAIN_DIGEST` (manifest-list digest) + and `COPY --from`s `/usr/bin/caddy` + `/crowdsec-out/{crowdsec,cscli,config}` + out of it. Every app-build workflow logs in to GHCR (`packages: read`) to pull + it; no `xcaddy` / CrowdSec compile runs on the app hot path. +- **Freshness guard:** `scripts/toolchain-key.sh` derives a content-addressed tag + (`caddy-crowdsec-`) from the two inline stage bodies + every consumed + version ARG (incl. the two pinned xcaddy plugins) + the digest-pinned + `golang`/`xx` bases + `.trivyignore`. `scripts/verify-toolchain-pin.sh` runs on + every PR (required check) and is **failure-closed** on trusted same-repo runs: + it fails if the Dockerfile's pinned tag/digest is stale for the current + recipe. When the daily rebuild produces a new digest, a bot opens + `bot/bump-toolchain-image` (base `development`). +- **Fork PR / bootstrap / offline fallback:** pass + `--build-arg CADDY_BUILDER_SRC=caddy-inline --build-arg CROWDSEC_BUILDER_SRC=crowdsec-inline` + (or `make build-offline`) to compile the byte-for-byte-identical recipe from + source instead of pulling the image. + +Operator runbook: `docs/ci/toolchain-image.md`. + ### Port Mapping | Port | Protocol | Purpose | Bind | @@ -1244,6 +1308,19 @@ services: # Frontend + Backend + Caddy in one container ``` +6. **Building the container image locally:** + + `docker build .` pulls the digest-pinned toolchain image + (`ghcr.io/wikid82/charon-toolchain`, ~30 MB) for the custom Caddy/CrowdSec + binaries — a `docker login ghcr.io` is required (the package is private). + Offline / air-gapped, or to compile the binaries from source instead: + + ```bash + make build-offline + # == docker build --build-arg CADDY_BUILDER_SRC=caddy-inline \ + # --build-arg CROWDSEC_BUILDER_SRC=crowdsec-inline . + ``` + ### Git Workflow **Branch Strategy:** diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 422b85340..35efc7198 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -261,6 +261,22 @@ git merge upstream/development git push origin development ``` +### Container builds on fork PRs + +CI builds the container image by pulling a digest-pinned prebuilt toolchain image +(`ghcr.io/wikid82/charon-toolchain`) that carries the custom Caddy/CrowdSec +binaries. That image is private to this repo, so **fork PRs cannot pull it** — +their CI compiles the Caddy/CrowdSec binaries from source instead (the +byte-for-byte identical recipe, ~14 minutes slower per image build). This is +automatic; you don't need to do anything. A maintainer re-running the trusted +same-repo checks exercises the fast prebuilt path before merge. + +To build locally without the pull (offline, or not logged in to GHCR): + +```bash +make build-offline +``` + ## Coding Standards ### Go Backend diff --git a/SECURITY.md b/SECURITY.md index 829612448..fca033d34 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -25,6 +25,40 @@ public disclosure. --- +## Build Integrity — Bundled Caddy / CrowdSec Toolchain + +Charon ships a custom Caddy v2 binary (built with `xcaddy` + in-place +transitive-dependency security patches) and a custom CrowdSec agent, built from +source. Both are produced by `.github/workflows/toolchain-image.yml` into the +**digest-pinned, multi-arch** image `ghcr.io/wikid82/charon-toolchain` and +`COPY --from`'d into the app image — the compile does **not** run on the app +build hot path. + +Keeping those bundled binaries patched is mechanised by: + +- a **daily** deterministic `--no-cache --pull` rebuild of the toolchain image + with a **blocking** Trivy CRITICAL/HIGH gate; a new digest or finding opens a + `feat(security)` bot PR and a failure opens a tracked issue; +- `scripts/verify-toolchain-pin.sh` — a **required, failure-closed** per-PR check + that fails if the Dockerfile's pinned toolchain tag/digest is stale for the + current pinned inputs (the two inline stage bodies, every consumed version ARG + including the two now-pinned xcaddy plugins, the digest-pinned `golang`/`xx` + bases, and `.trivyignore`). + +**Scope, stated precisely.** This covers **pinned-dependency** drift (any tracked +ARG or literal `go get x@vN` in the recipe) and **base-image** drift. It does +**not** close the pre-existing gap where an upstream *security fix to a +genuinely unpinned transitive Go dependency* is not picked up because nothing +raises the MVS lower bound — that is unchanged, and is closed only by a human +adding an explicit `go get @` pin (the recipe already carries ~40). + +Fork PRs, first-run bootstrap, and offline builds compile the byte-for-byte +identical recipe from source (`--build-arg CADDY_BUILDER_SRC=caddy-inline +--build-arg CROWDSEC_BUILDER_SRC=crowdsec-inline`, or `make build-offline`) — not +weaker, just slower and unpinned. See `docs/ci/toolchain-image.md`. + +--- + ## Known Vulnerabilities Last reviewed: 2026-09-02 diff --git a/docs/ci/toolchain-image.md b/docs/ci/toolchain-image.md new file mode 100644 index 000000000..87e617eb6 --- /dev/null +++ b/docs/ci/toolchain-image.md @@ -0,0 +1,86 @@ +# Bundled proxy toolchain image — operator runbook + +`ghcr.io/wikid82/charon-toolchain` is a multi-arch prebuilt image holding the +custom **Caddy v2** binary (built with `xcaddy` + in-place transitive-dependency +security patches) and the custom **CrowdSec** agent (`crowdsec` + `cscli`). The +app `Dockerfile` `COPY --from`s these instead of recompiling them (~14 min) on +every CI image build. + +## How it fits together + +| Piece | Role | +|---|---| +| `Dockerfile` `caddy-inline` / `crowdsec-inline` stages | The single source of truth for the build recipe. Compiled only by the toolchain workflow and the fork/offline fallback. | +| `Dockerfile` `toolchain-runtime` stage | `docker buildx build --target toolchain-runtime` → the publishable image. | +| `Dockerfile` `ARG CHARON_TOOLCHAIN_TAG` / `CHARON_TOOLCHAIN_DIGEST` | The pin. `DIGEST` is the arch-independent manifest-list (OCI index) digest. Bot-owned — do **not** hand-edit. | +| `Dockerfile` `ARG CADDY_BUILDER_SRC` / `CROWDSEC_BUILDER_SRC` | Selector: `toolchain-prebuilt` (default) or `caddy-inline` / `crowdsec-inline`. | +| `scripts/toolchain-key.sh` | Prints `caddy-crowdsec-<16hex>` — a content hash of the two inline stage bodies + every consumed version ARG (incl. the two pinned xcaddy plugins) + the digest-pinned `golang`/`xx` bases + `.trivyignore` + a `SCHEMA_VERSION`. | +| `scripts/verify-toolchain-pin.sh` | Required PR check. Failure-closed on trusted same-repo runs: needs `regctl` + `GHCR_READ_TOKEN`, resolves `:$KEY` in GHCR, and asserts the pinned digest matches. A fork PR degrades to a tag-only check with a `::warning::`. | +| `scripts/lib/dockerfile-stage.sh` | Shared `extract_stage` used by both scripts. | +| `.github/workflows/toolchain-image.yml` | Builds / publishes / scans. | +| `.github/workflows/security-weekly-rebuild.yml` | `workflow_call`s the above for the Tuesday full rebuild + MEDIUM/LOW JSON report. | + +## Determinism + +The build sets `--provenance=false --sbom=false`, a **fixed** +`SOURCE_DATE_EPOCH` (`1700000000`), and `rewrite-timestamp=true`. An unchanged +recipe (same `toolchain-key.sh` output) therefore always reproduces an +**identical** manifest-list digest. This is what stops `sync-pin-on-pr` from +looping. The toolchain image is an internal build *input* — the app image's own +provenance/SBOM attestations (in `docker-build.yml`) are separate and unaffected. + +## Triggers + +| Event | Behaviour | +|---|---| +| `schedule` (daily 06:00 UTC) | `--no-cache --pull` deterministic rebuild + blocking Trivy CRITICAL/HIGH gate + `:trivy-toolchain` SARIF. If the digest moved → opens `bot/bump-toolchain-image` (base `development`). | +| `workflow_dispatch` (`force_rebuild=true` default) | Same as `schedule`. Use this for an urgent out-of-band refresh. | +| `pull_request` touching a tracked input | **Skip-if-already-published:** if `:$KEY` already exists in GHCR, reuse that digest and do not rebuild. Same-repo PRs that genuinely moved a pin get the two-line `TAG`/`DIGEST` bump pushed onto the PR branch by `sync-pin-on-pr` (then re-run the `verify-toolchain-pin` check — `GITHUB_TOKEN` pushes don't auto-retrigger). Forks build `cacheonly` (recipe-compile validation only). | +| `workflow_call` (from `security-weekly-rebuild.yml`) | Forced deterministic rebuild + full report. | + +## Common tasks + +### Force an immediate refresh (e.g. an urgent upstream fix in a pinned dep) + +1. Bump the relevant `ARG` in the `Dockerfile` (this changes `toolchain-key.sh`). +2. `gh workflow run toolchain-image.yml -f force_rebuild=true` — publishes + `:` and (off the PR path) opens `bot/bump-toolchain-image`. +3. Merge the bot PR (or, on your own PR, let `sync-pin-on-pr` bump the pin and + re-run the freshness check). + +### Respond to the daily-rebuild failure issue (`🚨 Toolchain image rebuild ... failed`) + +1. Open the linked run. The blocking Trivy CRITICAL/HIGH gate or a compile + failure is the usual cause. +2. For a new CRITICAL/HIGH in a bundled binary: add an explicit + `go get @` pin in the relevant inline stage (the stages already + carry ~40), or a justified `.trivyignore` entry with an `exp:` review date. +3. `workflow_dispatch` the workflow again; merge the resulting bot PR. + +### Respond to a `verify-toolchain-pin` PR failure + +The message tells you the recomputed key vs the pinned tag. Either: +- your PR legitimately changed a tracked input → `workflow_dispatch` + `toolchain-image.yml` (or push and let `sync-pin-on-pr` handle it), then bump + `CHARON_TOOLCHAIN_TAG` / `CHARON_TOOLCHAIN_DIGEST`; or +- the pin was hand-edited / is stale → restore it to the bot-owned value. + +### Roll back the whole feature + +Revert the single merged commit. The `charon-toolchain` package stays in GHCR +unreferenced (`container-prune.yml` ages it out). The `Dockerfile` returns to +inline `caddy-builder` / `crowdsec-builder` with `--no-cache-filter`; +`security-weekly-rebuild.yml` returns to its prior behaviour. Security posture is +identical to before, and the app image content is byte-identical (same recipe). + +## One-time bootstrap notes + +- The GHCR package `charon-toolchain` is created on the first publish and is + **private**. GHCR auto-links packages pushed by Actions to the source repo, so + same-repo workflows pull it with `GITHUB_TOKEN` + `permissions: packages: read` + + an explicit `docker login ghcr.io`. A maintainer may optionally flip it to + public — not required. +- `Dockerfile:` `COPY scripts/ /app/scripts/` copies the new shell scripts into + the runtime image (a few KB), consistent with the dozens of `scripts/*.sh` + already shipped. No `.dockerignore` / `.gitignore` / `.codecov.yml` change is + needed (shell/bats/YAML carry no Go/TS coverage). From 75ca9c8c84629d7389700399bc2d88e95d1545ac Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Mon, 7 Sep 2026 21:33:50 -0400 Subject: [PATCH 08/19] chore(docker): exclude build-only scripts and bats tests from the image context MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Dockerfile` `COPY scripts/ /app/scripts/` copies the whole scripts directory into the runtime image. The toolchain helpers added by this PR are used only by .github/workflows/toolchain-image.yml and the quality-checks.yml verify-toolchain-pin / bats jobs from a plain checkout — never from inside a built container — so `.dockerignore` now excludes: scripts/tests/ scripts/toolchain-key.sh scripts/verify-toolchain-pin.sh scripts/lib/dockerfile-stage.sh Verified via a scratch `COPY scripts/` probe: those paths are absent from the build context while runtime scripts (e.g. db-recovery.sh) remain. `.dockerignore` uses blacklist semantics with no `!scripts/…` re-includes, so the plain excludes take effect. Also corrects the stale premise in docs/plans/current_spec.md §2.5 that claimed `scripts/` is not copied into the image. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .dockerignore | 11 +++++++++++ docs/plans/current_spec.md | 2 +- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/.dockerignore b/.dockerignore index fd65d4dc9..7816ea67f 100644 --- a/.dockerignore +++ b/.dockerignore @@ -158,6 +158,17 @@ docs/implementation/ # ----------------------------------------------------------------------------- dist/ +# ----------------------------------------------------------------------------- +# Build-only helper scripts — used by .github/workflows/toolchain-image.yml and +# the quality-checks.yml verify-toolchain-pin / bats jobs from a plain checkout, +# never from inside a built container. The Dockerfile's `COPY scripts/` would +# otherwise ship them into /app/scripts/ in the runtime image. +# ----------------------------------------------------------------------------- +scripts/tests/ +scripts/toolchain-key.sh +scripts/verify-toolchain-pin.sh +scripts/lib/dockerfile-stage.sh + # ----------------------------------------------------------------------------- # Tools (not needed in image) # ----------------------------------------------------------------------------- diff --git a/docs/plans/current_spec.md b/docs/plans/current_spec.md index 987445077..5563182f5 100644 --- a/docs/plans/current_spec.md +++ b/docs/plans/current_spec.md @@ -161,7 +161,7 @@ The composite action's own doc comment (`action.yml:16-33`) instructs CVE-scan c - Conventional commits; `(security)` scope only for genuine security work, subject line vague. The digest-bump and freshness-guard commits *are* security-relevant — use `feat(security):` / `fix(security):` with vague subjects (e.g. `feat(security): pin bundled proxy toolchain to a scanned prebuilt image`). - Weekly `nightly → main` promotion PRs merge via **merge commit**. This feature's PR targets `development` (normal flow) — **confirmed it does not touch `weekly-nightly-promotion.yml`** and imposes no new constraint on the promotion merge method. (`weekly-nightly-promotion.yml` carries the app image through unchanged; the toolchain digest pin travels with the Dockerfile like any other line.) - `ARCHITECTURE.md` §"Deployment Architecture / Multi-Stage Dockerfile" (`:1082`), §"Infrastructure" table (`:158`), §"Directory Structure" (`:286`), §"Layer 2: CrowdSec Integration" (`:780`) must be updated (§9). -- **Ignore-file check (CLAUDE.md "Ignore Files"):** the new files are `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh`, `scripts/tests/toolchain-key.bats`, `.github/workflows/toolchain-image.yml`, `docs/ci/toolchain-image.md`. `.dockerignore` already excludes `.github/`, `docs/`, `scripts/` is not copied into the image context by any `COPY` (the Dockerfile only `COPY`s `backend/`, `frontend/`, `.docker/`) → **no `.dockerignore` change needed**. `.gitignore` — these are source files that must be committed; none matches an existing ignore glob (`scripts/tests/` is new, not ignored) → **no `.gitignore` change needed**. `.codecov.yml` — shell/bats and YAML carry no Go/TS coverage; not in any coverage path → **no `.codecov.yml` change needed**. This is recorded explicitly per CLAUDE.md. +- **Ignore-file check (CLAUDE.md "Ignore Files"):** the new files are `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh`, `scripts/tests/toolchain-key.bats` (+ `verify-toolchain-pin.bats`, `helpers/toolchain_fixture.bash`), `.github/workflows/toolchain-image.yml`, `docs/ci/toolchain-image.md`. **Correction (Rev 2.1):** the earlier claim that `scripts/` is not copied into the image was wrong — `Dockerfile` `COPY scripts/ /app/scripts/` copies the whole directory into the runtime image (it already ships ~40 `scripts/*.sh` + a pre-existing `.bats`). These four build-only helpers are used only by `toolchain-image.yml` and the `quality-checks.yml` `verify-toolchain-pin` / bats jobs from a plain checkout — never from inside a built container — so **`.dockerignore` now excludes `scripts/tests/`, `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh`** (blacklist semantics, no `!scripts/…` re-includes to fight). `.github/` and `docs/` are already excluded, so `toolchain-image.yml` / `docs/ci/toolchain-image.md` never enter the context. `.gitignore` — these are source files that must be committed; none matches an existing ignore glob → **no `.gitignore` change**. `.codecov.yml` — shell/bats and YAML carry no Go/TS coverage → **no `.codecov.yml` change**. Recorded explicitly per CLAUDE.md. --- From e3b76d0438ea5f06e2e1f8654e3599cb3cadc523 Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Tue, 8 Sep 2026 02:41:29 -0400 Subject: [PATCH 09/19] docs(ci): reconcile toolchain-image bootstrap notes with the image-context exclusions - docs/ci/toolchain-image.md: the "one-time bootstrap notes" paragraph predated the `.dockerignore` follow-up and still said the new build-only scripts ship in the runtime image. Corrected: `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh` and `scripts/tests/` are excluded from the image build context via `.dockerignore`. - docs/security.md: added a one-line pointer from the novice-facing security doc to the new "Build Integrity" section in SECURITY.md. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- docs/ci/toolchain-image.md | 12 ++++++++---- docs/security.md | 4 ++++ 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/ci/toolchain-image.md b/docs/ci/toolchain-image.md index 87e617eb6..bed05d24e 100644 --- a/docs/ci/toolchain-image.md +++ b/docs/ci/toolchain-image.md @@ -80,7 +80,11 @@ identical to before, and the app image content is byte-identical (same recipe). same-repo workflows pull it with `GITHUB_TOKEN` + `permissions: packages: read` + an explicit `docker login ghcr.io`. A maintainer may optionally flip it to public — not required. -- `Dockerfile:` `COPY scripts/ /app/scripts/` copies the new shell scripts into - the runtime image (a few KB), consistent with the dozens of `scripts/*.sh` - already shipped. No `.dockerignore` / `.gitignore` / `.codecov.yml` change is - needed (shell/bats/YAML carry no Go/TS coverage). +- `Dockerfile:` `COPY scripts/ /app/scripts/` copies the whole `scripts/` + directory into the runtime image. The build-only helpers this feature adds — + `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, + `scripts/lib/dockerfile-stage.sh`, and `scripts/tests/` — are **excluded from + the image context via `.dockerignore`** (they run from a plain checkout in + `toolchain-image.yml` / `quality-checks.yml`, never from inside a container). + No `.gitignore` change (source files, must be committed); no `.codecov.yml` + change (shell/bats/YAML carry no Go/TS coverage). diff --git a/docs/security.md b/docs/security.md index 26c0cffb5..814fb4c8a 100644 --- a/docs/security.md +++ b/docs/security.md @@ -11,6 +11,10 @@ You can disable it in **System Settings → Optional Features** if you don't nee Want the quick reference? See . +> The Caddy and CrowdSec software bundled inside Charon is kept patched +> automatically. How that works is described in +> [SECURITY.md → Build Integrity](../SECURITY.md#build-integrity--bundled-caddy--crowdsec-toolchain). + --- ## What Is Cerberus? From ea5680e107f96ca480d7f928f2bd14d913ab387b Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Tue, 8 Sep 2026 03:06:46 -0400 Subject: [PATCH 10/19] docs(qa): replace QA report with prebuilt toolchain image audit --- docs/reports/qa_report.md | 508 ++++++++++++++++++++++++-------------- 1 file changed, 324 insertions(+), 184 deletions(-) diff --git a/docs/reports/qa_report.md b/docs/reports/qa_report.md index 001141696..8e23ebcc7 100644 --- a/docs/reports/qa_report.md +++ b/docs/reports/qa_report.md @@ -1,222 +1,362 @@ -# QA & Security Report — Uptime Monitoring at Scale +# QA & Security Report — Prebuilt Caddy/CrowdSec Toolchain Image -**Branch**: `feat/uptime-monitoring-scale` (16 commits ahead of `development`) +**PR**: #1300 — `feat(ci): prebuilt Caddy/CrowdSec toolchain image to fix Docker-build timeouts` +**Branch**: `feat/prebuilt-toolchain-image` → base `main` (draft) +**Branch tip audited**: `22e9c722` (working tree clean, rebased on `origin/main` `cc65e634`) **Reviewed by**: qa-security agent (final pipeline pass) -**Date**: 2026-08-27 -**Plan**: `docs/plans/current_spec.md` -**Prior review**: `docs/reports/supervisor_review.md` (implementation verdict: APPROVE) -**Diff**: 59 files, +11354 / −1307 (`git diff development...feat/uptime-monitoring-scale`) +**Date**: 2026-09-08 +**Spec**: `docs/plans/current_spec.md` (Rev 2 + §3.4.3 "Rev 2.1") --- -## Final verdict: **PASS** — ready to merge +## Verdict: PASS WITH FOLLOW-UPS -No blocking security or quality issues. All Definition-of-Done gates pass with real -numbers below. Three of the supervisor's six nice-to-haves were fixed in this pass -(comment-only, commit `b59115ab`); the remaining three are documented with rationale -for deferral. One local-environment note (stale git worktrees polluting the Trivy -scan — not part of this branch) is called out under Trivy. +Clear to bring PR #1300 out of draft. No blocking security or QA issues. Four +non-blocking follow-ups and two residual supply-chain risks the merger should +accept knowingly (enumerated at the end). ---- - -## 1. Security Audit - -### 1.1 SSRF — shared keep-alive HTTP client (`network/safeclient.go` + `uptime_check.go`) — PASS +The change is CI/build-infrastructure only — no Go or TypeScript application code, +no `backend/internal/models/**`, no GORM queries, no migrations, no frontend +surface. The new executable code is three shell scripts covered by a 17-test bats +suite. All CI checks on the tip are green. -| Check | Result | -|---|---| -| `safeDialer` re-resolves + re-validates the destination IP on **every new connection** | CONFIRMED. `safeDialer(&cfg)` is wired as `Transport.DialContext`; it runs `net.DefaultResolver.LookupIPAddr` + `IsPrivateIP`/`IsRFC1918` gating per dial. `WithKeepAlive` does **not** touch it. | -| Pooled idle connections can't reach a rebind-poisoned host | CONFIRMED. Keep-alive only flips `DisableKeepAlives` / `MaxIdleConns` (100) / `MaxIdleConnsPerHost` (4) / `IdleConnTimeout` (30s). An established TCP socket cannot be re-bound to a new IP; a reused idle conn keeps talking to the already-validated peer. `idleConnTimeout = 30s` bounds the revalidation-staleness window (spec §3.2.2 / N2). | -| Link-local / `169.254.169.254` / cloud-metadata / other reserved ranges blocked with keep-alive ON | CONFIRMED. `WithAllowRFC1918()` whitelists **only** `10/8`, `172.16/12`, `192.168/16` (`rfc1918CIDRs`). `169.254.0.0/16`, loopback (unless `WithAllowLocalhost`), `0.0.0.0/8`, `240.0.0.0/4`, `::1`, `fc00::/7`, `fe80::/10`, and Go's `IsLinkLocal*`/`IsMulticast`/`IsUnspecified` fast-path all still reject. Verified by `safeclient_test.go` (link-local + metadata blocked with keep-alive on; idle conn past `idleTimeout` not reused; redirects still not followed). | -| Redirects not followed | CONFIRMED. `newUptimeChecker` passes `WithMaxRedirects(0)` → `CheckRedirect` returns `http.ErrUseLastResponse`. Unchanged by keep-alive. | -| Layer-1 `ValidateExternalURL` still called per HTTP check | CONFIRMED. `uptimeChecker.probe` calls `security.ValidateExternalURL(monitor.URL, WithAllowLocalhost, WithAllowHTTP, WithTimeout(3s), WithAllowRFC1918)` before every `http/https` request. | -| Double DNS lookup (Layer 1 validate + Layer 2 dial) NOT collapsed | CONFIRMED. The `probe` comment explicitly notes "double-DNS accepted, spec §3.2.4". Layer 1 resolves in `ValidateExternalURL`; Layer 2 resolves again in `safeDialer`. The deliberate redundancy is intact. | +--- -TCP monitors dial `host:port` directly without URL validation — this is unchanged -legacy behaviour, scoped to admin-configured `RemoteServer` targets built from trusted -fields, and `RFC1918` is intentionally permitted there. No regression. +## 1. Build-integrity / CVE-recurrence guarantee — VERIFIED -### 1.2 SQL injection — raw SQL in the new query paths — PASS +`--no-cache-filter caddy-inline,crowdsec-inline` (the CVE-2026-84304 recurrence +guard that forced from-source rebuilds on every CVE-gate PR) is removed. Every +compensating link claimed in the brief exists and is wired: -| Location | Statement | Binding | +| Link | Where | Enforced? | |---|---|---| -| `uptime_pruner.go:178-185` | `DELETE FROM uptime_heartbeats WHERE id IN (SELECT id … WHERE created_at < ? ORDER BY id LIMIT ?)` | `cutoff` (`time.Time`) and `pruneChunkSize` (compile-time const `5000`) are both `?` binds. No `fmt.Sprintf`/concat. | -| `uptime_pruner.go:218` | `CREATE INDEX IF NOT EXISTS idx_heartbeat_monitor_created ON uptime_heartbeats (monitor_id, created_at)` | Static string literal, no interpolation. | -| `uptime_summary_service.go:172-181` (`recentBeatsSQL`) | `ROW_NUMBER() OVER (PARTITION BY monitor_id ORDER BY created_at DESC)` windowed, `WHERE created_at >= ?` … `WHERE rn <= ?` | `windowStart` and `uptimeSummaryMaxBeats` (const `60`) are `?` binds via `db.Raw(sql, windowStart, 60)`. The user's `?beats=` value never reaches SQL — it only slices the cached Go slice. | -| `uptime_summary_service.go:209-214` (`uptime24hSQL`) | grouped 24h up-ratio, `WHERE created_at >= ?` | `windowStart` `?`-bound. | -| `uptime_service.go:1272` `GetMonitorHistory` (`before` cursor) | `s.DB.Where("monitor_id = ?", id).Where("created_at < ?", before)` | GORM parameterised; `before` is a parsed `time.Time`, `id` a `?` bind. | +| (a) Content-hash key | `scripts/toolchain-key.sh` — SHA-256 over both inline stage bodies + 16 consumed version ARGs + the two pinned xcaddy plugins + `tonistiigi/xx` pin + digest-pinned `golang:*-alpine` bases + `sha256(.trivyignore)` + `SCHEMA_VERSION=2` | Yes — 10 bats tests assert determinism + per-input sensitivity + fail-loud on broken extraction | +| (b) Daily forced rebuild | `toolchain-image.yml` `on.schedule: '0 6 * * *'`; plan step sets `no_cache="--no-cache --pull"` for `schedule`/`force_rebuild`; `--target toolchain-runtime` recompiles `caddy-inline` + `crowdsec-inline` from source | Yes | +| (c) `verify-toolchain-pin.sh` per-PR check | `quality-checks.yml` job `verify-toolchain-pin` (installs regctl, maps trust env, runs the script) | Yes — passing on the tip; failure-closed on same-repo (see §2). **Branch-protection required-status enrolment is a merger check — see follow-up F1.** | +| (d) LABEL ↔ recipe-key check | `docker-build.yml` step "Verify pinned toolchain image matches the recipe (N5)" — `docker pull @PIN_DIGEST`, reads `io.charon.toolchain.key` LABEL, compares to freshly recomputed `toolchain-key.sh` | Yes | +| (e) Blocking weekly Trivy CRITICAL/HIGH | `toolchain-image.yml` `trivy-scan` job: `severity: CRITICAL,HIGH`, `exit-code: '1'`, `continue-on-error: ${{ github.event_name == 'pull_request' }}`. `security-weekly-rebuild.yml` reaches it via `workflow_call`, where `github.event_name` resolves to the caller's `schedule`/`workflow_dispatch` → `continue-on-error:false` → blocking | Yes | + +**Additional independent link** not in the brief: `docker-build.yml` `build-amd64` +/ `build-arm64` and `nightly-build.yml` pull the toolchain by **immutable +`@sha256:` digest** (`FROM ${CHARON_TOOLCHAIN_IMAGE}@${CHARON_TOOLCHAIN_DIGEST}`), +then the merged app image is Syft-SBOM'd, `actions/attest`-attested, Trivy- and +Grype-scanned, and Cosign-signed — so any poisoned content still has to survive +the app-image scan gates. + +**Doc-overclaim check — PASS.** Both `SECURITY.md` ("Build Integrity — Bundled +Caddy / CrowdSec Toolchain") and `ARCHITECTURE.md` ("Supply-chain hardening" +callout) retain the caveat verbatim and accurately: + +> "…it does **not** close the pre-existing gap where an upstream security fix to a +> genuinely *unpinned* transitive Go dependency is not picked up because nothing +> raises the MVS lower bound — that is unchanged, and is closed only by a human +> adding an explicit `go get @` pin (the recipe already carries ~40)." + +Both files scope the guarantee to "pinned-dependency drift and base-image drift" +only. No overclaim. `docs/ci/toolchain-image.md` "Roll back the whole feature" +correctly states security posture and app-image content are byte-identical to the +pre-PR inline path. -No dynamic SQL string construction anywhere in the new paths. +--- -### 1.3 Authorization — 3 new endpoints — PASS +## 2. `verify-toolchain-pin.sh` robustness — FAILURE-CLOSED, no bypass found + +**Trust classification** (`SAME_REPO`): defaults to `1` (trusted / failure-closed) +and only degrades to `0` (tag-only + `::warning::`) when +`GITHUB_EVENT_NAME == pull_request` **and** +`GITHUB_EVENT_PULL_REQUEST_HEAD_REPO_FULL_NAME != GITHUB_REPOSITORY`. Unknown / +unset → trusted. This is the safe polarity: the only way to *reach* the degraded +path is to be a genuine fork PR (whose token cannot read the private package +anyway); anything ambiguous fails closed. + +**Bypass attempts:** + +- **Env-var spoofing of `GITHUB_EVENT_PULL_REQUEST_HEAD_REPO_FULL_NAME`** — the + two env vars are mapped in each workflow from trusted GitHub contexts + (`${{ github.event.pull_request.head.repo.full_name }}`, `${{ github.repository }}`), + not from anything a fork PR author controls. A fork PR cannot alter the base + workflow that runs. A same-repo branch *could* edit `quality-checks.yml` to + mis-map them, but that requires write access (already a trusted actor) and is + visible in the PR diff. Not a new weakness. +- **Fork → convince script it's same-repo** — would only make it *stricter* + (failure-closed digest check); the fork runner has no `GITHUB_TOKEN` with + `packages:read` on the base repo, so it fails closed. No trust gained. +- **Same-repo → convince script it's a fork** — needs `head.repo.full_name != + repository`, impossible for a real same-repo PR without editing the workflow + (trusted-actor, diff-visible). +- **Hand-edited `CHARON_TOOLCHAIN_DIGEST` that still passes** — on the trusted + path the script resolves `:$KEY` via `regctl image digest` and requires + `REMOTE_DIGEST == PINNED_DIGEST`. The only digest that passes is the one GHCR + actually serves for that content-addressed tag. Defence-in-depth: `docker-build.yml` + LABEL check rejects a digest that points at a *different-recipe* toolchain + image. +- **TOCTOU between `imagetools inspect` / `regctl image digest` and the app + build's `FROM …@digest`** — not exploitable for injection. The app build + consumes an immutable `@sha256:` reference; re-tagging `:$KEY` afterwards cannot + change what `@digest` resolves to. Worst case is a spurious check failure + (false positive), never a silent poisoned pull. + +**`bats scripts/tests/` result: 17/17 PASS** (local, `Bats 1.13.0`; also green in +CI job "Toolchain key / freshness-guard scripts (bats)"). + +Failure modes **actually asserted** by `verify-toolchain-pin.bats`: + +| Assertion | Covered | +|---|---| +| fork PR + matching tag → `exit 0` + `::warning::Fork PR` | ✅ | +| mismatched tag (any trust level) → `exit 1`, actionable message | ✅ | +| same-repo `push` + `regctl` absent → `exit 1` (failure-closed) | ✅ | +| same-repo `push` + `GHCR_READ_TOKEN` unset → `exit 1` (failure-closed) | ✅ | +| same-repo PR + GHCR digest ≠ pinned digest → `exit 1` ("hand-edited or stale") | ✅ | +| same-repo PR + GHCR digest == pinned digest → `exit 0` ("verified (same-repo)") | ✅ | +| `workflow_dispatch` treated as trusted same-repo (fails closed on missing regctl) | ✅ | + +Failure-closed branches present in the script but **not** directly asserted (see +follow-up F2): + +- `PINNED_DIGEST` empty on a same-repo run → `exit 1`. +- `:$KEY` present but `regctl image digest` returns non-zero (unresolvable in + GHCR) → `exit 1` ("does not resolve in GHCR"). The bats `regctl` stub always + succeeds, so this specific exit path is uncovered. + +`toolchain-key.bats` (10 tests) covers determinism, whitespace-stability of edits +*outside* the two inline stages, and sensitivity to: a `go get` line inside +`caddy-inline`, `CADDY_VERSION`, `CADDY_GEOIP2_VERSION` (B4 plugin pin), the +digest-pinned `golang` base (N4), and `.trivyignore`; plus two fail-loud cases +(stage removed, stage truncated to a stub). Not asserted: sensitivity to a +`tonistiigi/xx` pin move, an `ALPINE_IMAGE` move, or a `SCHEMA_VERSION` bump — +all three *are* in the hashed input set; the gap is test-only (F2). -`routes.go:641-651` registers all three inside the `management` group (same -`RequireManagementAccess()` JWT/role guard as every existing `/uptime/*` route): +--- -- `GET /uptime/monitors/summary` → `uptimeHandler.Summary` -- `GET /uptime/monitors/:id/history` → `uptimeHandler.GetHistory` -- `GET /uptime/health` → `uptimeHandler.Health` +## 3. Determinism fix (§3.4.3 Rev 2.1) — does NOT weaken app-image posture + +`toolchain-image.yml` builds with `--provenance=false --sbom=false`, fixed +`SOURCE_DATE_EPOCH=1700000000`, `--output type=image,push=true,rewrite-timestamp=true`. + +- **App-image supply-chain posture is unaffected.** Verified: `docker-build.yml` + `merge-and-publish` generates the app image's own SBOM (`anchore/sbom-action` + syft `v1.51.1`, with a pinned-syft fallback), attests it (`actions/attest` + `v4.2.2`), and Cosign-signs the merged digest — all against the final `charon` + app-image digest, not the toolchain image. `nightly-build.yml` retains + `provenance: true` / `sbom: true`. `grep` across `.github/workflows/` for any + consumer of the toolchain image's attestations: **none** — nothing runs + `cosign verify-attestation` / SBOM-diff against `charon-toolchain`. Disabling + provenance/SBOM on an internal build *input* that nobody verifies is correct; + it is what makes "same recipe key ⇒ identical manifest-list digest" hold and + stops `sync-pin-on-pr` from looping. + +- **"Skip build if `:$KEY` already published"** (`toolchain-image.yml` "Decide + build plan"): on a **non-forced same-repo** run, if + `imagetools inspect ${TOOLCHAIN_IMAGE}:${KEY}` resolves, `should_build=false` + and the existing digest is reused / pinned. This does trust the current + content of the mutable `:$KEY` tag. Mitigations: (i) writing that tag requires + `packages: write` on the package = trusted maintainer; fork PRs never reach + this path (no login → `type=cacheonly`); (ii) the daily run is `forced=true`, + which bypasses skip-if-published, rebuilds deterministically, and — via + `open-bump-pr` — surfaces any digest discrepancy as a `feat(security)` bot PR, + so a poisoned tag self-heals within ~24 h; (iii) the app build ultimately pins + an immutable `@digest` and the resulting app image is Trivy/Grype-scanned and + signed. **Residual R1** (accept-knowingly): a maintainer-level credential + compromise could, within a one-day window, get a poisoned `:$KEY` digest + pinned via `sync-pin-on-pr` without that PR performing a from-source rebuild. + The pre-PR `--no-cache-filter` behaviour rebuilt from source on every CVE-gate + PR; this PR trades that for the daily deterministic rebuild + freshness guard. + +- **Who holds `packages: write` on `ghcr.io/wikid82/charon-toolchain`:** + - `toolchain-image.yml` → job `build-toolchain` (workflow-level + `permissions: packages: write`). GHCR login **and** push are gated + `if: steps.trust.outputs.same_repo == 'true'`; forks produce `type=cacheonly` + (no push). + - `security-weekly-rebuild.yml` → job `toolchain-rebuild`, which is + `uses: ./.github/workflows/toolchain-image.yml` with + `permissions: packages: write` (+ `contents/pull-requests/issues: write` for + the bump-PR job). Same underlying workflow; caller event is + `schedule`/`workflow_dispatch` ⇒ same-repo. + - `docker-build.yml` (`build-amd64`/`build-arm64`/`merge-and-publish`), + `nightly-build.yml`, `orthrus-build.yml` hold `packages: write` but target + the `charon` / `charon-agent` images — they only **read** (`FROM …@digest`) + the toolchain image, never push to it. + No fork-reachable job can write the toolchain package. -`/uptime/health` response body is exactly `{heartbeats_dropped, checks_enqueue_dropped, -queue_depth, worker_pool_size}` — four integer back-pressure counters. No monitor URLs, -target hosts, tokens, DB errors, or internal IPs. Nil-safe when the pool/ingester -haven't started (returns 0). Acceptable to expose to management-authenticated callers. +--- -### 1.4 Input validation — PASS +## 4. New third-party action `iarekylew00t/regctl-installer` — verified, low risk + +`quality-checks.yml` `verify-toolchain-pin` job: +`uses: iarekylew00t/regctl-installer@c2202c17a65fe59371c71ecc169c9e58c3710a15 # v4.0.16` + +- **SHA ↔ release: VERIFIED.** Annotated tag `v4.0.16` → tag object + `f14118b1…` → **points at commit `c2202c17a65fe59371c71ecc169c9e58c3710a15`** + (message "chore: Bumping version to v4.0.16", tagger 2026-08-05). The pin is + exact and matches the tag comment. +- **Action source at the pinned SHA:** `action.yml` is a compiled + `using: node24` / `main: dist/index.js` action. Declared purpose: download the + `regctl` release (default `latest` — here left default, so it resolves newest + at run time) and, with `verify: true` (default, left on), **cosign-verify the + downloaded binary's signature**. Inputs are `regctl-release`, `verify`, + `cache`, `token` (`${{ github.token }}` default) — all consistent with a + GitHub-API release downloader; nothing in `action.yml` indicates behaviour + beyond install + verify. `dist/index.js` is a minified bundle and was not + line-audited. +- **Blast radius:** runs only in the `verify-toolchain-pin` job, whose + `permissions` are `contents: read` + `packages: read` — no write scope, no + secrets beyond the read-only `GITHUB_TOKEN`. +- **`curl | sha256sum -c` vs this action:** an inline pinned-hash install would + remove a compiled-JS third-party action from the trust chain, but it also + drops the cosign signature check the action performs and needs manual hash + bumps (staleness risk). Given the minimal job permissions, **not materially + safer** — noting it (F3) as an optional hardening, not a defect. If adopted, + pin `regctl-release` to an exact version too (currently `latest`). -| Path | Enforcement | -|---|---| -| `SettingsHandler.UpdateSetting` (`uptime.*`) | `validateUptimeSetting` — `default_interval_seconds` ∈ [30, 86400], `worker_pool_size` ∈ [1, 200], `heartbeat_retention_days` ∈ [1, 3650]; non-integer or unknown `uptime.*` key → 400 `invalid_uptime_setting`. | -| `UptimeHandler.Create` | `0 < interval < 30` → 400 "interval must be at least 30 seconds"; `interval == 0` deferred to `CreateMonitor` write-time default; `type` bound `oneof=http tcp https`. | -| `UptimeService.UpdateMonitor` | positive sub-30 `interval` → `ErrIntervalTooLow` → 400; non-positive left for `clampInterval`; field whitelist (`max_retries`, `interval`, `enabled` only). | -| Auto-create sync paths (`SyncMonitors`, `SyncAndCheckForHost`, `SyncAndCheckForRemoteServer`) | create with `Interval: 0` → `clampInterval(0, cfg)` resolves the admin default; scheduler re-clamps every interval at scheduling time. | -| `Summary` `?beats=` | handler clamps to [1, 60]; service `clampBeats` clamps again. | -| `GetHistory` `?limit=` | non-positive/unparseable → default 60; service caps at `uptimeHistoryMaxLimit = 500`. | -| `GetHistory` `?before=` | non-empty + not RFC3339 → **400 "before must be an RFC3339 timestamp"** (not silently ignored). Empty → no cursor filter. | +--- -### 1.5 Resource exhaustion / DoS — PASS +## 5. Private `charon-toolchain` — every build path covered, forks still build -| Component | Control | -|---|---| -| Worker pool | `jobs` channel bounded at `uptimeQueueCapacity = 512`; `TryEnqueue` is non-blocking `select … default` → drop + `enqDropped` metric; scheduler leaves the job due and retries. No unbounded goroutines. Worker count from config, clamped 1-200. | -| Ingester | `results` channel bounded (`uptimeChannelBufferSize`); `Send` non-blocking `select … default` → `noteDropped(1)` + rate-limited warning + `DroppedCount()`. Batch discarded (and counted) after repeated flush failures so a DB fault can't stall the pipeline. | -| Pruner | 5000-row chunks; `time.Sleep` between chunks (50 ms steady / 250 ms until first clean pass); `ctx.Err()` check between chunks so it can be cut at any boundary; `wal_checkpoint(TRUNCATE)` only after ≥50k rows. Deliberately **not** in the ordered drain chain — never holds the single write connection across a shutdown. | -| Scheduler | `uptimeSchedulerMaxEnqueuePerTick = 200` caps host + monitor enqueues per tick (`hostPass` and `monitorPass` both truncate). Cold-start / past-due rows get a `jitterDuration` (crypto/rand, deterministic `maxD/2` fallback) spread over `uptimeSchedulerBackfillWindow` — prevents a restart stampede. | -| Summary endpoint | `recentBeatsSQL` window is always `now − 24h` and the per-monitor row cap is always the const `60` — **neither is user-controllable**. `loadMonitors` capped at 500. 30 s TTL cache in front. `?beats=` only trims the cached Go slice. Cannot be widened into an unbounded scan via query params. | - -### 1.6 Log injection — the two `go/log-injection` suppressions (`3c9ac3bd`) — PASS (legitimate) - -| Suppression entry | Sink | Logged value | Assessment | -|---|---|---|---| -| `remote_server_handler.go:142` | `logger.Log().WithError(syncErr).WithField("remote_server_id", id).Warn(...)` in the `Update` sync goroutine | `id uint` (bound from `server.ID`, a GORM numeric PK) | True false-positive. A Go `uint` cannot carry CR/LF/control chars. CodeQL flags it only because the enclosing `Update` also calls `c.ShouldBindJSON`, tainting the whole `server` struct; two taint paths reach the one line, so the SARIF has the finding twice. | -| `uptime_service.go:1475` | `logger.Log().WithField("remote_server_id", remoteServerID).Debug(...)` in `SyncAndCheckForRemoteServer` | `remoteServerID uint` argument (from `server.ID` at the `go …(server.ID)` call site) | Same — `uint`, no injectable payload. | - -Both YAML entries are well-formed: `rule_id`, `path`, `line`, `reason`, `added: 2026-08-27`, -`review_by: 2026-11-27` (not expired). Both have the primary in-source -`// codeql[go/log-injection]` annotation on the standalone line immediately above the -sink. A fresh local SARIF scan does not populate `result.suppressions` (documented -local-CLI limitation — see the pre-existing `go/cookie-secure-not-set` entry), which is -exactly what the machine-enforced ignore-list fallback exists for. - -### 1.7 Secrets / data exposure — PASS - -No monitor URLs, targets, tokens, or internal IPs newly logged at `info` or exposed in -an unauthenticated response. New info-level logs carry only counts -(`deleted`, `host_count`, `monitor_count`), string names, and status strings. The -pruner/ingester/scheduler log drop *totals*, never payloads. `/uptime/health` is -management-gated and body is counters only (§1.3). No Gotify tokens in logs, test -artifacts, screenshots, API examples, or URL query strings. - -### 1.8 GORM — PASS - -`./scripts/scan-gorm-security.sh --check`: **0 CRITICAL, 0 HIGH, 0 MEDIUM**, 2 pre-existing -INFO suggestions on `models/user.go` (missing FK indexes — not in this branch's scope). -No new raw-string query building. The ingester's coalesced `UPDATE` is -`tx.Model(&models.UptimeMonitor{}).Where("id = ?", id).Updates(map[string]any{...})` — -scoped by primary key, values passed as a bind map, one row per monitor. +**`uses: ./.github/actions/build-charon-image` — 6 call sites, all pass both +`builder-src` (fork ternary) and `ghcr-token`:** ---- +| Workflow | `builder-src` | `ghcr-token` | +|---|---|---| +| `security-pr.yml:158` | fork ternary → `inline` \| `prebuilt` | `secrets.GITHUB_TOKEN` | +| `supply-chain-pr.yml:253` | fork ternary | `secrets.GITHUB_TOKEN` | +| `cerberus-integration.yml:35` | fork ternary | `secrets.GITHUB_TOKEN` | +| `crowdsec-integration.yml:35` | fork ternary | `secrets.GITHUB_TOKEN` | +| `waf-integration.yml:35` | fork ternary | `secrets.GITHUB_TOKEN` | +| `rate-limit-integration.yml:35` | fork ternary | `secrets.GITHUB_TOKEN` | -## 2. Definition of Done — verification (re-run, real numbers) +All six also gained `permissions: packages: read`. The composite action logs in +to GHCR only `if: inputs.builder-src != 'inline' && inputs.ghcr-token != ''`, and +rejects an invalid `builder-src` with `exit 1`. -| Gate | Command | Result | +**Raw `docker buildx build` / `build-push-action` app-image paths:** + +| Workflow / job | Toolchain source | GHCR login | |---|---|---| -| Backend build | `cd backend && go build ./...` | **PASS** (exit 0) | -| Frontend build | `cd frontend && npm run build` | **PASS** (built in 2.23 s, exit 0) | -| Frontend type-check | `cd frontend && npm run type-check` | **PASS** (`tsc --noEmit`, exit 0) | -| Full backend suite | `cd backend && go test ./... -count=1` | **PASS** — every package `ok`, exit 0 | -| Flake check (`8b2277bb`) | `go test ./internal/api/handlers -run TestRemoteServerHandler -count=1` ×5 | **PASS 5/5** — `TestRemoteServerHandler_Update_SyncsLinkedMonitor` stable | -| Backend coverage | `CHARON_MIN_COVERAGE=85 bash scripts/go-test-coverage.sh` | **PASS** — statement 92.0 %, **line 88.7 %** (≥ 85 %) | -| Frontend coverage | `bash scripts/frontend-test-coverage.sh` | **PASS** — **lines 90.86 %** (statements 89.66 %, gate min 87 %) | -| Patch coverage | `bash scripts/local-patch-report.sh` | **PASS** — Overall **96.0 %**, Backend 96.4 %, Frontend 90.8 % (≥ 90 % overall). Artifacts: `test-results/local-patch-report.{md,json}` | -| GORM security scan | `./scripts/scan-gorm-security.sh --check` | **PASS** — 0 critical / 0 high / 0 medium | -| CodeQL Go | `skill-runner.sh security-scan-codeql` + `codeql-findings-gate.sh … go` | **PASS** — 0 errors / 0 warnings; 4 SARIF results, **all 4 suppressed, 0 blocking** | -| CodeQL JS | same run | **PASS** — 0 findings | -| Trivy | `skill-runner.sh security-scan-trivy` + targeted `trivy fs` on canonical manifests | **PASS for this branch** — see §2.1 | -| Targeted Playwright (firefox) | `npx playwright test tests/monitoring/uptime-monitoring-scale.spec.ts tests/monitoring/uptime-monitoring.spec.ts tests/a11y/uptime.a11y.spec.ts --project=firefox` | **PASS** — 29/29 passed (37.3 s) | - -CodeQL toolchain: `codeql 2.26.4` (≥ 2.26.0 required for this repo's query-pack pins) — -no `install-codeql.sh` needed. - -### 2.1 Trivy detail - -**This branch introduces zero HIGH/CRITICAL CVEs.** It touches no `go.mod`/`go.sum`, -no `package.json`/`package-lock.json`, no `Dockerfile`, no CI workflow -(`git diff --stat development...HEAD` on all of those is empty). - -Canonical working-tree manifests scan clean: - -| Target | HIGH/CRITICAL | -|---|---| -| `backend/go.mod` | 0 | -| `frontend/package-lock.json` | 0 | -| `package-lock.json` (root) | 0 | -| `agent/go.mod` | 0 | - -The `security-scan-trivy` skill exits non-zero, entirely due to two artifacts that are -**not part of this branch**: - -1. **Stale git worktrees in the working directory** — `.claude/worktrees/fix-banner-image/` - and `.claude/worktrees/fix-renovate-gin-lookup/` carry *older* lockfiles - (`axios 1.17.0` → `GHSA-gcfj-64vw-6mp9`, `react-router 7.17.0`, `form-data 4.0.5`, - `golang.org/x/net v0.55.0`, `golang.org/x/text v0.37.0`). The current branch's own - `frontend/package-lock.json` is already past these (scans 0). CLAUDE.md forbids - worktrees; these are leftover local cruft and a clean CI checkout will not see them. - **Recommendation (local hygiene, non-blocking):** `git worktree prune` / remove - `.claude/worktrees/`. -2. **`backend/internal/api/routes/keys/hecate-ca.key`** — pre-existing EC dev/test CA - fixture, already listed in `.trivyignore`, assessed in prior QA audits (2026-05-18, - 2026-06-03). Not committed to git history (`*.key` gitignored). Unchanged by this - branch. +| `docker-build.yml` `build-amd64` / `build-arm64` | `CADDY_BUILDER_SRC` / `CROWDSEC_BUILDER_SRC` env = fork ternary (`*-inline` for foreign head repo, else `toolchain-prebuilt`) | pre-existing "Log in to GitHub Container Registry" step (`secrets.GITHUB_TOKEN`) | +| `e2e-tests-split.yml` `build` | build-args fork ternary | added `Log in to GHCR` step, `if: image_source == 'build' && head.repo.full_name == github.repository` | +| `nightly-build.yml` | hard-coded `toolchain-prebuilt` (schedule/same-repo only — correct) | pre-existing login-action + `packages: write` | +| `toolchain-image.yml` | builds the toolchain itself (`--target toolchain-runtime`, `caddy-inline`/`crowdsec-inline` from source) | login `if: same_repo == 'true'` | +| `orthrus-build.yml` | builds `agent/Dockerfile` only — **does not use the root Dockerfile / toolchain image**; no change needed | n/a | + +**Fork PR path:** every ternary resolves `head.repo.full_name != '' && +head.repo.full_name != github.repository` → `caddy-inline` / `crowdsec-inline`, +i.e. compile the byte-identical recipe from source (~14 min). `make build-offline` +(new) does the same locally. No fork build path depends on pulling the private +image. **Fork PRs still build.** ✅ + +**Empty-head-repo guard:** the `head.repo.full_name != ''` conjunct means `push` +and non-PR events (empty `head.repo.full_name`) correctly resolve to +`toolchain-prebuilt`, not accidentally to `inline`. --- -## 3. Nice-to-have triage (supervisor's 6) - -| # | Finding | Disposition | -|---|---|---| -| **NI-2** | Shutdown-grace arithmetic: 25 s drain ctx vs. a theoretical `hardCap (20s) + notifyTimeout (10s)` on the `workerWG` path after C1. | **FIXED** (`b59115ab`) — comment at `main.go` `drainCtx` now explains why 25 s is sufficient: `appCancel()` runs first, so the probe ctx and the C1 dispatch ctx are born already-cancelled and unwind immediately; the only real bound left is the HTTP client's own 20 s timeout, which fits inside 25 s. No behaviour change; the analysis matches the supervisor's. | -| **NI-3** | `TestUptimeSummary_PerfBudget` seed is light (60 k rows) and the header comment's production-profile math is wrong ("~360 k"). | **FIXED** (`b59115ab`) — comment corrected: at the 30 s interval floor a 24 h window is 500 × 2880 ≈ **1.44 M** rows; the lean 60 k seed is stated plainly as a coarse guard that only catches a per-monitor-loop regression, not index-not-used / O(n²) slippage at scale. Seed **not** ballooned — 1.44 M rows would make the unit test multi-minute for little gain, and the p95 gate is already downgraded to "QA timing output, not hard-gated" (plan re-review S5). | -| **NI-4** | Stale "transient duplicate … C5 collapses the legacy path" comment at `uptime_check.go:56-59` (and the "mirrors checkMonitor's switch exactly" phrasing). | **FIXED** (`b59115ab`) — reworded to describe `probe()` as the sole probe switch; verified `UptimeService.checkMonitor` now calls `s.checker.probe` directly and carries no switch of its own. | -| **NI-1** | `last_notified_down` is carried in `monDebounce` (worker sets it on a down transition) and read back by `loadMonState`, but `flush()` never persists it for monitors. | **DEFERRED.** Not a regression — the column was unused pre-PR, and no monitor-level code consumes it today (only the host-level `host.LastNotifiedDown` re-notify damper is live). The in-memory field resets to a never-written value on every restart, which is cosmetically confusing but functionally inert. The two real fixes — (a) delete the vestigial field from `monDebounce`/`loadMonState`, or (b) thread it onto `CheckResult` and persist it — differ by whether a monitor-level re-notify damper is wanted, which is a product decision outside this QA pass. Low risk either way; safe to leave for a follow-up. | -| **NI-5** | `hostPass`/`monitorPass` only advance the schedule entry for IDs returned by the snapshot load; a row deleted between the due-scan and the snapshot load is never advanced and is re-selected every tick until the next `rescan()` (≤ 30 s). | **DEFERRED.** Harmless, self-healing churn — the stale ID produces at most a handful of wasted enqueues over ≤ 30 s, and enqueuing a job for a since-deleted monitor is itself a no-op downstream. The fix (advance-or-drop any due ID absent from the snapshot) adds branching to the scheduler's hot per-tick loop for a benefit that `rescan()` already delivers within one cycle. Not worth the change risk in a final pass. | -| **NI-6** | `dispatch()` wraps the non-blocking `NotifyMonitorDown` (map insert + `AfterFunc`) in the same goroutine + deadline-select harness as the blocking `NotifyMonitorUp` external send. | **DEFERRED.** Cosmetic — cost is one short-lived goroutine per down transition, bounded by worker count. Calling the down path inline would micro-optimise it but risks the C1 guarantee that *no* notification call can wedge a worker or `workerWG.Wait()`; keeping both paths on the identical bounded harness is the safer invariant. Leave as-is. | +## 6. Trivy — clean, no new suppressions + +- **`.trivyignore`: UNCHANGED in this PR.** `git log origin/main..tip -- .trivyignore` + → no commits; 257 non-blank lines, identical to `main`. **No new blanket + suppressions.** (`.trivyignore`'s `sha256` is itself a `toolchain-key.sh` + input, so any future edit forces a toolchain rebuild + re-pin.) +- **CI Trivy runs on the tip — all green:** + - "Trivy scan (toolchain image)" — `toolchain-image.yml` `trivy-scan`, + `CRITICAL,HIGH`, `exit-code 1` — **pass**. + - "Trivy Binary Scan" — `security-pr.yml` — **pass**. + - "Security Scan PR Image" — app image built from the pinned toolchain — + **pass**. + - "Verify Supply Chain" — **pass**; "grype" — **pass**; "Semgrep SAST" / + "Semgrep OSS" / "semgrep-cloud-platform" — **pass**. + - Top-level "Trivy" shows `NEUTRAL / skipping` — this is the pre-existing + always-on external check that no-ops on this event path, not a regression. + - Zero unignored CRITICAL/HIGH on both the toolchain image and the app image + built from it (the two `exit-code: '1'` gates passed). +- Local Trivy was **not** re-run: the toolchain image is a private GHCR package + and this environment has no GHCR credentials; CI ran it with proper auth. + Per CLAUDE.md this is a CI-scoped (`ci:`/`feat(ci)`) change and CI runs Trivy + unconditionally, so nothing is skipped. --- -## 4. Scope notes / observations (non-blocking) +## 7. Definition of Done (CI-scoped change) -- **Legacy `checkHost` / `markHostMonitorsDown` / `checkAllHosts` in `uptime_service.go`** - remain as the no-pool inline fallback. `markHostMonitorsDown` still does direct - `s.DB.Save` / `s.DB.Create` (GORM-parameterised, no injection risk). If a later pass - confirms these are unreachable in production (pool always wired), they're dead-code - removal candidates — out of scope here, and the supervisor already confirmed the - *new* files carry no dead code. -- **Stale `.claude/worktrees/` directories** — see §2.1; recommend pruning locally. +| DoD item | Status | +|---|---| +| Targeted Playwright E2E (touched specs) | **N/A to run locally** — no FE/BE/spec files changed. CI full E2E on tip `22e9c722` is **legit and complete**: "Prepare Application Image" (which now exercises the `toolchain-prebuilt` `COPY --from` path on a same-repo PR) **pass**; all shards **pass** — Chromium 1–4 + Security Enforcement, Firefox 1–4 + Security Enforcement, WebKit 1–4 + Security Enforcement; "E2E Test Results (Final)" **pass**. A broken pin/image would have failed the image build, not silently passed. | +| GORM security scan (`scan-gorm-security.sh`) | **N/A** — no `backend/internal/models/**`, no GORM queries, no migrations in the diff. | +| `local-patch-report.sh` / Go+TS patch coverage | **N/A** — no Go/TS lines changed. `codecov/patch` check on PR: **pass** (0 changed coverable lines). New executable code is shell, covered by 17 bats tests (see §2). | +| Frontend `npm run type-check` / `npm run build` / FE coverage 85% | **N/A** — no `frontend/` files touched. | +| Backend `go build ./...` / Go coverage 85% | **N/A** — no `backend/` files touched. "Backend (Go)" / "Agent (Go)" CI: **pass** (unchanged). | +| staticcheck / golangci-lint | **N/A** (no Go). Substituted by `shellcheck --severity=error` on the 3 new scripts — **clean locally** (also clean at default severity) and in CI job "Toolchain key / freshness-guard scripts (bats)"; `actionlint` on all 12 changed workflows — **clean locally** (`exit 0`). | +| CodeQL Go / JS | Green on tip ("CodeQL analysis (go)" + "(javascript-typescript)" **pass**). Effectively N/A (no Go/JS/TS source changed) but ran. | +| `bats scripts/tests/` | **17/17 PASS** locally + CI. | +| Build verification, `docker build` both `builder-src` modes | `toolchain-prebuilt` path: exercised & green across `build-amd64`, `build-arm64`, "Prepare Application Image", and all 6 integration image builds on this same-repo PR. `caddy-inline`/`crowdsec-inline` full-app path: **not exercised on a same-repo PR** (fork-only) — see **F4 / R2**. The inline *stage bodies themselves* are compiled from source on every `toolchain-image.yml` run (daily + tracked-path PRs) via `--target toolchain-runtime`, and passed on this PR ("Build & publish toolchain image" **pass**). | +| No debug leftovers in new scripts | **Clean** — `grep -nE 'TODO|FIXME|XXX|DEBUG|set -x|console.log|fmt.Print'` over the 3 scripts + 2 bats files + fixture helper → no matches. All three scripts use `set -euo pipefail`. | --- -## 5. Commits made during this QA pass - -| SHA | Message | Gate re-run | -|---|---|---| -| `b59115ab` | `chore(uptime): correct stale comments from QA triage` (NI-2 / NI-3 / NI-4, comment-only) | `go build ./...`, `go vet`, `go test ./internal/services ./cmd/api -count=1`, `make lint-staticcheck-only` (0 issues), `lefthook run pre-commit` (semgrep 0, golangci-lint-fast 0), `local-patch-report.sh` (96.0 % overall) — all green | +## Follow-ups (non-blocking) + +- **F1 — Confirm required-status enrolment.** Verify branch protection for `main` + (and `development`) lists **"Toolchain pin freshness (verify-toolchain-pin)"** + and **"Toolchain key / freshness-guard scripts (bats)"** as required checks. + The failure-closed guarantee in §1(c) / §2 only bites if the check is required; + the code is correct but enrolment is a repo-settings action outside this diff. +- **F2 — Close two bats coverage gaps** in `verify-toolchain-pin.bats`: (i) + same-repo run with `regctl` present but `image digest` failing (unresolvable + `:$KEY`) → `exit 1`; (ii) same-repo run with `CHARON_TOOLCHAIN_DIGEST` empty → + `exit 1`. And in `toolchain-key.bats`: sensitivity to a `tonistiigi/xx` pin + move and an `ALPINE_IMAGE` move (both are hashed inputs, currently untested). +- **F3 — (optional) `regctl-installer` hardening.** Either accept as-is (minimal + job perms, cosign verify on) or replace with a pinned-hash `curl | sha256sum -c` + install; if kept, pin `regctl-release` to an exact version rather than the + default `latest`. +- **F4 — Add periodic coverage of the offline/inline app build.** A scheduled or + label-gated job running `make build-offline` (or at minimum + `docker build --build-arg CADDY_BUILDER_SRC=caddy-inline + --build-arg CROWDSEC_BUILDER_SRC=crowdsec-inline --check .`) so the + `FROM ${CADDY_BUILDER_SRC} AS caddy-builder` selector + final-stage assembly on + the fork path can't silently rot between fork PRs. +- **F5 — Stale note in `docs/ci/toolchain-image.md`.** The "One-time bootstrap + notes" paragraph still says *"`COPY scripts/ /app/scripts/` copies the new + shell scripts into the runtime image … No `.dockerignore` … change is needed"*, + which the `.dockerignore` follow-up commit `22e9c722` (excludes + `scripts/tests/`, `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, + `scripts/lib/dockerfile-stage.sh` from the build context) now contradicts. + One-paragraph doc fix. + +## Residual supply-chain risk — accept knowingly + +- **R1 — one-day poisoned-tag window.** An actor with `packages: write` on + `ghcr.io/wikid82/charon-toolchain` (maintainer-level) could push a poisoned + image to the mutable `:$KEY` tag; a non-forced same-repo PR that recomputes the + same key would skip the rebuild and `sync-pin-on-pr` could pin that digest + without a from-source rebuild *in that PR*. Bounded by: fork PRs cannot reach + the path; the daily `--no-cache --pull` deterministic rebuild + `open-bump-pr` + self-heal within ~24 h; the resulting app image is still Trivy/Grype-scanned, + SBOM-attested and Cosign-signed. Net change vs pre-PR: the "every CVE-gate PR + rebuilds Caddy/CrowdSec from source" property is replaced by "daily + deterministic rebuild + per-PR freshness guard + immutable digest pin." +- **R2 — fork/offline `caddy-inline`+`crowdsec-inline` *whole-app* build is not + CI-exercised on same-repo PRs.** The stage bodies are compiled daily by + `toolchain-image.yml`; only the `FROM ${ARG} AS caddy-builder` indirection and + the final-stage COPY wiring on the inline path go unverified until a fork PR or + a manual `make build-offline`. Low severity (small surface, `toolchain-key.sh` + sanity-checks the stages exist and contain a build step). F4 closes it. +- **R3 — the toolchain image itself is digest-pinned but not Cosign-signed.** + Acceptable: it is built by the repo's own Actions, pulled by immutable digest, + and recipe→digest is bound by the freshness guard + LABEL check; the shipped + app image carries the signature/attestation. --- -## 6. Definition-of-Done checklist - -- [x] Targeted Playwright E2E (firefox) — 29/29 green -- [x] GORM security scan — 0 critical/high -- [x] Local patch coverage preflight — artifacts present, 96.0 % overall (≥ 90 %) -- [x] CodeQL Go + JS — 0 high/critical; 3 suppressed `go/log-injection` (2 YAML entries) are the only new suppressed Go findings, both legitimate `uint` false-positives -- [x] Trivy — 0 high/critical introduced by this branch -- [x] Lefthook pre-commit — clean on the QA commit -- [x] Staticcheck — 0 issues -- [x] Backend coverage — line 88.7 % (≥ 85 %) -- [x] Frontend coverage — lines 90.86 % (≥ 87 %) -- [x] Frontend type-check — clean -- [x] Backend + frontend build — clean -- [x] Full `go test ./... -count=1` — green, flake stabilised - -**Verdict: PASS — no blocking issues. Cleared to merge.** +## Scans run for this audit + +- `bats scripts/tests/toolchain-key.bats scripts/tests/verify-toolchain-pin.bats` → **17/17 pass** (`Bats 1.13.0`) +- `shellcheck --severity=error` + default severity on `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh` → **clean** +- `actionlint` on the 12 changed workflow files → **clean (exit 0)** +- Debug-leftover grep over the 3 scripts + 2 bats + fixture → **clean** +- `git log origin/main..tip -- .trivyignore` → **no changes** +- `gh api` verification: `regctl-installer` tag `v4.0.16` → commit `c2202c17…` → **exact match** +- `gh pr checks 1300` on tip `22e9c722` → **no failing / cancelled checks** (all pass or intentionally skipped) +- Diff review of `Dockerfile`, `toolchain-image.yml`, `build-charon-image/action.yml`, and the 11 other changed workflows; `SECURITY.md`, `ARCHITECTURE.md`, `docs/ci/toolchain-image.md` +- GORM security scan — **not run (N/A: no models/queries/migrations)** +- Local Trivy / CodeQL — **deferred to CI** (CI-scoped change; both ran green with proper credentials) From e06ff2a7ce873f49d69156cf7f381be0587b2e84 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 8 Sep 2026 07:30:53 +0000 Subject: [PATCH 11/19] chore(main): release 0.40.0 --- .release-please-manifest.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.release-please-manifest.json b/.release-please-manifest.json index 418b49e99..650781534 100644 --- a/.release-please-manifest.json +++ b/.release-please-manifest.json @@ -1,3 +1,3 @@ { - ".": "0.39.0" + ".": "0.40.0" } From 5bfe27cd5c8c22a6589dbda52161fbd0b34e8e97 Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Tue, 8 Sep 2026 03:34:28 -0400 Subject: [PATCH 12/19] ci: add consolidated integration-tests workflow (build once, fan out) Adds .github/workflows/integration-tests.yml, which builds the Charon image a single time via the shared build-charon-image composite action, uploads it as a 1-day run-scoped artifact, then fans out to parallel per-suite test jobs (cerberus / waf / rate-limit / crowdsec) that `docker load` that image instead of rebuilding it. Replaces four near-identical standalone workflows that each paid the image-build cost independently. Each suite job keeps its own test steps, debug-dump-on-failure and cleanup verbatim (each inspects different containers/ports/volumes). Job/check names are preserved so the diff is reviewable, but they now live under one workflow. Also guards the unconditional `docker build` in scripts/crowdsec_integration.sh behind a `docker image inspect charon:local` check, matching the sibling scripts (crowdsec_startup_test.sh, cerberus/waf/rate_limit_integration.sh) so the crowdsec job actually consumes the pre-built artifact rather than rebuilding the image a second time. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .github/workflows/cerberus-integration.yml | 124 ----- .github/workflows/crowdsec-integration.yml | 141 ------ .github/workflows/integration-tests.yml | 451 +++++++++++++++++++ .github/workflows/rate-limit-integration.yml | 119 ----- .github/workflows/waf-integration.yml | 106 ----- scripts/crowdsec_integration.sh | 8 +- 6 files changed, 457 insertions(+), 492 deletions(-) delete mode 100644 .github/workflows/cerberus-integration.yml delete mode 100644 .github/workflows/crowdsec-integration.yml create mode 100644 .github/workflows/integration-tests.yml delete mode 100644 .github/workflows/rate-limit-integration.yml delete mode 100644 .github/workflows/waf-integration.yml diff --git a/.github/workflows/cerberus-integration.yml b/.github/workflows/cerberus-integration.yml deleted file mode 100644 index 5bcb3998b..000000000 --- a/.github/workflows/cerberus-integration.yml +++ /dev/null @@ -1,124 +0,0 @@ -name: Cerberus Integration - -# Builds the Charon image locally via the shared build-charon-image composite action (GHA layer cache), then runs the Cerberus integration tests. -on: - workflow_dispatch: - inputs: - image_tag: - description: 'Docker image tag to test (e.g., pr-123-abc1234, latest)' - required: false - type: string - pull_request: - push: - branches: - - main - -# Prevent race conditions when PR is updated mid-test -# Cancels old test runs when new build completes with different SHA -concurrency: - group: ${{ github.workflow }}-${{ github.event.workflow_run.event || github.event_name }}-${{ github.event.workflow_run.head_branch || github.ref }} - cancel-in-progress: true - -permissions: - contents: read - packages: read - -jobs: - cerberus-integration: - name: Cerberus Security Stack Integration - runs-on: ubuntu-latest - timeout-minutes: 20 # 20m: same-repo runs COPY Caddy/CrowdSec from the pinned toolchain image (~2-4m build); fork PRs compile them inline (~14m) + test work, which sets the floor (B6). - - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - - name: Build Docker image (Local) - uses: ./.github/actions/build-charon-image - with: - # Fork PRs cannot pull the private toolchain image -> compile inline. - builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} - ghcr-token: ${{ secrets.GITHUB_TOKEN }} - - - name: Run Cerberus integration tests - id: cerberus-test - run: | - chmod +x scripts/cerberus_integration.sh - scripts/cerberus_integration.sh 2>&1 | tee cerberus-test-output.txt - exit "${PIPESTATUS[0]}" - - - name: Upload Container Logs on Failure - if: failure() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: cerberus-container-logs-${{ github.run_id }} - path: | - /tmp/charon-cerberus-test.log - /tmp/cerberus-backend.log - if-no-files-found: ignore - retention-days: 7 - - - name: Dump Debug Info on Failure - if: failure() - run: | - { - echo "## 🔍 Debug Information" - echo "" - - echo "### Container Status" - echo '```' - docker ps -a --filter "name=charon" --filter "name=cerberus" --filter "name=backend" 2>&1 || true - echo '```' - echo "" - - echo "### Security Status API" - echo '```json' - curl -s http://localhost:8480/api/v1/security/status 2>/dev/null | head -100 || echo "Could not retrieve security status" - echo '```' - echo "" - - echo "### Caddy Admin Config" - echo '```json' - curl -s http://localhost:2319/config 2>/dev/null | head -200 || echo "Could not retrieve Caddy config" - echo '```' - echo "" - - echo "### Charon Container Logs (last 100 lines)" - echo '```' - docker logs charon-cerberus-test 2>&1 | tail -100 || echo "No container logs available" - echo '```' - } >> "$GITHUB_STEP_SUMMARY" - - - name: Cerberus Integration Summary - if: always() - run: | - { - echo "## 🔱 Cerberus Integration Test Results" - if [ "${{ steps.cerberus-test.outcome }}" == "success" ]; then - echo "✅ **All Cerberus tests passed**" - echo "" - echo "### Test Results:" - echo '```' - grep -E "✓|PASS|TC-[0-9]|=== ALL" cerberus-test-output.txt || echo "See logs for details" - echo '```' - echo "" - echo "### Features Tested:" - echo "- WAF (Coraza) payload inspection" - echo "- Rate limiting enforcement" - echo "- Security handler ordering" - echo "- Legitimate traffic flow" - else - echo "❌ **Cerberus tests failed**" - echo "" - echo "### Failure Details:" - echo '```' - grep -E "✗|FAIL|Error|failed" cerberus-test-output.txt | head -30 || echo "See logs for details" - echo '```' - fi - } >> "$GITHUB_STEP_SUMMARY" - - - name: Cleanup - if: always() - run: | - docker rm -f charon-cerberus-test || true - docker rm -f cerberus-backend || true - docker volume rm charon_cerberus_test_data caddy_cerberus_test_data caddy_cerberus_test_config 2>/dev/null || true - docker network rm containers_default || true diff --git a/.github/workflows/crowdsec-integration.yml b/.github/workflows/crowdsec-integration.yml deleted file mode 100644 index d7945b103..000000000 --- a/.github/workflows/crowdsec-integration.yml +++ /dev/null @@ -1,141 +0,0 @@ -name: CrowdSec Integration - -# Builds the Charon image locally via the shared build-charon-image composite action (GHA layer cache), then runs the CrowdSec bouncer integration tests. -on: - workflow_dispatch: - inputs: - image_tag: - description: 'Docker image tag to test (e.g., pr-123-abc1234, latest)' - required: false - type: string - pull_request: - push: - branches: - - main - -# Prevent race conditions when PR is updated mid-test -# Cancels old test runs when new build completes with different SHA -concurrency: - group: ${{ github.workflow }}-${{ github.event.workflow_run.event || github.event_name }}-${{ github.event.workflow_run.head_branch || github.ref }} - cancel-in-progress: true - -permissions: - contents: read - packages: read - -jobs: - crowdsec-integration: - name: CrowdSec Bouncer Integration - runs-on: ubuntu-latest - timeout-minutes: 20 # 20m: same-repo runs COPY Caddy/CrowdSec from the pinned toolchain image (~2-4m build); fork PRs compile them inline (~14m) + test work, which sets the floor (B6). - - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - - name: Build Docker image (Local) - uses: ./.github/actions/build-charon-image - with: - ci: 'true' - # Fork PRs cannot pull the private toolchain image -> compile inline. - builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} - ghcr-token: ${{ secrets.GITHUB_TOKEN }} - - - name: Run CrowdSec integration tests - id: crowdsec-test - run: | - chmod +x .github/skills/scripts/skill-runner.sh - .github/skills/scripts/skill-runner.sh integration-test-crowdsec 2>&1 | tee crowdsec-test-output.txt - exit "${PIPESTATUS[0]}" - - - name: Run CrowdSec Startup and LAPI Tests - id: lapi-test - run: | - chmod +x .github/skills/scripts/skill-runner.sh - .github/skills/scripts/skill-runner.sh integration-test-crowdsec-startup 2>&1 | tee lapi-test-output.txt - exit "${PIPESTATUS[0]}" - - - name: Dump Debug Info on Failure - if: failure() - run: | - { - echo "## 🔍 Debug Information" - echo "" - - echo "### Container Status" - echo '```' - docker ps -a --filter "name=charon" --filter "name=crowdsec" 2>&1 || true - echo '```' - echo "" - - # Check which test container exists and dump its logs - if docker ps -a --filter "name=charon-crowdsec-startup-test" --format "{{.Names}}" | grep -q "charon-crowdsec-startup-test"; then - echo "### Charon Startup Test Container Logs (last 100 lines)" - echo '```' - docker logs charon-crowdsec-startup-test 2>&1 | tail -100 || echo "No container logs available" - echo '```' - elif docker ps -a --filter "name=charon-debug" --format "{{.Names}}" | grep -q "charon-debug"; then - echo "### Charon Container Logs (last 100 lines)" - echo '```' - docker logs charon-debug 2>&1 | tail -100 || echo "No container logs available" - echo '```' - fi - echo "" - - # Check for CrowdSec specific logs if LAPI test ran - if [ -f "lapi-test-output.txt" ]; then - echo "### CrowdSec LAPI Test Failures" - echo '```' - grep -E "✗ FAIL|✗ CRITICAL|CROWDSEC.*BROKEN" lapi-test-output.txt 2>&1 || echo "No critical failures found in LAPI test" - echo '```' - fi - } >> "$GITHUB_STEP_SUMMARY" - - - name: CrowdSec Integration Summary - if: always() - run: | - { - echo "## 🛡️ CrowdSec Integration Test Results" - - # CrowdSec Preset Integration Tests - if [ "${{ steps.crowdsec-test.outcome }}" == "success" ]; then - echo "✅ **CrowdSec Hub Presets: Passed**" - echo "" - echo "### Preset Test Results:" - echo '```' - grep -E "^✓|^===|^Pull|^Apply" crowdsec-test-output.txt || echo "See logs for details" - echo '```' - else - echo "❌ **CrowdSec Hub Presets: Failed**" - echo "" - echo "### Preset Failure Details:" - echo '```' - grep -E "^✗|Unexpected|Error|failed|FAIL" crowdsec-test-output.txt | head -20 || echo "See logs for details" - echo '```' - fi - - echo "" - - # CrowdSec Startup and LAPI Tests - if [ "${{ steps.lapi-test.outcome }}" == "success" ]; then - echo "✅ **CrowdSec Startup & LAPI: Passed**" - echo "" - echo "### LAPI Test Results:" - echo '```' - grep -E "^\[TEST\]|✓ PASS|Check [0-9]|CrowdSec LAPI" lapi-test-output.txt || echo "See logs for details" - echo '```' - else - echo "❌ **CrowdSec Startup & LAPI: Failed**" - echo "" - echo "### LAPI Failure Details:" - echo '```' - grep -E "✗ FAIL|✗ CRITICAL|Error|failed" lapi-test-output.txt | head -20 || echo "See logs for details" - echo '```' - fi - } >> "$GITHUB_STEP_SUMMARY" - - - name: Cleanup - if: always() - run: | - docker rm -f charon-debug || true - docker rm -f charon-crowdsec-startup-test || true - docker rm -f crowdsec || true - docker network rm containers_default || true diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml new file mode 100644 index 000000000..2c8cadf03 --- /dev/null +++ b/.github/workflows/integration-tests.yml @@ -0,0 +1,451 @@ +name: Integration Tests + +# Builds the Charon image ONCE via the shared build-charon-image composite action +# (GHA layer cache), uploads it as a short-lived, run-scoped artifact, then fans +# out to parallel per-suite test jobs that `docker load` that image — the image +# is built a single time per run, not once per suite. +# +# Replaces the former standalone workflows: +# cerberus-integration.yml -> job `cerberus` (was "Cerberus Security Stack Integration") +# waf-integration.yml -> job `waf` (was "Coraza WAF Integration") +# rate-limit-integration.yml -> job `rate-limit` (was "Rate Limiting Integration") +# crowdsec-integration.yml -> job `crowdsec` (was "CrowdSec Bouncer Integration") +on: + workflow_dispatch: + pull_request: + push: + branches: + - main + +# Prevent race conditions when a PR is updated mid-test; cancel superseded runs. +concurrency: + group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + packages: read + +jobs: + build: + name: Build Charon image + runs-on: ubuntu-latest + timeout-minutes: 20 # 20m: same-repo runs COPY Caddy/CrowdSec from the pinned toolchain image (~2-4m build); fork PRs compile them inline (~14m), which sets the floor (B6). Test-only fan-out jobs below are sized smaller. + + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - name: Build Docker image (Local) + uses: ./.github/actions/build-charon-image + with: + ci: 'true' + # Fork PRs cannot pull the private toolchain image -> compile inline. + builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} + ghcr-token: ${{ secrets.GITHUB_TOKEN }} + + - name: Save Charon image + run: docker save charon:local -o /tmp/charon-image.tar + + - name: Upload Charon image artifact + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: charon-integration-image + path: /tmp/charon-image.tar + retention-days: 1 # Only needed for the duration of this run's fan-out jobs. + if-no-files-found: error + + cerberus: + name: Cerberus Security Stack Integration + needs: build + runs-on: ubuntu-latest + timeout-minutes: 12 # test-only work; image is built upstream in `build`. + + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - name: Download Charon image artifact + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: charon-integration-image + path: /tmp + - name: Load Charon image + run: docker load -i /tmp/charon-image.tar + + - name: Run Cerberus integration tests + id: cerberus-test + run: | + chmod +x scripts/cerberus_integration.sh + scripts/cerberus_integration.sh 2>&1 | tee cerberus-test-output.txt + exit "${PIPESTATUS[0]}" + + - name: Upload Container Logs on Failure + if: failure() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: cerberus-container-logs-${{ github.run_id }} + path: | + /tmp/charon-cerberus-test.log + /tmp/cerberus-backend.log + if-no-files-found: ignore + retention-days: 7 + + - name: Dump Debug Info on Failure + if: failure() + run: | + { + echo "## 🔍 Debug Information" + echo "" + + echo "### Container Status" + echo '```' + docker ps -a --filter "name=charon" --filter "name=cerberus" --filter "name=backend" 2>&1 || true + echo '```' + echo "" + + echo "### Security Status API" + echo '```json' + curl -s http://localhost:8480/api/v1/security/status 2>/dev/null | head -100 || echo "Could not retrieve security status" + echo '```' + echo "" + + echo "### Caddy Admin Config" + echo '```json' + curl -s http://localhost:2319/config 2>/dev/null | head -200 || echo "Could not retrieve Caddy config" + echo '```' + echo "" + + echo "### Charon Container Logs (last 100 lines)" + echo '```' + docker logs charon-cerberus-test 2>&1 | tail -100 || echo "No container logs available" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + + - name: Cerberus Integration Summary + if: always() + run: | + { + echo "## 🔱 Cerberus Integration Test Results" + if [ "${{ steps.cerberus-test.outcome }}" == "success" ]; then + echo "✅ **All Cerberus tests passed**" + echo "" + echo "### Test Results:" + echo '```' + grep -E "✓|PASS|TC-[0-9]|=== ALL" cerberus-test-output.txt || echo "See logs for details" + echo '```' + echo "" + echo "### Features Tested:" + echo "- WAF (Coraza) payload inspection" + echo "- Rate limiting enforcement" + echo "- Security handler ordering" + echo "- Legitimate traffic flow" + else + echo "❌ **Cerberus tests failed**" + echo "" + echo "### Failure Details:" + echo '```' + grep -E "✗|FAIL|Error|failed" cerberus-test-output.txt | head -30 || echo "See logs for details" + echo '```' + fi + } >> "$GITHUB_STEP_SUMMARY" + + - name: Cleanup + if: always() + run: | + docker rm -f charon-cerberus-test || true + docker rm -f cerberus-backend || true + docker volume rm charon_cerberus_test_data caddy_cerberus_test_data caddy_cerberus_test_config 2>/dev/null || true + docker network rm containers_default || true + + waf: + name: Coraza WAF Integration + needs: build + runs-on: ubuntu-latest + timeout-minutes: 12 # test-only work; image is built upstream in `build`. + + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - name: Download Charon image artifact + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: charon-integration-image + path: /tmp + - name: Load Charon image + run: docker load -i /tmp/charon-image.tar + + - name: Run WAF integration tests + id: waf-test + run: | + chmod +x scripts/waf_integration.sh + scripts/waf_integration.sh 2>&1 | tee waf-test-output.txt + exit "${PIPESTATUS[0]}" + + - name: Dump Debug Info on Failure + if: failure() + run: | + { + echo "## 🔍 Debug Information" + echo "" + + echo "### Container Status" + echo '```' + docker ps -a --filter "name=charon" --filter "name=waf" 2>&1 || true + echo '```' + echo "" + + echo "### Caddy Admin Config" + echo '```json' + curl -s http://localhost:2119/config/ 2>/dev/null | head -200 || echo "Could not retrieve Caddy config" + echo '```' + echo "" + + echo "### Charon Container Logs (last 100 lines)" + echo '```' + docker logs charon-waf-test 2>&1 | tail -100 || echo "No container logs available" + echo '```' + echo "" + + echo "### WAF Ruleset Files" + echo '```' + docker exec charon-waf-test sh -c 'ls -la /app/data/caddy/coraza/rulesets/ 2>/dev/null && echo "---" && cat /app/data/caddy/coraza/rulesets/*.conf 2>/dev/null' || echo "No ruleset files found" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + + - name: WAF Integration Summary + if: always() + run: | + { + echo "## 🛡️ WAF Integration Test Results" + if [ "${{ steps.waf-test.outcome }}" == "success" ]; then + echo "✅ **All WAF tests passed**" + echo "" + echo "### Test Results:" + echo '```' + grep -E "^✓|^===|^Coraza" waf-test-output.txt || echo "See logs for details" + echo '```' + else + echo "❌ **WAF tests failed**" + echo "" + echo "### Failure Details:" + echo '```' + grep -E "^✗|Unexpected|Error|failed" waf-test-output.txt | head -20 || echo "See logs for details" + echo '```' + fi + } >> "$GITHUB_STEP_SUMMARY" + + - name: Cleanup + if: always() + run: | + docker rm -f charon-waf-test || true + docker rm -f waf-backend || true + docker network rm containers_default || true + + rate-limit: + name: Rate Limiting Integration + needs: build + runs-on: ubuntu-latest + timeout-minutes: 12 # test-only work; image is built upstream in `build`. + + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - name: Download Charon image artifact + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: charon-integration-image + path: /tmp + - name: Load Charon image + run: docker load -i /tmp/charon-image.tar + + - name: Run rate limit integration tests + id: ratelimit-test + run: | + chmod +x scripts/rate_limit_integration.sh + scripts/rate_limit_integration.sh 2>&1 | tee ratelimit-test-output.txt + exit "${PIPESTATUS[0]}" + + - name: Dump Debug Info on Failure + if: failure() + run: | + { + echo "## 🔍 Debug Information" + echo "" + + echo "### Container Status" + echo '```' + docker ps -a --filter "name=charon" --filter "name=ratelimit" --filter "name=backend" 2>&1 || true + echo '```' + echo "" + + echo "### Security Config API" + echo '```json' + curl -s http://localhost:8280/api/v1/security/config 2>/dev/null | head -100 || echo "Could not retrieve security config" + echo '```' + echo "" + + echo "### Security Status API" + echo '```json' + curl -s http://localhost:8280/api/v1/security/status 2>/dev/null | head -100 || echo "Could not retrieve security status" + echo '```' + echo "" + + echo "### Caddy Admin Config (rate_limit handlers)" + echo '```json' + curl -s http://localhost:2119/config/ 2>/dev/null | grep -A 20 '"handler":"rate_limit"' | head -30 || echo "Could not retrieve Caddy config" + echo '```' + echo "" + + echo "### Charon Container Logs (last 100 lines)" + echo '```' + docker logs charon-ratelimit-test 2>&1 | tail -100 || echo "No container logs available" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + + - name: Rate Limit Integration Summary + if: always() + run: | + { + echo "## ⏱️ Rate Limit Integration Test Results" + if [ "${{ steps.ratelimit-test.outcome }}" == "success" ]; then + echo "✅ **All rate limit tests passed**" + echo "" + echo "### Test Results:" + echo '```' + grep -E "✓|=== ALL|HTTP 429|HTTP 200" ratelimit-test-output.txt | head -30 || echo "See logs for details" + echo '```' + echo "" + echo "### Verified Behaviors:" + echo "- Requests within limit return HTTP 200" + echo "- Requests exceeding limit return HTTP 429" + echo "- Retry-After header present on blocked responses" + echo "- Rate limit window resets correctly" + else + echo "❌ **Rate limit tests failed**" + echo "" + echo "### Failure Details:" + echo '```' + grep -E "✗|FAIL|Error|failed|expected" ratelimit-test-output.txt | head -30 || echo "See logs for details" + echo '```' + fi + } >> "$GITHUB_STEP_SUMMARY" + + - name: Cleanup + if: always() + run: | + docker rm -f charon-ratelimit-test || true + docker rm -f ratelimit-backend || true + docker volume rm charon_ratelimit_data caddy_ratelimit_data caddy_ratelimit_config 2>/dev/null || true + docker network rm containers_default || true + + crowdsec: + name: CrowdSec Bouncer Integration + needs: build + runs-on: ubuntu-latest + timeout-minutes: 15 # test-only work (two suites); image is built upstream in `build`. + + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - name: Download Charon image artifact + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: charon-integration-image + path: /tmp + - name: Load Charon image + run: docker load -i /tmp/charon-image.tar + + - name: Run CrowdSec integration tests + id: crowdsec-test + run: | + chmod +x .github/skills/scripts/skill-runner.sh + .github/skills/scripts/skill-runner.sh integration-test-crowdsec 2>&1 | tee crowdsec-test-output.txt + exit "${PIPESTATUS[0]}" + + - name: Run CrowdSec Startup and LAPI Tests + id: lapi-test + run: | + chmod +x .github/skills/scripts/skill-runner.sh + .github/skills/scripts/skill-runner.sh integration-test-crowdsec-startup 2>&1 | tee lapi-test-output.txt + exit "${PIPESTATUS[0]}" + + - name: Dump Debug Info on Failure + if: failure() + run: | + { + echo "## 🔍 Debug Information" + echo "" + + echo "### Container Status" + echo '```' + docker ps -a --filter "name=charon" --filter "name=crowdsec" 2>&1 || true + echo '```' + echo "" + + # Check which test container exists and dump its logs + if docker ps -a --filter "name=charon-crowdsec-startup-test" --format "{{.Names}}" | grep -q "charon-crowdsec-startup-test"; then + echo "### Charon Startup Test Container Logs (last 100 lines)" + echo '```' + docker logs charon-crowdsec-startup-test 2>&1 | tail -100 || echo "No container logs available" + echo '```' + elif docker ps -a --filter "name=charon-debug" --format "{{.Names}}" | grep -q "charon-debug"; then + echo "### Charon Container Logs (last 100 lines)" + echo '```' + docker logs charon-debug 2>&1 | tail -100 || echo "No container logs available" + echo '```' + fi + echo "" + + # Check for CrowdSec specific logs if LAPI test ran + if [ -f "lapi-test-output.txt" ]; then + echo "### CrowdSec LAPI Test Failures" + echo '```' + grep -E "✗ FAIL|✗ CRITICAL|CROWDSEC.*BROKEN" lapi-test-output.txt 2>&1 || echo "No critical failures found in LAPI test" + echo '```' + fi + } >> "$GITHUB_STEP_SUMMARY" + + - name: CrowdSec Integration Summary + if: always() + run: | + { + echo "## 🛡️ CrowdSec Integration Test Results" + + # CrowdSec Preset Integration Tests + if [ "${{ steps.crowdsec-test.outcome }}" == "success" ]; then + echo "✅ **CrowdSec Hub Presets: Passed**" + echo "" + echo "### Preset Test Results:" + echo '```' + grep -E "^✓|^===|^Pull|^Apply" crowdsec-test-output.txt || echo "See logs for details" + echo '```' + else + echo "❌ **CrowdSec Hub Presets: Failed**" + echo "" + echo "### Preset Failure Details:" + echo '```' + grep -E "^✗|Unexpected|Error|failed|FAIL" crowdsec-test-output.txt | head -20 || echo "See logs for details" + echo '```' + fi + + echo "" + + # CrowdSec Startup and LAPI Tests + if [ "${{ steps.lapi-test.outcome }}" == "success" ]; then + echo "✅ **CrowdSec Startup & LAPI: Passed**" + echo "" + echo "### LAPI Test Results:" + echo '```' + grep -E "^\[TEST\]|✓ PASS|Check [0-9]|CrowdSec LAPI" lapi-test-output.txt || echo "See logs for details" + echo '```' + else + echo "❌ **CrowdSec Startup & LAPI: Failed**" + echo "" + echo "### LAPI Failure Details:" + echo '```' + grep -E "✗ FAIL|✗ CRITICAL|Error|failed" lapi-test-output.txt | head -20 || echo "See logs for details" + echo '```' + fi + } >> "$GITHUB_STEP_SUMMARY" + + - name: Cleanup + if: always() + run: | + docker rm -f charon-debug || true + docker rm -f charon-crowdsec-startup-test || true + docker rm -f crowdsec || true + docker network rm containers_default || true diff --git a/.github/workflows/rate-limit-integration.yml b/.github/workflows/rate-limit-integration.yml deleted file mode 100644 index 1bad9e9f1..000000000 --- a/.github/workflows/rate-limit-integration.yml +++ /dev/null @@ -1,119 +0,0 @@ -name: Rate Limit integration - -# Builds the Charon image locally via the shared build-charon-image composite action (GHA layer cache), then runs the rate limiting integration tests. -on: - workflow_dispatch: - inputs: - image_tag: - description: 'Docker image tag to test (e.g., pr-123-abc1234, latest)' - required: false - type: string - pull_request: - push: - branches: - - main - -# Prevent race conditions when PR is updated mid-test -# Cancels old test runs when new build completes with different SHA -concurrency: - group: ${{ github.workflow }}-${{ github.event.workflow_run.event || github.event_name }}-${{ github.event.workflow_run.head_branch || github.ref }} - cancel-in-progress: true - -permissions: - contents: read - packages: read - -jobs: - rate-limit-integration: - name: Rate Limiting Integration - runs-on: ubuntu-latest - timeout-minutes: 20 # 20m: same-repo runs COPY Caddy/CrowdSec from the pinned toolchain image (~2-4m build); fork PRs compile them inline (~14m) + test work, which sets the floor (B6). - - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - - name: Build Docker image (Local) - uses: ./.github/actions/build-charon-image - with: - # Fork PRs cannot pull the private toolchain image -> compile inline. - builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} - ghcr-token: ${{ secrets.GITHUB_TOKEN }} - - - name: Run rate limit integration tests - id: ratelimit-test - run: | - chmod +x scripts/rate_limit_integration.sh - scripts/rate_limit_integration.sh 2>&1 | tee ratelimit-test-output.txt - exit "${PIPESTATUS[0]}" - - - name: Dump Debug Info on Failure - if: failure() - run: | - { - echo "## 🔍 Debug Information" - echo "" - - echo "### Container Status" - echo '```' - docker ps -a --filter "name=charon" --filter "name=ratelimit" --filter "name=backend" 2>&1 || true - echo '```' - echo "" - - echo "### Security Config API" - echo '```json' - curl -s http://localhost:8280/api/v1/security/config 2>/dev/null | head -100 || echo "Could not retrieve security config" - echo '```' - echo "" - - echo "### Security Status API" - echo '```json' - curl -s http://localhost:8280/api/v1/security/status 2>/dev/null | head -100 || echo "Could not retrieve security status" - echo '```' - echo "" - - echo "### Caddy Admin Config (rate_limit handlers)" - echo '```json' - curl -s http://localhost:2119/config/ 2>/dev/null | grep -A 20 '"handler":"rate_limit"' | head -30 || echo "Could not retrieve Caddy config" - echo '```' - echo "" - - echo "### Charon Container Logs (last 100 lines)" - echo '```' - docker logs charon-ratelimit-test 2>&1 | tail -100 || echo "No container logs available" - echo '```' - } >> "$GITHUB_STEP_SUMMARY" - - - name: Rate Limit Integration Summary - if: always() - run: | - { - echo "## ⏱️ Rate Limit Integration Test Results" - if [ "${{ steps.ratelimit-test.outcome }}" == "success" ]; then - echo "✅ **All rate limit tests passed**" - echo "" - echo "### Test Results:" - echo '```' - grep -E "✓|=== ALL|HTTP 429|HTTP 200" ratelimit-test-output.txt | head -30 || echo "See logs for details" - echo '```' - echo "" - echo "### Verified Behaviors:" - echo "- Requests within limit return HTTP 200" - echo "- Requests exceeding limit return HTTP 429" - echo "- Retry-After header present on blocked responses" - echo "- Rate limit window resets correctly" - else - echo "❌ **Rate limit tests failed**" - echo "" - echo "### Failure Details:" - echo '```' - grep -E "✗|FAIL|Error|failed|expected" ratelimit-test-output.txt | head -30 || echo "See logs for details" - echo '```' - fi - } >> "$GITHUB_STEP_SUMMARY" - - - name: Cleanup - if: always() - run: | - docker rm -f charon-ratelimit-test || true - docker rm -f ratelimit-backend || true - docker volume rm charon_ratelimit_data caddy_ratelimit_data caddy_ratelimit_config 2>/dev/null || true - docker network rm containers_default || true diff --git a/.github/workflows/waf-integration.yml b/.github/workflows/waf-integration.yml deleted file mode 100644 index b455efc5f..000000000 --- a/.github/workflows/waf-integration.yml +++ /dev/null @@ -1,106 +0,0 @@ -name: WAF integration - -# Builds the Charon image locally via the shared build-charon-image composite action (GHA layer cache), then runs the Coraza WAF integration tests. -on: - workflow_dispatch: - inputs: - image_tag: - description: 'Docker image tag to test (e.g., pr-123-abc1234, latest)' - required: false - type: string - pull_request: - push: - branches: - - main - -# Prevent race conditions when PR is updated mid-test -# Cancels old test runs when new build completes with different SHA -concurrency: - group: ${{ github.workflow }}-${{ github.event.workflow_run.event || github.event_name }}-${{ github.event.workflow_run.head_branch || github.ref }} - cancel-in-progress: true - -permissions: - contents: read - packages: read - -jobs: - waf-integration: - name: Coraza WAF Integration - runs-on: ubuntu-latest - timeout-minutes: 20 # 20m: same-repo runs COPY Caddy/CrowdSec from the pinned toolchain image (~2-4m build); fork PRs compile them inline (~14m) + test work, which sets the floor (B6). - - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - - name: Build Docker image (Local) - uses: ./.github/actions/build-charon-image - with: - # Fork PRs cannot pull the private toolchain image -> compile inline. - builder-src: ${{ (github.event.pull_request.head.repo.full_name != '' && github.event.pull_request.head.repo.full_name != github.repository) && 'inline' || 'prebuilt' }} - ghcr-token: ${{ secrets.GITHUB_TOKEN }} - - - name: Run WAF integration tests - id: waf-test - run: | - chmod +x scripts/waf_integration.sh - scripts/waf_integration.sh 2>&1 | tee waf-test-output.txt - exit "${PIPESTATUS[0]}" - - - name: Dump Debug Info on Failure - if: failure() - run: | - { - echo "## 🔍 Debug Information" - echo "" - - echo "### Container Status" - echo '```' - docker ps -a --filter "name=charon" --filter "name=waf" 2>&1 || true - echo '```' - echo "" - - echo "### Caddy Admin Config" - echo '```json' - curl -s http://localhost:2119/config/ 2>/dev/null | head -200 || echo "Could not retrieve Caddy config" - echo '```' - echo "" - - echo "### Charon Container Logs (last 100 lines)" - echo '```' - docker logs charon-waf-test 2>&1 | tail -100 || echo "No container logs available" - echo '```' - echo "" - - echo "### WAF Ruleset Files" - echo '```' - docker exec charon-waf-test sh -c 'ls -la /app/data/caddy/coraza/rulesets/ 2>/dev/null && echo "---" && cat /app/data/caddy/coraza/rulesets/*.conf 2>/dev/null' || echo "No ruleset files found" - echo '```' - } >> "$GITHUB_STEP_SUMMARY" - - - name: WAF Integration Summary - if: always() - run: | - { - echo "## 🛡️ WAF Integration Test Results" - if [ "${{ steps.waf-test.outcome }}" == "success" ]; then - echo "✅ **All WAF tests passed**" - echo "" - echo "### Test Results:" - echo '```' - grep -E "^✓|^===|^Coraza" waf-test-output.txt || echo "See logs for details" - echo '```' - else - echo "❌ **WAF tests failed**" - echo "" - echo "### Failure Details:" - echo '```' - grep -E "^✗|Unexpected|Error|failed" waf-test-output.txt | head -20 || echo "See logs for details" - echo '```' - fi - } >> "$GITHUB_STEP_SUMMARY" - - - name: Cleanup - if: always() - run: | - docker rm -f charon-waf-test || true - docker rm -f waf-backend || true - docker network rm containers_default || true diff --git a/scripts/crowdsec_integration.sh b/scripts/crowdsec_integration.sh index 9875132d0..798bdc4fd 100755 --- a/scripts/crowdsec_integration.sh +++ b/scripts/crowdsec_integration.sh @@ -20,8 +20,12 @@ if ! command -v docker >/dev/null 2>&1; then exit 1 fi -echo "Building charon:local image..." -docker build -t charon:local . +if ! docker image inspect charon:local >/dev/null 2>&1; then + echo "Building charon:local image..." + docker build -t charon:local . +else + echo "Using existing charon:local image" +fi docker rm -f charon-debug >/dev/null 2>&1 || true if ! docker network inspect containers_default >/dev/null 2>&1; then From 900956667640b20425120a6a616ba45c2f8a0f02 Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Tue, 8 Sep 2026 03:34:48 -0400 Subject: [PATCH 13/19] ci: update in-tree references to the consolidated integration workflow Points docs and instructions that named the four deleted workflow files (cerberus-integration.yml / waf-integration.yml / rate-limit-integration.yml / crowdsec-integration.yml) at .github/workflows/integration-tests.yml and its per-suite jobs instead. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .github/instructions/testing.instructions.md | 2 +- .../crowdsec_integration_failure_analysis.md | 2 +- docs/issues/manual_test_workflow_triggers.md | 15 ++++----------- 3 files changed, 6 insertions(+), 13 deletions(-) diff --git a/.github/instructions/testing.instructions.md b/.github/instructions/testing.instructions.md index ad5360b0b..271a86eb8 100644 --- a/.github/instructions/testing.instructions.md +++ b/.github/instructions/testing.instructions.md @@ -93,7 +93,7 @@ This step: - Test requests routing through Caddy proxy with full middleware - **Port: 80 (User Traffic via Caddy)** - **Location: `backend/integration/` with `//go:build integration` tag** -- **CI: Runs in separate workflows (cerberus-integration.yml, waf-integration.yml, etc.)** +- **CI: Runs in `integration-tests.yml` (one shared image build, then parallel `cerberus` / `waf` / `rate-limit` / `crowdsec` suite jobs)** ### Two Modes: Docker vs Vite diff --git a/docs/analysis/crowdsec_integration_failure_analysis.md b/docs/analysis/crowdsec_integration_failure_analysis.md index ea0548514..c0536aa84 100644 --- a/docs/analysis/crowdsec_integration_failure_analysis.md +++ b/docs/analysis/crowdsec_integration_failure_analysis.md @@ -203,7 +203,7 @@ fi - `Dockerfile` (lines 218-310): CrowdSec builder and fallback stages - `.docker/docker-entrypoint.sh` (lines 120-230): CrowdSec initialization -- `.github/workflows/crowdsec-integration.yml`: CI workflow +- `.github/workflows/integration-tests.yml` (`crowdsec` job): CI workflow - `scripts/crowdsec_integration.sh`: Legacy integration test - `.github/skills/integration-test-crowdsec-scripts/run.sh`: Modern test wrapper diff --git a/docs/issues/manual_test_workflow_triggers.md b/docs/issues/manual_test_workflow_triggers.md index 3053f70c0..cd37d4ad2 100644 --- a/docs/issues/manual_test_workflow_triggers.md +++ b/docs/issues/manual_test_workflow_triggers.md @@ -11,10 +11,7 @@ Verify that all CI/CD workflows trigger correctly on feature branches and provid # Scope - `dry-run-history-rewrite.yml` (Modified) -- `cerberus-integration.yml` -- `crowdsec-integration.yml` -- `waf-integration.yml` -- `rate-limit-integration.yml` +- `integration-tests.yml` (consolidated: builds the Charon image once, then fans out to the `cerberus` / `waf` / `rate-limit` / `crowdsec` suite jobs) - `e2e-tests-split.yml` # Test Steps @@ -28,13 +25,9 @@ Verify that all CI/CD workflows trigger correctly on feature branches and provid ## 2. Integration Tests (Dual Mode Verification) - [ ] Using the same branch `feature/test-workflow-triggers`. -- [ ] Verify the following workflows start immediately (building locally): - - [ ] `Cerberus Integration` - - [ ] `CrowdSec Integration` - - [ ] `Coraza WAF Integration` - - [ ] `Rate Limiting Integration` -- [ ] Inspect the logs of one of them. -- [ ] Confirm it executes the "Build Docker image (Local)" step and *skips* the "Pull Docker image from registry" step. +- [ ] Verify the `Integration Tests` workflow starts immediately (building locally). +- [ ] Confirm its `Build Charon image` job runs the "Build Docker image (Local)" step exactly once. +- [ ] Confirm the `Cerberus Security Stack Integration`, `Coraza WAF Integration`, `Rate Limiting Integration` and `CrowdSec Bouncer Integration` jobs each `needs: build`, download the `charon-integration-image` artifact and `docker load` it instead of rebuilding. ## 3. Supply Chain (Split Verification) - [ ] Verify `Supply Chain Security (PR)` starts on the feature branch push. From 6f38be5c7a9583f59140a293c236f616d086f5f1 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 8 Sep 2026 07:51:39 +0000 Subject: [PATCH 14/19] chore(docker): sync toolchain image pin to caddy-crowdsec-9eb9862f44b9e769 --- Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index 89f0d9469..987d895e6 100644 --- a/Dockerfile +++ b/Dockerfile @@ -19,8 +19,8 @@ ARG CHARON_TOOLCHAIN_IMAGE=ghcr.io/wikid82/charon-toolchain # NOT Renovate-tracked (a content-hash tag has no series to follow, N7) — the # toolchain-image.yml bot owns these two lines. DIGEST is the arch-independent # manifest-list (OCI index) digest, so one pin covers linux/amd64 + linux/arm64. -ARG CHARON_TOOLCHAIN_TAG=caddy-crowdsec-1efe7f19fa52a512 -ARG CHARON_TOOLCHAIN_DIGEST=sha256:6575f4c6a9f76074870c64df9dd4c9ebee812342f37f52ae5ef8f511ba9f8f00 +ARG CHARON_TOOLCHAIN_TAG=caddy-crowdsec-9eb9862f44b9e769 +ARG CHARON_TOOLCHAIN_DIGEST=sha256:b41e571d5951bbfc3daa3dccdca033ad9dee535a8e720ac7e3b0bce338f223b2 # Stage selector — default consumes the prebuilt toolchain image (no compile). # Fork PRs / bootstrap / offline builds pass From 30f954ceff99e3163629002063ca804a0e7344e9 Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Tue, 8 Sep 2026 04:14:56 -0400 Subject: [PATCH 15/19] fix(ci): restore the toolchain pin to match main's recipe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverts 6f38be5c, a stray `sync-pin-on-pr` push from the bot-authored main -> development propagation PR #1307. That job checked out the PR head branch — which for a propagation PR IS `main` — recomputed the toolchain key against the merge ref (which pulls in development's gRPC 1.83.2 bump, a toolchain-key input) and committed the resulting `caddy-crowdsec-9eb9862f44b9e769` pin directly onto `main`. `main`'s actual recipe still has `GRPC_VERSION=1.83.1`, whose key is `caddy-crowdsec-1efe7f19fa52a512` (@ sha256:6575f4c6…), the digest #1300's docker-build actually pulled. The mismatch fails `verify-toolchain-pin` on every PR against `main`. Verified on this branch: - scripts/toolchain-key.sh -> caddy-crowdsec-1efe7f19fa52a512 - imagetools inspect -> sha256:6575f4c6a9f76074870c64df9dd4c9ebee812342f37f52ae5ef8f511ba9f8f00 Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index 987d895e6..89f0d9469 100644 --- a/Dockerfile +++ b/Dockerfile @@ -19,8 +19,8 @@ ARG CHARON_TOOLCHAIN_IMAGE=ghcr.io/wikid82/charon-toolchain # NOT Renovate-tracked (a content-hash tag has no series to follow, N7) — the # toolchain-image.yml bot owns these two lines. DIGEST is the arch-independent # manifest-list (OCI index) digest, so one pin covers linux/amd64 + linux/arm64. -ARG CHARON_TOOLCHAIN_TAG=caddy-crowdsec-9eb9862f44b9e769 -ARG CHARON_TOOLCHAIN_DIGEST=sha256:b41e571d5951bbfc3daa3dccdca033ad9dee535a8e720ac7e3b0bce338f223b2 +ARG CHARON_TOOLCHAIN_TAG=caddy-crowdsec-1efe7f19fa52a512 +ARG CHARON_TOOLCHAIN_DIGEST=sha256:6575f4c6a9f76074870c64df9dd4c9ebee812342f37f52ae5ef8f511ba9f8f00 # Stage selector — default consumes the prebuilt toolchain image (no compile). # Fork PRs / bootstrap / offline builds pass From 2e750b4eff95371775dadec952a33830994b54ca Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Tue, 8 Sep 2026 04:15:45 -0400 Subject: [PATCH 16/19] fix(ci): stop sync-pin-on-pr from committing to long-lived branches MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `sync-pin-on-pr` job in toolchain-image.yml pushes a recomputed TAG/DIGEST pin onto `github.event.pull_request.head.ref`. Its only guard against unwanted runs was `github.actor != 'github-actions[bot]'`, which does not fire for the main -> development auto-propagation PR (#1307): that PR's `pull_request` event runs under a non-bot actor even though the PR itself is bot-authored and its head ref is `main`. Result: the job recomputed the key against the propagation merge ref (which drags in development's gRPC bump) and committed `caddy-crowdsec-9eb9862f44b9e769` straight to `main`, breaking `verify-toolchain-pin` repo-wide (see the companion revert in this PR). Adds three guards to the job `if:`: - refuse any protected head ref (main / development / nightly / feature/beta-release) outright — this job has no business rewriting a long-lived branch in place; - also skip when the PR *author* is github-actions[bot], not just when the triggering actor is; - (existing actor guard retained). Legitimate toolchain-key moves on `development` are still handled by the `open-bump-pr` job, which opens a reviewed bot PR rather than pushing in place. No workflow-`if:` test harness exists in-tree (bats covers the scripts, not YAML guards); actionlint passes. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .github/workflows/toolchain-image.yml | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/.github/workflows/toolchain-image.yml b/.github/workflows/toolchain-image.yml index b24938d58..c280ab440 100644 --- a/.github/workflows/toolchain-image.yml +++ b/.github/workflows/toolchain-image.yml @@ -314,10 +314,26 @@ jobs: # commit when the Dockerfile pin genuinely changes. With the deterministic # build above (same key => same digest) this essentially never fires unless a # tracked pin actually moved on the PR. + # + # Hard guards (2026-09-08 incident): this job pushes to + # `github.event.pull_request.head.ref`, so it must NEVER run when that head + # is a long-lived branch. The main -> development auto-propagation PR (#1307) + # has head ref `main`; its merge ref pulls development's gRPC bump into the + # key computation, and the resulting pin got committed straight to `main`. + # `github.actor` was not the bot on that event (propagation runs under a + # human/PAT identity), so the actor guard alone did not catch it — also + # check the PR author, and refuse any protected head ref outright. A genuine + # key move on `development` is handled by `open-bump-pr` (a reviewed bot PR), + # not by this in-place sync. if: >- github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'github-actions[bot]' && + github.event.pull_request.user.login != 'github-actions[bot]' && + github.event.pull_request.head.ref != 'main' && + github.event.pull_request.head.ref != 'development' && + github.event.pull_request.head.ref != 'nightly' && + github.event.pull_request.head.ref != 'feature/beta-release' && needs.build-toolchain.outputs.digest != '' runs-on: ubuntu-latest permissions: From 6cc3a1c07adac044c1419c2441b24b6ce0fd6467 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 8 Sep 2026 13:40:30 +0000 Subject: [PATCH 17/19] chore(main): release 0.40.1 --- .release-please-manifest.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.release-please-manifest.json b/.release-please-manifest.json index 650781534..235806fee 100644 --- a/.release-please-manifest.json +++ b/.release-please-manifest.json @@ -1,3 +1,3 @@ { - ".": "0.40.0" + ".": "0.40.1" } From f16974086a13eaacd0c4553c082ddfff551da9c9 Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Tue, 8 Sep 2026 03:57:09 -0400 Subject: [PATCH 18/19] ci: bump integration image artifact retention to 3 days 1-day retention breaks "Re-run failed jobs" on a run older than 24h now that the suite jobs depend on the upstream build job's artifact instead of building the image themselves. 3 days covers realistic re-run windows at negligible cost. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .github/workflows/integration-tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml index 2c8cadf03..361b090e2 100644 --- a/.github/workflows/integration-tests.yml +++ b/.github/workflows/integration-tests.yml @@ -50,7 +50,7 @@ jobs: with: name: charon-integration-image path: /tmp/charon-image.tar - retention-days: 1 # Only needed for the duration of this run's fan-out jobs. + retention-days: 3 # Outlives the run so "Re-run failed jobs" still works >24h later (suite jobs no longer self-contain the build). if-no-files-found: error cerberus: From 0423373381d325ac01b444f3c8026f6ff9b14626 Mon Sep 17 00:00:00 2001 From: Wikid82 Date: Tue, 8 Sep 2026 10:04:07 -0400 Subject: [PATCH 19/19] ci: use chore prefix for the toolchain bump-bot commits and PRs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `open-bump-pr` job in toolchain-image.yml opened its daily digest-refresh PR with a `feat(security):` title and commit message. `feat:` makes release-please cut a minor release on every merge to `main`, so a routine toolchain-digest refresh was bumping the version each time. Switch the bot PR title and commit-message to `chore(docker): refresh bundled proxy toolchain image`, matching the sibling `sync-pin-on-pr` commit style (`chore(docker): sync toolchain image pin to …`). The `security` label is kept; only the conventional-commit prefix changes. Also updates docs/plans/current_spec.md so its bot-PR example and commit-convention note no longer contradict the workflow. Affects future bot PRs only. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS --- .github/workflows/toolchain-image.yml | 4 ++-- docs/plans/current_spec.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/toolchain-image.yml b/.github/workflows/toolchain-image.yml index c280ab440..4681c6024 100644 --- a/.github/workflows/toolchain-image.yml +++ b/.github/workflows/toolchain-image.yml @@ -426,14 +426,14 @@ jobs: base: development branch: bot/bump-toolchain-image delete-branch: true - title: "feat(security): refresh bundled proxy toolchain image" + title: "chore(docker): refresh bundled proxy toolchain image" labels: | dependencies automated docker security commit-message: | - feat(security): refresh bundled proxy toolchain image + chore(docker): refresh bundled proxy toolchain image Rebuilds the prebuilt Caddy/CrowdSec toolchain image so the shipped binaries pick up upstream fixes, and bumps the digest pin in the diff --git a/docs/plans/current_spec.md b/docs/plans/current_spec.md index 5563182f5..b2411dd01 100644 --- a/docs/plans/current_spec.md +++ b/docs/plans/current_spec.md @@ -158,7 +158,7 @@ The composite action's own doc comment (`action.yml:16-33`) instructs CVE-scan c ### 2.5 Constraints from `CLAUDE.md` / `ARCHITECTURE.md` - All frontend in `frontend/`, backend in `backend/` — unaffected (this is CI/build only). -- Conventional commits; `(security)` scope only for genuine security work, subject line vague. The digest-bump and freshness-guard commits *are* security-relevant — use `feat(security):` / `fix(security):` with vague subjects (e.g. `feat(security): pin bundled proxy toolchain to a scanned prebuilt image`). +- Conventional commits; `(security)` scope only for genuine security work, subject line vague. The initial-pin and freshness-guard commits *are* security-relevant — use `feat(security):` / `fix(security):` with vague subjects (e.g. `feat(security): pin bundled proxy toolchain to a scanned prebuilt image`). The routine daily digest-refresh bot PR uses **`chore(docker):`** — `feat:` there makes release-please cut a minor release on every refresh. - Weekly `nightly → main` promotion PRs merge via **merge commit**. This feature's PR targets `development` (normal flow) — **confirmed it does not touch `weekly-nightly-promotion.yml`** and imposes no new constraint on the promotion merge method. (`weekly-nightly-promotion.yml` carries the app image through unchanged; the toolchain digest pin travels with the Dockerfile like any other line.) - `ARCHITECTURE.md` §"Deployment Architecture / Multi-Stage Dockerfile" (`:1082`), §"Infrastructure" table (`:158`), §"Directory Structure" (`:286`), §"Layer 2: CrowdSec Integration" (`:780`) must be updated (§9). - **Ignore-file check (CLAUDE.md "Ignore Files"):** the new files are `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh`, `scripts/tests/toolchain-key.bats` (+ `verify-toolchain-pin.bats`, `helpers/toolchain_fixture.bash`), `.github/workflows/toolchain-image.yml`, `docs/ci/toolchain-image.md`. **Correction (Rev 2.1):** the earlier claim that `scripts/` is not copied into the image was wrong — `Dockerfile` `COPY scripts/ /app/scripts/` copies the whole directory into the runtime image (it already ships ~40 `scripts/*.sh` + a pre-existing `.bats`). These four build-only helpers are used only by `toolchain-image.yml` and the `quality-checks.yml` `verify-toolchain-pin` / bats jobs from a plain checkout — never from inside a built container — so **`.dockerignore` now excludes `scripts/tests/`, `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh`** (blacklist semantics, no `!scripts/…` re-includes to fight). `.github/` and `docs/` are already excluded, so `toolchain-image.yml` / `docs/ci/toolchain-image.md` never enter the context. `.gitignore` — these are source files that must be committed; none matches an existing ignore glob → **no `.gitignore` change**. `.codecov.yml` — shell/bats and YAML carry no Go/TS coverage → **no `.codecov.yml` change**. Recorded explicitly per CLAUDE.md. @@ -571,7 +571,7 @@ open-bump-pr: # event == schedule | workflow_dispatch | workflow_ - peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1 base: development branch: bot/bump-toolchain-image # updated in place if already open - title: "feat(security): refresh bundled proxy toolchain image" + title: "chore(docker): refresh bundled proxy toolchain image" labels: dependencies, automated, docker, security body: old→new digest, Trivy CRITICAL/HIGH summary, verification checklist - on failure: actions/github-script → open issue "🚨 Toolchain image rebuild failed"