sec: admin auth, embed-backfill durability, and embed-queue hardening (M23, L20, L21, L22)

[TeoSlayer]

Addresses the open cosift rows from the findings register: M23, L20, L21, L22.

Each change has a test that fails on main and passes here.

M23 — embed-backfill never persisted its vectors

handleEmbedBackfill called s.hnsw.AddPassage for every chunk it embedded, which only mutates the in-memory graph. Nothing wrote the graph back to Pebble, so a restart discarded the entire backfill.

Snapshot the graph via s.hnsw.Persist once the worker pool drains, on a detached context so a client disconnect can't abort the write. The response now carries persisted (and persist_error when it fails) instead of silently reporting success.

Test: TestEmbedBackfillPersistsHNSW runs the handler against an empty graph and reloads via index.LoadHNSW — on main the store holds no HNSW meta at all.

L20 — non-constant-time peer/admin token compare

The pebble HTTP path compared the request's bearer credential against cfg.Cluster.PeerAuthToken with !=, which short-circuits on the first differing byte. internal/server/http.go already used subtle.ConstantTimeCompare; this path diverged, in 24 places.

Added peerTokenOK / bearerToken in serve_helpers.go and routed every gate through it, including the two odd variants in feedback.go and querylog.go. The empty-token single-node default is preserved.

Tests: TestPeerTokenOK (accept/reject matrix incl. prefix and suffix cases), TestRequireAdminGate, and TestAdminGatesUseConstantTimeCompare — a source guard that fails if a raw comparison reappears.

L21 — data race on zombieDebugLogged

The bounded diagnostic counter behind COSIFT_ZOMBIE_RECLAIM=1 was a plain int read and incremented from every crawl worker. Now an atomic.Int64, with the bound expressed as Add(1) <= 5.

Test: TestZombieReclaimDiagnosticCounterConcurrent crawls 60 pages at MaxConcurrent=16 — reproduces reliably under -race on main (3/3 runs).

Note: the test uses a locally-defined embedder rather than the package's stubEmbedder, whose own call counter is unsynchronized and only safe at MaxConcurrent=1.

L22 — embed queue dropped passages and lost in-flight jobs

Two separate losses on the decoupled path:

1. Backpressure. The producer did a non-blocking send and counted the job as dropped the instant the buffer was full — a brief embedder slowdown cost that document's passages outright. Producers now retry with a bounded wait (COSIFT_EMBED_ENQUEUE_WAIT_MS, default 5s) before giving up.
2. Shutdown-cancel. Embed workers ran on runCtx, which terminator cancels as soon as the frontier goes quiet. Everything still queued then failed with a cancelled context. Workers now run on a context detached from that cancellation, with a per-job timeout (COSIFT_EMBED_JOB_SECONDS, default 120s) and a post-crawl drain deadline (COSIFT_EMBED_DRAIN_SECONDS, default 300s) so shutdown wall-clock stays bounded when the embedder is slow or unreachable. Work abandoned past the deadline is counted as dropped and left for embed-backfill.

The per-job body moved into runEmbedJob so the timeout context is scoped cleanly.

Test: TestEmbedPoolFinishesQueuedJobsAfterCrawlContextCancel crawls 40 pages against a 100ms embedder that honors cancellation. On main: 26 of 40 jobs fail. Here all 40 complete. Plus focused tests for the enqueue wait budget and the drain deadline.

Verification

go build ./... and go test -race -count=1 ./... are green across all 17 packages.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 58.57988% with 70 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/crawler/crawler.go	52.77%	25 Missing and 9 partials ⚠️
cmd/cosift/serve_crawl.go	50.00%	19 Missing and 2 partials ⚠️
cmd/cosift/serve_admin.go	77.50%	6 Missing and 3 partials ⚠️
cmd/cosift/feedback.go	0.00%	3 Missing ⚠️
cmd/cosift/querylog.go	0.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!