Readiness stays 500 for 3h after a transient DB error during boot (TrackerScriptCache refresh_all never retried)

[rosnovsky]

Past Issues Searched

Searched for TrackerScriptCache / tracker_script_cache / readiness; nothing matching.

Version

Community Edition v3.2.1 (also checked master: same code in lib/plausible/cache.ex and system_controller.ex).

What happened

A single transient DB error during boot makes /api/health (readiness) return 500 for 3 hours, while the app otherwise serves traffic fine. Under Kubernetes with the documented health probe this becomes an infinite restart loop.

Sequence, from logs:

1. At boot Plausible.Cache.Warmer runs refresh_all for every cache at once, plus Oban starts. With a 10-connection pool and a DB a few hundred ms away, the pool hit DBConnection.ConnectionError ... queue_timeout after ~6s.
2. Plausible.Cache.refresh/3 catches that and only logs Error refreshing 'tracker_script_cache' - ... (cache.ex ~L233). The cache stays empty.
3. TrackerScriptCache is in @critical_caches for readiness. ready?/1 returns false because size() == 0 but count_all() == 9. Readiness responds 500 {"sessions":"ok","postgres":"ok","clickhouse":"ok","sites_cache":null}.
4. The next refresh_all for tracker_script_cache is scheduled ~3 hours out (interval 10805730 in the warmer init log). refresh_updated_recently runs every 30s but only picks up rows updated since the last run, so it never repairs a cache that missed the initial load.
5. If the deployment's startup/liveness probe points at /api/health, the pod is killed long before 3h elapse, and the cycle repeats on every boot. I saw 56 consecutive restarts.

Plausible.Site.Cache and Plausible.Shield.IPRuleCache have the same shape but shorter refresh_all intervals (15 min / 3 min), so they self-heal before anyone notices.

Expected

A failed boot-time refresh_all should retry (short backoff) instead of waiting for the next multi-hour cycle, or refresh_updated_recently should fall back to a full refresh while ready?/1 is false.

Workaround

bin/plausible rpc 'Plausible.Site.TrackerScriptCache.refresh_all()' inside the container makes readiness green immediately. For Kubernetes, point startup/liveness at /api/system/health/live and keep /api/health for readiness only.