fix(keeper): surface oracle staleness in /health's top-level status - #368
fix(keeper): surface oracle staleness in /health's top-level status#368Morenikeoa wants to merge 1 commit into
Conversation
An external oracle outage (DexScreener + Jupiter both down) already pauses affected HYPERP markets via stalePausedMarkets, correctly visible on GET /pause-status. But /health's top-level `status` field was computed by computeHealthStatus() using only crank cadence and liquidation-scan liveness -- timeSinceLastOracle was surfaced as a raw informational JSON field but never fed into the status decision, and getStaleMarkets() was never consulted by /health at all. If all external oracle sources go down simultaneously while the crank loop keeps ticking on schedule (cranking with the on-chain fallback price), /health reports status: "ok" / HTTP 200 indefinitely. An operator or orchestrator alerting only on /health's top-level status -- the natural single endpoint to watch -- would miss a real, ongoing oracle outage that /pause-status already correctly tracks. Adds an optional stalePausedMarketCount input to computeHealthStatus() (defaults to 0 -- every existing caller/test is unaffected), wired from index.ts's existing stalePausedMarkets set. Mirrors the existing crank/ liquidation-scan staleness pattern: a partial outage degrades "ok" to "degraded"; every tracked market being stale-paused escalates to "down"; an already-"down" status is never de-escalated by a partial outage. BUG-109 from a clean-room Phase 4 audit pass.
|
Warning Review limit reached
More reviews will be available in 43 minutes and 11 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Problem
An external oracle outage (DexScreener + Jupiter both down) already pauses affected HYPERP markets via
stalePausedMarkets(src/index.ts), correctly visible onGET /pause-status. But/health's top-levelstatusfield is computed bycomputeHealthStatus()(src/lib/health-status.ts) using only crank cadence and liquidation-scan liveness —timeSinceLastOracleis surfaced as a raw informational JSON field in the/healthresponse but never feeds into thestatusdecision, andoracleService.getStaleMarkets()is never consulted by/healthat all.Production Impact
If all external oracle sources go down simultaneously while the crank loop keeps ticking on schedule (cranking with the on-chain fallback price, capped at
ON_CHAIN_FALLBACK_MAX_MS),/healthreportsstatus: "ok"/ HTTP 200 indefinitely. An operator or orchestrator alerting only on/health's top-level status — the natural single endpoint to watch for uptime/paging — would miss a real, ongoing oracle outage that/pause-statusalready correctly tracks. The two endpoints disagree on health for the same condition.Fix
Added an optional
stalePausedMarketCountinput tocomputeHealthStatus(), defaulting to0so every existing caller/test is unaffected. Wired fromindex.ts's existingstalePausedMarketsset size. Mirrors the existing crank/liquidation-scan staleness pattern already in this function (theGH#2025block):"ok"→"degraded"."down"."down"status (e.g. from stalled crank) is never de-escalated by a partial oracle outage.Proof of Fix
New tests in
tests/lib/health-status.test.ts(BUG-109: oracle staleness gates status):stalePausedMarketCountis a no-op (confirms zero behavior change for existing callers)"ok"to"degraded""down""down"status is not de-escalatedmarketsTracked === 0still short-circuits to"ok"regardlessTest Output
pnpm build— clean, zero errors.