Skip to content

fix(keeper): surface oracle staleness in /health's top-level status - #368

Open
Morenikeoa wants to merge 1 commit into
dcccrypto:mainfrom
Morenikeoa:fix/health-status-oracle-staleness
Open

fix(keeper): surface oracle staleness in /health's top-level status#368
Morenikeoa wants to merge 1 commit into
dcccrypto:mainfrom
Morenikeoa:fix/health-status-oracle-staleness

Conversation

@Morenikeoa

Copy link
Copy Markdown
Contributor

Problem

An external oracle outage (DexScreener + Jupiter both down) already pauses affected HYPERP markets via stalePausedMarkets (src/index.ts), correctly visible on GET /pause-status. But /health's top-level status field is computed by computeHealthStatus() (src/lib/health-status.ts) using only crank cadence and liquidation-scan liveness — timeSinceLastOracle is surfaced as a raw informational JSON field in the /health response but never feeds into the status decision, and oracleService.getStaleMarkets() is never consulted by /health at all.

Production Impact

If all external oracle sources go down simultaneously while the crank loop keeps ticking on schedule (cranking with the on-chain fallback price, capped at ON_CHAIN_FALLBACK_MAX_MS), /health reports status: "ok" / HTTP 200 indefinitely. An operator or orchestrator alerting only on /health's top-level status — the natural single endpoint to watch for uptime/paging — would miss a real, ongoing oracle outage that /pause-status already correctly tracks. The two endpoints disagree on health for the same condition.

Fix

Added an optional stalePausedMarketCount input to computeHealthStatus(), defaulting to 0 so every existing caller/test is unaffected. Wired from index.ts's existing stalePausedMarkets set size. Mirrors the existing crank/liquidation-scan staleness pattern already in this function (the GH#2025 block):

  • A partial outage (some but not all tracked markets stale-paused) degrades "ok""degraded".
  • Every tracked market being stale-paused escalates to "down".
  • An already-"down" status (e.g. from stalled crank) is never de-escalated by a partial oracle outage.

Proof of Fix

New tests in tests/lib/health-status.test.ts (BUG-109: oracle staleness gates status):

  • Omitted/zero stalePausedMarketCount is a no-op (confirms zero behavior change for existing callers)
  • Partial outage degrades "ok" to "degraded"
  • Total outage (every tracked market stale-paused) escalates to "down"
  • An already-"down" status is not de-escalated
  • marketsTracked === 0 still short-circuits to "ok" regardless

Test Output

 Test Files  1 passed (1)
      Tests  15 passed (15)

pnpm build — clean, zero errors.

An external oracle outage (DexScreener + Jupiter both down) already pauses
affected HYPERP markets via stalePausedMarkets, correctly visible on
GET /pause-status. But /health's top-level `status` field was computed by
computeHealthStatus() using only crank cadence and liquidation-scan
liveness -- timeSinceLastOracle was surfaced as a raw informational JSON
field but never fed into the status decision, and getStaleMarkets() was
never consulted by /health at all.

If all external oracle sources go down simultaneously while the crank loop
keeps ticking on schedule (cranking with the on-chain fallback price),
/health reports status: "ok" / HTTP 200 indefinitely. An operator or
orchestrator alerting only on /health's top-level status -- the natural
single endpoint to watch -- would miss a real, ongoing oracle outage that
/pause-status already correctly tracks.

Adds an optional stalePausedMarketCount input to computeHealthStatus()
(defaults to 0 -- every existing caller/test is unaffected), wired from
index.ts's existing stalePausedMarkets set. Mirrors the existing crank/
liquidation-scan staleness pattern: a partial outage degrades "ok" to
"degraded"; every tracked market being stale-paused escalates to "down";
an already-"down" status is never de-escalated by a partial outage.

BUG-109 from a clean-room Phase 4 audit pass.
@coderabbitai

coderabbitai Bot commented Jun 27, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@Morenikeoa, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 43 minutes and 11 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: dac9a23c-2c77-40d7-804d-a315ae18ac4b

📥 Commits

Reviewing files that changed from the base of the PR and between 8ee810d and 074d285.

📒 Files selected for processing (3)
  • src/index.ts
  • src/lib/health-status.ts
  • tests/lib/health-status.test.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant