Skip to content

Commit d023e33

Browse files
committed
fix(webapp): ignore stale ClickHouse gauges in the live queue blocks
Gauges are only emitted while a queue is active, so a drained queue's newest bucket holds its last non-zero reading. Trust the CH gauge only when its newest bucket is recent; past that, fall back to the loader's Redis/PG value instead of lingering on a stale count.
1 parent 96914d1 commit d023e33

1 file changed

Lines changed: 16 additions & 4 deletions

File tree

  • apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.queues_.$queueParam

apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.queues_.$queueParam/route.tsx

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -906,6 +906,10 @@ function KeyDrilldown({
906906
// signals the queue is stuck, not just busy.
907907
const OLDEST_WAIT_WARNING_MS = 5 * 60_000;
908908

909+
// How recent the newest ClickHouse gauge bucket must be to drive the live blocks. Above the 10s
910+
// bucket + pipeline lag; past it we treat the queue as idle and fall back to the loader value.
911+
const LIVE_GAUGE_FRESH_MS = 90_000;
912+
909913
function QueueStats({
910914
queue,
911915
environmentConcurrencyLimit,
@@ -951,11 +955,19 @@ function QueueStats({
951955
refreshIntervalMs: 15_000,
952956
}
953957
);
958+
// Gauges are only emitted while the queue is active, so a drained queue's newest bucket is a past
959+
// one holding its last non-zero reading. Trust the CH gauge only when its newest bucket is recent
960+
// (covers the 10s bucket + pipeline lag); once it ages out we fall back to the loader's live
961+
// Redis/PG value instead of lingering on a stale count.
954962
const latest = liveRows.length > 0 ? liveRows[liveRows.length - 1] : undefined;
955-
const runningLive = latest ? toNumber(latest.running) : null;
956-
const queuedLive = latest ? toNumber(latest.queued) : null;
957-
const limitLive = latest ? toNumber(latest.q_limit) : null;
958-
const ckWaitLive = latest ? toNumber(latest.ck_wait) : null;
963+
const latestBucketMs = latest ? clickhouseTimeToMs(latest.t) : NaN;
964+
const liveFresh =
965+
Number.isFinite(latestBucketMs) && Date.now() - latestBucketMs < LIVE_GAUGE_FRESH_MS;
966+
const fresh = latest && liveFresh ? latest : undefined;
967+
const runningLive = fresh ? toNumber(fresh.running) : null;
968+
const queuedLive = fresh ? toNumber(fresh.queued) : null;
969+
const limitLive = fresh ? toNumber(fresh.q_limit) : null;
970+
const ckWaitLive = fresh ? toNumber(fresh.ck_wait) : null;
959971

960972
// Prefer CH once it has landed; loader values before that.
961973
const runningDisplay = runningLive ?? queue.running;

0 commit comments

Comments
 (0)