Skip to content

fix(metrics): never block a /metrics scrape on DuckDB - #320

Merged
vishr merged 3 commits into
mainfrom
fix/metrics-nonblocking
Oct 16, 2026
Merged

vishr merged 3 commits into
mainfrom
fix/metrics-nonblocking

Conversation

@vishr

@vishr vishr commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Summary

An 8-vCPU benchmark of v2026.10.7-alpha aborted when a /metrics scrape exceeded the harness's 15 s client timeout. In the run's request log, 1,303 scrapes had a p50 of 4.2 ms, but 21 took over a second and the slowest took 15.9 s.

Every scrape ran SELECT ... FROM duckdb_memory() on the read pool. Its 5 s context did not bound the wait, because the query waited inside the engine behind long writes. One 15.5 s scrape returned at the instant a version-rollup INSERT was interrupted.

  • Memory sampling. A background sampler reads DuckDB memory every 5 s, with its own bounded context, on one dedicated connection from the shared connector. A blocked memory query therefore never takes read or write capacity.
    • Scrapes serve the last complete sample.
    • fanout_duckdb_memory_last_success_timestamp_seconds exposes the sample's age.
    • Repeated failures are logged once per streak.
  • Other collectors. Pool statistics and the compaction backlog come from cached or atomic snapshots, so no scrape waits on a pool, a lock or the engine.

Test plan

  • just check (pre-push)
  • just test-race
  • Regression tests:
    • with the memory query blocked, or the pool and catalog held, a scrape returns in under 100 ms (measured 0.2-0.6 ms);
    • memory sampling needs no read or write pool capacity;
    • the stall test runs -count=5, including under -race.

vishr added 3 commits October 16, 2026 08:58
An 8-vCPU benchmark aborted when a /metrics scrape exceeded the
harness's 15 s timeout. Most scrapes took about 4 ms, but 21 took over
a second and the slowest took 15.9 s. Each scrape queried
duckdb_memory() on the read pool. Its 5 s context did not bound the
wait, because the query waited inside the engine behind long writes:
one 15.5 s scrape returned at the instant a version-rollup INSERT was
interrupted.

- A background sampler reads DuckDB memory every 5 s with its own
  bounded context, and scrapes serve the last complete sample.
  fanout_duckdb_memory_last_success_timestamp_seconds shows when that
  sample is stale, and repeated failures are logged once per streak.
- Pool statistics and the compaction backlog are read from cached
  snapshots, so no scrape waits on a pool, a lock or the engine.

Scrapes now return in 0.2-0.6 ms while DuckDB is blocked.

Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
The background memory sampler borrowed a read-pool connection every
5 s. A memory query blocked inside the engine could therefore hold a
request slot for as long as it waited, and the stall test that asserts
no read connection is in use during a write raced with it. The sampler
now uses one dedicated connection from the shared connector, so it
never takes read or write capacity.

Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
Review fixes for the background DuckDB memory sampler:

- Close waits at most one second for the sampler. If the engine ignores
  cancellation, handles stay alive until the abandoned query returns,
  so native code never runs against a closed instance.
- fanout_duckdb_memory_bytes is exported only while the sample is at
  most two intervals old. A failed or hung sampler therefore drops the
  series instead of reporting stale values as current; the last-success
  timestamp is always exported.
- Pool gauges read live db.Stats() again. The pool cache, its sampler
  and its join are removed.
- Each read uses the sampling interval as its timeout, the next read
  starts one interval after the previous one finishes, and a successful
  sample is kept even if its deadline passes after the query returns.
- Connection initialization runs SET TimeZone='UTC' with the context the
  driver passes to Connect, so connections opened after startup still
  initialize.

Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
@vishr
vishr merged commit 89bf51b into main Oct 16, 2026
14 checks passed
@vishr
vishr deleted the fix/metrics-nonblocking branch October 16, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant