Repository navigation
fix(metrics): never block a /metrics scrape on DuckDB - #320
Merged
Merged
Conversation
An 8-vCPU benchmark aborted when a /metrics scrape exceeded the harness's 15 s timeout. Most scrapes took about 4 ms, but 21 took over a second and the slowest took 15.9 s. Each scrape queried duckdb_memory() on the read pool. Its 5 s context did not bound the wait, because the query waited inside the engine behind long writes: one 15.5 s scrape returned at the instant a version-rollup INSERT was interrupted. - A background sampler reads DuckDB memory every 5 s with its own bounded context, and scrapes serve the last complete sample. fanout_duckdb_memory_last_success_timestamp_seconds shows when that sample is stale, and repeated failures are logged once per streak. - Pool statistics and the compaction backlog are read from cached snapshots, so no scrape waits on a pool, a lock or the engine. Scrapes now return in 0.2-0.6 ms while DuckDB is blocked. Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
The background memory sampler borrowed a read-pool connection every 5 s. A memory query blocked inside the engine could therefore hold a request slot for as long as it waited, and the stall test that asserts no read connection is in use during a write raced with it. The sampler now uses one dedicated connection from the shared connector, so it never takes read or write capacity. Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
Review fixes for the background DuckDB memory sampler: - Close waits at most one second for the sampler. If the engine ignores cancellation, handles stay alive until the abandoned query returns, so native code never runs against a closed instance. - fanout_duckdb_memory_bytes is exported only while the sample is at most two intervals old. A failed or hung sampler therefore drops the series instead of reporting stale values as current; the last-success timestamp is always exported. - Pool gauges read live db.Stats() again. The pool cache, its sampler and its join are removed. - Each read uses the sampling interval as its timeout, the next read starts one interval after the previous one finishes, and a successful sample is kept even if its deadline passes after the query returns. - Connection initialization runs SET TimeZone='UTC' with the context the driver passes to Connect, so connections opened after startup still initialize. Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An 8-vCPU benchmark of v2026.10.7-alpha aborted when a
/metricsscrape exceeded the harness's 15 s client timeout. In the run's request log, 1,303 scrapes had a p50 of 4.2 ms, but 21 took over a second and the slowest took 15.9 s.Every scrape ran
SELECT ... FROM duckdb_memory()on the read pool. Its 5 s context did not bound the wait, because the query waited inside the engine behind long writes. One 15.5 s scrape returned at the instant a version-rollup INSERT was interrupted.fanout_duckdb_memory_last_success_timestamp_secondsexposes the sample's age.Test plan
just check(pre-push)just test-race-count=5, including under-race.