diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a677a44c6..be8276a80 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -115,6 +115,8 @@ jobs: - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 - name: just test run: just test + - name: Compile opt-in read benchmarks + run: just readbench-check race-shards: name: Race shards diff --git a/docs/proposals/0006-read-performance.md b/docs/proposals/0006-read-performance.md new file mode 100644 index 000000000..c57c224e7 --- /dev/null +++ b/docs/proposals/0006-read-performance.md @@ -0,0 +1,670 @@ +# Read performance under sustained ingest + +Status: Draft +Date: 2026-11-14 +Pull request: none yet + +## Summary + +Qualify Pitot by the ingest rate it can sustain while dashboards, searches and +trace investigation remain responsive. Keep the existing representative-data +targets of 500 ms p95 per warm panel and 1,500 ms p95 for a complete twelve-panel +dashboard. Add tail-latency, freshness and offered-load accounting to the +benchmark before treating any new run as evidence of mixed-load capacity. + +The partial-window trace-candidate experiment was rejected: its SQL-kernel gain +did not survive completed-query measurement. The current implementation +includes exact checkpoint-conflict retries, explicit captured file lists, earlier +startup maintenance, service-rollup footer pruning and fresh fixed-window +coalescing. Projection-based histogram expansion has been reintroduced +with those changes after failing earlier backlog comparisons. +An atomically published committed service watermark removes the standalone +preflight lookup that blocked all five panel slots in a captured stall. + +That runtime combination is **not qualified**: repeated runs still produced +backend busy responses and missed generator arrivals. A three-table analytical +cache-layout change is now retained after a +source-backed checkpoint lock diagnosis and deterministic native proof. Its +90-second comparison had no backend refusals, but two generator misses still +fail offered-load acceptance. Repeated representative qualification remains +outstanding. Faster successful reads do not erase earlier refusals; neither these local experiments nor the proposed numerical goals +establish deployed mixed-load capacity. + +### Fresh fixed-window requests: share one later replacement + +The fresh-result path exposed a separate, deterministic source of duplicate +work. When callers bypass stored results and wait for an older flight, its +replacement inherited the initiating caller's original arrival stamp. Other +waiting callers could therefore reject that replacement even though it was +registered after all their arrivals. For an absolute query window, they +requested the same data interval and could safely share that newer read. + +The cache now records registration time separately for explicitly fixed-window +panel computations. Both the initiating computation and the joining caller +must have clock-independent windows. Effective request time overrides control +eligibility, and the full-admission-queue path uses the same rule. Every read +snapshot starts after registration. A caller arriving after registration must +still wait for another fresh computation. Original stamps continue to control +relative-window matching and stored-result TTL; no limits or TTLs are increased. +Generic cached operations and relative windows retain their previous behavior. + +A real Executor regression holds an old native snapshot, commits late telemetry +and schedules two fresh callers before one replacement is registered. Before +the fix it executes three queries; after the fix it executes two. Both fresh +callers observe the late record, and the old response retains its original +snapshot. Relative-window controls retain three queries and their distinct +window ends. Additional cases protect request time overrides, later arrivals, +canceling one shared waiter, and joining through a full admission queue. +Mixed-load evidence for this fix is recorded separately from the earlier SQL +experiments and benchmark-control versions. + +## Follow-through: measured read experiments and maintenance fix + +Astra, AGY Carbon and Claude Fable were consulted independently. Their source +reviews identified hypotheses, not measured gains. We selected complete-query +profiling and a fixed-rate mixed-load experiment, including raw distinct-trace +panels, rather than accepting single-kernel timings or changing admission limits. +The other retention, footer-cache, compaction-layout and admission proposals +remain unimplemented pending measurements and correctness/resource review. + +### Duration quantiles: measured gain, current candidate under evaluation + +The projection candidate expands each row's map entries in a SELECT projection, +replacing correlated lateral `FROM source, unnest(map_entries(hist))`. Grouping, +totals, cumulative counts and the existing quantile interpolation remain the +same. Inputs have fixed internal schemas, so the new entry alias does not +collide with user measure aliases. No physical format, cache schema, resource +budget, admission policy or approximation change is involved. + +A paired full-Executor experiment used one canonical generated snapshot, +disabled result caching, three warmups and 100 interleaved samples per variant. +Complete results matched across nine shapes. Extra canonical records made +otherwise undefined equal-value LIMIT ties unambiguous instead of weakening +the result comparison. Representative p95 measurements: + +| Executor shape | Previous SQL | Projection UNNEST | +| --- | --- | --- | +| Latency time series | 14.92 ms | 8.33 ms | +| Quantile stat | 15.27 ms | 8.32 ms | +| Multiple quantiles | 17.61 ms | 10.35 ms | +| Ranked quantiles | 30.64 ms | 15.02 ms | +| Limited quantile table | 26.78 ms | 9.57 ms | +| Trends | 33.13 ms | 17.19 ms | +| Previous-period comparison | 23.48 ms | 14.64 ms | +| Protected raw-count control | 17.81 ms | 17.72 ms | +| Twelve-panel executor request | 60.07 ms | 54.61 ms | + +An independent latency repeat measured 15.67 to 8.52 ms p95. Native profiles +explain the improvement: the lateral delimiter/hash join and INOUT_FUNCTION +are replaced by plain UNNEST, expanding the same 54,630 bin rows into the same +9,072 grouped bins and 144 result rows. Native peak buffer memory decreased +from 113.6 to 64.3 MB, with zero temporary-directory spill in both variants. +These engine-buffer figures are distinct from whole-process RSS. Native +correctness tests cover empty/null maps, zero bins, multiple quantiles, totals, +folded groups and raw-vs-rollup parity. + +### Checkpointing: allow an older reader to drain + +A native regression reproduced a maintenance failure even while both writer +gates were held: an older read snapshot plus newer committed WAL work makes +non-FORCE CHECKPOINT return its transaction-conflict diagnostic. READ ONLY +transactions exhibit the same native behavior. The previous assumption that +writer gates alone prevented this refusal was incorrect. + +Checkpointing now retains both gates and one writer connection, retrying only +the exact typed conflict from the pinned engine. Retry admission and waits +have a 250 ms budget with context-aware 10 ms intervals. Native checkpoint I/O +keeps the original caller deadline; it is not given a new 250 ms execution cap. +Cancellation, unrelated errors and exhausted retries remain explicit failures. +Metrics count one logical checkpoint, including retry wait, rather than one +failure per transient attempt. No FORCE operation interrupts active readers. + +Tests prove success after the old reader drains, both gates remaining held, +bounded explicit exhaustion, cancellation, unrelated-error propagation and +unchanged native execution deadlines. The tradeoff is up to 250 ms additional +contention wait holding writer gates; mixed acceptance must inspect ingestion, +cache/compaction debt, freshness and memory as well as read latency. + +### Mixed-load methodology + +The opt-in mixed harness uses a fixed populated 24-hour history, warmed +analytical and completed-batch caches, the real Executor, and background +ingestion/rollups/compaction. Requests explicitly bypass result caching; normal +in-flight coalescing remains enabled. Each arrival is one of a single quantile +panel, a four-panel request, or the existing varied twelve-panel dashboard with +three raw distinct-trace panels. Exact counts and rankings are checked. + +The historical v5 comparison fixes 4 GB native memory, four native threads, eight read +connections, 32 client workers and queue capacity 64 across all rates. It ran +on the local Apple M3 Max, not an isolated four-vCPU host. It offered 20, 40, 80 +and 160 operations/s for 15 seconds each, with 2,048 spans and 2,048 logs per +live batch at five commits/s. Actual accepted ingestion is reported separately +from the 20,480-item/s target. Maintenance runs at an explicitly accelerated +five-second interval to expose overlap; the product default remains one hour. + +The scheduler records every offered slot, missed arrivals, client queue-full +events, backend busy responses, errors and during-load/drain completions. +Successful latency starts at scheduled arrival. Background operational errors +invalidate a run; shutdown cancellation does not. Peak RSS, cache markers, +compaction debt and analytical lag are reported separately. Fewer than 1,000 +successful samples per kind cannot establish the proposed p99 qualification. +Short synthetic probes do not replace representative production replay or a +long sustained qualification run. + +### Current runtime candidates and mixed-load result + +Startup maintenance now starts before the initial analytical rollup rather +than waiting for it to finish. Regular regressions hold the analytical gate +and verify independent compaction, exact raw reads, eventual completed-cache +and analytical convergence, retention and cancellation. An opt-in cold-start +guard also exercises compaction into an output larger than the incremental +cache row budget before the analytical gate is released. These are lifecycle +and correctness guards, not capacity measurements. + +Service rollup first discovers affected buckets from its captured catalog, then +rebinds that **same captured catalog** using each signal's event-time footer +bounds for the complete affected-minute interval. Tests protect late records +repairing an old minute, disjoint span/log/metric bounds, unknown statistics, +nanosecond half-open boundaries and captured inputs surviving publication, +compaction and retirement. This does not refresh the catalog mid-transaction +or replace event time with ingestion time. + +After earlier maintenance and explicit file lists, projection histogram SQL +and fixed-window coalescing were combined for new mixed-load checks. With +786,432 historical spans and 786,432 logs, initial compaction debt of 257, +40 scheduled operations/s and approximately 20,200 accepted ingest items/s, +the first combined run completed 600/600 reads; single- and twelve-panel p95 +were 10.4 and 71.5 ms. That success did not repeat cleanly: a later run completed +597/600 successfully, with transient backend busy responses, single-panel p95 +12.514 ms, twelve-panel p95 136.971 ms and accepted ingestion 20,207 items/s. +Successful-read tails are encouraging, but the refusals fail the zero-error +goal and each short run has insufficient per-kind samples for p99 qualification. + +The committed-service-watermark candidate publishes only after successful +service-rollup, startup rewind or skip-to-latest commits. It leaves all +snapshot-bound coverage SQL intact. Five regression tests pass with the race +detector, including exhausted read-pool access, cancellation, uncommitted and +failed refreshes, successful retries, restart and concurrent readers. The old +getter fails the exhausted-pool regression. + +The indexed combination's 90-second result still failed offered-load acceptance. +Its comparison with the subsequent cache-layout change is recorded below; +neither run establishes an isolated watermark speedup. + +### Checkpoint scan lock and retained analytical cache layout + +Pinned DuckDB 2's scan initialization can acquire the database's shared vacuum +lock before deciding whether an ART index can answer a filtered scan. A +compound-key index can therefore introduce a wait even when the final operation +is a sequential scan. The acquisition precedes the index-probe outcome in +[TableScanInitGlobal](https://github.com/duckdb/duckdb/blob/v2.0.0/src/function/table/table_scan.cpp#L937-L987). +A full checkpoint acquires the exclusive side and retains its guard across the +checkpoint in +[CreateCheckpoint](https://github.com/duckdb/duckdb/blob/v2.0.0/src/storage/storage_manager.cpp#L703-L754). +This supplies a concrete mechanism for the observed checkpoint-associated +scan-initialization waits; it does not attribute every busy response to that lock. + +The regular native regression +[checkpoint_scan_lock_test.go](../../internal/engine/checkpoint_scan_lock_test.go) +uses persisted, identical NOT NULL tables, with a compound primary key on only +the control. After committed WAL-producing updates, ordinary CHECKPOINT enters +a test-only one-second debug phase. A newly flushed WAL checkpoint record +establishes the phase; the control's native execution frame establishes that +it has started. The heap range scan returns all 96 expected rows while both +checkpoint and indexed control remain pending. The control returns the same +96 rows about one second later. No guessed startup delay or production setting +change establishes this result. It proves the physical-design distinction, +not a mixed-load operating envelope. + +The retained implementation removes compound primary keys only from `span_rollup`, +`log_rollup` and `endpoint_rollup`, preserving every key column's NOT NULL +constraint. It moves uniqueness responsibility to the existing serialized, +transactional delete-and-rebuild algorithm: delete the complete affected keys, +then insert aggregates grouped by each complete key. It therefore removes the +scan-only tables' index-probe path rather than changing query results or +admission limits. Tests check complete-key uniqueness and full raw-aggregate +parity across late repair, publication retries, repeated refreshes, failed +insertion rollback, affected-key chunks and endpoint overflow/literal-route +collisions. Other indexed cache tables retain their existing constraints. + +The first 30-second overlay run completed 1,200/1,200 offered reads, with +single-panel p95 8.523 ms and twelve-panel p95 63.610 ms. A subsequent local +90-second comparison used the same frozen 40-operations/s workload and resource +budgets as the prior indexed run, with the other fixes held constant: + +| 90-second local result | Compound primary keys | Three scan-only heaps | +| --- | --- | --- | +| Offered / dispatched / successful | 3,600 / 3,592 / 3,583 | 3,600 / 3,598 / 3,598 | +| Backend busy / generator misses | 9 / 8 | 0 / 2 | +| Single-panel p95 / p99 | 32.23 / 91.61 ms | 12.02 / 16.45 ms | +| Four-panel p95 / p99 | 50.47 / 143.73 ms | 15.15 / 21.58 ms | +| Twelve-panel p95 / p99 | 275.21 / 543.80 ms | 74.14 / 106.39 ms | +| Accepted ingest items/s | 20,480 | 20,480 | +| Ending compaction debt | 0 | 0 | +| Ending analytical lag | 1,000 ms | 1,401 ms | +| Process-lifetime peak RSS, including setup | 1,171 MiB | 1,075 MiB | + +Neither run reported background operational errors. Each read kind has enough +successful samples for the p99 sample guard; percentiles describe successes +only. The heap run's two unissued generator arrivals still make the complete +harness **fail** zero-loss offered-load acceptance. Misses are distinct from +backend busy and do not identify a backend capacity ceiling. These single local +before/after runs support the mechanism and retained change, but establish +neither a stable 40/s operating envelope nor repeated representative capacity. + +The disposable cache versions advance to read-cache 19 and analytical 5. +Upgrading triggers a one-time rebuild of derived caches while preserving +canonical telemetry and anomaly history. A new upgrade regression checks those +boundaries and passes on the retained schema. The full engine and query suites, +targeted race tests, opt-in harness compilation, generated-document checks, +formatting and behavioral test-name checks pass. The cold-start guard also +passes with 262,145 exact raw/candidate rows, complete cache markers and +watermarks, and no retired contributions. These correctness checks do not +change the mixed run's offered-load failure classification. + +### Controlled broad-file source comparison + +The native control holds the selected file set and pinned transaction constant, +uses three warmups and 100 interleaved executions per strategy and scope, and +compares the previous broad glob plus filename predicate with an explicit +captured list. The fixture uses 32 canonical spans per file, four native +threads and a 4 GB native memory budget. All-scope results were: + +| Captured files | Glob median / p95 | Explicit-list median / p95 | Mean process CPU, glob / list | +| --- | --- | --- | --- | +| 64 | 12.06 / 13.69 ms | 11.85 / 12.69 ms | 52.7 / 47.1 ms | +| 384 | 124.95 / 139.82 ms | 53.25 / 60.74 ms | 409.2 / 254.5 ms | +| 1,024 | 652.30 / 694.46 ms | 119.68 / 127.28 ms | 1,741.9 / 672.3 ms | + +Service-scoped results showed the same broad-selection improvement. Every +execution matched the complete Go oracle, including exact distinct traces, +nanosecond bounds, duration and error totals. This measures native preparation, +execution and row consumption; it excludes Go snapshot preparation, HTTP, +ingestion, maintenance and browser time. Process CPU comes from getrusage; +unavailable native profile fields provide no operator attribution. + +Runtime now uses the explicit captured list for broad selections as well as +narrow ones. Lifecycle tests retain the old captured source across late +publication, compaction and retention while independent current reads change. +UTF-8 paths, SQL escaping and empty-source schemas remain protected. The +controlled native gain supports this change; it alone does not prove a +mixed-load capacity gain. + +### Historical comparisons and measurement limitations + +Earlier projection-only tests improved small-history single-panel tails and +completed 1,200/1,200 requests at 80 operations/s. Both variants failed at +160/s. Larger-history backlog A/B/B/A runs instead produced 8 and 1 backend +busy responses with previous SQL versus 50 and 14 with projection SQL; +twelve-panel p95 was 490–644 versus 570–987 ms. With history compacted before +timing, all four runs completed 600 requests and projection improved individual +panels. Projection was rejected under that scheduling/source configuration; +its current reintroduction requires new paired evidence. A weighted-histogram +alternative increased high-diversity CPU and memory without a latency benefit +and remains rejected. The partial-window trace rewrite below also remains +rejected. + +The first harness versions omitted the application's foreground-pressure +callback for bounded background yielding. Correcting that wiring did not remove +catch-up failures. Explicit lists alone also failed to eliminate refusals in +initial mixed probes. A metadata-cache-only experiment failed its mixed probe +and is not retained. Those observations must not be described as clean current +baselines or proof that any one component caused all contention. + +Subsequent measurements keep read-pressure wiring and resource budgets fixed, +record offered slots, refusal timing and maintenance state, and use an +automatically cleaned temporary fixture. RSS is process-lifetime peak getrusage +RSS including setup, distinct from earlier phase-sampled current RSS. Substantial +unrelated workstation CPU/I/O and varying generator misses limit causal claims +from small differences. Typed native interruptions are excluded only after +explicit background shutdown intent and cancellation; running-phase interrupts +and unrelated shutdown errors still invalidate a run. + +### Retained-file preparation experiment + +A separate bounded native experiment held selected data constant at four +files, 128 spans and 64 traces while adding 256, 1,024 and 4,096 unrelated +canonical files. Raw complete-query median rose from 3.97 to 4.86 ms; +completed-candidate median rose from 5.15 to 6.86 ms. Candidate binding rose +from 0.88 to 2.63 ms while bound native execution stayed approximately +4.3–4.6 ms. This identifies a modest preparation cost at larger file counts, +not a demonstrated need for a new cache. The twenty-sample exploratory stages +are not additive and exclude HTTP, ingestion and cold-disk behavior. Marker +and snapshot reuse must preserve same-transaction retirement and scope repair; +no runtime metadata-cache change is retained from this experiment. + +## Earlier trace-candidate experiment: rejected + +Astra reviewed the approach and required a real completed-query comparison +against raw telemetry before retaining a runtime change. The implementation +adds that lifecycle oracle and an opt-in four-way plan comparison. The oracle +compares identity, order, error status and aggregate bounds across namespaces, +services, signed nanosecond boundaries, uncached/acknowledged late publications, +compaction and retention. A surviving late trace prevents empty lifecycle +comparisons from passing vacuously. + +The diagnostic fixture uses 20,000 shuffled trace identities, 80,000 parts, +four real Parquet batches, mixed-service traces, and boundary spans on both +sides of the window. DuckDB 2.0.0 uses two threads and a 244.1 MiB resolved +memory cap. Each plan has three warmups and 100 interleaved samples. Every +variant must match the raw ranking before timing. These numbers time execution +of the fully bound completed query on one pinned snapshot; they exclude +binding, HTTP and browser time. + +| Boundary traces | Service scope | Existing inline p95 | Rewrite inline p95 | Rewrite shared p95 | +| --- | --- | --- | --- | --- | +| 1% | All services | 12.34 ms | 13.30 ms | 12.61 ms | +| 1% | API, including mixed-service traces | 15.86 ms | 21.20 ms | 19.47 ms | +| 10% | All services | 12.64 ms | 13.56 ms | 12.79 ms | +| 10% | API, including mixed-service traces | 16.41 ms | 23.13 ms | 21.37 ms | + +Sharing the old candidate expression also provided no consistent material gain. +The rewrite increased mixed-service p95 by roughly 34–41% with inlining; +sharing still left it slower than the existing plan. Therefore neither the +rewrite nor a materialization-policy change is retained in product code. +The earlier standalone SQL measurements below remain exploratory history, +not an achieved improvement. The opt-in experiment preserves the rejected +alternative so future changes can reproduce this result: + +```sh +rtk bash scripts/with-duckdb.sh go test ./internal/engine -tags=filebench \ + -run '^TestTraceCandidateWindowPlanComparisonBenchmark$' -count=1 -v +``` + +Benchmark-tool repairs are independent of that negative performance result: +the format-5 replay and tagged compilation path are repaired in this repository, +and scheduled-arrival accounting and per-kind HTTP tail gates are implemented +in Pitot Bench. HTTP dashboard completion remains distinct from browser last-panel +paint. Short sweeps with fewer than 1,000 successes per kind cannot qualify +the proposed p99 goal; they must retain an insufficient-evidence verdict. + +The repaired executor microbenchmark completed with 100 warm samples per shape +on generated data: 196,608 spans and 196,608 logs across 192 Parquet files, no +metric data. Analytical summaries were populated before timing and the in-memory +result cache was disabled. On an Apple M3 Max (macOS ARM64), DuckDB 2.0.0 with +14 threads, a 20.5 GiB resolved memory cap and 16 read connections reported: + +| Generated-fixture shape | p95 executor latency | +| --- | --- | +| Distinct trace count, all services | 17.46 ms | +| Distinct trace count, one service | 18.42 ms | +| Distinct trace count, query-variable selection | 16.93 ms | +| Span count | 2.08 ms | +| Log count | 2.42 ms | +| Per-service duration p95 | 13.28 ms | +| Complete twelve-panel executor request | 55.32 ms | + +No measured shape exceeded the microbenchmark's reported 500 ms panel target. +The fixture is small, synthetic, warm and local; these are not improvements +over a measured earlier product version, representative-capacity evidence, +browser paint times or sustained-ingest results. Metric shapes were skipped +because the generated fixture contains no metrics. The improvement here is +that the harness builds and measures populated current-code query paths. + +## Motivation and evidence + +The [published November 6 run](https://github.com/labstack/pitot-bench/tree/main/published/2026-11-06-ccx33x2-v2026.11.0-alpha) +measured v2026.11.0-alpha at 507,311 mixed items/s on a Hetzner ccx33 with a +separate generator. That is an ingest-only result using synthetic telemetry. +Its unit is individual telemetry items, whereas the historical 20/s read load +counts user operations, including dashboards that issue multiple queries. +Twenty operations/s was a selected test point, not a measured read ceiling. +Read capacity also depends on the scanned history and query shape; dividing +500,000 by 20 does not compare equivalent work. +[Issue #419](https://github.com/labstack/pitot/issues/419) recorded read p95 of +7.4 seconds, errors and shedding during sustained partial-load ingest. +[PR #423](https://github.com/labstack/pitot/pull/423) subsequently fixed the +service drill-down scan responsible for those failures. Its local median +measurements do not establish current sustained-load p95 or p99. + +This investigation inspected product revision +`74f2157dd1fa2981ca326fdf7aadaecb315e1762` and benchmark revision +`04ae89d2597789bb7bd7d67c001beda7b1cdba3f`. Historical issue descriptions are +leads to verify against current code, rather than descriptions of today's +architecture. No representative workload was run in this investigation. + +## Goals and non-goals + +Qualify a declared hardware, retained dataset and query mix together. Protect +interactive investigations while maintaining bounded compaction debt, cache +backlog, analytical lag and process memory. Publish failures alongside latency. + +These goals do not promise a latency for arbitrary SQL, arbitrary retention, +unbounded traces or every hardware size. They do not authorize changing storage +semantics, approximating exact queries, provisioning benchmark hosts, or raising +admission/time limits to make an existing workload pass. + +## Design + +### Proposed acceptance envelope + +Use an 8-vCPU/32-GB host and separate generator for continuity. Record disk, +engine version, all resource limits, retained rows/bytes/files, actual populated +event-time bounds, cardinality, query limits and cache state. Populate the full +24-hour fixture before claiming a 24-hour read result. + +The first qualification load is 20 scheduled mixed user operations/s. A dashboard +refresh is one user operation that fans out into panels and annotations; it is +not one panel query. Publish both operation and statement counts. Keep the +existing workload for comparison and add separate first-page log/trace search +coverage. Freeze the exact queries, limits, mix and selectivity before testing. + +| Read class | Baseline p95 / p99 | Stretch p95 / p99 | +| --- | --- | --- | +| Warm individual dashboard panel | 500 ms / 1 s | 250 ms / 500 ms | +| Twelve-panel, 24-hour dashboard, trigger through last visible panel painted | 1.5 s / 3 s | 1 s / 2 s | +| Bounded trace-ID detail lookup | 300 ms / 750 ms | 150 ms / 300 ms | +| Selective drill-down or bounded first search page | 750 ms / 1.5 s | 300 ms / 750 ms | + +Only the panel and dashboard baseline p95 values are existing acceptance +criteria in [testing.md](../testing.md#dashboard-performance-measurements). +The other values are proposed engineering goals; attainability under mixed +load is unproven. CI's calibrated 1,800 ms dashboard ceiling remains separate. +Report cold-engine, cold-result-cache and warm-result-cache runs separately. +Large scans get a fixed query-specific budget and separate completion reporting. + +Within the qualified envelope, require zero observed valid-request errors, +timeouts or shedding, correct results, no dropped acknowledged data or +restarts, and the scheduled operation rate to be sustained. Short tests with +zero errors are observations, not production availability guarantees. + +Measure capacity jointly: report the highest accepted ingest rate that also +meets these read goals. Seventy percent of isolated ingest throughput is a +test point, not established headroom. Under deliberate overload, report early +retryable refusals, successful goodput, bounded queues/memory and recovery +time separately; refused operations never count as fast successful reads. + +### Freshness and correctness + +Use uniquely identified sentinel records to test visibility after durable +acknowledgement through raw reads with result-cache bypass. Record publication +to visibility latency and fail missing acknowledged data. Existing raw-cache +and uncached-tail visibility semantics must remain exact. + +The default result-cache TTL is ten seconds. Report cached result window age, +cache outcome and refresh cadence with latency; a fast cached response does +not prove a newly ingested event is visible. Respect the configured TTL rather +than assuming every deployment uses the default. Measure analytical watermark +lag separately from raw visibility and event-time lateness. Set a numerical +analytical freshness goal after recording current scheduling and recovery +behavior; do not invent an unsupported sub-cycle freshness promise. + +### Measurement requirements and implemented repairs + +Scheduled-arrival accounting and per-kind HTTP gates are implemented in the +benchmark working tree; published legacy runs retain their original semantics. +Browser measurement and representative qualification remain outstanding. + +1. Start client latency at the operation's scheduled arrival, retaining + scheduler queue wait, HTTP time, retries and completion. The inspected baseline + [scheduler](https://github.com/labstack/pitot-bench/blob/04ae89d2597789bb7bd7d67c001beda7b1cdba3f/internal/bench/query.go#L56) + starts timing after dequeue. Track successful, failed, timed-out and shed + populations separately, with all offered operations as the denominator. + Derive expected arrivals from elapsed time and the configured rate so dropped + scheduler ticks cannot silently reduce demand. Separate completions during + load from work drained after the load stops. +2. Add per-operation p95/p99 gates. The inspected baseline + [read gate](https://github.com/labstack/pitot-bench/blob/04ae89d2597789bb7bd7d67c001beda7b1cdba3f/internal/bench/runner.go#L560) + checks throughput, failures and shedding without enforcing tail latency. + A mixed percentile must not hide a slow drill-down behind cached dashboards. +3. Add browser trigger-to-last-panel paint measurements. Preserve existing + minimums of 100 warm samples per panel shape and 20 dashboard samples for + p95; obtain at least 1,000 samples per class before treating p99 as a useful + qualification statistic. State sample counts and retain failed samples. +4. Record response-relative freshness, expected nonempty results and correctness + checks. A present watermark field, empty panel or cache hit is insufficient. +5. Make representative performance-gate failures exit unsuccessfully. The + current panel microbenchmark reports target violations without failing + its outer benchmark; its small generated fixture is not a capacity test. + +Server panel execution histograms exclude admission and cache hits. Decompose +end-to-end latency into scheduler, admission, snapshot/bind, execution, +serialization/transfer and rendering where measurable. Use existing pool-wait, +statement, write-gate, cache, compaction and rollup metrics before adding new +instrumentation. Do not subtract independently aggregated percentiles. + +### Prioritized optimization experiments + +| Priority | Current code evidence and experiment | Prediction and acceptance condition | +| --- | --- | --- | +| 1 | Repeat the retained combination against its indexed baseline with early maintenance, exact captured lists, coalescing, projection expansion and committed watermark held constant. | Repeated complete offered load must pass correctness, zero refusals, ingest/freshness, debt and memory gates before claiming mixed-load improvement. | +| 2 | [exec.go](../../internal/query/exec.go): profile main frame, trends, annotations, totals, comparisons and units before sharing compatible work. | Reduce full-request occupancy while preserving totals, comparisons, annotations, limits and partial-result behavior. | +| 3 | [batch_cache.go](../../internal/engine/batch_cache.go) and [file_snapshot.go](../../internal/engine/file_snapshot.go): extend the measured fixed-selection retention control only if preparation becomes material. | Reuse must preserve transactionally consistent markers, late publication and compaction/retention repair; current modest preparation costs do not justify speculative metadata caching. | +| 4 | [pressure.go](../../internal/readpressure/pressure.go) and [duck.go](../../internal/engine/duck.go): correlate background work with refusal timing, CPU, disk and tails before changing cadence or bounds. | Improvement must persist without increasing long-run debt, analytical lag or peak RSS. Separate SQL handles do not reserve CPU or I/O. | + +These are falsifiable opportunities, not diagnosed bottlenecks. Interactive +reservations, bounded queues, sibling-panel limits, separate writer/parser +handles, coalescing, incremental caches and full-path filename pruning already +exist. Do not propose them as missing. Preserve namespace/trace identity, +nanosecond window boundaries, typed attributes, redaction, late publications, +uncached active files and exact mixed-scope results in every experiment. + +### Execution and publication + +1. With repaired benchmark accounting and gates, capture a current-code + baseline on a pinned representative dataset. Archive exact source identity, + harness settings, raw samples, generated SQL and profiles privately until + reviewed for public release. +2. Sweep ingest-only, reads-only and mixed loads at 30%, 50%, 70% and 85% of + the separately measured ingest ceiling, then deliberate saturation. At each + stage record actual offered and accepted ingest rather than the nominal + generator setting. Do not change read/admission limits between comparisons. +3. Run the candidate qualification point for thirty minutes, independently + repeat it three times and run a one-hour soak. The harness's sweep repeat + count does not repeat its sustained phase. Include normal compaction, + rollups, late arrivals and a separate retention/restart recovery exercise. +4. Change one measured cost at a time, starting with qualification of the + current candidate combination and isolated follow-up comparisons. Keep data, + resource budgets and offered arrivals fixed for A/B comparisons. Report correctness, goodput, p95/p99, cache/freshness, RSS, + debt and recovery together. +5. Publish the qualified joint ingest/read operating envelope and its limits, + alongside the isolated ingest ceiling. Update public performance claims + only after the evidence passes the declared gates. + +## Alternatives + +Increasing connections or native threads can increase contention; benchmark +before changing defaults. Longer cache TTL can improve apparent latency while +delaying visibility. Pausing compaction can hide debt until a short run ends. +Approximate rollups cannot replace exact raw-query semantics silently. + +DuckDB recommends profiling workload shape, pushdown and repeated-query costs; +many small concurrent queries are not its primary optimization goal. Preserve +its memory-governed asynchronous I/O defaults unless measurements justify a +change. See [DuckDB 2 workload tuning](https://duckdb.org/docs/2.0/guides/performance/how_to_tune_workloads). + +## Risks and open questions + +Trace-candidate rewrites have substantial correctness risk at scope/window and +publication boundaries. Shared preparation can retain stale file references. +Background throttling can exchange visible latency for hidden maintenance debt. +Prepared-query reuse must account for snapshot-specific file bindings. + +Before public qualification, agree on representative row/file counts, realistic +attribute cardinality, populated retention, trace-size limits, search +selectivity and dashboard count. Determine the analytical freshness budget and +the minimum accepted ingest rate the product should advertise with reads active. +Numerical goals here are proposed acceptance criteria rather than an industry +standard. User-oriented latency, errors, freshness and correctness follow the +measurement approach in [Google SRE's SLO guidance](https://sre.google/workbook/implementing-slos/). + +## Benchmark tooling and follow-up controls + +### Repaired opt-in benchmark + +An initial invocation before the repair failed to build: + +```sh +rtk bash scripts/with-duckdb.sh go test ./internal/query -tags=readbench \ + -run '^$' -bench BenchmarkPanelQueries24Hours -benchtime=100x -count=1 -v +``` + +The failure is `backlog_pressure_test.go:77: unknown field ServiceVersion in +struct literal of type telemetry.Span`. Without `-tags=readbench`, the same +benchmark selection runs no benchmarks and can misleadingly print `PASS`. + +The repair moves that fixture's service version to the current +typed resource attributes (`Resource: map[string]any{"service.version": version}`), +and the tagged suite now compiles. The replay loader in +`panel_bench_data_test.go` also checked `m.Version != 4`, and replay output +constructed version-4 metadata, despite the format-5-only product. The replay +path now uses the authoritative current +format contract, including physical schemas, trace indexes and metadata +validation; changing the version number alone does not prove it works. +The clipping writer recomputes format-5 spill footer counts for retained rows. +Source-preservation tests cover normal input and captures containing compacted +and pruned files. Replay preserves retirement sidecars on a private copy and +tiles only its active catalog; a live root compaction transaction is rejected +before replay. A reusable bounded verifier checks copied and generated batches. +A compile-only `readbench-check` recipe now runs in CI to prevent drift. + +This failure is in optional measurement tooling. It does not establish a defect +in ordinary product query execution. That failed attempt produced no timings; +the repaired generated-fixture measurements are recorded above. + +The panel microbenchmark also supplies an explicit environment to configuration +loading; exporting `PITOT_DUCKDB_*` settings does not currently change its +resource budget. Add deliberate test-only controls before claiming a knob A/B +comparison. Likewise, the managed external harness strips inherited `PITOT_` +variables: exporting `PITOT_PPROF_ENABLED` alone will not enable managed-server +profiling. Product pprof support already exists; the harness needs explicit +diagnostic forwarding and private artifact collection. + +### Retain negative results and profile the complete path + +The initial isolated partial-trace SQL kernel improved an unusually favorable +trace-grouped fixture with one boundary trace, but its gain shrank when 10% of +traces crossed the boundary. The later completed-query oracle and timings above +rejected that rewrite. It is historical evidence for requiring full-path +measurement, not the next proposed production change. + +For repeated dashboard reads, use DuckDB 2's supported profiling interface: +discover `duckdb_available_metrics()` and select available `tracked_metrics`. +Capture actual scan rows/groups, intermediate rows/bytes, planning time, native +CPU and peak buffer/temp usage. The old `custom_profiling_settings` interface +is not the current profiling recipe. Keep internal profiling on a dedicated +trusted connection without opening arbitrary-query configuration access. +See [profiling](https://duckdb.org/docs/2.0/dev/profiling) and +[metrics](https://duckdb.org/docs/2.0/dev/metrics). + +Test sharing a filtered projection only after observing duplicate scan work. +DuckDB already materializes many multiply referenced CTEs and can push partial +aggregation below joins. Forcing materialization everywhere can block useful +filters or increase memory. See [CTE behavior](https://duckdb.org/docs/2.0/sql/query_syntax/with). + +### A small tuning matrix instead of guessed defaults + +If CPU contention remains after reducing demonstrated query work, compare the +current global worker count against half that count at fixed admitted read +load. Hold the memory budget, queries, ingestion, maintenance and cache policy +constant. `threads` governs the native instance globally, rather than being +multiplied per connection. Record the tradeoff between small interactive reads +and large scans instead of treating more threads as automatically faster. +See [configuration](https://duckdb.org/docs/2.0/configuration/overview). + +As proposed experiment gates, require at least a 10% p95 improvement across +repeated paired runs, no more than a 5% regression in protected read/ingest +workloads, unchanged results and bounded memory, debt and analytical lag. +These percentages are acceptance choices, not predicted gains. Retain default +memory-governed read-ahead; a positive fixed depth removes that budget. +See [DuckDB 2 I/O](https://duckdb.org/docs/2.0/data/io). diff --git a/docs/proposals/README.md b/docs/proposals/README.md index 9597260a0..385fac3f7 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -37,5 +37,6 @@ Earlier proposals predate the template and keep their original structure. | 0003 | [Milestone 5: correct metrics and the system page](0003-agent-dashboards-m5.md) | Implemented | | 0004 | [Full UI without an LLM key, and PitotQL](0004-pitotql.md) | Accepted | | 0005 | [Dedicated attribute columns and batch format 5](0005-dedicated-attribute-columns.md) | Accepted | +| 0006 | [Read performance under sustained ingest](0006-read-performance.md) | Draft | Settled decisions that a proposal leads to are recorded in [../adr](../adr/README.md). diff --git a/docs/testing.md b/docs/testing.md index 4344d2fdd..17d2292ee 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -167,9 +167,9 @@ used by the dashboard. Retain failures and timeout counts alongside latency; do make a slow or failed run disappear by retrying until it passes. These targets are acceptance criteria, not guarantees for arbitrary telemetry volumes. -The executor compiles ordinary structured aggregates against raw -telemetry. Its disposable read-cache version 4 stores completed-batch trace -candidates and parts, not general minute aggregates. Service maps combine +The executor uses analytical minute summaries for supported span/log aggregates, +filters and dimensions; other shapes read raw telemetry. The disposable trace +read cache stores completed-batch candidates and parts. Service maps combine analytical rollups with a raw client-span query for external peers. That query materializes the filtered client projection once for reuse by peer and edge aggregates; its child-span and observed-service checks also read spans. Health @@ -177,6 +177,52 @@ panels use analytical rollups. Rollup data has separate watermark lag and approx semantics. Do not assume those rollups answer arbitrary raw counts, rates or quantiles exactly, or report every dashboard query as cached. +`just readbench-check` compiles the opt-in native read harness in CI without +running its large fixtures. Run the local executor microbenchmark explicitly: + +```sh +rtk bash scripts/with-duckdb.sh go test ./internal/query -tags=readbench \ + -run '^$' -bench '^BenchmarkPanelQueries24Hours$' -benchtime=100x -count=1 -v +``` + +Without the `readbench` build tag this selection runs no benchmarks. The default +generated fixture is regression data, not capacity evidence. Representative +replay uses `PITOT_PANEL_BENCH_DATA_DIR` and must pass format-5 verification. +Executor measurements exclude browser paint and sustained external ingest; +record result-cache policy and analytical warmup with their results. + +An opt-in mixed-load probe exercises the Executor while ingestion, analytical +rollups, completed-batch caching, compaction and maintenance run concurrently: + +```sh +rtk env PITOT_MIXED_READ=1 PITOT_MIXED_RATE=40 PITOT_MIXED_SCALE=4 \ + PITOT_MIXED_SEED_STATE=settled \ + bash scripts/with-duckdb.sh go test ./internal/query -tags=readbench \ + -run '^TestMixedReadLoad$' -count=1 -timeout=3m -v +``` + +Use `PITOT_MIXED_SEED_STATE=backlog` to include initial compaction catch-up. +Freeze `PITOT_MIXED_END` to the same in-retention UTC RFC3339 timestamp for paired +runs. The default probe lasts 15 seconds; `PITOT_MIXED_DURATION` accepts up to +five minutes. A 90-second run at 40 requests/sec offers 1,200 requests per kind, +allowing p99 reporting when at least 1,000 complete successfully. These remain +local synthetic measurements, not a sustained production capacity qualification. +Its rate counts user requests, including multi-panel requests, whereas ingest +rates count telemetry records. Report missed arrivals and backend refusals +alongside latency; a completed probe does not by itself qualify RSS, debt or +freshness. See [the read-performance investigation](proposals/0006-read-performance.md). + +The separate `filebench` experiment compares the complete bound trace-ranking +query across candidate expressions and materialization policies: + +```sh +rtk bash scripts/with-duckdb.sh go test ./internal/engine -tags=filebench \ + -run '^TestTraceCandidateWindowPlanComparisonBenchmark$' -count=1 -v +``` + +It checks rankings against raw spans before collecting interleaved warm samples +on one pinned file snapshot. Its timings exclude binding and user-interface work. + The native tests exercise whole-window totals, bounded newest buckets and width-aware 24-hour bucket selection: diff --git a/internal/engine/batch_cache.go b/internal/engine/batch_cache.go index 62bcc2270..855f8ede7 100644 --- a/internal/engine/batch_cache.go +++ b/internal/engine/batch_cache.go @@ -25,7 +25,7 @@ import ( // anomaly_log is not one of them: it holds detector output that nothing can // rebuild, so the drop keeps it. Its table is created once; any change to its // schema must be a separate, explicit migration step. -const readCacheVersion = 18 +const readCacheVersion = 19 const ( readCacheBatchLimit = 64 diff --git a/internal/engine/checkpoint_retry_test.go b/internal/engine/checkpoint_retry_test.go new file mode 100644 index 000000000..9c004e323 --- /dev/null +++ b/internal/engine/checkpoint_retry_test.go @@ -0,0 +1,287 @@ +package engine + +import ( + "context" + "database/sql" + "database/sql/driver" + "errors" + "fmt" + "runtime" + "strings" + "testing" + "time" + + "github.com/DATA-DOG/go-sqlmock" + duckdb "github.com/duckdb/duckdb-go/v2" + "github.com/labstack/pitot/internal/engine/writegate" + "github.com/labstack/pitot/internal/metrics" + "github.com/labstack/pitot/internal/queryrows" + "github.com/labstack/pitot/internal/telemetry" + telemetrystore "github.com/labstack/pitot/internal/telemetry/store" + "github.com/prometheus/client_golang/prometheus/testutil" +) + +func checkpointOldReader(t *testing.T, d *Duck, repo *telemetrystore.Repository) queryrows.Rows { + t.Helper() + at := time.Date(2026, 11, 1, 12, 0, 0, 0, time.UTC) + if err := repo.Commit(t.Context(), telemetrystore.Batch{ID: "checkpoint-reader", Logs: []telemetry.Log{{TimeUnixNanos: at.UnixNano(), EventUnixNanos: at.UnixNano(), IngestedAt: at.UnixNano(), Body: "checkpoint reader"}}}); err != nil { + t.Fatal(err) + } + if _, err := d.writer().ExecContext(t.Context(), "CREATE TABLE checkpoint_progress(value INTEGER); INSERT INTO checkpoint_progress VALUES (0)"); err != nil { + t.Fatal(err) + } + w := queryrows.Window{Start: at, End: at.Add(time.Minute)} + rows, err := d.QueryContext(queryrows.WithWindow(t.Context(), w), "SELECT count(*) FROM logs WHERE time>=?::TIMESTAMP_NS::TIMESTAMPTZ_NS AND time time.Second { + t.Fatalf("250ms retry budget was not bounded: %s", elapsed) + } + if after := testutil.ToFloat64(metrics.DuckDBCheckpointFailures); after != before+1 { + t.Fatalf("attempts counted as logical checkpoint failures: before=%v after=%v", before, after) + } + _, _, failures, healthErr := d.MaintenanceHealth() + if failures != 1 || !errors.Is(healthErr, context.DeadlineExceeded) { + t.Fatalf("checkpoint exhaustion not reflected in maintenance health: failures=%d error=%v", failures, healthErr) + } +} + +func TestCheckpointRetryMatchesOnlyPinnedNativeDiagnostic(t *testing.T) { + busy := &duckdb.Error{Type: duckdb.ErrorTypeTransaction, Msg: checkpointBusyMessage} + for _, check := range []struct { + name string + err error + want bool + }{ + {"native", busy, true}, + {"wrapped native", fmt.Errorf("wrapped: %w", busy), true}, + {"untyped same text", errors.New(checkpointBusyMessage), false}, + {"different transaction failure", &duckdb.Error{Type: duckdb.ErrorTypeTransaction, Msg: "TransactionContext Error: unrelated failure"}, false}, + {"different native type", &duckdb.Error{Type: duckdb.ErrorTypeOutOfMemory, Msg: checkpointBusyMessage}, false}, + {"nil", nil, false}, + } { + t.Run(check.name, func(t *testing.T) { + if got := checkpointRetryable(check.err); got != check.want { + t.Fatalf("retryable=%t want=%t for %v", got, check.want, check.err) + } + }) + } +} + +func TestCheckpointRetriesConflictWithoutCountingTransientFailure(t *testing.T) { + db, mock, err := sqlmock.New() + if err != nil { + t.Fatal(err) + } + defer db.Close() + d := &Duck{DB: db, writeDB: db} + for range 2 { + mock.ExpectExec("^CHECKPOINT$").WillReturnError(&duckdb.Error{Type: duckdb.ErrorTypeTransaction, Msg: checkpointBusyMessage}) + } + mock.ExpectExec("^CHECKPOINT$").WillReturnResult(sqlmock.NewResult(0, 0)) + before := testutil.ToFloat64(metrics.DuckDBCheckpointFailures) + if err := d.checkpoint(t.Context()); err != nil { + t.Fatal(err) + } + if after := testutil.ToFloat64(metrics.DuckDBCheckpointFailures); after != before { + t.Fatalf("successful logical checkpoint counted retry attempts as errors: before=%v after=%v", before, after) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +func TestCheckpointDoesNotRetryUnrelatedError(t *testing.T) { + db, mock, err := sqlmock.New() + if err != nil { + t.Fatal(err) + } + defer db.Close() + d := &Duck{DB: db, writeDB: db} + injected := errors.New("injected checkpoint I/O failure") + mock.ExpectExec("^CHECKPOINT$").WillReturnError(injected) + if err := d.checkpoint(t.Context()); !errors.Is(err, injected) { + t.Fatalf("unrelated failure was retried or hidden: %v", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +func TestCheckpointRetainsCallerExecutionBudget(t *testing.T) { + ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second) + defer cancel() + wantDeadline, _ := ctx.Deadline() + db := sql.OpenDB(checkpointExecutionConnector{execute: func(actual context.Context) (driver.Result, error) { + deadline, ok := actual.Deadline() + if !ok || deadline != wantDeadline { + return nil, fmt.Errorf("native checkpoint execution deadline=%s; want unchanged caller deadline=%s", deadline, wantDeadline) + } + return driver.RowsAffected(0), nil + }}) + defer db.Close() + d := &Duck{DB: db, writeDB: db} + // Inspect the context at the driver seam, without sleeping for a slow I/O + // operation: the 250ms retry budget must never cap native execution. + if err := d.checkpoint(ctx); err != nil { + t.Fatal(err) + } +} + +func TestCheckpointNativeExecutionHonorsCallerCancellation(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + entered := make(chan struct{}) + db := sql.OpenDB(checkpointExecutionConnector{execute: func(actual context.Context) (driver.Result, error) { + close(entered) + <-actual.Done() + return nil, actual.Err() + }}) + defer func() { cancel(); db.Close() }() + d := &Duck{DB: db, writeDB: db} + result := make(chan error, 1) + go func() { result <- d.checkpoint(ctx) }() + select { + case <-entered: + cancel() + case <-time.After(time.Second): + t.Fatal("native execution was not reached") + } + select { + case err := <-result: + if !errors.Is(err, context.Canceled) { + t.Fatalf("native caller cancellation hidden: %v", err) + } + case <-time.After(time.Second): + t.Fatal("native execution ignored caller cancellation") + } +} + +type checkpointExecutionConnector struct { + execute func(context.Context) (driver.Result, error) +} + +func (c checkpointExecutionConnector) Connect(context.Context) (driver.Conn, error) { + return checkpointExecutionConn(c), nil +} +func (checkpointExecutionConnector) Driver() driver.Driver { return checkpointExecutionDriver{} } + +type checkpointExecutionDriver struct{} + +func (checkpointExecutionDriver) Open(string) (driver.Conn, error) { + return nil, errors.New("checkpoint test requires its connector") +} + +type checkpointExecutionConn struct { + execute func(context.Context) (driver.Result, error) +} + +func (checkpointExecutionConn) Close() error { return nil } +func (checkpointExecutionConn) Prepare(string) (driver.Stmt, error) { + return nil, errors.New("checkpoint test does not prepare statements") +} +func (checkpointExecutionConn) Begin() (driver.Tx, error) { + return nil, errors.New("checkpoint test does not begin transactions") +} +func (c checkpointExecutionConn) ExecContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Result, error) { + if query != "CHECKPOINT" || len(args) != 0 { + return nil, fmt.Errorf("unexpected checkpoint execution: %s", query) + } + return c.execute(ctx) +} diff --git a/internal/engine/checkpoint_scan_lock_test.go b/internal/engine/checkpoint_scan_lock_test.go new file mode 100644 index 000000000..e3b06ee51 --- /dev/null +++ b/internal/engine/checkpoint_scan_lock_test.go @@ -0,0 +1,194 @@ +package engine + +import ( + "context" + "database/sql" + "fmt" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + "time" +) + +// This is a physical-design control, independent of production cache schemas. +// In the pinned engine an indexed scan considers ART selection under the shared +// vacuum lock even when its compound index cannot answer the range predicate. +// A FULL checkpoint owns the exclusive lock during its debug sleep. The heap +// has the same rows and NOT NULL columns, but no index-selection lock to acquire. +func TestCheckpointCompoundIndexScanWaitsWhileHeapScanProgresses(t *testing.T) { + path := filepath.Join(t.TempDir(), "scan-lock.duckdb") + db, err := sql.Open("duckdb", path) + if err != nil { + t.Fatal(err) + } + db.SetMaxOpenConns(4) + ctx, cancel := context.WithTimeout(t.Context(), 8*time.Second) + // Cancel active statements before closing the shared native database. + defer func() { cancel(); _ = db.Close() }() + var version string + if err := db.QueryRowContext(ctx, "SELECT version()").Scan(&version); err != nil { + t.Fatal(err) + } + if version != "v2.0.0" { + t.Fatalf("proof requires pinned engine v2.0.0, got %q", version) + } + for _, statement := range []string{ + "SET threads=2", + "SET memory_limit='256MB'", + "CREATE TABLE scan_heap(k BIGINT NOT NULL, part BIGINT NOT NULL, value BIGINT NOT NULL)", + "CREATE TABLE scan_compound(k BIGINT NOT NULL, part BIGINT NOT NULL, value BIGINT NOT NULL, PRIMARY KEY(k,part))", + "INSERT INTO scan_heap SELECT i,0,i FROM range(256) AS r(i)", + "INSERT INTO scan_compound SELECT * FROM scan_heap", + "CHECKPOINT", + "UPDATE scan_heap SET value=value+1", + "UPDATE scan_compound SET value=value+1", + } { + if _, err := db.ExecContext(ctx, statement); err != nil { + t.Fatalf("%s: %v", statement, err) + } + } + checkpointConn, err := db.Conn(ctx) + if err != nil { + t.Fatal(err) + } + defer checkpointConn.Close() + if _, err := checkpointConn.ExecContext(ctx, "SET debug_checkpoint_sleep_ms=1000"); err != nil { + t.Fatal(err) + } + heapConn, err := db.Conn(ctx) + if err != nil { + t.Fatal(err) + } + defer heapConn.Close() + compoundConn, err := db.Conn(ctx) + if err != nil { + t.Fatal(err) + } + defer compoundConn.Close() + const query = "SELECT count(*) FROM %s WHERE k>=? AND k baseline.Size() { + break + } + if err != nil && !os.IsNotExist(err) { + t.Fatal(err) + } + select { + case result := <-checkpointDone: + t.Fatalf("checkpoint returned before flushed WAL checkpoint-record barrier: %v", result.err) + case <-ctx.Done(): + t.Fatalf("checkpoint-record barrier: %v", ctx.Err()) + case <-ticker.C: + } + } + // The WAL record is appended/flushed by WALStartCheckpoint before the + // configured debug sleep. No guessed delay is used to start either read. + readsFinished := make(chan struct{}) + go func() { + defer close(readsFinished) + start := make(chan struct{}) + inner := make(chan struct{}) + go func() { + defer close(inner) + close(start) + var count int + err := checkpointCompoundRangeScan(ctx, compoundConn, &count) + compoundDone <- outcome{count: count, at: time.Now(), err: err} + }() + <-start + // Observe the control in a native call before issuing the heap query, + // so an unscheduled control goroutine cannot pass this proof vacuously. + buffer := make([]byte, 1<<20) + for { + entered := false + for stack := range strings.SplitSeq(string(buffer[:runtime.Stack(buffer, true)]), "\n\n") { + if strings.Contains(stack, "checkpointCompoundRangeScan") && strings.Contains(stack, "_Cfunc_duckdb_execute_pending") { + entered = true + break + } + } + if entered { + break + } + select { + case <-ctx.Done(): + heapDone <- outcome{err: ctx.Err(), at: time.Now()} + <-inner + return + case <-ticker.C: + } + } + var count int + err := heapConn.QueryRowContext(ctx, fmt.Sprintf(query, "scan_heap"), 32, 128).Scan(&count) + heapDone <- outcome{count: count, at: time.Now(), err: err} + <-inner + }() + defer func() { cancel(); <-readsFinished }() + heap := <-heapDone + if heap.err != nil { + t.Fatal(heap.err) + } + select { + case checkpoint := <-checkpointDone: + t.Fatalf("checkpoint returned before heap-progress barrier: %v", checkpoint.err) + default: + } + select { + case compound := <-compoundDone: + t.Fatalf("compound-index control completed during checkpoint-active heap progress: count=%d error=%v", compound.count, compound.err) + default: + } + checkpoint := <-checkpointDone + compound := <-compoundDone + for name, result := range map[string]outcome{"heap": heap, "compound": compound} { + if result.err != nil || result.count != 96 { + t.Fatalf("%s count=%d want96: %v", name, result.count, result.err) + } + } + if checkpoint.err != nil { + t.Fatal(checkpoint.err) + } + if !heap.at.Before(checkpoint.at) { + t.Fatal("heap range scan failed to complete during the checkpoint-active interval") + } + // Native lock release precedes ExecContext returning to Go, so relative + // Go return timestamps cannot safely order the final release and reader. + t.Logf("heap completed while checkpoint/control pending; both exact counts=%d/%d; compound_minus_heap=%s", heap.count, compound.count, compound.at.Sub(heap.at)) +} + +// A distinct frame lets the barrier observe this control inside native code. +func checkpointCompoundRangeScan(ctx context.Context, conn *sql.Conn, count *int) error { + return conn.QueryRowContext(ctx, "SELECT count(*) FROM scan_compound WHERE k>=? AND k %v", before, got) + } + if _, err := d.DB.ExecContext(t.Context(), `ALTER TABLE span_rollup DROP COLUMN incompatible`); err != nil { + t.Fatal(err) + } + if _, err := d.refreshServiceRollup(t.Context()); err != nil { + t.Fatal(err) + } + if got := assertCommittedWatermarkParity(t, d); !got.After(before) { + t.Fatalf("successful retry did not publish new watermark: %v -> %v", before, got) + } +} + +func TestCommittedWatermarkRestoresRewoundStartupState(t *testing.T) { + cfg := config.Config{DataDir: t.TempDir(), DuckDBMemory: "256MB", DuckDBThreads: 2, DuckDBMaxConns: 2} + repo, err := telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + t.Fatal(err) + } + d, err := NewDuck(t.Context(), cfg, repo) + if err != nil { + repo.Close() + t.Fatal(err) + } + t.Cleanup(func() { _ = d.Close(); _ = repo.Close() }) + commitWatermarkSpan(t, repo, "durable") + if _, err := d.refreshServiceRollup(t.Context()); err != nil { + t.Fatal(err) + } + want := assertCommittedWatermarkParity(t, d) + // A failed publication can advance the persisted empty tail beyond the + // durable repository tip. Open rewinds that tail before exposing readers. + if _, err := d.DB.ExecContext(t.Context(), `UPDATE rollup_state SET last_ingested_unix_nano=? WHERE cache_key=?`, want.Add(time.Hour).UnixNano(), serviceRollupStateKey); err != nil { + t.Fatal(err) + } + if err := d.Close(); err != nil { + t.Fatal(err) + } + if err := repo.Close(); err != nil { + t.Fatal(err) + } + repo, err = telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + t.Fatal(err) + } + d, err = NewDuck(t.Context(), cfg, repo) + if err != nil { + t.Fatal(err) + } + if got := assertCommittedWatermarkParity(t, d); !got.Equal(want) { + t.Fatalf("restart exposed unrewound horizon: got=%v want=%v", got, want) + } +} + +func TestCommittedWatermarkInitializesSkipToLatest(t *testing.T) { + cfg := config.Config{DataDir: t.TempDir(), DuckDBMemory: "256MB", DuckDBThreads: 2, DuckDBMaxConns: 2, RollupSkipToLatest: true} + repo, err := telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + t.Fatal(err) + } + defer repo.Close() + commitWatermarkSpan(t, repo, "skip") + horizon, _, _ := repo.PublicationBounds() + d, err := NewDuck(t.Context(), cfg, repo) + if err != nil { + t.Fatal(err) + } + defer d.Close() + if got := assertCommittedWatermarkParity(t, d); got.UnixNano() != horizon { + t.Fatalf("skip horizon=%v want=%d", got, horizon) + } + var rows int + if err := d.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM span_rollup`).Scan(&rows); err != nil || rows != 0 { + t.Fatalf("skip unexpectedly aggregated rows=%d: %v", rows, err) + } +} + +func TestCommittedWatermarkConcurrentReadersAndRefreshes(t *testing.T) { + d, repo := newBatchCacheTest(t) + stop := make(chan struct{}) + errs := make(chan error, 4) + var ready, readers sync.WaitGroup + ready.Add(4) + readers.Add(4) + for range 4 { + go func() { + defer readers.Done() + ready.Done() + var previous time.Time + ticker := time.NewTicker(time.Millisecond) + defer ticker.Stop() + for { + got, err := d.RollupWatermark(t.Context()) + if err != nil || got.Before(previous) { + errs <- fmt.Errorf("reader watermark=%v previous=%v err=%v", got, previous, err) + return + } + previous = got + select { + case <-stop: + return + case <-ticker.C: + } + } + }() + } + finish := sync.OnceFunc(func() { close(stop); readers.Wait() }) + defer finish() + ready.Wait() + for i := range 4 { + commitWatermarkSpan(t, repo, fmt.Sprintf("concurrent-%d", i)) + if _, err := d.refreshServiceRollup(t.Context()); err != nil { + t.Fatal(err) + } + assertCommittedWatermarkParity(t, d) + } + finish() + close(errs) + for err := range errs { + t.Error(err) + } +} + +func commitWatermarkSpan(t *testing.T, repo *telemetrystore.Repository, id string) { + t.Helper() + at := time.Now().UTC().Truncate(time.Minute).Add(-time.Hour) + if err := repo.Commit(t.Context(), telemetrystore.Batch{ID: id, Spans: []telemetry.Span{{Namespace: "ns", ServiceName: "svc", TraceID: id, SpanID: id, StartUnixNanos: at.UnixNano(), EndUnixNanos: at.Add(time.Millisecond).UnixNano(), DurationMS: 1}}}); err != nil { + t.Fatal(err) + } +} + +func assertCommittedWatermarkParity(t *testing.T, d *Duck) time.Time { + t.Helper() + var stored int64 + if err := d.DB.QueryRowContext(t.Context(), `SELECT coalesce((SELECT last_ingested_unix_nano FROM rollup_state WHERE cache_key=?),0)`, serviceRollupStateKey).Scan(&stored); err != nil { + t.Fatal(err) + } + got, err := d.RollupWatermark(t.Context()) + want := time.Time{} + if stored != 0 { + want = time.Unix(0, stored).UTC() + } + if err != nil || !got.Equal(want) { + t.Fatalf("published watermark=%v persisted=%v: %v", got, want, err) + } + return got +} diff --git a/internal/engine/duck.go b/internal/engine/duck.go index ad95d3304..8bb93d7a2 100644 --- a/internal/engine/duck.go +++ b/internal/engine/duck.go @@ -87,6 +87,9 @@ type Duck struct { publicationBounds func() (horizon, published, minimum int64) // Cached under writeGate so an idle pass executes no SQL at all. rollupWatermarks map[string]int64 + // Standalone panel preflight reads a committed freshness lower bound without + // borrowing a native connection. Snapshot-bound coverage still reads SQL. + committedServiceWatermark atomic.Int64 // serviceChunk narrows the service rollup's ingest window after it runs // out of memory. Guarded by writeGate. serviceChunk rollupChunk @@ -128,7 +131,7 @@ const ( // Rollup progress keys in rollup_state. readCacheVersion drops that table // with the rollups, so a version change also restarts their backfill; the // keys need no version of their own. - rollup_schema_version = 4 + rollup_schema_version = 5 serviceRollupStateKey = "service_rollup_v3" edgeRollupStateKey = "edge_rollup_v2" edgeRollupSubCursorKey = "edge_rollup_v2_substart" @@ -355,6 +358,7 @@ func (d *Duck) skipRollupToLatest(ctx context.Context) error { return err } d.rollupWatermarks = map[string]int64{serviceRollupStateKey: horizon, edgeRollupStateKey: horizon} + d.committedServiceWatermark.Store(horizon) return nil } @@ -558,23 +562,83 @@ func (d *Duck) tryLockBackground() func() { func (d *Duck) writer() *sql.DB { return d.writeDB } // checkpoint folds the rollup cache's WAL back into its file, and says how long -// that took. A checkpoint blocks new transactions for its duration, so it is a -// candidate explanation for a stalled query and has to be measurable. +// that took. Checkpoint work can contend with reads, so it is a candidate +// explanation for a stalled query and has to be measurable. // -// The non-FORCE form refuses while another *write* transaction is open. Every -// domain is serialized by its own gate. Maintenance holds both gates before -// calling this function, so a refusal should not happen — the counter is there -// to say so rather than to be assumed. Only a checkpoint that ran is timed: a -// refusal returns in microseconds and would drag the distribution towards zero, -// which is the opposite of what the histogram is for. -func (d *Duck) checkpoint(ctx context.Context) error { +// Maintenance holds both write gates throughout this operation. The pinned +// engine can still return its transaction-conflict diagnostic when an older +// reader remains open after a newer write commits. Give that reader a bounded +// opportunity to drain; keep one writer connection and admit no new writers. +// Other errors are not retried. Metrics describe the logical operation: one +// failure on exhaustion/error, or one successful duration including retry wait. +func (d *Duck) checkpoint(ctx context.Context) (retErr error) { start := time.Now() - if _, err := d.writer().ExecContext(ctx, "CHECKPOINT"); err != nil { - metrics.DuckDBCheckpointFailures.Inc() + defer func() { + if retErr != nil { + metrics.DuckDBCheckpointFailures.Inc() + } else { + metrics.DuckDBCheckpoint.Observe(time.Since(start).Seconds()) + } + }() + conn, err := d.writer().Conn(ctx) + if err != nil { + if ctx.Err() != nil { + return ctx.Err() + } return err } - metrics.DuckDBCheckpoint.Observe(time.Since(start).Seconds()) - return nil + defer conn.Close() + // This bounds retry admission/wait only. A successful native checkpoint can + // legitimately take longer, and retains the caller's execution budget. + retryCtx, cancel := context.WithTimeout(ctx, checkpointRetryBudget) + defer cancel() + var lastConflict error + budgetError := func() error { + if ctx.Err() != nil { + return ctx.Err() + } + return fmt.Errorf("checkpoint retry budget exhausted after %s: %w", checkpointRetryBudget, errors.Join(lastConflict, retryCtx.Err())) + } + for { + if err := ctx.Err(); err != nil { + return err + } + if lastConflict != nil && retryCtx.Err() != nil { + return budgetError() + } + _, err := conn.ExecContext(ctx, "CHECKPOINT") + if ctx.Err() != nil { + return ctx.Err() + } + if err == nil { + return nil + } + if !checkpointRetryable(err) { + return err + } + lastConflict = err + if retryCtx.Err() != nil { + return budgetError() + } + timer := time.NewTimer(checkpointRetryDelay) + select { + case <-retryCtx.Done(): + timer.Stop() + return budgetError() + case <-timer.C: + } + } +} + +const ( + checkpointRetryBudget = 250 * time.Millisecond + checkpointRetryDelay = 10 * time.Millisecond + checkpointBusyMessage = "TransactionContext Error: Cannot CHECKPOINT: there are other write transactions active. Try using FORCE CHECKPOINT to wait until all active transactions are finished" +) + +func checkpointRetryable(err error) bool { + var native *duckdb.Error + return errors.As(err, &native) && native.Type == duckdb.ErrorTypeTransaction && native.Error() == checkpointBusyMessage } // DefaultNamespace returns empty string so queries search all namespaces. @@ -583,6 +647,15 @@ func (d *Duck) DefaultNamespace() string { } func (d *Duck) RunRollups(ctx context.Context) { + // Retire file debt while the initial analytical pass catches up too. Waiting + // for that pass leaves foreground reads on the larger startup catalog. + maintenanceDone := make(chan struct{}) + go func() { + defer close(maintenanceDone) + d.runMaintenanceLoop(ctx) + }() + defer func() { <-maintenanceDone }() + // rollupOnce can return rows AND an error (one cache committed while another // failed), so telemetry is recorded unconditionally — otherwise the rollup // throughput metric flatlines on an instance whose service rollup is @@ -597,12 +670,6 @@ func (d *Duck) RunRollups(ctx context.Context) { slog.Info("startup rollup complete", "rows", rows, "duration", time.Since(start)) } d.updateParquetStats(ctx) - maintenanceDone := make(chan struct{}) - go func() { - defer close(maintenanceDone) - d.runMaintenanceLoop(ctx) - }() - defer func() { <-maintenanceDone }() readCacheDone := make(chan struct{}) go func() { defer close(readCacheDone) @@ -1003,7 +1070,11 @@ func (d *Duck) refreshServiceRollup(ctx context.Context) (affected int64, err er return 0, err } defer func() { _ = tx.Rollback() }() - unbind, err := d.bindRollupSnapshot(ctx, tx) + var captured []telemetry.BatchMetadata + if d.repository != nil && d.repository.Parquet != nil { + captured = d.repository.Parquet.BatchMetadata() + } + unbind, err := d.bindRollupSnapshotFiles(ctx, tx, captured, nil) if err != nil { return 0, fmt.Errorf("service_rollup/snapshot: %w", err) } @@ -1027,6 +1098,13 @@ func (d *Duck) refreshServiceRollup(ctx context.Context) (affected int64, err er // delete or insert any buckets. Keep exact signed nanoseconds in Go. if lower.Valid && upper.Valid { lo, hi := time.Unix(0, lower.Int64).UTC(), time.Unix(0, upper.Int64).UTC() + // Every recomputation below reads this event interval. Avoid repeatedly + // preparing unrelated retained files after affected-key discovery, while + // keeping the original captured catalog and complete affected buckets. + window := queryrows.Window{Start: lo, End: hi} + if _, err := d.bindRollupSnapshotFiles(ctx, tx, captured, &window); err != nil { + return 0, fmt.Errorf("service_rollup/affected_snapshot: %w", err) + } if _, err := tx.ExecContext(ctx, serviceRollupDeleteSQL, lo, hi); err != nil { return 0, fmt.Errorf("service_rollup/delete: %w", err) } @@ -1062,6 +1140,7 @@ func (d *Duck) refreshServiceRollup(ctx context.Context) (affected int64, err er } result = metrics.RollupSuccess d.rollupWatermarks[serviceRollupStateKey] = newWatermark + d.committedServiceWatermark.Store(newWatermark) updateRollupProgress(metrics.RollupService, true, newWatermark, published) recordedRows = rows @@ -1421,6 +1500,7 @@ func (d *Duck) loadRollupWatermarks(ctx context.Context) error { return err } d.rollupWatermarks = watermarks + d.committedServiceWatermark.Store(watermarks[serviceRollupStateKey]) return nil } @@ -2405,13 +2485,14 @@ func (d *Duck) PublishParquet(ctx context.Context, publish func(context.Context) // ---- Queries for API ---- -// RollupWatermark is the ingest horizon shared by service_rollup and span_rollup. -// Read persisted state instead of the writer-owned in-memory cursor. +// RollupWatermark is the committed ingest horizon shared by service_rollup and +// span_rollup. This standalone preflight precedes the panel's data snapshot; +// snapshot-bound coverage checks read persisted state in their own transaction. func (d *Duck) RollupWatermark(ctx context.Context) (time.Time, error) { - var stamp int64 - if err := d.QueryRowScan(ctx, []any{&stamp}, `SELECT coalesce((SELECT last_ingested_unix_nano FROM rollup_state WHERE cache_key=?),0)`, serviceRollupStateKey); err != nil { + if err := ctx.Err(); err != nil { return time.Time{}, err } + stamp := d.committedServiceWatermark.Load() if stamp == 0 { return time.Time{}, nil } diff --git a/internal/engine/file_snapshot.go b/internal/engine/file_snapshot.go index 97cd6c13e..c985e0207 100644 --- a/internal/engine/file_snapshot.go +++ b/internal/engine/file_snapshot.go @@ -49,32 +49,11 @@ func (d *Duck) physicalSource(signal string, batches []telemetry.BatchMetadata) return "SELECT " + rawProjection(s) + " FROM read_parquet([" + strings.Join(files, ",") + "], union_by_name=false, hive_partitioning=false)" } -// snapshotSource uses a glob when pruning saves little. The filename predicate -// admits exactly the captured batches' files, including when ingest publishes a new file -// or compaction renames an output in after metadata capture, or a retired -// input is deleted. DuckDB prunes the other files by name without opening -// them, but binds the schema from the first file listed, so the list starts -// with the schema batch, which is never deleted while the process runs. -// Narrow subsets keep explicit lists and footer pruning. -func (d *Duck) snapshotSource(signal string, selected, active []telemetry.BatchMetadata) string { - count := 0 - for _, b := range active { - _, n := batchTime(b, signal) - if n > 0 { - count++ - } - } - if len(selected) < 64 || len(selected)*5 < count*4 { - return d.physicalSource(signal, selected) - } - paths := make([]string, 0, len(selected)) - for _, b := range selected { - paths = append(paths, filepath.ToSlash(filepath.Join(d.repository.Parquet.BatchesDir(), b.ID+telemetry.BatchSuffix, signal+".parquet"))) - } - s, _ := telemetry.LookupSignal(signal) - // Compare whole names: DuckDB prunes a glob by a filename predicate only - // when it can evaluate it per file, which a computed batch ID defeats. - return "SELECT " + rawProjection(s) + " FROM read_parquet(" + parquetGlob(d.repository.Parquet.Dir(), signal) + ", union_by_name=false, hive_partitioning=false) WHERE filename IN (" + idsSQL(paths) + ")" +// snapshotSource enumerates the captured files even for broad selections. +// A glob plus filename IN becomes expensive at larger file counts. The caller's +// pin keeps these exact immutable paths available through row consumption. +func (d *Duck) snapshotSource(signal string, selected, _ []telemetry.BatchMetadata) string { + return d.physicalSource(signal, selected) } func (d *Duck) batchIDColumn(signal, filename string) string { @@ -87,7 +66,7 @@ func cleanSource(signal, physical string) string { return cleanSelect(signal, "("+physical+")") } -// parquetGlob lists one signal's files, schema batch first; see snapshotSource. +// parquetGlob lists the schema batch first for stable view binding. func parquetGlob(parquetDir, signal string) string { batches := filepath.Join(parquetDir, "batches") return "[" + sqlLiteral(filepath.ToSlash(filepath.Join(batches, telemetry.SchemaBatch, signal+".parquet"))) + "," + @@ -103,11 +82,20 @@ func (d *Duck) bindRollupSnapshot(ctx context.Context, tx *sql.Tx) (func() error if d.repository == nil || d.repository.Parquet == nil { return func() error { return nil }, nil } - batches := d.repository.Parquet.BatchMetadata() + return d.bindRollupSnapshotFiles(ctx, tx, d.repository.Parquet.BatchMetadata(), nil) +} + +// Discovery binds the full captured catalog. Reaggregation may narrow that +// same catalog to its known event-time interval; the caller retains the pin +// and transaction throughout both bindings. Each signal uses its own footer. +func (d *Duck) bindRollupSnapshotFiles(ctx context.Context, tx *sql.Tx, batches []telemetry.BatchMetadata, window *queryrows.Window) (func() error, error) { + if d.repository == nil || d.repository.Parquet == nil { + return func() error { return nil }, nil + } for _, s := range telemetry.Signals { selected := make([]telemetry.BatchMetadata, 0, len(batches)) for _, b := range batches { - if _, n := b.Signal(s); n > 0 { + if _, n := b.Signal(s); n > 0 && (window == nil || overlapping(b, s.Name, *window)) { selected = append(selected, b) } } diff --git a/internal/engine/file_snapshot_test.go b/internal/engine/file_snapshot_test.go index 8bc8aa458..d8399c587 100644 --- a/internal/engine/file_snapshot_test.go +++ b/internal/engine/file_snapshot_test.go @@ -397,20 +397,19 @@ func TestReadCacheSemanticVersionRebuildsAllContributions(t *testing.T) { } } -func TestBroadGlobRetainsCapturedFileSet(t *testing.T) { +func TestBroadSelectionRetainsCapturedFilesAcrossPublicationAndRetirement(t *testing.T) { d, repo := newBatchCacheTest(t) - at := time.Now().UTC() + at := time.Now().UTC().Add(-time.Hour) for i := range 64 { - if err := repo.Commit(t.Context(), telemetrystore.Batch{ID: fmt.Sprintf("glob-%d", i), Spans: []telemetry.Span{{TraceID: "trace", SpanID: fmt.Sprint(i), StartUnixNanos: at.UnixNano(), IngestedAt: at.UnixNano()}}}); err != nil { + if err := commitRestoredFixture(repo, t.Context(), telemetrystore.Batch{ID: fmt.Sprintf("glob-%d", i), Spans: []telemetry.Span{{TraceID: "trace", SpanID: fmt.Sprint(i), StartUnixNanos: at.UnixNano(), IngestedAt: at.UnixNano()}}}); err != nil { t.Fatal(err) } } captured := repo.Parquet.BatchMetadata() + pin := d.pinParquet() + defer d.unpinParquet(pin) source := d.snapshotSource("spans", captured, captured) - if !strings.Contains(source, "*.batch") { - t.Fatal("broad selection did not use glob") - } - if err := repo.Commit(t.Context(), telemetrystore.Batch{ID: "new-after-capture", Spans: []telemetry.Span{{TraceID: "new", StartUnixNanos: at.UnixNano(), IngestedAt: at.UnixNano()}}}); err != nil { + if err := repo.Commit(t.Context(), telemetrystore.Batch{ID: "new-after-capture", Spans: []telemetry.Span{{TraceID: "new", StartUnixNanos: at.UnixNano(), IngestedAt: at.Add(time.Hour).UnixNano()}}}); err != nil { t.Fatal(err) } var count int @@ -424,6 +423,29 @@ func TestBroadGlobRetainsCapturedFileSet(t *testing.T) { if err := d.DB.QueryRow("SELECT count(*) FROM (" + narrow + ")").Scan(&count); err != nil || count != 1 { t.Fatalf("narrow rows=%d: %v", count, err) } + current := func(want int) { + t.Helper() + if err := d.QueryRowScan(t.Context(), []any{&count}, "SELECT count(*) FROM spans"); err != nil || count != want { + t.Fatalf("current rows=%d want=%d: %v", count, want, err) + } + } + current(65) + if merged, err := repo.CompactParquet(t.Context(), d, 64); err != nil || merged < 2 { + t.Fatalf("compact captured inputs=%d: %v", merged, err) + } + current(65) + if err := d.DB.QueryRow("SELECT count(*) FROM (" + source + ")").Scan(&count); err != nil || count != 64 { + t.Fatalf("pinned captured rows after compaction=%d: %v", count, err) + } + // Include the newly published batch's actual ingestion stamp as well as + // the restored inputs, so the live catalog is completely retired. + if _, err := repo.PruneParquetPass(t.Context(), d, time.Now().Add(time.Minute).UnixNano(), 128, time.Minute); err != nil { + t.Fatal("prune captured inputs:", err) + } + current(0) + if err := d.DB.QueryRow("SELECT count(*) FROM (" + source + ")").Scan(&count); err != nil || count != 64 { + t.Fatalf("pinned captured rows after retention=%d: %v", count, err) + } } func TestBatchFilenameIdentityWithRelativeDataDir(t *testing.T) { diff --git a/internal/engine/panel_rollup_keys_test.go b/internal/engine/panel_rollup_keys_test.go new file mode 100644 index 000000000..07bdd417a --- /dev/null +++ b/internal/engine/panel_rollup_keys_test.go @@ -0,0 +1,357 @@ +package engine + +import ( + "fmt" + "reflect" + "strings" + "testing" + "time" + + "github.com/labstack/pitot/internal/config" + "github.com/labstack/pitot/internal/telemetry" + telemetrystore "github.com/labstack/pitot/internal/telemetry/store" +) + +var panelRollupKeys = []struct { + table string + keys string +}{ + {"span_rollup", "namespace,bucket,service,kind,status"}, + {"log_rollup", "namespace,bucket,service,severity"}, + {"endpoint_rollup", "namespace,bucket,service,kind,status,route,overflow"}, +} + +// derivedRollupKeys also dropped their ART primary keys. Their INSERT SQL is +// too complex to re-derive from raw rows here, so they are guarded for schema +// shape (no index, NOT NULL keys) and for key uniqueness after each refresh, +// not for full raw-population equivalence. +var derivedRollupKeys = []struct { + table string + keys string +}{ + {"service_rollup", "namespace,bucket,service"}, + {"edge_rollup", "namespace,bucket,caller,callee,edge_type"}, +} + +func TestPanelRollupKeysKeepNotNullWithoutIndexes(t *testing.T) { + d, _ := newBatchCacheTest(t) + for _, table := range panelRollupKeys { + t.Run(table.table, func(t *testing.T) { + var constraints, indexes int + if err := d.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM duckdb_constraints() WHERE table_name=? AND constraint_type IN ('PRIMARY KEY','UNIQUE')`, table.table).Scan(&constraints); err != nil { + t.Fatal(err) + } + if err := d.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM duckdb_indexes() WHERE table_name=?`, table.table).Scan(&indexes); err != nil { + t.Fatal(err) + } + if constraints != 0 || indexes != 0 { + t.Errorf("scan-only summary retains key constraints/indexes=%d/%d", constraints, indexes) + } + keys := strings.Split(table.keys, ",") + for nullAt, key := range keys { + values := make([]any, len(keys)) + for i, name := range keys { + values[i] = "" + if name == "bucket" { + values[i] = time.Now().UTC().Truncate(time.Minute) + } else if name == "overflow" { + values[i] = false + } + } + values[nullAt] = nil + query := "INSERT INTO " + table.table + " (" + table.keys + ") VALUES (" + strings.TrimSuffix(strings.Repeat("?,", len(keys)), ",") + ")" + if _, err := d.DB.ExecContext(t.Context(), query, values...); err == nil || !strings.Contains(err.Error(), "NOT NULL") || !strings.Contains(err.Error(), table.table+"."+key) { + t.Errorf("NULL %s did not fail its NOT NULL constraint: %v", key, err) + } + } + }) + } +} + +func TestDerivedRollupKeysDropIndexesKeepNotNull(t *testing.T) { + d, _ := newBatchCacheTest(t) + for _, table := range derivedRollupKeys { + t.Run(table.table, func(t *testing.T) { + var constraints, indexes int + if err := d.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM duckdb_constraints() WHERE table_name=? AND constraint_type IN ('PRIMARY KEY','UNIQUE')`, table.table).Scan(&constraints); err != nil { + t.Fatal(err) + } + if err := d.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM duckdb_indexes() WHERE table_name=?`, table.table).Scan(&indexes); err != nil { + t.Fatal(err) + } + if constraints != 0 || indexes != 0 { + t.Errorf("derived summary retains key constraints/indexes=%d/%d", constraints, indexes) + } + keys := strings.Split(table.keys, ",") + for nullAt, key := range keys { + values := make([]any, len(keys)) + for i, name := range keys { + values[i] = "" + if name == "bucket" { + values[i] = time.Now().UTC().Truncate(time.Minute) + } + } + values[nullAt] = nil + query := "INSERT INTO " + table.table + " (" + table.keys + ") VALUES (" + strings.TrimSuffix(strings.Repeat("?,", len(keys)), ",") + ")" + if _, err := d.DB.ExecContext(t.Context(), query, values...); err == nil || !strings.Contains(err.Error(), "NOT NULL") || !strings.Contains(err.Error(), table.table+"."+key) { + t.Errorf("NULL %s did not fail its NOT NULL constraint: %v", key, err) + } + } + }) + } +} + +func TestPanelRollupKeysSurviveLateRepairRetryAndRollback(t *testing.T) { + d, repo := newBatchCacheTest(t) + at := time.Now().UTC().Truncate(time.Minute).Add(-time.Hour) + batch := panelRollupKeyBatch("initial", at, 1, true) + if err := repo.Commit(t.Context(), batch); err != nil { + t.Fatal(err) + } + refreshPanelRollupKeys(t, d) + // An idempotent publication retry and repeated refresh cannot append a + // second aggregate for a previously processed complete key. + if err := repo.Commit(t.Context(), batch); err != nil { + t.Fatal(err) + } + refreshPanelRollupKeys(t, d) + refreshPanelRollupKeys(t, d) + for _, table := range panelRollupKeys { + if _, err := d.DB.ExecContext(t.Context(), "CREATE TABLE before_"+table.table+" AS SELECT * FROM "+table.table); err != nil { + t.Fatal(err) + } + } + before, err := d.RollupWatermark(t.Context()) + if err != nil { + t.Fatal(err) + } + if err := repo.Commit(t.Context(), panelRollupKeyBatch("late", at.Add(20*time.Second), 1, true)); err != nil { + t.Fatal(err) + } + // Endpoint insertion follows the span aggregate insertion. Failure here + // must roll back the shared transaction, including all earlier deletions. + if _, err := d.DB.ExecContext(t.Context(), `ALTER TABLE endpoint_rollup ADD COLUMN incompatible BOOLEAN`); err != nil { + t.Fatal(err) + } + if _, err := d.refreshServiceRollup(t.Context()); err == nil { + t.Fatal("incompatible endpoint insertion unexpectedly committed") + } + if _, err := d.DB.ExecContext(t.Context(), `ALTER TABLE endpoint_rollup DROP COLUMN incompatible`); err != nil { + t.Fatal(err) + } + for _, table := range panelRollupKeys { + var differences int + query := "SELECT count(*) FROM ((SELECT * FROM " + table.table + " EXCEPT ALL SELECT * FROM before_" + table.table + ") UNION ALL (SELECT * FROM before_" + table.table + " EXCEPT ALL SELECT * FROM " + table.table + "))" + if err := d.DB.QueryRowContext(t.Context(), query).Scan(&differences); err != nil || differences != 0 { + t.Fatalf("rollback changed %s: differences=%d err=%v", table.table, differences, err) + } + } + if got, err := d.RollupWatermark(t.Context()); err != nil || !got.Equal(before) { + t.Fatalf("rollback changed watermark: %v -> %v: %v", before, got, err) + } + refreshPanelRollupKeys(t, d) + refreshPanelRollupKeys(t, d) + var collisionGroups int + if err := d.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM endpoint_rollup WHERE route='Other routes (overflow)'`).Scan(&collisionGroups); err != nil || collisionGroups != 2 { + t.Fatalf("literal route and overflow groups=%d want2: %v", collisionGroups, err) + } +} + +func TestPanelRollupKeysStayUniqueAcrossAffectedChunks(t *testing.T) { + d, repo := newBatchCacheTest(t) + at := time.Now().UTC().Truncate(time.Minute).Add(-time.Hour) + // Each service-minute belongs to one chunk even though every service has + // multiple observations. Cross the actual production key-chunk boundary. + for _, id := range []string{"first", "late"} { + if err := repo.Commit(t.Context(), panelRollupKeyBatch(id, at, affectedKeyChunkSize+1, false)); err != nil { + t.Fatal(err) + } + refreshPanelRollupKeys(t, d) + } + refreshPanelRollupKeys(t, d) +} + +func TestPanelRollupKeysUpgradePreviousCacheOnReopen(t *testing.T) { + cfg := config.Config{DataDir: t.TempDir(), DuckDBMemory: "256MB", DuckDBThreads: 2, DuckDBMaxConns: 2} + repo, err := telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + t.Fatal(err) + } + d, err := NewDuck(t.Context(), cfg, repo) + if err != nil { + _ = repo.Close() + t.Fatal(err) + } + t.Cleanup(func() { + if d != nil { + _ = d.Close() + } + if repo != nil { + _ = repo.Close() + } + }) + at := time.Now().UTC().Truncate(time.Minute).Add(-time.Hour) + batch := panelRollupKeyBatch("previous-cache", at, 2, false) + if err := repo.Commit(t.Context(), batch); err != nil { + t.Fatal(err) + } + refreshPanelRollupKeys(t, d) + if _, err := d.RefreshReadCaches(t.Context()); err != nil { + t.Fatal(err) + } + if got := assertCommittedWatermarkParity(t, d); got.IsZero() { + t.Fatal("fixture did not warm analytical progress") + } + metadata := repo.Parquet.BatchMetadata() + for _, table := range append(append([]struct{ table, keys string }{}, panelRollupKeys...), derivedRollupKeys...) { + // Recreate the exact previous physical key design, rather than only + // changing a version number on an already unindexed cache. + if _, err := d.DB.ExecContext(t.Context(), "ALTER TABLE "+table.table+" ADD PRIMARY KEY ("+table.keys+")"); err != nil { + t.Fatal(err) + } + } + for _, statement := range []string{ + `UPDATE read_cache_version SET version=18`, + `UPDATE rollup_schema_version SET version=4`, + `INSERT INTO anomaly_log VALUES ('upgrade','svc','latency',now(),now(),'witness','preserve me')`, + `INSERT INTO read_edge_buckets VALUES ('upgrade',now(),10)`, + `INSERT INTO edge_rollup (namespace,bucket,caller,callee,calls) VALUES ('upgrade',now(),'a','b',1)`, + `INSERT INTO version_rollup VALUES ('upgrade','svc','v1',now(),now())`, + `INSERT INTO version_rollup_batches VALUES ('previous-cache',1)`, + `INSERT INTO rollup_state VALUES ('edge_rollup_v2',1,now()),('edge_rollup_v2_substart',1,now()),('edge_rollup_v2_subwindow_end',2,now())`, + } { + if _, err := d.DB.ExecContext(t.Context(), statement); err != nil { + t.Fatal(err) + } + } + var markers int + if err := d.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM read_batches`).Scan(&markers); err != nil || markers != 1 { + t.Fatalf("fixture markers=%d: %v", markers, err) + } + if err := d.Close(); err != nil { + t.Fatal(err) + } + d = nil + if err := repo.Close(); err != nil { + t.Fatal(err) + } + repo = nil + repo, err = telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + t.Fatal(err) + } + d, err = NewDuck(t.Context(), cfg, repo) + if err != nil { + t.Fatal(err) + } + for table, want := range map[string]int{"read_cache_version": 19, "rollup_schema_version": 5} { + var got int + if err := d.DB.QueryRowContext(t.Context(), "SELECT version FROM "+table).Scan(&got); err != nil || got != want { + t.Fatalf("reopened %s=%d want=%d: %v", table, got, want, err) + } + } + for _, table := range []string{"span_rollup", "log_rollup", "endpoint_rollup", "service_rollup", "edge_rollup", "receiver_rollup", "resource_rollup", "receiver_entity_rollup", "rollup_state", "version_rollup", "version_rollup_batches", "read_batches", "read_trace_parts", "read_trace_candidates", "read_resource_inventory", "read_resource_seconds", "read_metric_names", "read_edge_buckets"} { + var count int + if err := d.DB.QueryRowContext(t.Context(), "SELECT count(*) FROM "+table).Scan(&count); err != nil || count != 0 { + t.Fatalf("previous %s survived rebuild: count=%d: %v", table, count, err) + } + } + if got := assertCommittedWatermarkParity(t, d); !got.IsZero() { + t.Fatalf("previous in-memory progress survived reopen: %v", got) + } + var anomalies int + if err := d.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM anomaly_log WHERE namespace='upgrade' AND service='svc'`).Scan(&anomalies); err != nil || anomalies != 1 { + t.Fatalf("anomaly witness lost: count=%d: %v", anomalies, err) + } + if got := repo.Parquet.BatchMetadata(); !reflect.DeepEqual(got, metadata) { + t.Fatal("cache upgrade changed canonical batch metadata") + } + var spans, logs int + if err := d.QueryRowScan(t.Context(), []any{&spans, &logs}, `SELECT (SELECT count(*) FROM spans),(SELECT count(*) FROM logs)`); err != nil || spans != len(batch.Spans) || logs != len(batch.Logs) { + t.Fatalf("canonical rows after reopen=%d/%d: %v", spans, logs, err) + } + for _, table := range append(append([]struct{ table, keys string }{}, panelRollupKeys...), derivedRollupKeys...) { + var indexes int + if err := d.DB.QueryRowContext(t.Context(), `SELECT (SELECT count(*) FROM duckdb_constraints() WHERE table_name=? AND constraint_type IN ('PRIMARY KEY','UNIQUE'))+(SELECT count(*) FROM duckdb_indexes() WHERE table_name=?)`, table.table, table.table).Scan(&indexes); err != nil || indexes != 0 { + t.Fatalf("reopened %s key indexes=%d: %v", table.table, indexes, err) + } + for _, key := range strings.Split(table.keys, ",") { + var required int + if err := d.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM duckdb_constraints() WHERE table_name=? AND constraint_type='NOT NULL' AND list_contains(constraint_column_names,?)`, table.table, key).Scan(&required); err != nil || required != 1 { + t.Fatalf("reopened %s.%s NOT NULL constraints=%d: %v", table.table, key, required, err) + } + } + } + refreshPanelRollupKeys(t, d) + if _, backlog, err := d.refreshReadCaches(t.Context()); err != nil || backlog { + t.Fatalf("rebuilt cache failed to converge: backlog=%v err=%v", backlog, err) + } + if err := d.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM read_batches WHERE batch_id='previous-cache'`).Scan(&markers); err != nil || markers != 1 { + t.Fatalf("rebuilt markers=%d: %v", markers, err) + } + var candidates int + if err := d.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM read_trace_candidates WHERE min_start=?`, at.UnixNano()).Scan(&candidates); err != nil || candidates != len(batch.Spans) { + t.Fatalf("rebuilt exact candidates=%d: %v", candidates, err) + } + if got := assertCommittedWatermarkParity(t, d); got.IsZero() { + t.Fatal("rebuilt analytical progress did not advance") + } +} + +func panelRollupKeyBatch(id string, at time.Time, services int, overflow bool) telemetrystore.Batch { + batch := telemetrystore.Batch{ID: id} + for service := range services { + routes := []string{"/a", "/b"} + if overflow { + routes = []string{"Other routes (overflow)", "Other routes (overflow)", "Other routes (overflow)"} + for i := range endpointRouteLimit + 2 { + routes = append(routes, fmt.Sprintf("/route/%03d", i)) + } + } + for i, route := range routes { + ns, svc := "ns'", fmt.Sprintf("svc-%04d", service) + duration := float64(i % 17) + batch.Spans = append(batch.Spans, telemetry.Span{Namespace: ns, ServiceName: svc, TraceID: fmt.Sprintf("%s-%d-%d", id, service, i), SpanID: "span", StartUnixNanos: at.UnixNano(), EndUnixNanos: at.Add(time.Duration(duration) * time.Millisecond).UnixNano(), DurationMS: duration, Kind: "SPAN_KIND_SERVER", StatusCode: "STATUS_CODE_OK", Attributes: map[string]any{"http.route": route}}) + severity := "INFO" + if i%2 == 0 { + severity = "ERROR" + } + batch.Logs = append(batch.Logs, telemetry.Log{Namespace: ns, ServiceName: svc, TimeUnixNanos: at.UnixNano(), Severity: severity, Body: id}) + } + } + return batch +} + +func refreshPanelRollupKeys(t *testing.T, d *Duck) { + t.Helper() + if _, err := d.refreshServiceRollup(t.Context()); err != nil { + t.Fatal(err) + } + for _, table := range append(append([]struct{ table, keys string }{}, panelRollupKeys...), derivedRollupKeys...) { + var duplicates int + query := "SELECT count(*) FROM (SELECT " + table.keys + " FROM " + table.table + " GROUP BY " + table.keys + " HAVING count(*)<>1)" + if err := d.DB.QueryRowContext(t.Context(), query).Scan(&duplicates); err != nil || duplicates != 0 { + t.Fatalf("%s duplicate complete keys=%d: %v", table.table, duplicates, err) + } + } + // Independent raw populations check all keys, counts, duration statistics + // and every histogram bin, not only the number of persisted aggregates. + hist := `histogram(CASE WHEN duration_ms>0 THEN coalesce(try_cast(ceil(log2(duration_ms)*8) AS INTEGER),2147483647) ELSE -2147483648 END)` + expected := map[string]string{ + "span_rollup": `SELECT namespace,date_trunc('minute',start_time::TIMESTAMP_NS) bucket,service,coalesce(kind,'') kind,coalesce(status,'') status,count(*) spans,sum(duration_ms) duration_sum,min(duration_ms) duration_min,max(duration_ms) duration_max,` + hist + ` hist FROM spans GROUP BY ALL`, + "log_rollup": `SELECT namespace,date_trunc('minute',time::TIMESTAMP_NS) bucket,service,coalesce(severity,'') severity,count(*) logs FROM logs GROUP BY ALL`, + "endpoint_rollup": fmt.Sprintf(`WITH eligible AS ( + SELECT namespace,date_trunc('minute',start_time::TIMESTAMP_NS) bucket,service,coalesce(kind,'') kind,coalesce(status,'') status,attributes['http.route']::VARCHAR route,duration_ms FROM spans WHERE attributes['http.route']::VARCHAR<>'' + ), populations AS (SELECT namespace,bucket,service,route,count(*) n FROM eligible GROUP BY ALL), + ranked AS (SELECT *,row_number() OVER(PARTITION BY namespace,bucket,service ORDER BY n DESC,route) rank FROM populations), + folded AS (SELECT e.* EXCLUDE(route),CASE WHEN r.rank<=%d THEN e.route ELSE 'Other routes (overflow)' END route,r.rank>%d overflow FROM eligible e JOIN ranked r USING(namespace,bucket,service,route)) + SELECT namespace,bucket,service,kind,status,route,overflow,count(*) spans,sum(duration_ms) duration_sum,min(duration_ms) duration_min,max(duration_ms) duration_max,%s hist FROM folded GROUP BY ALL`, endpointRouteLimit, endpointRouteLimit, hist), + } + for _, table := range panelRollupKeys { + var differences int + query := "WITH expected AS (" + expected[table.table] + "), differences AS ((SELECT * FROM expected EXCEPT ALL SELECT * FROM " + table.table + ") UNION ALL (SELECT * FROM " + table.table + " EXCEPT ALL SELECT * FROM expected)) SELECT count(*) FROM differences" + if err := d.QueryRowScan(t.Context(), []any{&differences}, query); err != nil || differences != 0 { + t.Fatalf("%s raw/summary differences=%d: %v", table.table, differences, err) + } + } +} diff --git a/internal/engine/rollup_snapshot_window_test.go b/internal/engine/rollup_snapshot_window_test.go new file mode 100644 index 000000000..032251815 --- /dev/null +++ b/internal/engine/rollup_snapshot_window_test.go @@ -0,0 +1,179 @@ +package engine + +import ( + "database/sql" + "math" + "testing" + "time" + + "github.com/labstack/pitot/internal/engine/writegate" + "github.com/labstack/pitot/internal/queryrows" + "github.com/labstack/pitot/internal/telemetry" + telemetrystore "github.com/labstack/pitot/internal/telemetry/store" +) + +func TestRollupSnapshotWindowRepairsOldMinuteFromAllCapturedFiles(t *testing.T) { + d, repo := newBatchCacheTest(t) + at := time.Now().UTC().Truncate(time.Minute).Add(-24 * time.Hour) + commit := func(id string, spanAt, otherAt time.Time, value float64) { + t.Helper() + if err := repo.Commit(t.Context(), rollupSnapshotBatch(id, spanAt, otherAt, value)); err != nil { + t.Fatal(err) + } + } + commit("old", at.Add(10*time.Second), at.Add(10*time.Second), 1) + commit("unrelated", at.Add(-2*time.Hour), at.Add(-2*time.Hour), 99) + if _, err := d.refreshServiceRollup(t.Context()); err != nil { + t.Fatal(err) + } + before := d.rollupWatermarks[serviceRollupStateKey] + // Only the span is in another hour. The new log and receiver point still + // repair the old minute using their own signal's footer bounds. + commit("late", at.Add(-time.Hour), at.Add(30*time.Second), 3) + if _, err := d.refreshServiceRollup(t.Context()); err != nil { + t.Fatal(err) + } + if d.rollupWatermarks[serviceRollupStateKey] <= before { + t.Fatal("late publication did not advance the analytical watermark") + } + var spans, logs, names int + if err := d.DB.QueryRowContext(t.Context(), `SELECT spans,log_count,metric_count FROM service_rollup WHERE bucket=?::TIMESTAMP_NS AND service='svc'`, at).Scan(&spans, &logs, &names); err != nil || spans != 1 || logs != 2 || names != 1 { + t.Fatalf("repaired old minute spans/logs/metric names=%d/%d/%d: %v", spans, logs, names, err) + } + var first, last, increase, seconds float64 + if err := d.DB.QueryRowContext(t.Context(), `SELECT first_point.value,last_point.value,counter_increase,counter_seconds FROM receiver_rollup WHERE bucket=?::TIMESTAMP_NS`, at).Scan(&first, &last, &increase, &seconds); err != nil || first != 1 || last != 3 || increase != 2 || seconds != 20 { + t.Fatalf("receiver repair first/last/increase/seconds=%g/%g/%g/%g: %v", first, last, increase, seconds, err) + } + var resources, entities int + if err := d.DB.QueryRowContext(t.Context(), `SELECT (SELECT count(*) FROM resource_rollup WHERE bucket=?::TIMESTAMP_NS),(SELECT count(*) FROM receiver_entity_rollup WHERE bucket=?::TIMESTAMP_NS)`, at, at).Scan(&resources, &entities); err != nil || resources != 3 || entities != 1 { + t.Fatalf("repaired resource/entity summaries=%d/%d: %v", resources, entities, err) + } +} + +func TestRollupSnapshotWindowUsesSignalBoundsAndIncludesUnknown(t *testing.T) { + d, repo := newBatchCacheTest(t) + lo := time.Date(2026, 1, 1, 12, 0, 0, 7, time.UTC) + hi := lo.Add(time.Minute) + for _, point := range []struct { + id string + at time.Time + }{{"before", lo.Add(-time.Nanosecond)}, {"lower", lo}, {"inside", hi.Add(-time.Nanosecond)}, {"upper", hi}, {"unknown", hi.Add(time.Hour)}} { + if err := repo.Commit(t.Context(), rollupSnapshotBatch(point.id, point.at, point.at, 1)); err != nil { + t.Fatal(err) + } + } + if err := repo.Commit(t.Context(), rollupSnapshotBatch("disjoint", lo.Add(-time.Hour), lo, 1)); err != nil { + t.Fatal(err) + } + pin := d.pinParquet() + defer d.unpinParquet(pin) + captured := repo.Parquet.BatchMetadata() + for i := range captured { + if captured[i].ID == "unknown" { + // Optional footer statistics may be unavailable. Change only this + // captured descriptor; every underlying format-5 file remains valid. + captured[i].SpanTime.Known = false + captured[i].LogTime.Known = false + captured[i].MetricTime.Known = false + } + } + tx, err := d.writer().BeginTx(t.Context(), nil) + if err != nil { + t.Fatal(err) + } + defer tx.Rollback() + unbind, err := d.bindRollupSnapshotFiles(t.Context(), tx, captured, nil) + if err != nil { + t.Fatal(err) + } + assertRollupSnapshotCounts(t, tx, 6, 6, 6) + window := queryrows.Window{Start: lo, End: hi} + if _, err := d.bindRollupSnapshotFiles(t.Context(), tx, captured, &window); err != nil { + t.Fatal(err) + } + // Bounds select whole files. Unknown bounds retain their out-of-window + // row, and SQL predicates still remove it during aggregation. + assertRollupSnapshotCounts(t, tx, 3, 4, 4) + var spans, logs, metrics int + if err := tx.QueryRowContext(t.Context(), `SELECT + (SELECT count(*) FROM spans WHERE start_time>=?::TIMESTAMP_NS::TIMESTAMPTZ_NS AND start_time=?::TIMESTAMP_NS::TIMESTAMPTZ_NS AND time=?::TIMESTAMP_NS::TIMESTAMPTZ_NS AND time=?::TIMESTAMP_NS::TIMESTAMPTZ_NS AND start_time= w.End.UnixNano() || scoped && (span.Namespace != "prod" || span.ServiceName != "api") { + continue + } + key := [2]string{span.Namespace, span.ServiceName} + row := groups[key] + if row == nil { + row = &broadSnapshotRow{Namespace: span.Namespace, Service: span.ServiceName, MinStart: span.StartUnixNanos, MaxStart: span.StartUnixNanos} + groups[key], traces[key] = row, map[string]bool{} + } + row.Rows++ + row.MinStart, row.MaxStart = min(row.MinStart, span.StartUnixNanos), max(row.MaxStart, span.StartUnixNanos) + row.DurationNanos += span.EndUnixNanos - span.StartUnixNanos + if span.StatusCode == "STATUS_CODE_ERROR" { + row.Errors++ + } + traces[key][span.TraceID] = true + } + result := make([]broadSnapshotRow, 0, len(groups)) + for key, row := range groups { + row.Traces = int64(len(traces[key])) + result = append(result, *row) + } + sort.Slice(result, func(i, j int) bool { + if result[i].Namespace != result[j].Namespace { + return result[i].Namespace < result[j].Namespace + } + return result[i].Service < result[j].Service + }) + return result +} + +func readBroadSnapshotRows(ctx context.Context, tx *sql.Tx, statement string, w queryrows.Window) ([]broadSnapshotRow, error) { + rows, err := tx.QueryContext(ctx, statement, w.Start, w.End) + if err != nil { + return nil, err + } + defer rows.Close() + var result []broadSnapshotRow + for rows.Next() { + var row broadSnapshotRow + if err := rows.Scan(&row.Namespace, &row.Service, &row.Rows, &row.Traces, &row.MinStart, &row.MaxStart, &row.DurationNanos, &row.Errors); err != nil { + return nil, err + } + result = append(result, row) + } + if err := rows.Err(); err != nil { + return nil, err + } + return result, rows.Close() +} + +func compareBroadSnapshotSources(t *testing.T, ctx context.Context, d *Duck, batches []telemetry.BatchMetadata, w queryrows.Window, oracle []telemetry.Span) { + t.Helper() + conn, pin, err := d.readConn(ctx) + if err != nil { + t.Fatal(err) + } + var tx *sql.Tx + defer func() { endRead(conn, tx); d.unpinParquet(pin) }() + tx, err = beginRead(ctx, conn) + if err != nil { + t.Fatal(err) + } + // Freeze the previous broad-source expression explicitly so this control + // remains useful if runtime snapshotSource later changes its selection policy. + paths := make([]string, 0, len(batches)) + for _, batch := range batches { + paths = append(paths, filepath.ToSlash(filepath.Join(d.repository.Parquet.BatchesDir(), batch.ID+telemetry.BatchSuffix, "spans.parquet"))) + } + signal, _ := telemetry.LookupSignal("spans") + glob := "SELECT " + rawProjection(signal) + " FROM read_parquet(" + parquetGlob(d.repository.Parquet.Dir(), "spans") + ", union_by_name=false, hive_partitioning=false) WHERE filename IN (" + idsSQL(paths) + ")" + list := d.physicalSource("spans", append([]telemetry.BatchMetadata(nil), batches...)) + for _, scoped := range []bool{false, true} { + scope := "all" + predicate := "" + if scoped { + scope, predicate = "prod_api", " AND namespace='prod' AND service='api'" + } + want := broadSnapshotOracle(oracle, w, scoped) + var statements [2]string + for i, source := range []string{glob, list} { + statements[i] = `SELECT namespace,service,count(*)::BIGINT,count(DISTINCT trace_id)::BIGINT, + min(epoch_ns(start_time)),max(epoch_ns(start_time)),sum(end_unix_nano-start_unix_nano)::BIGINT, + sum(CASE WHEN status='STATUS_CODE_ERROR' THEN 1 ELSE 0 END)::BIGINT + FROM (` + cleanSource("spans", source) + `) WHERE start_time>=?::TIMESTAMP_NS::TIMESTAMPTZ_NS AND start_time 0 + if len(operators) == 0 { + keys := make([]string, 0, len(profile)) + for key := range profile { + keys = append(keys, key) + } + sort.Strings(keys) + result["root_keys"] = keys + result["reason"] = "native EXPLAIN JSON operator layout was not recognized; use measured process CPU instead" + } + return result +} diff --git a/internal/engine/startup_cold_bench_test.go b/internal/engine/startup_cold_bench_test.go new file mode 100644 index 000000000..f63d0a4c0 --- /dev/null +++ b/internal/engine/startup_cold_bench_test.go @@ -0,0 +1,212 @@ +//go:build filebench + +package engine + +import ( + "context" + "fmt" + "path/filepath" + "runtime" + "sync" + "syscall" + "testing" + "time" + + "github.com/labstack/pitot/internal/config" + "github.com/labstack/pitot/internal/engine/writegate" + "github.com/labstack/pitot/internal/queryrows" + "github.com/labstack/pitot/internal/telemetry" + telemetrystore "github.com/labstack/pitot/internal/telemetry/store" +) + +// This opt-in guard exercises cold startup correctness with one compacted file +// larger than the complete-batch read-cache budget. It is not capacity evidence. +func TestColdStartupCompactsOversizedCacheOutput(t *testing.T) { + began := time.Now() + defer func() { + var usage syscall.Rusage + if err := syscall.Getrusage(syscall.RUSAGE_SELF, &usage); err != nil { + t.Logf("cold_startup elapsed=%s process_peak_error=%v", time.Since(began), err) + return + } + divisor := float64(1 << 20) + if runtime.GOOS == "linux" { + divisor = 1024 + } + t.Logf("cold_startup elapsed=%s process_peak_mib=%.3f peak_scope=whole_test_process_including_fixture platform=%s/%s capacity_evidence=false", time.Since(began), float64(usage.Maxrss)/divisor, runtime.GOOS, runtime.GOARCH) + }() + ctx, cancel := context.WithTimeout(t.Context(), 3*time.Minute) + defer cancel() + cfg := config.Config{DataDir: filepath.Join(t.TempDir(), "cold-state"), DuckDBMemory: "4GB", DuckDBThreads: 4, DuckDBMaxConns: 4, RollupInterval: time.Second, MaintenanceInterval: time.Hour, RetentionDays: 30} + repo, err := telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + t.Fatal(err) + } + // Two otherwise-equal resources retain distinct integer/string identities. + resources := []map[string]any{ + {"generation": int64(1), "ready": true, "nested": map[string]any{"labels": []any{"a", int64(2)}}, "literal.dot": []byte{0, 255}}, + {"generation": "1", "ready": true, "nested": map[string]any{"labels": []any{"a", int64(2)}}, "literal.dot": []byte{0, 255}}, + } + at := time.Now().UTC().Truncate(time.Hour).Add(-time.Hour + time.Minute) + const inputs = 8 + const rowsPerInput = 32768 + const historical = inputs * rowsPerInput + const want = historical + 1 + if historical <= readCacheRowBudget { + t.Fatalf("fixture no longer exceeds read-cache budget: rows=%d budget=%d", historical, readCacheRowBudget) + } + inputIDs := make([]string, 0, inputs) + for batchIndex := range inputs { + id := fmt.Sprintf("cold-input-%02d", batchIndex) + inputIDs = append(inputIDs, id) + batch := telemetrystore.Batch{ID: id, Spans: make([]telemetry.Span, rowsPerInput)} + for i := range batch.Spans { + n := at.Add(time.Duration(batchIndex)*time.Second + time.Duration(i)*time.Nanosecond).UnixNano() + identity := fmt.Sprintf("cold-%02d-%05d", batchIndex, i) + batch.Spans[i] = telemetry.Span{Namespace: "cold_startup", ServiceName: "api", TraceID: identity, SpanID: identity, StartUnixNanos: n, EndUnixNanos: n + 1_000_000, DurationMS: 1, Kind: "SPAN_KIND_SERVER", StatusCode: "STATUS_CODE_OK", IngestedAt: at.UnixNano(), Resource: resources[i%2]} + } + if err := commitRestoredFixture(repo, ctx, batch); err != nil { + repo.Close() + t.Fatal(err) + } + } + sentinel := at.Add(time.Hour).UnixNano() + if err := commitRestoredFixture(repo, ctx, telemetrystore.Batch{ID: "cold-newest", Spans: []telemetry.Span{{Namespace: "cold_startup", ServiceName: "api", TraceID: "cold-newest", SpanID: "sentinel", StartUnixNanos: sentinel, EndUnixNanos: sentinel + 1_000_000, DurationMS: 1, Kind: "SPAN_KIND_SERVER", StatusCode: "STATUS_CODE_OK", IngestedAt: sentinel, Resource: resources[0]}}}); err != nil { + repo.Close() + t.Fatal(err) + } + // Reopen the canonical published fixture as a real restored repository. No + // query cache, synthetic DB rows or warmed complete-batch markers are supplied. + if err := repo.Close(); err != nil { + t.Fatal(err) + } + repo, err = telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + t.Fatal(err) + } + d, err := NewDuck(ctx, cfg, repo) + if err != nil { + repo.Close() + t.Fatal(err) + } + t.Cleanup(func() { + if err := d.Close(); err != nil { + t.Error(err) + } + if err := repo.Close(); err != nil { + t.Error(err) + } + }) + var marked int + if err := d.DB.QueryRowContext(ctx, "SELECT count(*) FROM read_batches").Scan(&marked); err != nil || marked != 0 { + t.Fatalf("fixture is not cold: markers=%d error=%v", marked, err) + } + horizon, _, _ := repo.PublicationBounds() + window := queryrows.Window{Start: at.Add(-time.Second), End: at.Add(time.Hour + time.Second)} + unlock := sync.OnceFunc(d.writeGate.Lock(writegate.RollupService)) + work, stop := context.WithCancel(ctx) + done := make(chan struct{}) + startupBegan := time.Now() + go func() { defer close(done); d.RunRollups(work) }() + join := func() { + t.Helper() + timer := time.NewTimer(10 * time.Second) + defer timer.Stop() + select { + case <-done: + case <-timer.C: + t.Error("cold startup RunRollups failed to join after cancellation") + } + } + t.Cleanup(func() { stop(); unlock(); join() }) + var output telemetry.BatchMetadata + coldStartupAwait(t, ctx, "oversized compacted output before analytical gate release", func(context.Context) (bool, error) { + active := repo.Parquet.BatchMetadata() + for _, batch := range active { + if batch.Generation > 0 && batch.Spans > readCacheRowBudget { + output = batch + return len(active) == 2, nil + } + } + return false, nil + }) + if output.Spans != historical { + t.Fatalf("compacted output spans=%d want=%d", output.Spans, historical) + } + // Raw immutable reads must remain exact before any analytical write succeeds. + coldStartupAssertRaw(t, ctx, d, want) + t.Logf("cold_startup early_compaction=%s output_rows=%d cache_row_budget=%d analytical_gate_held=true", time.Since(startupBegan), output.Spans, readCacheRowBudget) + unlock() + coldStartupAwait(t, ctx, "active cache markers and service/edge watermarks", func(ctx context.Context) (bool, error) { + markers, err := batchMarkers(ctx, d.DB) + if err != nil { + return false, err + } + active := repo.Parquet.BatchMetadata() + if len(markers) != len(active) { + return false, nil + } + for _, batch := range active { + if !markers[batch.ID] { + return false, nil + } + } + var complete int + err = d.DB.QueryRowContext(ctx, `SELECT count(*) FROM rollup_state WHERE cache_key IN (?,?) AND last_ingested_unix_nano>=?`, serviceRollupStateKey, edgeRollupStateKey, horizon).Scan(&complete) + return complete == 2, err + }) + coldStartupAssertRaw(t, ctx, d, want) + traceWindow := window + traceWindow.Kind = queryrows.TraceCandidateRead + traceWindow.Namespace = "cold_startup" + traceWindow.Service = "api" + var candidates, tail int + if err := d.QueryRowScan(queryrows.WithWindow(ctx, traceWindow), []any{&candidates, &tail}, `SELECT (SELECT count(DISTINCT trace_id) FROM (SELECT trace_id FROM trace_candidates UNION ALL SELECT trace_id FROM trace_tail)),(SELECT count(*) FROM trace_tail)`); err != nil || candidates != want || tail != 0 { + t.Fatalf("candidate traces=%d want=%d uncached_tail=%d error=%v", candidates, want, tail, err) + } + assertInventoryOracle(t, d, window, 2) + var integerResources, stringResources, readyResources int + if err := d.QueryRowScan(inventoryContext(ctx, window), []any{&integerResources, &stringResources, &readyResources}, `SELECT count(*) FILTER (WHERE variant_type(resource,'generation')='INT64'),count(*) FILTER (WHERE variant_type(resource,'generation')='VARCHAR'),count(*) FILTER (WHERE resource['ready']::BOOLEAN) FROM resource_inventory`); err != nil || integerResources != 1 || stringResources != 1 || readyResources != 2 { + t.Fatalf("typed resources integer=%d string=%d ready=%d error=%v", integerResources, stringResources, readyResources, err) + } + for _, table := range []string{"read_batches", "read_trace_parts", "read_resource_inventory", "read_resource_seconds", "read_metric_names"} { + var retired int + if err := d.DB.QueryRowContext(ctx, "SELECT count(*) FROM "+table+" WHERE batch_id IN ("+idsSQL(inputIDs)+")").Scan(&retired); err != nil || retired != 0 { + t.Fatalf("retired contributions table=%s rows=%d error=%v", table, retired, err) + } + } + var candidateRows, rolledSpans int + if err := d.DB.QueryRowContext(ctx, `SELECT (SELECT count(*) FROM read_trace_candidates),(SELECT coalesce(sum(spans),0) FROM span_rollup)`).Scan(&candidateRows, &rolledSpans); err != nil || candidateRows != want || rolledSpans != want { + t.Fatalf("cache candidates=%d rollup spans=%d want=%d error=%v", candidateRows, rolledSpans, want, err) + } + stop() + join() + t.Logf("cold_startup converged=%s active_batches=%d candidate_rows=%d raw_rows=%d cache_markers=complete watermarks=complete retired_contributions=0", time.Since(startupBegan), len(repo.Parquet.BatchMetadata()), candidateRows, want) +} + +func coldStartupAwait(t *testing.T, ctx context.Context, label string, ready func(context.Context) (bool, error)) { + t.Helper() + ticker := time.NewTicker(50 * time.Millisecond) + defer ticker.Stop() + for { + ok, err := ready(ctx) + if err != nil { + t.Fatalf("%s: %v", label, err) + } + if ok { + return + } + select { + case <-ctx.Done(): + t.Fatalf("no progress: %s: %v", label, ctx.Err()) + case <-ticker.C: + } + } +} +func coldStartupAssertRaw(t *testing.T, ctx context.Context, d *Duck, want int) { + t.Helper() + var spans, traces int + if err := d.QueryRowScan(ctx, []any{&spans, &traces}, `SELECT count(*),count(DISTINCT trace_id) FROM spans`); err != nil || spans != want || traces != want { + t.Fatalf("raw spans=%d traces=%d want=%d error=%v", spans, traces, want, err) + } +} diff --git a/internal/engine/startup_maintenance_test.go b/internal/engine/startup_maintenance_test.go new file mode 100644 index 000000000..73cb59de1 --- /dev/null +++ b/internal/engine/startup_maintenance_test.go @@ -0,0 +1,182 @@ +package engine + +import ( + "context" + "fmt" + "sync" + "testing" + "time" + + "github.com/labstack/pitot/internal/engine/writegate" + "github.com/labstack/pitot/internal/queryrows" + "github.com/labstack/pitot/internal/telemetry" + telemetrystore "github.com/labstack/pitot/internal/telemetry/store" +) + +// Initial analytical work must not prevent independent Parquet maintenance +// from reducing the catalog. All production gates and budgets remain in use. +func TestStartupMaintenanceProgressesBeforeInitialRollup(t *testing.T) { + for _, state := range []struct { + name string + warm bool + expired bool + }{ + {name: "cold"}, + {name: "warm", warm: true}, + {name: "cold_with_retention", expired: true}, + } { + t.Run(state.name, func(t *testing.T) { + d, repo, window, horizon := startupMaintenanceFixture(t, state.expired) + if state.warm { + if _, err := d.rollupOnce(t.Context()); err != nil { + t.Fatal(err) + } + } + unlock, stop, done := startMaintenanceBehindAnalyticalGate(t, d) + startupMaintenanceAwait(t, "catalog compaction while initial analytical gate is held", func(context.Context) (bool, error) { + active := repo.Parquet.BatchMetadata() + return len(active) == 2 && (active[0].Generation > 0 || active[1].Generation > 0), nil + }) + // Before any initial analytical write, raw reads already see exactly + // the retained canonical rows through the new compacted catalog. + assertStartupRawRows(t, d, 9) + unlock() + startupMaintenanceAwait(t, "complete active markers and analytical watermarks", func(ctx context.Context) (bool, error) { + markers, err := batchMarkers(ctx, d.DB) + if err != nil { + return false, err + } + active := repo.Parquet.BatchMetadata() + if len(markers) != len(active) { + return false, nil + } + for _, batch := range active { + if !markers[batch.ID] { + return false, nil + } + } + var complete int + err = d.DB.QueryRowContext(ctx, `SELECT count(*) FROM rollup_state WHERE cache_key IN (?,?) AND last_ingested_unix_nano>=?`, serviceRollupStateKey, edgeRollupStateKey, horizon).Scan(&complete) + return complete == 2, err + }) + assertStartupRawRows(t, d, 9) + var traces int + window.Kind = queryrows.TraceCandidateRead + window.Namespace, window.Service = "startup", "api" + if err := d.QueryRowScan(queryrows.WithWindow(t.Context(), window), []any{&traces}, `SELECT count(DISTINCT trace_id) FROM (SELECT trace_id FROM trace_candidates UNION ALL SELECT trace_id FROM trace_tail)`); err != nil || traces != 9 { + t.Fatalf("candidate traces=%d want9: %v", traces, err) + } + assertInventoryOracle(t, d, window, 3) + var rolled int + if err := d.DB.QueryRowContext(t.Context(), `SELECT coalesce(sum(spans),0) FROM span_rollup`).Scan(&rolled); err != nil || rolled != 9 { + t.Fatalf("retained rollup spans=%d want9: %v", rolled, err) + } + stop() + awaitStartupStopped(t, done) + }) + } +} + +func TestStartupMaintenanceCancellationDuringInitialGateWait(t *testing.T) { + d, repo, _, _ := startupMaintenanceFixture(t, false) + unlock, stop, done := startMaintenanceBehindAnalyticalGate(t, d) + startupMaintenanceAwait(t, "maintenance starts independently of initial gate wait", func(context.Context) (bool, error) { + active := repo.Parquet.BatchMetadata() + return len(active) == 2 && (active[0].Generation > 0 || active[1].Generation > 0), nil + }) + // Cancellation happens while the analytical gate is held. Its existing + // non-contextual Lock still needs the owner to release it; after release, + // RunRollups must join both background loops before returning. + stop() + unlock() + awaitStartupStopped(t, done) +} + +func startupMaintenanceFixture(t *testing.T, expired bool) (*Duck, *telemetrystore.Repository, queryrows.Window, int64) { + t.Helper() + d, repo := newBatchCacheTest(t) // 256MB, two native threads, two read connections. + d.cfg.RollupInterval = time.Second + d.cfg.MaintenanceInterval = time.Hour + d.cfg.RetentionDays = 1 + at := time.Now().UTC().Truncate(time.Hour).Add(-time.Hour + time.Minute) + put := func(id string, stamp time.Time) { + t.Helper() + n := stamp.UnixNano() + resource := map[string]any{"generation": int64(1), "ready": true} + metric := telemetry.Metric{Namespace: "startup", ServiceName: "api", Name: "startup.gauge", Type: "gauge", Value: 1, TimeUnixNanos: n, EventUnixNanos: n, IngestedAt: n, Resource: resource} + metric.SeriesID = telemetry.MetricSeriesID(metric) + batch := telemetrystore.Batch{ID: id, + Spans: []telemetry.Span{{Namespace: "startup", ServiceName: "api", TraceID: id, SpanID: id, StartUnixNanos: n, EndUnixNanos: n + 1_000_000, DurationMS: 1, Kind: "SPAN_KIND_SERVER", StatusCode: "STATUS_CODE_OK", IngestedAt: n, Resource: resource}}, + Logs: []telemetry.Log{{Namespace: "startup", ServiceName: "api", TimeUnixNanos: n, EventUnixNanos: n, IngestedAt: n, Severity: "INFO", Body: id, Resource: resource}}, Metrics: []telemetry.Metric{metric}} + if err := commitRestoredFixture(repo, t.Context(), batch); err != nil { + t.Fatal(err) + } + } + for i := range 8 { + put(fmt.Sprintf("startup-%d", i), at.Add(time.Duration(i)*time.Nanosecond)) + } + // A partial newest-hour group waits for the production file ceiling. + // This newer sentinel makes the eight historical inputs eligible unchanged. + put("newest", at.Add(time.Hour)) + if expired { + put("expired", at.Add(-48*time.Hour)) + } + // Restored fixtures preserve a deterministic same-hour ingestion clock. + d.publicationBounds = restoredPublicationBounds(repo) + return d, repo, queryrows.Window{Start: at.Add(-time.Second), End: at.Add(time.Hour + time.Second)}, at.Add(time.Hour).UnixNano() +} + +func startMaintenanceBehindAnalyticalGate(t *testing.T, d *Duck) (func(), context.CancelFunc, <-chan struct{}) { + t.Helper() + unlock := sync.OnceFunc(d.writeGate.Lock(writegate.RollupService)) + ctx, stop := context.WithCancel(t.Context()) + done := make(chan struct{}) + go func() { defer close(done); d.RunRollups(ctx) }() + t.Cleanup(func() { + stop() + unlock() + awaitStartupStopped(t, done) + }) + return unlock, stop, done +} + +func startupMaintenanceAwait(t *testing.T, label string, ready func(context.Context) (bool, error)) { + t.Helper() + ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second) + defer cancel() + ticker := time.NewTicker(time.Millisecond) + defer ticker.Stop() + for { + ok, err := ready(ctx) + if err != nil { + t.Fatalf("%s: %v", label, err) + } + if ok { + return + } + select { + case <-ctx.Done(): + t.Fatalf("no progress: %s: %v", label, ctx.Err()) + case <-ticker.C: + } + } +} + +func awaitStartupStopped(t *testing.T, done <-chan struct{}) { + t.Helper() + timer := time.NewTimer(2 * time.Second) + defer timer.Stop() + select { + case <-done: + case <-timer.C: + t.Error("RunRollups did not join its background loops after cancellation") + } +} + +func assertStartupRawRows(t *testing.T, d *Duck, want int) { + t.Helper() + var spans, logs, metrics int + if err := d.QueryRowScan(t.Context(), []any{&spans, &logs, &metrics}, `SELECT (SELECT count(*) FROM spans),(SELECT count(*) FROM logs),(SELECT count(*) FROM metrics)`); err != nil || spans != want || logs != want || metrics != want { + t.Fatalf("raw rows=%d/%d/%d want%d each: %v", spans, logs, metrics, want, err) + } +} diff --git a/internal/engine/trace_candidate_window_bench_test.go b/internal/engine/trace_candidate_window_bench_test.go new file mode 100644 index 000000000..04ec4ce63 --- /dev/null +++ b/internal/engine/trace_candidate_window_bench_test.go @@ -0,0 +1,186 @@ +//go:build filebench + +package engine + +import ( + "fmt" + "reflect" + "sort" + "strings" + "testing" + "time" + + "github.com/labstack/pitot/internal/queryrows" + "github.com/labstack/pitot/internal/telemetry" + telemetrystore "github.com/labstack/pitot/internal/telemetry/store" +) + +// Exercise the real snapshot binder, scoped candidate repair, raw tail and +// completed-query kernel. This is an exploratory local benchmark, not an HTTP +// or mixed-ingest capacity gate. Run unchanged before and after a query edit. +func TestTraceCandidateWindowPlanComparisonBenchmark(t *testing.T) { + const traces = 20000 + for _, boundaryPercent := range []int{1, 10} { + t.Run(fmt.Sprintf("boundary_%d_percent", boundaryPercent), func(t *testing.T) { + d, repo := newBatchCacheTest(t) + base := time.Date(2026, 11, 1, 12, 0, 0, 0, time.UTC) + for part := range 4 { + batch := telemetrystore.Batch{ID: fmt.Sprintf("part_%d", part)} + for i := range traces { + // Permute identities independently of time and scope. Some + // traces span services, so the scoped index must be repaired. + id := uint64(i+1) * 0x9e3779b97f4a7c15 + service := "api" + if i%7 == 0 && part == 0 { + service = "db" + } + start := base.Add(time.Duration(i%1000+part) * time.Millisecond) + if i%100 < boundaryPercent && part == 0 { + start = base.Add(-time.Second) + if (i/100)%2 == 0 { + start = base.Add(3 * time.Second) + } + } + status := "STATUS_CODE_OK" + if i%13 == 0 && part == 0 { + status = "STATUS_CODE_ERROR" + } + batch.Spans = append(batch.Spans, telemetry.Span{Namespace: "prod", ServiceName: service, + TraceID: fmt.Sprintf("%032x", id), SpanID: fmt.Sprintf("%016x", part+1), + StartUnixNanos: start.UnixNano(), EndUnixNanos: start.Add(time.Millisecond).UnixNano(), + DurationMS: 1, StatusCode: status, IngestedAt: base.UnixNano()}) + } + if err := repo.Commit(t.Context(), batch); err != nil { + t.Fatal(err) + } + } + if _, err := d.RefreshReadCaches(t.Context()); err != nil { + t.Fatal(err) + } + var version, memory string + var threads int + if err := d.DB.QueryRow("SELECT version(),current_setting('memory_limit'),current_setting('threads')").Scan(&version, &memory, &threads); err != nil { + t.Fatal(err) + } + t.Logf("engine=%s memory=%s threads=%d traces=%d parts=%d boundary_percent=%d", version, memory, threads, traces, traces*4, boundaryPercent) + for _, service := range []string{"", "api"} { + w := queryrows.Window{Start: base, End: base.Add(2 * time.Second), Namespace: "prod", Service: service, Kind: queryrows.TraceCandidateRead} + compareCompletedTraceWindowPlans(t, d, w) + } + }) + } +} + +func compareCompletedTraceWindowPlans(t *testing.T, d *Duck, w queryrows.Window) { + t.Helper() + const samples = 100 + pin := d.repository.Parquet.Pin() + defer d.repository.Parquet.Unpin(pin) + tx, err := d.DB.BeginTx(t.Context(), nil) + if err != nil { + t.Fatal(err) + } + defer tx.Rollback() + compiled, err := d.compileSnapshot(t.Context(), tx, queryrows.CompletedTraceQuery) + if err != nil { + t.Fatal(err) + } + batches := d.repository.Parquet.BatchMetadata() + sources := map[string]string{} + if err := d.aggregateSources(t.Context(), tx, batches, w, false, sources); err != nil { + t.Fatal(err) + } + markers, err := batchMarkers(t.Context(), tx) + if err != nil { + t.Fatal(err) + } + var ids []string + for _, b := range batches { + if markers[b.ID] && overlapping(b, "spans", w) { + ids = append(ids, b.ID) + } + } + scope := "batch_id IN (" + idsSQL(ids) + ") AND namespace=" + sqlLiteral(w.Namespace) + if w.Service != "" { + scope += " AND service=" + sqlLiteral(w.Service) + } + full := fmt.Sprintf("min_start>=%d AND max_start<%d", w.Start.UnixNano(), w.End.UnixNano()) + index, err := d.traceCandidateSource(t.Context(), tx, batches, markers, w) + if err != nil { + t.Fatal(err) + } + partial := "SELECT * FROM (" + index + ") WHERE NOT (" + full + ") AND " + fmt.Sprintf("max_start>=%d AND min_start<%d", w.Start.UnixNano(), w.End.UnixNano()) + // Keep both experimental expressions here so a negative result remains + // reproducible without changing product code or its materialization policy. + bodies := []string{ + `SELECT namespace,trace_id,min(min_start) AS min_start,max(max_end) AS max_end,max(has_error) AS has_error FROM read_trace_parts WHERE ` + scope + ` AND ` + full + ` GROUP BY namespace,trace_id`, + "SELECT namespace,trace_id,min_start,max_end,has_error FROM (" + index + ") WHERE " + full + + ` UNION ALL SELECT namespace,trace_id,min(min_start),max(max_end),max(has_error) FROM read_trace_parts WHERE ` + scope + ` AND ` + full + + ` AND (namespace,trace_id) IN (SELECT namespace,trace_id FROM (` + partial + `)) GROUP BY namespace,trace_id`, + } + if sources["trace_candidates"] != bodies[0] { + t.Fatal("benchmark baseline no longer matches the production candidate expression") + } + var queries, names []string + for i, body := range bodies { + sources["trace_candidates"] = body + bound, err := bindCompiledSnapshot(compiled, sources) + if err != nil { + t.Fatal(err) + } + marker := compiled.names["trace_candidates"] + " AS NOT MATERIALIZED (" + if strings.Count(bound, marker) != 1 { + t.Fatal("benchmark requires the current candidate inlining policy") + } + queries = append(queries, bound, strings.Replace(bound, marker, compiled.names["trace_candidates"]+" AS MATERIALIZED (", 1)) + names = append(names, fmt.Sprintf("expression_%d_inline", i), fmt.Sprintf("expression_%d_shared", i)) + } + args := []any{w.Start, w.End, w.Namespace, w.Namespace, w.Service, w.Service, 20} + run := func(query string) []completedTraceWindowRow { + rows, err := tx.QueryContext(t.Context(), query, args...) + if err != nil { + t.Fatal(err) + } + defer rows.Close() + var out []completedTraceWindowRow + for rows.Next() { + var row completedTraceWindowRow + if err := rows.Scan(&row.namespace, &row.trace, &row.hasError); err != nil { + t.Fatal(err) + } + out = append(out, row) + } + if err := rows.Err(); err != nil || len(out) != 20 { + t.Fatalf("completed rows=%d error=%v", len(out), err) + } + return out + } + rawWindow := w + rawWindow.Kind = queryrows.RawRead + want := completedTraceWindowRows(t, d, rawWindow, rawCompletedTraceWindowQuery, args) + // The actual completed-query API projects identities/errors. The separate + // lifecycle oracle compares its internal bounds as well. + for i := range want { + want[i].minStart, want[i].maxEnd = 0, 0 + } + for _, query := range queries { + for range 3 { + if got := run(query); !reflect.DeepEqual(got, want) { + t.Fatalf("experimental query changed ranking: got=%v want=%v", got, want) + } + } + } + timings := make([][]float64, len(queries)) + for sample := range samples { + for offset := range queries { + i := (sample + offset) % len(queries) + started := time.Now() + run(queries[i]) + timings[i] = append(timings[i], float64(time.Since(started))/float64(time.Millisecond)) + } + } + for i, values := range timings { + sort.Float64s(values) + t.Logf("scope_service=%q variant=%s bound_execution_only=true samples=%d median_ms=%.3f p95_ms=%.3f", w.Service, names[i], samples, (values[49]+values[50])/2, values[94]) + } +} diff --git a/internal/engine/trace_candidate_window_test.go b/internal/engine/trace_candidate_window_test.go new file mode 100644 index 000000000..62557f82e --- /dev/null +++ b/internal/engine/trace_candidate_window_test.go @@ -0,0 +1,147 @@ +package engine + +import ( + "fmt" + "reflect" + "strings" + "testing" + "time" + + "github.com/labstack/pitot/internal/queryrows" + "github.com/labstack/pitot/internal/telemetry" + telemetrystore "github.com/labstack/pitot/internal/telemetry/store" +) + +// Compare the complete ranking query with raw rows, not merely the candidate +// source: boundary candidates and raw tail overlap and must merge exactly. +func TestCompletedTraceWindowsMatchRawSpans(t *testing.T) { + d, repo := newBatchCacheTest(t) + span := func(namespace, service, trace string, start, end int64, failed bool) telemetry.Span { + status := "STATUS_CODE_OK" + if failed { + status = "STATUS_CODE_ERROR" + } + return telemetry.Span{Namespace: namespace, ServiceName: service, TraceID: trace, + SpanID: fmt.Sprintf("%s-%s-%d", namespace, service, start), StartUnixNanos: start, + EndUnixNanos: end, DurationMS: float64(end-start) / 1e6, StatusCode: status, IngestedAt: 1} + } + commit := func(id string, spans ...telemetry.Span) { + t.Helper() + for i := range spans { + if id == "interior" { + spans[i].IngestedAt = 2 + } else if id == "late" { + spans[i].IngestedAt = 3 + } + } + if err := commitRestoredFixture(repo, t.Context(), telemetrystore.Batch{ID: id, Spans: spans}); err != nil { + t.Fatal(err) + } + } + // The file straddles both boundaries, but includes an entirely contained + // trace and an interior part of a trace whose other parts cross the start. + commit("boundaries", + span("prod", "api", "contained", 0, 5, false), + span("prod", "api", "partial", -15, -10, true), + span("prod", "api", "partial", -5, -1, false), + span("prod", "api", "mixed", 1, 7, false), + span("prod", "db", "mixed", -15, 200, true), + span("alpha", "api", "collision", -3, 4, false), + span("beta", "api", "collision", 1, 2, true), + span("prod", "api", "at_start", -10, -8, false), + span("prod", "api", "at_end", 10, 30, true), + ) + commit("interior", + span("prod", "api", "partial", 0, 9, false), + span("prod", "api", "contained", 3, 7, false), + span("prod", "db", "mixed", 4, 5, true), + ) + if _, err := d.RefreshReadCaches(t.Context()); err != nil { + t.Fatal(err) + } + check := func(stage string) { + t.Helper() + for _, scope := range [][2]string{{"", ""}, {"prod", ""}, {"prod", "api"}, {"prod", "db"}, {"alpha", "api"}, {"beta", "api"}} { + for _, bounds := range [][2]int64{{-10, 10}, {-5, 4}, {0, 1}, {-20, 30}} { + for _, limit := range []int{1, 3, 20} { + w := queryrows.Window{Start: time.Unix(0, bounds[0]).UTC(), End: time.Unix(0, bounds[1]).UTC(), Namespace: scope[0], Service: scope[1], Kind: queryrows.TraceCandidateRead} + args := []any{w.Start, w.End, scope[0], scope[0], scope[1], scope[1], limit} + // Expose the kernel's aggregate bounds as well as its public + // identity/error projection, without replacing any merge logic. + query := strings.Replace(queryrows.CompletedTraceQuery, "SELECT namespace,trace_id,has_error FROM candidates", "SELECT namespace,trace_id,has_error,min_start,max_end FROM candidates", 1) + got := completedTraceWindowRows(t, d, w, query, args) + w.Kind = queryrows.RawRead + want := completedTraceWindowRows(t, d, w, rawCompletedTraceWindowQuery, args) + if !reflect.DeepEqual(got, want) { + t.Fatalf("%s scope=%v bounds=%v limit=%d: cached=%v raw=%v", stage, scope, bounds, limit, got, want) + } + } + } + } + } + check("cached") + commit("late", span("prod", "api", "partial", -1, 20, true), span("prod", "api", "new", 2, 8, false)) + check("uncached late publication") + if _, err := d.RefreshReadCaches(t.Context()); err != nil { + t.Fatal(err) + } + check("acknowledged late publication") + for i := range 8 { + commit(fmt.Sprintf("filler_%d", i), span("prod", "api", fmt.Sprintf("filler_%d", i), -1, 0, false)) + } + if _, err := d.RefreshReadCaches(t.Context()); err != nil { + t.Fatal(err) + } + if n, err := repo.CompactParquet(t.Context(), d, 8); err != nil || n == 0 { + t.Fatalf("compaction inputs=%d error=%v", n, err) + } + check("compaction before retired marker cleanup") + if n, err := repo.PruneParquet(t.Context(), d, 3, 8); err != nil || n == 0 { + t.Fatalf("retention removed=%d error=%v", n, err) + } + check("retention before marker cleanup") + var surviving int + if err := d.QueryRowScan(t.Context(), []any{&surviving}, `SELECT count(*) FROM spans WHERE trace_id='new'`); err != nil || surviving != 1 { + t.Fatalf("late trace must survive retention: rows=%d error=%v", surviving, err) + } + if _, err := d.RefreshReadCaches(t.Context()); err != nil { + t.Fatal(err) + } + check("retired marker cleanup") +} + +const rawCompletedTraceWindowQuery = `SELECT namespace,trace_id, + max(CASE WHEN status='STATUS_CODE_ERROR' THEN 1 ELSE 0 END) AS has_error, + min(start_unix_nano),max(end_unix_nano) + FROM spans WHERE start_time >= $1::TIMESTAMP_NS::TIMESTAMPTZ_NS + AND start_time < $2::TIMESTAMP_NS::TIMESTAMPTZ_NS + AND ($3='' OR namespace=$4) AND ($5='' OR service=$6) AND trace_id<>'' + GROUP BY namespace,trace_id + ORDER BY has_error DESC,max(end_unix_nano)-min(start_unix_nano) DESC,trace_id,namespace LIMIT $7` + +type completedTraceWindowRow struct { + namespace, trace string + hasError int + minStart, maxEnd int64 +} + +func completedTraceWindowRows(t testing.TB, d *Duck, w queryrows.Window, sql string, args []any) []completedTraceWindowRow { + t.Helper() + rows, err := d.QueryContext(queryrows.WithWindow(t.Context(), w), sql, args...) + if err != nil { + t.Fatal(err) + } + defer rows.Close() + var out []completedTraceWindowRow + for rows.Next() { + var row completedTraceWindowRow + if err := rows.Scan(&row.namespace, &row.trace, &row.hasError, &row.minStart, &row.maxEnd); err != nil { + t.Fatal(err) + } + out = append(out, row) + } + if err := rows.Err(); err != nil { + t.Fatal(err) + } + return out +} diff --git a/internal/engine/views.go b/internal/engine/views.go index 937e79a85..332f08ac1 100644 --- a/internal/engine/views.go +++ b/internal/engine/views.go @@ -8,11 +8,16 @@ import ( "github.com/labstack/pitot/internal/telemetry" ) +// service_rollup drops its compound ART primary key for the same reason as the +// scan summaries below. refreshServiceRollup replaces each affected scope in one +// serialized transaction and its INSERT groups by the complete logical key, so +// uniqueness holds without an index DuckDB would probe under the checkpoint +// vacuum lock. Keep every key column NOT NULL. const createServiceRollupTable = ` CREATE TABLE IF NOT EXISTS service_rollup ( - namespace TEXT, - bucket TIMESTAMP, - service TEXT, + namespace TEXT NOT NULL, + bucket TIMESTAMP NOT NULL, + service TEXT NOT NULL, spans BIGINT, -- Spans that describe work this service performed, rather than a call it was -- waiting on. A bucket with none of them has no latency of its own to report, @@ -22,26 +27,29 @@ CREATE TABLE IF NOT EXISTS service_rollup ( p95_ms DOUBLE, error_rate DOUBLE, log_count BIGINT DEFAULT 0, - metric_count BIGINT DEFAULT 0, - PRIMARY KEY (namespace, bucket, service) + metric_count BIGINT DEFAULT 0 );` // spanHistogramZero is outside the index range of positive finite durations. const spanHistogramZero = -2147483648 +// These scan-oriented summaries intentionally have no ART primary key. +// refreshServiceRollup serializes complete affected-scope replacement in one +// transaction, and each INSERT groups by the complete logical key. Keep every +// key column NOT NULL. A compound ART cannot accelerate these scans, but the +// pinned engine probes it under a vacuum lock that waits on full checkpoints. const createSpanRollupTable = ` CREATE TABLE IF NOT EXISTS span_rollup ( - namespace TEXT, - bucket TIMESTAMP_NS, - service TEXT, - kind TEXT, - status TEXT, + namespace TEXT NOT NULL, + bucket TIMESTAMP_NS NOT NULL, + service TEXT NOT NULL, + kind TEXT NOT NULL, + status TEXT NOT NULL, spans BIGINT, duration_sum DOUBLE, duration_min DOUBLE, duration_max DOUBLE, - hist MAP(INTEGER, UBIGINT), - PRIMARY KEY (namespace, bucket, service, kind, status) + hist MAP(INTEGER, UBIGINT) );` // Endpoint summaries retain at most 128 routes per service-minute plus one @@ -50,31 +58,36 @@ const endpointRouteLimit = 128 const createEndpointRollupTable = ` CREATE TABLE IF NOT EXISTS endpoint_rollup ( - namespace TEXT, bucket TIMESTAMP_NS, service TEXT, kind TEXT, status TEXT, - route TEXT, overflow BOOLEAN, spans BIGINT, duration_sum DOUBLE, - duration_min DOUBLE, duration_max DOUBLE, hist MAP(INTEGER, UBIGINT), - PRIMARY KEY (namespace,bucket,service,kind,status,route,overflow) + namespace TEXT NOT NULL, bucket TIMESTAMP_NS NOT NULL, service TEXT NOT NULL, kind TEXT NOT NULL, status TEXT NOT NULL, + route TEXT NOT NULL, overflow BOOLEAN NOT NULL, spans BIGINT, duration_sum DOUBLE, + duration_min DOUBLE, duration_max DOUBLE, hist MAP(INTEGER, UBIGINT) );` const createLogRollupTable = ` CREATE TABLE IF NOT EXISTS log_rollup ( - namespace TEXT, bucket TIMESTAMP_NS, service TEXT, severity TEXT, - logs BIGINT, PRIMARY KEY(namespace,bucket,service,severity) + namespace TEXT NOT NULL, bucket TIMESTAMP_NS NOT NULL, service TEXT NOT NULL, severity TEXT NOT NULL, + logs BIGINT );` +// edge_rollup also drops its ART primary key. Its pass deletes the affected +// window and reinserts it in one serialized transaction. The INSERT is a +// UNION ALL of three branches whose edge_type literals ('call', 'messaging', +// 'peer') are disjoint, and each branch groups by (namespace, bucket, caller, +// callee), so the complete key stays unique without the index. A new branch +// MUST use a distinct edge_type and group by the full key, or uniqueness +// breaks silently. Keep every key column NOT NULL. const createEdgeRollupTable = ` CREATE TABLE IF NOT EXISTS edge_rollup ( - namespace TEXT, - bucket TIMESTAMP, - caller TEXT, - callee TEXT, + namespace TEXT NOT NULL, + bucket TIMESTAMP NOT NULL, + caller TEXT NOT NULL, + callee TEXT NOT NULL, calls BIGINT, avg_ms DOUBLE, error_rate DOUBLE, - edge_type TEXT DEFAULT 'call', + edge_type TEXT NOT NULL DEFAULT 'call', p50_ms DOUBLE DEFAULT 0, - p95_ms DOUBLE DEFAULT 0, - PRIMARY KEY (namespace, bucket, caller, callee, edge_type) + p95_ms DOUBLE DEFAULT 0 );` const createRollupStateTable = ` diff --git a/internal/query/backlog_pressure_test.go b/internal/query/backlog_pressure_test.go index e439c2cae..e905289af 100644 --- a/internal/query/backlog_pressure_test.go +++ b/internal/query/backlog_pressure_test.go @@ -74,7 +74,7 @@ func pressureBatch(batch int, base time.Time, live bool) telemetrystore.Batch { } service := fmt.Sprintf("svc%d", i%6) trace := fmt.Sprintf("%s-trace-%d-%d", prefix, batch, i/4) - b.Spans = append(b.Spans, telemetry.Span{Namespace: "pressure", ServiceName: service, TraceID: trace, SpanID: fmt.Sprintf("s%d", i), StartUnixNanos: n, EndUnixNanos: n + int64(i%1000+1)*1000000, DurationMS: float64(i%1000 + 1), Kind: "SPAN_KIND_SERVER", StatusCode: "STATUS_CODE_OK", ServiceVersion: version, IngestedAt: ingested}) + b.Spans = append(b.Spans, telemetry.Span{Namespace: "pressure", ServiceName: service, TraceID: trace, SpanID: fmt.Sprintf("s%d", i), StartUnixNanos: n, EndUnixNanos: n + int64(i%1000+1)*1000000, DurationMS: float64(i%1000 + 1), Kind: "SPAN_KIND_SERVER", StatusCode: "STATUS_CODE_OK", Resource: map[string]any{"service.version": version}, IngestedAt: ingested}) if i < 192 { b.Logs = append(b.Logs, telemetry.Log{Namespace: "pressure", ServiceName: service, TraceID: trace, EventUnixNanos: n, TimeUnixNanos: n, IngestedAt: ingested, Body: "request complete", Severity: "INFO"}) } @@ -559,7 +559,9 @@ func pressureExactCount(t *testing.T, r Result, expected int64) bool { func TestBacklogCountsReadWholeWindowMeasureTotals(t *testing.T) { ctx, cancel := context.WithTimeout(t.Context(), time.Minute) defer cancel() - d, _, start, end := pressurePanelDuck(ctx, t, 1) + d, repo, start, end := pressurePanelDuck(ctx, t, 1) + horizon, _, _ := repo.PublicationBounds() + panelBenchWarmDashboard(t, d, horizon) results, err := NewExecutor(d, 30).Run(ctx, RunRequest{Dashboard: pressureSpec(start, end)}) if err != nil { t.Fatal(err) @@ -603,7 +605,9 @@ func (e pressurePanelEngine) QueryContext(ctx context.Context, q string, args .. func TestBacklogPanelErrorsIdentifyActualSQLAndKeepNativeCause(t *testing.T) { ctx, cancel := context.WithTimeout(t.Context(), time.Minute) defer cancel() - d, _, start, end := pressurePanelDuck(ctx, t, 1) + d, repo, start, end := pressurePanelDuck(ctx, t, 1) + horizon, _, _ := repo.PublicationBounds() + panelBenchWarmDashboard(t, d, horizon) q := "SELECT missing_pressure_column FROM spans WHERE start_time>=?::TIMESTAMP_NS::TIMESTAMPTZ_NS AND start_time= 100 && len(spec.Panels) == 1 && spec.Panels[0].ID == "latency" { + for i, engine := range engines { + var key, plan string + if err := d.DB.QueryRowContext(b.Context(), "EXPLAIN (ANALYZE, FORMAT JSON) "+engine.query, engine.args...).Scan(&key, &plan); err != nil { + b.Fatal(err) + } + var compact bytes.Buffer + if err := json.Compact(&compact, []byte(plan)); err != nil { + b.Fatal(err) + } + b.Logf("native_profile_variant=%d %s=%s", i, key, compact.String()) + } + } +} diff --git a/internal/query/histogram_expansion_test.go b/internal/query/histogram_expansion_test.go new file mode 100644 index 000000000..757d71545 --- /dev/null +++ b/internal/query/histogram_expansion_test.go @@ -0,0 +1,69 @@ +package query + +import ( + "math" + "strings" + "testing" +) + +// Expanding an empty map must not remove the aggregate group's count or turn +// an absent quantile into zero. Zero-duration bins remain real measurements. +func TestHistogramExpansionPreservesEmptyGroupsAndZeroBins(t *testing.T) { + d, _ := newTestEngine(t) + var query strings.Builder + query.WriteString(`WITH input(service,spans,hist) AS (VALUES + ('empty',2::BIGINT,map([]::INTEGER[],[]::UBIGINT[])), + ('null',3::BIGINT,NULL::MAP(INTEGER,UBIGINT)), + ('zero',4::BIGINT,map([-2147483648],[4]::UBIGINT[])), + ('positive',5::BIGINT,map([8,16],[2,3]::UBIGINT[])))`) + from := histogramAggregate(&query, "checked", "input", + []string{"service", "sum(spans)::DOUBLE AS n", "NULL::DOUBLE AS p50", "NULL::DOUBLE AS p95"}, + []string{"service", "n", "p50", "p95"}, []string{"service"}, false, + map[int]Measure{2: {Q: .5}, 3: {Q: .95}}) + query.WriteString(" SELECT service,n,p50,p95 FROM " + from + " ORDER BY service") + rows, err := d.DB.QueryContext(t.Context(), query.String()) + if err != nil { + t.Fatal(err) + } + defer rows.Close() + lower := math.Exp2(15.0 / 8) + want := []struct { + service string + count float64 + median, p95 *float64 + }{ + {service: "empty", count: 2}, {service: "null", count: 3}, + {service: "positive", count: 5, median: histogramTestFloat(lower + (4-lower)/6), p95: histogramTestFloat(lower + (4-lower)*11/12)}, + {service: "zero", count: 4, median: histogramTestFloat(0), p95: histogramTestFloat(0)}, + } + for _, expected := range want { + if !rows.Next() { + t.Fatalf("missing group %s: %v", expected.service, rows.Err()) + } + var service string + var count float64 + var median, p95 *float64 + if err := rows.Scan(&service, &count, &median, &p95); err != nil { + t.Fatal(err) + } + if service != expected.service || count != expected.count { + t.Fatalf("group=%s count=%g want=%s/%g", service, count, expected.service, expected.count) + } + for i, pair := range [][2]*float64{{median, expected.median}, {p95, expected.p95}} { + if pair[0] == nil || pair[1] == nil { + if pair[0] != pair[1] { + t.Fatalf("%s quantile%d null changed", service, i) + } + continue + } + if math.Abs(*pair[0]-*pair[1]) > 1e-12 { + t.Fatalf("%s quantile%d=%g want=%g", service, i, *pair[0], *pair[1]) + } + } + } + if rows.Next() || rows.Err() != nil { + t.Fatalf("unexpected groups/error: %v", rows.Err()) + } +} + +func histogramTestFloat(value float64) *float64 { return &value } diff --git a/internal/query/histogram_weighted_bench_test.go b/internal/query/histogram_weighted_bench_test.go new file mode 100644 index 000000000..ba2001f1e --- /dev/null +++ b/internal/query/histogram_weighted_bench_test.go @@ -0,0 +1,348 @@ +//go:build readbench + +package query + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "reflect" + "regexp" + "sort" + "strconv" + "strings" + "sync" + "testing" + "time" + + "github.com/labstack/pitot/internal/config" + "github.com/labstack/pitot/internal/engine" + "github.com/labstack/pitot/internal/queryrows" + telemetrystore "github.com/labstack/pitot/internal/telemetry/store" +) + +// This experiment rewrites only the compiler's known histogram-bin fragment. +// It is deliberately test-only: repeated-map compression is a hypothesis until +// both full-executor and mixed-load comparisons clear the observed regression. +var weightedHistogramFragment = regexp.MustCompile(`, ([a-zA-Z0-9_]+)_bins AS \(SELECT (.*?)e.key AS __hist_k,sum\(e.value\)::DOUBLE AS __hist_n FROM \(SELECT \*,unnest\(map_entries\(hist\)\) AS e FROM ([a-zA-Z0-9_]+)\) __hist_entries(.*?)\), ([a-zA-Z0-9_]+)_cdf AS`) + +func histogramExperimentSQL(query, variant string) (string, error) { + for _, source := range []string{"base", "folded"} { + lateral := "FROM " + source + ",unnest(map_entries(hist)) entries(e)" + projected := "FROM (SELECT *,unnest(map_entries(hist)) AS e FROM " + source + ") __hist_entries" + if variant == "lateral" { + query = strings.ReplaceAll(query, projected, lateral) + } else { + query = strings.ReplaceAll(query, lateral, projected) + } + } + if variant == "lateral" || variant == "projection" { + return query, nil + } + if variant != "weighted" { + return "", fmt.Errorf("unknown histogram experiment %q", variant) + } + var rewriteErr error + matched := 0 + query = weightedHistogramFragment.ReplaceAllStringFunc(query, func(fragment string) string { + match := weightedHistogramFragment.FindStringSubmatch(fragment) + prefix, hidden, source := match[1], match[2], match[3] + if prefix != match[5] { + rewriteErr = fmt.Errorf("histogram CTE prefix mismatch") + return fragment + } + marker, rest, ok := strings.Cut(hidden, " AS __hist_group,") + if !ok { + rewriteErr = fmt.Errorf("unrecognized histogram group marker %q", hidden) + return fragment + } + var groups, projected []string + for i := 0; rest != ""; i++ { + expression, remaining, ok := strings.Cut(rest, fmt.Sprintf(" AS __hist_key_%d,", i)) + if !ok { + rewriteErr = fmt.Errorf("unrecognized histogram grouping expression %q", rest) + return fragment + } + alias := fmt.Sprintf("__hist_input_%d", i) + groups = append(groups, alias) + projected = append(projected, expression+" AS "+alias) + rest = remaining + } + totals := strings.HasPrefix(marker, "grouping(") + if marker != "0" && !totals || totals && len(groups) == 0 { + rewriteErr = fmt.Errorf("unrecognized histogram total marker %q", marker) + return fragment + } + newHidden := []string{"0 AS __hist_group"} + if totals { + newHidden[0] = "grouping(" + groups[0] + ") AS __hist_group" + } + for i, group := range groups { + newHidden = append(newHidden, group+fmt.Sprintf(" AS __hist_key_%d", i)) + } + all := append(append([]string(nil), groups...), "e.key") + grouping := " GROUP BY " + strings.Join(all, ",") + if totals { + rest := append(append([]string(nil), groups[1:]...), "e.key") + grouping = " GROUP BY GROUPING SETS ((" + strings.Join(all, ",") + "),(" + strings.Join(rest, ",") + "))" + } + projected = append(projected, "hist", "count(*) AS __hist_mult") + compressed := "SELECT " + strings.Join(projected, ",") + " FROM " + source + " GROUP BY ALL" + matched++ + return fmt.Sprintf(", %s_bins AS (SELECT %s,e.key AS __hist_k,sum(e.value::HUGEINT*__hist_mult)::DOUBLE AS __hist_n FROM (SELECT *,unnest(map_entries(hist)) AS e FROM (%s) __hist_compressed) __hist_entries%s), %s_cdf AS", prefix, strings.Join(newHidden, ","), compressed, grouping, prefix) + }) + if rewriteErr != nil { + return "", rewriteErr + } + if strings.Contains(query, "__hist_k") && matched == 0 { + return "", fmt.Errorf("histogram experiment did not match generated bins") + } + return query, nil +} + +type weightedHistogramEngine struct { + *engine.Duck + variant string + mu sync.Mutex + query string + args []any +} + +func (e *weightedHistogramEngine) QueryContext(ctx context.Context, query string, args ...any) (queryrows.Rows, error) { + query, err := histogramExperimentSQL(query, e.variant) + if err != nil { + return nil, err + } + if strings.Contains(query, "__panel_hist_bins") { + e.mu.Lock() + e.query, e.args = query, append([]any(nil), args...) + e.mu.Unlock() + } + return e.Duck.QueryContext(ctx, query, args...) +} + +func TestWeightedHistogramExperimentPreservesBinsAndQuantiles(t *testing.T) { + d, _ := newTestEngine(t) + input := `WITH base(bucket,service,spans,hist) AS (VALUES + (0,'empty',2::BIGINT,map([]::INTEGER[],[]::UBIGINT[])), + (1,'empty',3::BIGINT,map([]::INTEGER[],[]::UBIGINT[])), + (0,'null',4::BIGINT,NULL::MAP(INTEGER,UBIGINT)), + (1,'null',5::BIGINT,NULL::MAP(INTEGER,UBIGINT)), + (0,'zero',6::BIGINT,map([-2147483648],[6]::UBIGINT[])), + (1,'zero',6::BIGINT,map([-2147483648],[6]::UBIGINT[])), + (0,'positive',5::BIGINT,map([8,16],[2,3]::UBIGINT[])), + (0,'positive',5::BIGINT,map([8,16],[2,3]::UBIGINT[])), + (1,'positive',5::BIGINT,map([16,8],[3,2]::UBIGINT[])), + (1,NULL,5::BIGINT,map([8,16],[2,3]::UBIGINT[])), + (1,'',5::BIGINT,map([8,16],[2,3]::UBIGINT[])), + (0,'huge',1::BIGINT,map([8],[9223372036854775808]::UBIGINT[])), + (0,'huge',1::BIGINT,map([8],[9223372036854775808]::UBIGINT[])), + (0,'huge',1::BIGINT,map([8],[9223372036854775808]::UBIGINT[])))` + for _, shape := range []struct { + name string + groups []string + totals bool + }{ + {"whole_window", nil, false}, + {"service", []string{"coalesce(service,'')"}, false}, + {"bucket_totals", []string{"bucket"}, true}, + {"bucket_service_totals", []string{"bucket", "coalesce(service,'')"}, true}, + } { + t.Run(shape.name, func(t *testing.T) { + var query strings.Builder + query.WriteString(input) + selects := []string{"sum(spans)::DOUBLE AS n", "NULL::DOUBLE AS p50", "NULL::DOUBLE AS p95"} + aliases := []string{"n", "p50", "p95"} + for i, group := range shape.groups { + alias := fmt.Sprintf("g%d", i) + selects, aliases = append(selects, group+" AS "+alias), append(aliases, alias) + } + histogramAggregate(&query, "checked", "base", selects, aliases, shape.groups, shape.totals, map[int]Measure{1: {Q: .5}, 2: {Q: .95}}) + for _, relation := range []string{"checked_bins", "checked"} { + var want [][]any + for _, variant := range []string{"lateral", "projection", "weighted"} { + sql, err := histogramExperimentSQL(query.String()+" SELECT * FROM "+relation+" ORDER BY ALL", variant) + if err != nil { + t.Fatal(err) + } + rows, err := d.DB.QueryContext(t.Context(), sql) + if err != nil { + t.Fatalf("%s/%s: %v", relation, variant, err) + } + columns, err := rows.Columns() + if err != nil { + rows.Close() + t.Fatal(err) + } + var got [][]any + for rows.Next() { + values, pointers := make([]any, len(columns)), make([]any, len(columns)) + for i := range values { + pointers[i] = &values[i] + } + if err := rows.Scan(pointers...); err != nil { + rows.Close() + t.Fatal(err) + } + got = append(got, values) + } + err = rows.Err() + closeErr := rows.Close() + if err != nil || closeErr != nil { + t.Fatalf("rows=%v close=%v", err, closeErr) + } + if variant == "lateral" { + want = got + } else if !reflect.DeepEqual(want, got) { + t.Fatalf("%s/%s: bins/results changed: want=%v got=%v", relation, variant, want, got) + } + } + } + }) + } +} + +func BenchmarkWeightedHistogramComparison(b *testing.B) { + scale := 4 + if raw := os.Getenv("PITOT_HISTOGRAM_SCALE"); raw != "" { + var err error + scale, err = strconv.Atoi(raw) + if err != nil || scale < 1 || scale > 4 { + b.Fatal("PITOT_HISTOGRAM_SCALE must be 1..4") + } + } + for _, diversity := range []string{"low", "high"} { + b.Run(diversity, func(b *testing.B) { + b.StopTimer() + cfg := config.Config{DataDir: filepath.Join(panelBenchScratch(b), "state"), RetentionDays: 30, DuckDBMemory: "4GB", DuckDBThreads: 4, DuckDBMaxConns: 8} + repo, err := telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + b.Fatal(err) + } + b.Cleanup(func() { repo.Close() }) + end := time.Date(2026, 11, 13, 0, 0, 0, 0, time.UTC) + start := end.Add(-24 * time.Hour) + for index := range 96 * scale { + at := start.Add(time.Minute + time.Duration(index)*(24*time.Hour-4*time.Minute)/time.Duration(96*scale)) + batch := mixedReadBatch(fmt.Sprintf("histogram-%06d", index), at, 2048) + for i := range batch.Spans { + n := at.Add(time.Duration(i) * (24*time.Hour - 4*time.Minute) / time.Duration(96*scale*2048)).UnixNano() + duration := batch.Spans[i].DurationMS + if diversity == "high" { + duration = float64((i%6 + 1) * (1 + (i*7919+index*409)%1000)) + } + batch.Spans[i].StartUnixNanos, batch.Spans[i].EndUnixNanos = n, n+int64(duration*1e6) + batch.Spans[i].DurationMS = duration + batch.Logs[i].EventUnixNanos, batch.Logs[i].TimeUnixNanos = n, n + } + if err := repo.Commit(b.Context(), batch); err != nil { + b.Fatal(err) + } + } + d, err := engine.NewDuck(b.Context(), cfg, repo) + if err != nil { + b.Fatal(err) + } + b.Cleanup(func() { d.Close() }) + horizon, _, _ := repo.PublicationBounds() + panelBenchWarmDashboard(b, d, horizon) + var version, memory string + var threads int + if err := d.DB.QueryRowContext(b.Context(), "SELECT version(),current_setting('memory_limit'),current_setting('threads')").Scan(&version, &memory, &threads); err != nil { + b.Fatal(err) + } + panels := []Panel{ + {ID: "service_p95", Viz: "table", Query: &Spec{From: "spans", Measures: []string{"p95(duration_ms)"}, By: []string{"service"}}}, + {ID: "latency", Viz: "timeseries", Query: &Spec{From: "spans", Measures: []string{"p95(duration_ms)"}, Bucket: "auto"}}, + {ID: "totals", Viz: "stat", Query: &Spec{From: "spans", Measures: []string{"p95(duration_ms)"}}}, + {ID: "multiple", Viz: "timeseries", Query: &Spec{From: "spans", Measures: []string{"p50(duration_ms)", "p95(duration_ms)", "p99(duration_ms)", "avg(duration_ms)"}, Bucket: "auto"}}, + {ID: "rank", Viz: "timeseries", Query: &Spec{From: "spans", Measures: []string{"p95(duration_ms)"}, By: []string{"service"}, Bucket: "auto"}, Options: &Options{Top: 3}}, + {ID: "trends", Viz: "table", Query: &Spec{From: "spans", Measures: []string{"p95(duration_ms)"}, By: []string{"service"}, Limit: 3}, Options: &Options{Columns: []ColumnFormat{{Field: "p95", Format: "sparkline"}}}}, + } + for _, panel := range panels { + panel.Title = panel.ID + spec := Dashboard{Name: "Histogram compression experiment", Time: Time{From: &start, To: &end}, Panels: []Panel{panel}} + b.Run(panel.ID, func(b *testing.B) { + b.Logf("diversity=%s scale=%d spans=%d logs=%d batches=%d engine=%s memory=%s threads=%d connections=8 result_cache_ttl=0 analytical=warm background=off", diversity, scale, 96*scale*2048, 96*scale*2048, len(repo.Parquet.BatchMetadata()), version, memory, threads) + compareWeightedHistograms(b, d, spec, end) + }) + } + }) + } +} + +func compareWeightedHistograms(b *testing.B, d *engine.Duck, spec Dashboard, end time.Time) { + b.StopTimer() + names := []string{"lateral", "projection", "weighted"} + engines, executors := make([]*weightedHistogramEngine, len(names)), make([]*Executor, len(names)) + for i, name := range names { + engines[i] = &weightedHistogramEngine{Duck: d, variant: name} + executors[i] = NewExecutor(engines[i], 30) + executors[i].SetCacheTTL(0) + executors[i].now = func() time.Time { return end } + b.Cleanup(executors[i].Close) + } + run := func(i int) []Result { + results, err := executors[i].Run(WithoutCachedResults(b.Context()), RunRequest{Dashboard: spec}) + if err != nil || len(results) != len(spec.Panels) { + b.Fatalf("variant=%s results=%d error=%v", names[i], len(results), err) + } + for j := range results { + result := &results[j] + if result.Status != StatusOK || result.Frame == nil || result.Frame.Rows == 0 || result.partial || result.Cached || result.AnnotationError != "" { + b.Fatalf("variant=%s panel=%s status=%s error=%s partial=%t cached=%t annotation=%s", names[i], result.ID, result.Status, result.Error, result.partial, result.Cached, result.AnnotationError) + } + result.ElapsedMS, result.SQL = 0, "" + } + return results + } + want := run(0) + for range 3 { + for i := range names { + if got := run(i); !reflect.DeepEqual(want, got) { + a, _ := json.Marshal(want) + z, _ := json.Marshal(got) + b.Fatalf("variant=%s changed complete result: want=%s got=%s", names[i], a, z) + } + } + } + for i := range names { + if engines[i].query == "" || i > 0 && engines[i].query == engines[i-1].query { + b.Fatal("histogram experiment did not capture distinct native queries") + } + } + timings := make([][]float64, len(names)) + b.ResetTimer() + b.StartTimer() + for sample := range b.N { + for offset := range names { + i := (sample + offset) % len(names) + started := time.Now() + run(i) + timings[i] = append(timings[i], float64(time.Since(started))/float64(time.Millisecond)) + } + } + b.StopTimer() + for i, values := range timings { + sort.Float64s(values) + p95 := values[(95*len(values)+99)/100-1] + b.ReportMetric(p95, names[i]+"_p95_ms") + b.Logf("variant=%s samples=%d p50_ms=%.3f p95_ms=%.3f full_executor=true", names[i], len(values), values[(50*len(values)+99)/100-1], p95) + } + if b.N >= 100 && spec.Panels[0].ID == "service_p95" { + for i, e := range engines { + var key, plan string + if err := d.DB.QueryRowContext(b.Context(), "EXPLAIN (ANALYZE, FORMAT JSON) "+e.query, e.args...).Scan(&key, &plan); err != nil { + b.Fatal(err) + } + var compact bytes.Buffer + if err := json.Compact(&compact, []byte(plan)); err != nil { + b.Fatal(err) + } + b.Logf("native_profile_variant=%s %s=%s", names[i], key, compact.String()) + } + } +} diff --git a/internal/query/mixed_read_bench_test.go b/internal/query/mixed_read_bench_test.go new file mode 100644 index 000000000..3066dfaff --- /dev/null +++ b/internal/query/mixed_read_bench_test.go @@ -0,0 +1,1224 @@ +//go:build readbench + +package query + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "log" + "log/slog" + "math" + "os" + "os/exec" + "path/filepath" + "runtime" + "sort" + "strconv" + "strings" + "sync" + "sync/atomic" + "syscall" + "testing" + "time" + + duckdb "github.com/duckdb/duckdb-go/v2" + "github.com/labstack/pitot/internal/config" + "github.com/labstack/pitot/internal/engine" + "github.com/labstack/pitot/internal/metrics" + "github.com/labstack/pitot/internal/telemetry" + telemetrystore "github.com/labstack/pitot/internal/telemetry/store" + dto "github.com/prometheus/client_model/go" +) + +// This deliberately small synthetic workload is an experiment, not production +// capacity evidence. One arrival is one executor request, not one panel or SQL. + +const mixedReadClientWorkers = 32 +const mixedReadClientQueue = 64 + +var mixedReadKinds = [...]string{"single_p95", "batch_four", "batch_twelve"} + +type mixedReadJob struct { + slot int + scheduled time.Time +} +type mixedReadKind struct { + Offered, Missed, ClientQueueFull, Dispatched, Success, BackendBusy, Errors, DuringLoad, DuringDrain, CachedPanels int + E2EMS []float64 `json:"-"` + QueueMS []float64 `json:"-"` + P50MS, P95MS, P99MS, QueueP95MS float64 + // P99Qualified reports sample sufficiency only, not a passed latency or loss gate. + P99Qualified bool +} +type mixedReadReport struct { + RSSMethod string + ParquetMetadataCache bool + ObjectCacheBytes int64 + ExpectedShutdownInterrupts, ExpectedShutdownCancellations int + ResultCacheOutcomes map[string]float64 + NativeQueryStatementCountDelta uint64 + DiagnosticCountersReset bool + + SeedState, WindowStart, WindowEnd string + InitialGenerationCounts map[uint32]int + + IngestRowsPerSignalPerBatch, IngestCommitTargetPerSecond, IngestTargetItemsPerSecond int + PeakActiveWorkers int64 + GeneratorLimited bool + BackgroundErrors []string + MaintenanceInterval, MaintenanceMode, CachePreparation string + + OperationUnit string + ArrivalIntervalNS float64 + EngineVersion, Platform, EffectiveMemory string + InitialActiveBatches, InitialCompactionDebt int + InitialReadCacheMarkers int64 + InitialRollupLagMS float64 + SingleP95BudgetMS, BatchP95BudgetMS float64 + + Methodology string + Rate int + DurationSeconds float64 + Scale, SeedSpans, SeedLogs, Threads, Connections, Workers, QueueCapacity int + Memory, CacheTTL string + CacheBypass bool + Kinds map[string]*mixedReadKind + IngestAcceptedBatches, IngestAcceptedRows, IngestErrors int64 + IngestRowsPerSecond, PeakRSSMiB float64 + RSSError string + ActiveBatches, CompactionDebt int + ReadCacheMarkers int64 + RollupLagMS float64 + Errors []string +} + +type mixedReadLogState struct { + sync.Mutex + failures []string + background context.Context + stopping atomic.Bool + shutdownInterrupts, shutdownCancellations int +} +type mixedReadLogHandler struct { + slog.Handler + state *mixedReadLogState + attrs []slog.Attr +} + +// Classify the entire error tree so an interrupt joined with an unrelated +// failure is never excused by shutdown. Ordinary wrappers retain their cause. +func mixedReadShutdownError(err error) (expected, interrupt bool) { + if native, ok := err.(*duckdb.Error); ok { + return native.Type == duckdb.ErrorTypeInterrupt, native.Type == duckdb.ErrorTypeInterrupt + } + if err == context.Canceled { + return true, false + } + if joined, ok := err.(interface{ Unwrap() []error }); ok { + causes := joined.Unwrap() + if len(causes) == 0 { + return false, false + } + for _, cause := range causes { + allowed, native := mixedReadShutdownError(cause) + if !allowed { + return false, false + } + interrupt = interrupt || native + } + return true, interrupt + } + if wrapped, ok := err.(interface{ Unwrap() error }); ok { + return mixedReadShutdownError(wrapped.Unwrap()) + } + return false, false +} + +func (h mixedReadLogHandler) Enabled(ctx context.Context, level slog.Level) bool { + return level >= slog.LevelWarn || h.Handler.Enabled(ctx, level) +} +func (h mixedReadLogHandler) Handle(ctx context.Context, r slog.Record) error { + if r.Level >= slog.LevelWarn { + stopping := h.state.stopping.Load() && h.state.background != nil && h.state.background.Err() == context.Canceled + seenError, unexpected, interrupted := false, false, false + details := "" + inspect := func(a slog.Attr) bool { + if e, ok := a.Value.Any().(error); ok { + seenError = true + expected, native := mixedReadShutdownError(e) + if !stopping || !expected { + unexpected = true + } + interrupted = interrupted || native + } + details += " " + a.Key + "=" + a.Value.String() + return true + } + for _, a := range h.attrs { + inspect(a) + } + r.Attrs(inspect) + h.state.Lock() + if seenError && !unexpected { + if interrupted { + h.state.shutdownInterrupts++ + } else { + h.state.shutdownCancellations++ + } + } else if len(h.state.failures) < 10 { + h.state.failures = append(h.state.failures, r.Message+details) + } + h.state.Unlock() + } + if h.Handler.Enabled(ctx, r.Level) { + return h.Handler.Handle(ctx, r) + } + return nil +} +func (h mixedReadLogHandler) WithAttrs(attrs []slog.Attr) slog.Handler { + return mixedReadLogHandler{Handler: h.Handler.WithAttrs(attrs), state: h.state, attrs: append(append([]slog.Attr(nil), h.attrs...), attrs...)} +} +func (h mixedReadLogHandler) WithGroup(name string) slog.Handler { + return mixedReadLogHandler{Handler: h.Handler.WithGroup(name), state: h.state, attrs: h.attrs} +} + +// Never wrap slog.Default().Handler(): its built-in bridge calls log.Output, +// and SetDefault redirects that standard logger back into the new handler. +func mixedReadCaptureLogs(background context.Context) (*mixedReadLogState, func()) { + oldLogger := slog.Default() + oldWriter, oldFlags := log.Writer(), log.Flags() + state := &mixedReadLogState{background: background} + slog.SetDefault(slog.New(mixedReadLogHandler{Handler: slog.NewTextHandler(os.Stderr, nil), state: state})) + return state, func() { slog.SetDefault(oldLogger); log.SetOutput(oldWriter); log.SetFlags(oldFlags) } +} + +func mixedReadWarmCache(t *testing.T, d *engine.Duck, repo *telemetrystore.Repository) { + t.Helper() + ctx, cancel := context.WithTimeout(t.Context(), 5*time.Minute) + defer cancel() + target := int64(len(repo.Parquet.BatchMetadata())) + var before int64 + for { + if _, err := d.RefreshReadCaches(ctx); err != nil { + t.Fatal("completed read cache preparation:", err) + } + var marked int64 + if err := d.DB.QueryRowContext(ctx, "SELECT count(*) FROM read_batches").Scan(&marked); err != nil { + t.Fatal(err) + } + if marked == target { + return + } + if marked <= before || marked > target { + t.Fatalf("completed read cache preparation made no progress: marked=%d target=%d", marked, target) + } + before = marked + } +} + +type mixedReadWorkerActivity struct{ active, peak atomic.Int64 } + +func (a *mixedReadWorkerActivity) start() { + current := a.active.Add(1) + for previous := a.peak.Load(); current > previous; previous = a.peak.Load() { + if a.peak.CompareAndSwap(previous, current) { + break + } + } +} +func (a *mixedReadWorkerActivity) stop() { a.active.Add(-1) } + +type mixedReadDiagnosticSnapshot struct { + cache map[string]float64 + statements uint64 +} + +func mixedReadDiagnostics() (mixedReadDiagnosticSnapshot, error) { + snapshot := mixedReadDiagnosticSnapshot{cache: map[string]float64{}} + for _, outcome := range []string{"hit", "stale", "miss", "coalesced"} { + metric := &dto.Metric{} + if err := metrics.QueryCacheRequests.WithLabelValues(outcome).Write(metric); err != nil { + return snapshot, err + } + snapshot.cache[outcome] = metric.GetCounter().GetValue() + } + metric := &dto.Metric{} + if err := metrics.DuckDBStatement.Write(metric); err != nil { + return snapshot, err + } + snapshot.statements = metric.GetHistogram().GetSampleCount() + return snapshot, nil +} +func mixedReadDiagnosticDelta(before, after mixedReadDiagnosticSnapshot) (map[string]float64, uint64, bool) { + outcomes := map[string]float64{} + reset := false + for _, outcome := range []string{"hit", "stale", "miss", "coalesced"} { + delta := after.cache[outcome] - before.cache[outcome] + if delta < 0 { + reset = true + delta = 0 + } + outcomes[outcome] = delta + } + var statements uint64 + if after.statements >= before.statements { + statements = after.statements - before.statements + } else { + reset = true + } + return outcomes, statements, reset +} + +func mixedReadRate(raw string) (int, error) { + if raw == "" { + return 20, nil + } + rate, err := strconv.Atoi(raw) + if err != nil || rate < 1 || rate > 320 { + return 0, errors.New("PITOT_MIXED_RATE must be an integer from 1 to 320 read operations per second") + } + return rate, nil +} + +func mixedReadIngestRows(raw string) (int, error) { + if raw == "" { + return 2048, nil + } + rows, err := strconv.Atoi(raw) + if err != nil || rows < 128 || rows > 8192 { + return 0, errors.New("PITOT_MIXED_INGEST_ROWS must be an integer from 128 to 8192 per signal per batch") + } + return rows, nil +} + +func mixedReadSeedConfig(rawState, rawEnd string, now time.Time) (string, time.Time, error) { + state := rawState + if state == "" { + state = "backlog" + } + if state != "backlog" && state != "settled" { + return "", time.Time{}, errors.New("PITOT_MIXED_SEED_STATE must be backlog or settled") + } + end := now.UTC().Truncate(time.Minute).Add(-time.Minute) + if rawEnd != "" { + var err error + end, err = time.Parse(time.RFC3339Nano, rawEnd) + if err != nil { + return "", time.Time{}, errors.New("PITOT_MIXED_END must be RFC3339Nano") + } + end = end.UTC() + } + if end.After(now) || end.Add(-24*time.Hour).Before(now.Add(-30*24*time.Hour)) { + return "", time.Time{}, errors.New("PITOT_MIXED_END must place the complete 24h window inside retention and not in the future") + } + return state, end, nil +} +func mixedReadCompactionProgress(before, after, merged, debt int) (bool, error) { + if merged == 0 { + if debt != 0 { + return false, fmt.Errorf("seed compaction stopped with debt=%d", debt) + } + return true, nil + } + if merged < 0 || after >= before { + return false, fmt.Errorf("seed compaction made no progress: before=%d after=%d merged=%d", before, after, merged) + } + return false, nil +} +func mixedReadSettleSeed(t *testing.T, repo *telemetrystore.Repository, d *engine.Duck) { + t.Helper() + ctx, cancel := context.WithTimeout(t.Context(), 5*time.Minute) + defer cancel() + passes := len(repo.Parquet.BatchMetadata()) + 1 + for range passes { + before := len(repo.Parquet.BatchMetadata()) + n, err := repo.CompactParquet(ctx, d, 128) + if err != nil { + t.Fatal("settle seed compaction:", err) + } + done, err := mixedReadCompactionProgress(before, len(repo.Parquet.BatchMetadata()), n, repo.CompactionDebt()) + if err != nil { + t.Fatal(err) + } + if done { + return + } + } + t.Fatal("seed compaction exceeded bounded pass count") +} + +func mixedReadOffset(slot, rate int) time.Duration { + return time.Duration(int64(slot) * int64(time.Second) / int64(rate)) +} +func mixedReadTotal(duration time.Duration, rate int) int { + return int((int64(duration)*int64(rate) + int64(time.Second) - 1) / int64(time.Second)) +} +func mixedReadDue(elapsed time.Duration, rate, total int) int { + if elapsed < 0 { + return 0 + } + // Offsets are floor(slot*1s/rate). Use the same rational schedule here; + // rounding an interval first would offer an extra slot for rates such as 3. + return min(total, int(((int64(elapsed)+1)*int64(rate)-1)/int64(time.Second))+1) +} + +func mixedReadPercentile(values []float64, percentile int) float64 { + if len(values) == 0 { + return 0 + } + copyOf := append([]float64(nil), values...) + sort.Float64s(copyOf) + return copyOf[(percentile*len(copyOf)+99)/100-1] +} + +// Getrusage measures this process without starting ps or inspecting other +// processes. Its high-water mark includes fixture preparation, not just load. +func mixedReadPeakRSS() (float64, error) { + var usage syscall.Rusage + if err := syscall.Getrusage(syscall.RUSAGE_SELF, &usage); err != nil { + return 0, err + } + divisor := float64(1024) + if runtime.GOOS == "darwin" { + divisor = 1 << 20 // Darwin reports bytes; Linux reports KiB. + } + return float64(usage.Maxrss) / divisor, nil +} +func mixedReadBusy(err error, results []Result) bool { + if errors.Is(err, ErrBusy) { + return true + } + for _, r := range results { + if r.Status == StatusError && r.Error == BusyMessage { + return true + } + } + return false +} + +func mixedReadKindGate(k *mixedReadKind, budget float64) error { + if k.Offered != k.Missed+k.ClientQueueFull+k.Dispatched || k.Dispatched != k.Success+k.BackendBusy+k.Errors || k.Success != k.DuringLoad+k.DuringDrain { + return errors.New("operation accounting does not reconcile") + } + if k.Success == 0 || k.Missed+k.ClientQueueFull+k.BackendBusy+k.Errors > 0 { + return errors.New("offered work did not all complete successfully") + } + if k.P95MS > budget { + return fmt.Errorf("E2E p95 %.3f ms exceeds %.3f ms", k.P95MS, budget) + } + return nil +} + +// Every successful operation checks exact counts and rank, including during +// ingestion/compaction. Live writes fall outside the immutable oracle window. +func mixedReadOracle(results []Result, panels int, expected int) error { + if len(results) != panels { + return fmt.Errorf("results=%d want=%d", len(results), panels) + } + for _, r := range results { + if r.Cached { + return fmt.Errorf("panel=%s served cached result under explicit bypass", r.ID) + } + if r.Status != StatusOK || r.Frame == nil || r.Frame.Rows == 0 || r.AnnotationError != "" { + return fmt.Errorf("panel=%s status=%s error=%s annotation=%s", r.ID, r.Status, r.Error, r.AnnotationError) + } + switch { + case r.ID == "traces_all" || r.ID == "traces_selected" || r.ID == "traces_checkout": + want := expected / 4 + if r.ID == "traces_checkout" { + want = expected / 2048 * 342 + } + found := false + for i, c := range r.Frame.Columns { + if c.Role == "measure" { + if i >= len(r.Frame.Values) || len(r.Frame.Values[i]) != 1 { + return fmt.Errorf("panel=%s missing scalar distinct trace count", r.ID) + } + value, err := strconv.ParseFloat(fmt.Sprint(r.Frame.Values[i][0]), 64) + if err != nil || value != float64(want) { + return fmt.Errorf("panel=%s traces=%v want=%d", r.ID, r.Frame.Values[i][0], want) + } + found = true + } + } + if !found { + return fmt.Errorf("panel=%s missing distinct trace measure", r.ID) + } + case strings.HasPrefix(r.ID, "count"), strings.HasPrefix(r.ID, "logs"): + found := false + for i, c := range r.Frame.Columns { + if c.Role == "measure" && c.Name == "count" && i < len(r.Frame.Totals) { + v, err := strconv.ParseFloat(fmt.Sprint(r.Frame.Totals[i]), 64) + if err != nil || v != float64(expected) { + return fmt.Errorf("panel=%s total=%v want=%d", r.ID, r.Frame.Totals[i], expected) + } + found = true + } + } + if !found { + return fmt.Errorf("panel=%s missing count total", r.ID) + } + case strings.HasPrefix(r.ID, "p95"): + found := false + for i, c := range r.Frame.Columns { + if c.Name == "service" && len(r.Frame.Values[i]) > 0 { + if r.Frame.Values[i][0] != "svc5" { + return fmt.Errorf("panel=%s first=%v want svc5", r.ID, r.Frame.Values[i][0]) + } + found = true + } + } + if !found { + return fmt.Errorf("panel=%s missing service ranking", r.ID) + } + } + } + return nil +} + +func mixedReadSpecs(start, end time.Time) []Dashboard { + four := pressureSpec(start, end) + twelve := panelBenchmarkDashboard(panelBenchData{Start: start, End: end, Services: []string{"svc0", "svc1", "svc2", "svc3", "svc4", "svc5"}, Namespaces: []string{"mixed_synthetic"}}) + twelve.Name = "Synthetic varied twelve panel mixed load" + single := four + single.Name = "Synthetic single p95 mixed load" + single.Panels = []Panel{four.Panels[2]} + return []Dashboard{single, four, twelve} +} + +func mixedReadBatch(id string, start time.Time, rows int) telemetrystore.Batch { + batch := telemetrystore.Batch{ID: id} + for i := range rows { + service := i % 6 + n := start.Add(time.Duration(i) * time.Nanosecond).UnixNano() + duration := int64(service+1) * 10_000_000 + batch.Spans = append(batch.Spans, telemetry.Span{Namespace: "mixed_synthetic", ServiceName: fmt.Sprintf("svc%d", service), TraceID: fmt.Sprintf("%s-trace-%d", id, i/4), SpanID: fmt.Sprint(i), StartUnixNanos: n, EndUnixNanos: n + duration, DurationMS: float64(service+1) * 10, Kind: "SPAN_KIND_SERVER", StatusCode: "STATUS_CODE_OK", Resource: map[string]any{"service.version": "synthetic"}}) + batch.Logs = append(batch.Logs, telemetry.Log{Namespace: "mixed_synthetic", ServiceName: fmt.Sprintf("svc%d", service), TraceID: fmt.Sprintf("%s-trace-%d", id, i/4), EventUnixNanos: n, TimeUnixNanos: n, Body: "synthetic mixed-load event", Severity: "INFO"}) + } + return batch +} + +func TestMixedReadLoad(t *testing.T) { + if os.Getenv("PITOT_MIXED_READ") != "1" { + t.Skip("opt-in synthetic mixed read/load experiment") + } + rate, err := mixedReadRate(os.Getenv("PITOT_MIXED_RATE")) + if err != nil { + t.Fatal(err) + } + ingestRows, err := mixedReadIngestRows(os.Getenv("PITOT_MIXED_INGEST_ROWS")) + if err != nil { + t.Fatal(err) + } + seedState, windowEnd, err := mixedReadSeedConfig(os.Getenv("PITOT_MIXED_SEED_STATE"), os.Getenv("PITOT_MIXED_END"), time.Now().UTC()) + if err != nil { + t.Fatal(err) + } + duration := 15 * time.Second + if raw := os.Getenv("PITOT_MIXED_DURATION"); raw != "" { + var err error + duration, err = time.ParseDuration(raw) + if err != nil || duration < time.Second || duration > 5*time.Minute { + t.Fatal("PITOT_MIXED_DURATION must be between 1s and 5m") + } + } + scale := 1 + if raw := os.Getenv("PITOT_MIXED_SCALE"); raw != "" { + var err error + scale, err = strconv.Atoi(raw) + if err != nil || scale < 1 || scale > 16 { + t.Fatal("PITOT_MIXED_SCALE must be 1..16") + } + } + singleBudget, batchBudget := 500.0, 1500.0 + for name, target := range map[string]*float64{"PITOT_MIXED_SINGLE_P95_MS": &singleBudget, "PITOT_MIXED_BATCH_P95_MS": &batchBudget} { + if raw := os.Getenv(name); raw != "" { + v, err := strconv.ParseFloat(raw, 64) + if err != nil || math.IsNaN(v) || math.IsInf(v, 0) || v <= 0 { + t.Fatalf("invalid %s", name) + } + *target = v + } + } + root := t.TempDir() + cfg := config.Config{DataDir: filepath.Join(root, "state"), RetentionDays: 30, DuckDBMemory: "4GB", DuckDBThreads: 4, DuckDBMaxConns: 8, RollupInterval: time.Second, MaintenanceInterval: 5 * time.Second} + repo, err := telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + t.Fatal(err) + } + defer repo.Close() + end := windowEnd + start := end.Add(-24 * time.Hour) + // 96 evenly spaced batches populate the complete day. Keep a minute margin + // on each edge; per-service duration makes svc5 the unambiguous p95 winner. + for batchIndex := range 96 * scale { + at := start.Add(time.Minute + time.Duration(batchIndex)*(24*time.Hour-4*time.Minute)/time.Duration(96*scale)) + batch := mixedReadBatch(fmt.Sprintf("mixed-seed-%06d", batchIndex), at, 2048) + // Spread the 2048 rows over this batch's portion of the day. + for i := range batch.Spans { + n := at.Add(time.Duration(i) * (24*time.Hour - 4*time.Minute) / time.Duration(96*scale*2048)).UnixNano() + batch.Spans[i].StartUnixNanos = n + batch.Spans[i].EndUnixNanos = n + int64(batch.Spans[i].DurationMS*1e6) + batch.Logs[i].EventUnixNanos = n + batch.Logs[i].TimeUnixNanos = n + } + if err := repo.Commit(t.Context(), batch); err != nil { + t.Fatal(err) + } + } + d, err := engine.NewDuck(t.Context(), cfg, repo) + if err != nil { + t.Fatal(err) + } + defer d.Close() + if seedState == "settled" { + mixedReadSettleSeed(t, repo, d) + } + horizon, _, _ := repo.PublicationBounds() + panelBenchWarmDashboard(t, d, horizon) + mixedReadWarmCache(t, d, repo) + var probe *mixedReadProbe + var queryEngine Engine = d + if os.Getenv("PITOT_MIXED_DIAGNOSTICS") == "1" { + probe = &mixedReadProbe{duck: d, walPath: cfg.QueryDuckDBPath() + ".wal", stackPath: os.Getenv("PITOT_MIXED_STACK_FILE")} + queryEngine = &mixedReadObservedEngine{Duck: d, probe: probe} + } + executor := NewExecutor(queryEngine, 30) + if probe != nil { + probe.executor = executor + } + d.SetReadPressure(executor.Pressure) + executor.SetCacheTTL(10 * time.Second) + defer executor.Close() + specs := mixedReadSpecs(start, end) + expected := 96 * scale * 2048 + for _, spec := range specs { + results, err := executor.Run(WithoutCachedResults(t.Context()), RunRequest{Dashboard: spec}) + if err != nil { + t.Fatal(err) + } + if err := mixedReadOracle(results, len(spec.Panels), expected); err != nil { + t.Fatal("preflight oracle:", err) + } + } + report := mixedReadReport{Methodology: "synthetic_fixed_rate_fresh_executor_frozen_exact_shapes_seed_control_v8_native_rss_tempdir_not_production_capacity", MaintenanceInterval: "5s", MaintenanceMode: "accelerated_probe_product_default_1h", CachePreparation: "analytical_and_completed_read_caches_warm", Rate: rate, OperationUnit: "executor_request_not_individual_panels", ArrivalIntervalNS: float64(time.Second) / float64(rate), DurationSeconds: duration.Seconds(), Scale: scale, SeedSpans: expected, SeedLogs: expected, Threads: 4, Connections: 8, Workers: mixedReadClientWorkers, QueueCapacity: mixedReadClientQueue, Memory: "4GB", CacheTTL: "10s", CacheBypass: true, Kinds: map[string]*mixedReadKind{}} + for _, kind := range mixedReadKinds { + report.Kinds[kind] = &mixedReadKind{} + } + report.SeedState = seedState + report.WindowStart = start.Format(time.RFC3339Nano) + report.WindowEnd = end.Format(time.RFC3339Nano) + report.InitialGenerationCounts = map[uint32]int{} + for _, m := range repo.Parquet.BatchMetadata() { + report.InitialGenerationCounts[m.Generation]++ + } + report.IngestRowsPerSignalPerBatch = ingestRows + report.IngestCommitTargetPerSecond = 5 + report.IngestTargetItemsPerSecond = ingestRows * 2 * 5 + report.SingleP95BudgetMS = singleBudget + report.BatchP95BudgetMS = batchBudget + report.Platform = runtime.GOOS + "/" + runtime.GOARCH + report.RSSMethod = "process_lifetime_peak_getrusage_including_setup" + if err := d.DB.QueryRowContext(t.Context(), "SELECT version(),current_setting('memory_limit')").Scan(&report.EngineVersion, &report.EffectiveMemory); err != nil { + t.Fatal(err) + } + report.InitialActiveBatches = len(repo.Parquet.BatchMetadata()) + if err := d.DB.QueryRowContext(t.Context(), "SELECT current_setting('parquet_metadata_cache')::BOOLEAN").Scan(&report.ParquetMetadataCache); err != nil { + t.Fatal(err) + } + report.InitialCompactionDebt = repo.CompactionDebt() + if err := d.DB.QueryRowContext(t.Context(), "SELECT count(*) FROM read_batches").Scan(&report.InitialReadCacheMarkers); err != nil { + t.Fatal(err) + } + initialStamp, err := d.RollupWatermark(t.Context()) + if err != nil { + t.Fatal(err) + } + if !initialStamp.IsZero() { + report.InitialRollupLagMS = max(0, float64(horizon-initialStamp.UnixNano())/1e6) + } + + phaseDiagnostics, err := mixedReadDiagnostics() + if err != nil { + t.Fatal(err) + } + phaseCtx, stopPhase := context.WithTimeout(t.Context(), duration) + defer stopPhase() + background, stopBackground := context.WithCancel(t.Context()) + defer stopBackground() + logState, restoreLogs := mixedReadCaptureLogs(background) + defer restoreLogs() + backgroundDone := make(chan struct{}) + go func() { defer close(backgroundDone); d.RunRollups(background) }() + var accepted, acceptedRows, ingestErrors atomic.Int64 + ingestDone := make(chan struct{}) + go func() { + defer close(ingestDone) + ticker := time.NewTicker(time.Second / 5) + defer ticker.Stop() + for index := 0; ; index++ { + select { + case <-phaseCtx.Done(): + return + case <-ticker.C: + } + batch := mixedReadBatch(fmt.Sprintf("mixed-live-%06d", index), time.Now().UTC(), ingestRows) + if err := repo.Commit(phaseCtx, batch); err != nil { + if phaseCtx.Err() == nil { + ingestErrors.Add(1) + } + return + } + accepted.Add(1) + acceptedRows.Add(int64(len(batch.Spans) + len(batch.Logs))) + } + }() + rssDone := make(chan struct{}) + go func() { + defer close(rssDone) + ticker := time.NewTicker(time.Second) + defer ticker.Stop() + for { + select { + case <-background.Done(): + return + case <-ticker.C: + probe.record("sample", time.Now(), false) + rss, err := mixedReadPeakRSS() + if err != nil { + report.RSSError = err.Error() + } else { + report.PeakRSSMiB = max(report.PeakRSSMiB, rss) + } + } + } + }() + jobs := make(chan mixedReadJob, mixedReadClientQueue) + var workers sync.WaitGroup + var activity mixedReadWorkerActivity + var lock sync.Mutex + arrivalStart := time.Now() + if probe != nil { + probe.setStart(arrivalStart) + } + cutoff := arrivalStart.Add(duration) + readCtx, stopReads := context.WithDeadline(t.Context(), cutoff.Add(15*time.Second)) + defer stopReads() + for range mixedReadClientWorkers { + workers.Add(1) + go func() { + defer workers.Done() + for job := range jobs { + activity.start() + kind := mixedReadKinds[job.slot%len(mixedReadKinds)] + spec := specs[job.slot%len(specs)] + began := time.Now() + request, cancel := context.WithTimeout(WithoutCachedResults(readCtx), 15*time.Second) + results, err := executor.Run(request, RunRequest{Dashboard: spec}) + cancel() + finished := time.Now() + probe.record(kind, began, mixedReadBusy(err, results)) + semanticErr := error(nil) + if err == nil { + semanticErr = mixedReadOracle(results, len(spec.Panels), expected) + } + lock.Lock() + k := report.Kinds[kind] + switch { + case mixedReadBusy(err, results): + k.BackendBusy++ + case err != nil || semanticErr != nil: + k.Errors++ + if len(report.Errors) < 5 { + report.Errors = append(report.Errors, fmt.Sprintf("kind=%s error=%v oracle=%v", kind, err, semanticErr)) + } + default: + k.Success++ + k.E2EMS = append(k.E2EMS, float64(finished.Sub(job.scheduled))/float64(time.Millisecond)) + k.QueueMS = append(k.QueueMS, float64(began.Sub(job.scheduled))/float64(time.Millisecond)) + if finished.Before(cutoff) { + k.DuringLoad++ + } else { + k.DuringDrain++ + } + for _, r := range results { + if r.Cached { + k.CachedPanels++ + } + } + } + lock.Unlock() + activity.stop() + } + }() + } + total := mixedReadTotal(duration, rate) + for next := 0; next < total; { + now := time.Now() + if !now.Before(cutoff) { + for ; next < total; next++ { + k := report.Kinds[mixedReadKinds[next%3]] + lock.Lock() + k.Offered++ + k.Missed++ + lock.Unlock() + } + break + } + due := mixedReadDue(now.Sub(arrivalStart), rate, total) + if due <= next { + timer := time.NewTimer(time.Until(arrivalStart.Add(mixedReadOffset(next, rate)))) + select { + case <-phaseCtx.Done(): + case <-timer.C: + } + timer.Stop() + continue + } + for ; next < due-1; next++ { + lock.Lock() + k := report.Kinds[mixedReadKinds[next%3]] + k.Offered++ + k.Missed++ + lock.Unlock() + } + job := mixedReadJob{slot: next, scheduled: arrivalStart.Add(mixedReadOffset(next, rate))} + lock.Lock() + k := report.Kinds[mixedReadKinds[next%3]] + k.Offered++ + select { + case jobs <- job: + k.Dispatched++ + default: + k.ClientQueueFull++ + } + lock.Unlock() + next++ + } + close(jobs) + <-phaseCtx.Done() // Keep background work running for the full offered interval. + workers.Wait() + stopPhase() + <-ingestDone + logState.stopping.Store(true) + stopBackground() + <-backgroundDone + <-rssDone + probe.report(t) + afterDiagnostics, err := mixedReadDiagnostics() + if err != nil { + t.Fatal(err) + } + report.ResultCacheOutcomes, report.NativeQueryStatementCountDelta, report.DiagnosticCountersReset = mixedReadDiagnosticDelta(phaseDiagnostics, afterDiagnostics) + logState.Lock() + report.BackgroundErrors = append([]string(nil), logState.failures...) + report.ExpectedShutdownInterrupts = logState.shutdownInterrupts + report.ExpectedShutdownCancellations = logState.shutdownCancellations + logState.Unlock() + report.PeakActiveWorkers = activity.peak.Load() + report.IngestAcceptedBatches = accepted.Load() + report.IngestAcceptedRows = acceptedRows.Load() + report.IngestErrors = ingestErrors.Load() + report.IngestRowsPerSecond = float64(acceptedRows.Load()) / duration.Seconds() + report.ActiveBatches = len(repo.Parquet.BatchMetadata()) + report.CompactionDebt = repo.CompactionDebt() + if err := d.DB.QueryRowContext(t.Context(), "SELECT coalesce(sum(memory_usage_bytes),0)::BIGINT FROM duckdb_memory() WHERE tag='OBJECT_CACHE'").Scan(&report.ObjectCacheBytes); err != nil { + t.Fatal(err) + } + stamp, err := d.RollupWatermark(t.Context()) + if err != nil { + t.Fatal(err) + } + horizon, _, _ = repo.PublicationBounds() + if !stamp.IsZero() { + report.RollupLagMS = max(0, float64(horizon-stamp.UnixNano())/1e6) + } + if err := d.DB.QueryRowContext(t.Context(), "SELECT count(*) FROM read_batches").Scan(&report.ReadCacheMarkers); err != nil { + t.Fatal(err) + } + for _, kind := range mixedReadKinds { + k := report.Kinds[kind] + k.P50MS = mixedReadPercentile(k.E2EMS, 50) + k.P95MS = mixedReadPercentile(k.E2EMS, 95) + k.P99MS = mixedReadPercentile(k.E2EMS, 99) + k.QueueP95MS = mixedReadPercentile(k.QueueMS, 95) + k.P99Qualified = len(k.E2EMS) >= 1000 + } + for _, k := range report.Kinds { + report.GeneratorLimited = report.GeneratorLimited || k.ClientQueueFull > 0 || k.Missed > 0 + } + encoded, err := json.Marshal(report) + if err != nil { + t.Fatal(err) + } + t.Logf("MIXED_READ_REPORT %s", encoded) + offered := 0 + for i, kind := range mixedReadKinds { + k := report.Kinds[kind] + offered += k.Offered + report.GeneratorLimited = report.GeneratorLimited || k.ClientQueueFull > 0 || k.Missed > 0 + budget := batchBudget + if i == 0 { + budget = singleBudget + } + if err := mixedReadKindGate(k, budget); err != nil { + t.Errorf("%s: %v", kind, err) + } + } + if len(report.BackgroundErrors) > 0 { + t.Errorf("background operational errors invalidate run: %v", report.BackgroundErrors) + } + if offered != total { + t.Errorf("offered=%d want=%d", offered, total) + } + if report.IngestAcceptedBatches == 0 || report.IngestErrors != 0 { + t.Errorf("background ingest accepted=%d errors=%d", report.IngestAcceptedBatches, report.IngestErrors) + } +} + +func TestMixedReadScheduleAndGates(t *testing.T) { + if !mixedReadBusy(nil, []Result{{Status: StatusError, Error: BusyMessage}}) { + t.Fatal("panel busy result misclassified") + } + if got := mixedReadDue(175*time.Millisecond, 20, 20); got != 4 { + t.Fatalf("delayed wake due=%d", got) + } + if got := mixedReadDue(time.Second, 20, 20); got != 20 { + t.Fatalf("cutoff due=%d", got) + } + k := &mixedReadKind{Offered: 3, Missed: 1, Dispatched: 2, Success: 2, DuringLoad: 1, DuringDrain: 1, P95MS: 1} + if err := mixedReadKindGate(k, 500); err == nil { + t.Fatal("scheduler misses passed") + } + k.Offered = 2 + k.Missed = 0 + k.P95MS = 501 + if err := mixedReadKindGate(k, 500); err == nil { + t.Fatal("slow operation kind passed") + } + k.P95MS = 1 + k.Dispatched = 3 + if err := mixedReadKindGate(k, 500); err == nil { + t.Fatal("broken accounting passed") + } + k.Dispatched = 2 + if err := mixedReadKindGate(k, 500); err != nil { + t.Fatal(err) + } + if p := mixedReadPercentile([]float64{1, 2, 3, 100}, 99); p != 100 { + t.Fatalf("tail=%v", p) + } + frame := &Frame{Rows: 1, Columns: []Column{{Name: "service", Role: "dimension"}}, Values: [][]any{{"svc0"}}} + if err := mixedReadOracle([]Result{{ID: "p95", Status: StatusOK, Frame: frame}}, 1, 100); err == nil { + t.Fatal("wrong rank passed") + } + countFrame := &Frame{Rows: 1, Columns: []Column{{Name: "time", Role: "time"}, {Name: "count", Role: "measure"}}, Values: [][]any{{int64(0)}, {99.0}}, Totals: []any{nil, 99.0}} + if err := mixedReadOracle([]Result{{ID: "count", Status: StatusOK, Frame: countFrame}}, 1, 100); err == nil { + t.Fatal("wrong exact total passed") + } + countFrame.Totals[1] = 100.0 + if err := mixedReadOracle([]Result{{ID: "count", Status: StatusOK, Frame: countFrame}}, 1, 100); err != nil { + t.Fatal(err) + } + frame.Values[0][0] = "svc5" + if err := mixedReadOracle([]Result{{ID: "p95", Status: StatusOK, Frame: frame}}, 1, 100); err != nil { + t.Fatal(err) + } +} + +func TestMixedReadConfigurableRateSchedule(t *testing.T) { + for _, rate := range []int{3, 20, 40, 80, 160, 320} { + t.Run(fmt.Sprint(rate), func(t *testing.T) { + if parsed, err := mixedReadRate(fmt.Sprint(rate)); err != nil || parsed != rate { + t.Fatalf("rate=%d error=%v", parsed, err) + } + duration := 15 * time.Second + total := mixedReadTotal(duration, rate) + if total != 15*rate { + t.Fatalf("offered=%d want=%d", total, 15*rate) + } + // Every scheduled slot is due exactly at its offset, and not one ns before. + for slot := range total { + at := mixedReadOffset(slot, rate) + if got := mixedReadDue(at, rate, total); got != slot+1 { + t.Fatalf("slot=%d due=%d", slot, got) + } + if got := mixedReadDue(at-time.Nanosecond, rate, total); got != slot { + t.Fatalf("slot=%d early due=%d", slot, got) + } + } + offered := 0 + missed := 0 + dispatched := 0 + for next := 0; next < total; { + due := mixedReadDue(mixedReadOffset(min(total-1, next+3), rate), rate, total) + offered += due - next + missed += due - next - 1 + dispatched++ + next = due + } + if offered != total || offered != missed+dispatched { + t.Fatalf("offered=%d missed=%d dispatched=%d", offered, missed, dispatched) + } + }) + } + for _, raw := range []string{"0", "-1", "321", "2.5", "invalid"} { + if _, err := mixedReadRate(raw); err == nil { + t.Fatalf("invalid rate %q accepted", raw) + } + } + if rate, err := mixedReadRate(""); err != nil || rate != 20 { + t.Fatalf("default=%d error=%v", rate, err) + } +} + +func TestMixedReadCapturesOperationalBackgroundErrors(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + state := &mixedReadLogState{background: ctx} + state.stopping.Store(true) + h := mixedReadLogHandler{Handler: slog.NewTextHandler(io.Discard, nil), state: state} + canceled := slog.NewRecord(time.Now(), slog.LevelWarn, "stopping", 0) + canceled.AddAttrs(slog.Any("err", context.Canceled)) + if err := h.Handle(context.Background(), canceled); err != nil { + t.Fatal(err) + } + failed := slog.NewRecord(time.Now(), slog.LevelWarn, "telemetry maintenance failed", 0) + failed.AddAttrs(slog.Any("err", errors.New("cannot CHECKPOINT while another write is active"))) + if err := h.Handle(context.Background(), failed); err != nil { + t.Fatal(err) + } + if len(state.failures) != 1 || !strings.Contains(state.failures[0], "CHECKPOINT") { + t.Fatalf("background failures=%v", state.failures) + } +} + +func TestMixedReadArrivalWorkerActivity(t *testing.T) { + var activity mixedReadWorkerActivity + for range 3 { + activity.start() + } + activity.stop() + activity.stop() + activity.start() + if got := activity.peak.Load(); got != 3 { + t.Fatalf("peak active=%d want3", got) + } + if got := activity.active.Load(); got != 2 { + t.Fatalf("active=%d want2", got) + } + if mixedReadClientWorkers != 32 || mixedReadClientQueue != 64 { + t.Fatal("capacity sweep generator envelope changed") + } +} + +func TestMixedReadDefaultLogBridge(t *testing.T) { + if os.Getenv("PITOT_MIXED_LOG_CHILD") == "1" { + originalWriter, originalFlags := log.Writer(), log.Flags() + state, restore := mixedReadCaptureLogs(context.Background()) + slog.Info("synthetic bridge info") + log.Print("synthetic standard log bridge") + slog.Warn("synthetic maintenance failed", "err", errors.New("synthetic CHECKPOINT conflict")) + restore() + if log.Writer() != originalWriter || log.Flags() != originalFlags { + t.Fatal("standard log configuration not restored") + } + slog.Info("synthetic restored slog bridge") + log.Print("synthetic restored standard log") + if len(state.failures) != 1 { + t.Fatalf("captured=%v", state.failures) + } + return + } + ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, os.Args[0], "-test.run=^TestMixedReadDefaultLogBridge$", "-test.count=1") + cmd.Env = append(os.Environ(), "PITOT_MIXED_LOG_CHILD=1") + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("default log bridge subprocess error=%v timeout=%v output=%s", err, ctx.Err(), out) + } +} + +func TestMixedReadVariedShapesAndTraceOracle(t *testing.T) { + start := time.Unix(0, 0).UTC() + specs := mixedReadSpecs(start, start.Add(24*time.Hour)) + if len(specs) != 3 || len(specs[0].Panels) != 1 || len(specs[1].Panels) != 4 || len(specs[2].Panels) != 12 { + t.Fatal("unexpected operation/panel shape counts") + } + tracePanels := 0 + for _, p := range specs[2].Panels { + if strings.HasPrefix(p.ID, "traces_") { + tracePanels++ + } + } + if tracePanels != 3 || len(specs[2].Variables) != 1 { + t.Fatal("varied dashboard lost raw distinct-trace shapes or selected variable") + } + for _, test := range []struct { + id string + want float64 + }{{"traces_all", 512}, {"traces_checkout", 342}, {"traces_selected", 512}} { + frame := &Frame{Rows: 1, Columns: []Column{{Name: "trace_count", Role: "measure"}}, Values: [][]any{{test.want}}} + result := []Result{{ID: test.id, Status: StatusOK, Frame: frame}} + if err := mixedReadOracle(result, 1, 2048); err != nil { + t.Fatal(err) + } + frame.Values[0][0] = test.want + 1 + if err := mixedReadOracle(result, 1, 2048); err == nil { + t.Fatalf("incorrect %s count passed", test.id) + } + } +} +func TestMixedReadIngestScale(t *testing.T) { + for _, test := range []struct { + raw string + want int + }{{"", 2048}, {"128", 128}, {"2048", 2048}, {"8192", 8192}} { + if got, err := mixedReadIngestRows(test.raw); err != nil || got != test.want { + t.Fatalf("input=%q rows=%d error=%v", test.raw, got, err) + } + } + for _, raw := range []string{"0", "127", "8193", "invalid"} { + if _, err := mixedReadIngestRows(raw); err == nil { + t.Fatalf("invalid %q accepted", raw) + } + } + at := time.Now().UTC() + batch := mixedReadBatch("live-test", at, 128) + for i, s := range batch.Spans { + if s.StartUnixNanos != at.UnixNano()+int64(i) || batch.Logs[i].EventUnixNanos != s.StartUnixNanos { + t.Fatal("live timestamps extend beyond nanosecond offsets") + } + } +} + +func TestMixedReadSeedControls(t *testing.T) { + now := time.Date(2026, 11, 14, 12, 0, 0, 0, time.UTC) + state, end, err := mixedReadSeedConfig("", "", now) + if err != nil || state != "backlog" || !end.Equal(now.Add(-time.Minute)) { + t.Fatalf("default state=%s end=%s error=%v", state, end, err) + } + fixed := now.Add(-time.Hour).Add(17 * time.Nanosecond) + state, end, err = mixedReadSeedConfig("settled", fixed.Format(time.RFC3339Nano), now) + if err != nil || state != "settled" || !end.Equal(fixed) { + t.Fatalf("fixed state=%s end=%s error=%v", state, end, err) + } + for _, test := range []struct{ state, end string }{{"invalid", ""}, {"backlog", "invalid"}, {"settled", now.Add(time.Second).Format(time.RFC3339Nano)}, {"settled", now.Add(-30 * 24 * time.Hour).Format(time.RFC3339Nano)}} { + if _, _, err := mixedReadSeedConfig(test.state, test.end, now); err == nil { + t.Fatalf("invalid seed control passed: %+v", test) + } + } + if done, err := mixedReadCompactionProgress(4, 2, 3, 0); err != nil || done { + t.Fatalf("positive merge done=%t error=%v", done, err) + } + if done, err := mixedReadCompactionProgress(2, 2, 0, 0); err != nil || !done { + t.Fatalf("settled done=%t error=%v", done, err) + } + if _, err := mixedReadCompactionProgress(2, 2, 0, 1); err == nil { + t.Fatal("residual debt passed") + } + if _, err := mixedReadCompactionProgress(2, 2, 1, 0); err == nil { + t.Fatal("nonprogressing merge passed") + } +} + +func TestMixedReadSettledSeedTinyFixture(t *testing.T) { + if os.Getenv("PITOT_MIXED_SETTLE_SMOKE") != "1" { + t.Skip("opt-in native seed-settlement smoke; run only with timing slot") + } + cfg := config.Config{DataDir: panelBenchScratch(t), RetentionDays: 30, DuckDBMemory: "4GB", DuckDBThreads: 4, DuckDBMaxConns: 8, RollupInterval: time.Second} + repo, err := telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + t.Fatal(err) + } + defer repo.Close() + at := time.Now().UTC().Add(-time.Minute) + for i := range 128 { + if err := repo.Commit(t.Context(), mixedReadBatch(fmt.Sprintf("settle-smoke-%03d", i), at, 4)); err != nil { + t.Fatal(err) + } + } + d, err := engine.NewDuck(t.Context(), cfg, repo) + if err != nil { + t.Fatal(err) + } + defer d.Close() + mixedReadSettleSeed(t, repo, d) + metadata := repo.Parquet.BatchMetadata() + if len(metadata) >= 128 || repo.CompactionDebt() != 0 { + t.Fatalf("settlement failed: batches=%d debt=%d", len(metadata), repo.CompactionDebt()) + } + var spans, logs int64 + if err := d.QueryRowScan(t.Context(), []any{&spans, &logs}, "SELECT (SELECT count(*) FROM spans),(SELECT count(*) FROM logs)"); err != nil { + t.Fatal(err) + } + if spans != 512 || logs != 512 { + t.Fatalf("settlement changed rows: spans=%d logs=%d", spans, logs) + } +} + +func TestMixedReadDiagnosticCounterDeltas(t *testing.T) { + before := mixedReadDiagnosticSnapshot{cache: map[string]float64{"hit": 10, "miss": 20, "coalesced": 30, "stale": 1}, statements: 100} + after := mixedReadDiagnosticSnapshot{cache: map[string]float64{"hit": 12, "miss": 25, "coalesced": 33, "stale": 1}, statements: 109} + outcomes, statements, reset := mixedReadDiagnosticDelta(before, after) + if reset || statements != 9 || outcomes["hit"] != 2 || outcomes["miss"] != 5 || outcomes["coalesced"] != 3 || outcomes["stale"] != 0 { + t.Fatalf("outcomes=%v statements=%d reset=%t", outcomes, statements, reset) + } + outcomes, statements, reset = mixedReadDiagnosticDelta(after, before) + if !reset || statements != 0 { + t.Fatalf("reset statements=%d reset=%t", statements, reset) + } + for _, delta := range outcomes { + if delta < 0 { + t.Fatal("negative diagnostic counter delta") + } + } +} + +func TestMixedReadShutdownInterruptClassification(t *testing.T) { + interrupt := &duckdb.Error{Type: duckdb.ErrorTypeInterrupt, Msg: "interrupted"} + unrelated := errors.New("cannot CHECKPOINT while another write is active") + for _, test := range []struct { + name string + stopping, canceled bool + err error + expected bool + interruptCount int + }{ + {"running_interrupt", false, false, interrupt, false, 0}, + {"canceled_without_intent", false, true, interrupt, false, 0}, + {"intent_before_cancel", true, false, interrupt, false, 0}, + {"intentional_interrupt", true, true, fmt.Errorf("rollup: %w", interrupt), true, 1}, + {"poststop_unrelated", true, true, unrelated, false, 0}, + {"joined_unrelated", true, true, errors.Join(interrupt, unrelated), false, 0}, + {"poststop_other_native", true, true, &duckdb.Error{Type: duckdb.ErrorTypeInvalidInput, Msg: "bad input"}, false, 0}, + {"intentional_go_cancel", true, true, context.Canceled, true, 0}, + } { + t.Run(test.name, func(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + if test.canceled { + cancel() + } + state := &mixedReadLogState{background: ctx} + state.stopping.Store(test.stopping) + var raw strings.Builder + h := mixedReadLogHandler{Handler: slog.NewTextHandler(&raw, nil), state: state} + record := slog.NewRecord(time.Now(), slog.LevelError, "background rollup failed", 0) + record.AddAttrs(slog.Any("err", test.err)) + if err := h.Handle(context.Background(), record); err != nil { + t.Fatal(err) + } + if (len(state.failures) == 0) != test.expected || state.shutdownInterrupts != test.interruptCount { + t.Fatalf("failures=%v shutdown_interrupts=%d", state.failures, state.shutdownInterrupts) + } + if !strings.Contains(raw.String(), "background rollup failed") { + t.Fatal("raw error log was suppressed") + } + }) + } +} diff --git a/internal/query/mixed_read_diagnostics_test.go b/internal/query/mixed_read_diagnostics_test.go new file mode 100644 index 000000000..dd2316e01 --- /dev/null +++ b/internal/query/mixed_read_diagnostics_test.go @@ -0,0 +1,133 @@ +//go:build readbench + +package query + +import ( + "context" + "encoding/json" + "os" + "runtime" + "strings" + "sync" + "syscall" + "testing" + "time" + + "github.com/labstack/pitot/internal/engine" + "github.com/labstack/pitot/internal/queryrows" +) + +// Optional bounded observations distinguish engine work from admission wait. +// Keep SQL and telemetry values out of the diagnostic output. +type mixedReadObservation struct { + Kind string + StartMS, DurationMS float64 + Busy bool + WALBytes int64 + CPUSeconds float64 + // These counts describe completion time, not peak occupancy. + Batches, ActivePanels, WaitingPanels, PoolInUse int +} + +type mixedReadProbe struct { + sync.Mutex + start time.Time + events []mixedReadObservation + executor *Executor + duck *engine.Duck + walPath string + dropped int + stackPath string + stackOnce sync.Once +} + +func (p *mixedReadProbe) setStart(at time.Time) { p.Lock(); p.start = at; p.Unlock() } + +func (p *mixedReadProbe) record(kind string, start time.Time, busy bool) { + if p == nil { + return + } + p.Lock() + origin := p.start + p.Unlock() + if origin.IsZero() { + return + } + duration := time.Since(start) + if kind != "sample" && !busy && duration < 25*time.Millisecond { + return + } + event := mixedReadObservation{Kind: kind, StartMS: float64(start.Sub(origin)) / float64(time.Millisecond), DurationMS: float64(duration) / float64(time.Millisecond), Busy: busy} + if info, err := os.Stat(p.walPath); err == nil { + event.WALBytes = info.Size() + } + var usage syscall.Rusage + if syscall.Getrusage(syscall.RUSAGE_SELF, &usage) == nil { + event.CPUSeconds = float64(usage.Utime.Sec+usage.Stime.Sec) + float64(usage.Utime.Usec+usage.Stime.Usec)/1e6 + } + a := p.executor.admission + a.mu.Lock() + event.Batches, event.ActivePanels, event.WaitingPanels = a.batches, a.active[panelClass], len(a.queues[panelClass]) + a.mu.Unlock() + event.PoolInUse = p.duck.DB.Stats().InUse + p.Lock() + if len(p.events) < 4096 { + p.events = append(p.events, event) + } else { + p.dropped++ + } + p.Unlock() + if busy && p.stackPath != "" { + p.stackOnce.Do(func() { + buf := make([]byte, 1<<20) + n := runtime.Stack(buf, true) + _ = os.WriteFile(p.stackPath, buf[:n], 0600) + }) + } +} + +func (p *mixedReadProbe) report(t *testing.T) { + if p == nil { + return + } + p.Lock() + defer p.Unlock() + encoded, err := json.Marshal(p.events) + if err != nil { + t.Fatal(err) + } + t.Logf("MIXED_READ_DIAGNOSTICS origin_utc=%s dropped=%d events=%s", p.start.UTC().Format(time.RFC3339Nano), p.dropped, encoded) +} + +type mixedReadObservedEngine struct { + *engine.Duck + probe *mixedReadProbe +} + +func (e *mixedReadObservedEngine) QueryContext(ctx context.Context, query string, args ...any) (queryrows.Rows, error) { + start := time.Now() + kind := "engine_other" + if strings.Contains(query, "__hist_") { + kind = "engine_histogram" + } else if strings.Contains(strings.ToLower(query), "distinct") { + kind = "engine_distinct" + } + rows, err := e.Duck.QueryContext(ctx, query, args...) + if err != nil { + e.probe.record(kind, start, false) + return nil, err + } + return &mixedReadObservedRows{Rows: rows, done: func() { e.probe.record(kind, start, false) }}, nil +} + +type mixedReadObservedRows struct { + queryrows.Rows + once sync.Once + done func() +} + +func (r *mixedReadObservedRows) Close() error { + err := r.Rows.Close() + r.once.Do(r.done) + return err +} diff --git a/internal/query/panel_bench_data_test.go b/internal/query/panel_bench_data_test.go index 8cf08322a..51f5d3356 100644 --- a/internal/query/panel_bench_data_test.go +++ b/internal/query/panel_bench_data_test.go @@ -3,7 +3,9 @@ package query import ( + "context" "crypto/sha256" + "database/sql" "encoding/binary" "encoding/json" "errors" @@ -13,20 +15,25 @@ import ( "os" "path/filepath" "sort" + "strconv" "strings" "testing" "time" "github.com/labstack/pitot/internal/config" + native "github.com/labstack/pitot/internal/duckdb" "github.com/labstack/pitot/internal/engine" "github.com/labstack/pitot/internal/queryrows" "github.com/labstack/pitot/internal/telemetry" + "github.com/labstack/pitot/internal/telemetry/semconv" telemetrystore "github.com/labstack/pitot/internal/telemetry/store" "github.com/parquet-go/parquet-go" "github.com/parquet-go/parquet-go/compress/zstd" "github.com/zeebo/xxh3" ) +const panelBenchFormat = 5 + type panelBenchData struct { Dir string `json:"-"` SourceHash string @@ -75,6 +82,13 @@ func panelBenchmarkMetricNames(tb testing.TB, engine Engine, data panelBenchData // Do not open the controller's repository: Open can clean staging and rewrite // its schema batch. Validate every metadata record before copying anything. func panelReplayMetadata(input string) ([]telemetry.BatchMetadata, error) { + // A live compaction transaction can depend on staging outside published batches. + // Require a quiescent capture; committed retirement sidecars are copied below. + if _, err := os.Lstat(filepath.Join(input, "COMPACTION.json")); err == nil { + return nil, errors.New("replay needs a quiescent capture without a live COMPACTION.json marker") + } else if !errors.Is(err, os.ErrNotExist) { + return nil, err + } entries, err := os.ReadDir(filepath.Join(input, "parquet", "batches")) if err != nil { return nil, err @@ -95,7 +109,7 @@ func panelReplayMetadata(input string) ([]telemetry.BatchMetadata, error) { if err := json.Unmarshal(raw, &m); err != nil { return nil, err } - if m.Version != 4 { + if m.Version != panelBenchFormat { return nil, fmt.Errorf("%w: version %d", telemetry.ErrUnsupportedBatchFormat, m.Version) } if err := telemetry.ValidateBatchID(m.ID); err != nil { @@ -112,6 +126,19 @@ func panelReplayMetadata(input string) ([]telemetry.BatchMetadata, error) { return out, nil } +// Preserve committed catalog state when copying a captured batch. +func panelReplaySidecars(tb testing.TB, input, id string, names []string) []string { + tb.Helper() + for _, name := range []string{"replaces.json", "pruned"} { + if _, err := os.Lstat(filepath.Join(input, "parquet", "batches", id+telemetry.BatchSuffix, name)); err == nil { + names = append(names, name) + } else if !errors.Is(err, os.ErrNotExist) { + tb.Fatal(err) + } + } + return names +} + // Copy accepted batches byte for byte before opening a repository. Like the // replay harness, input is the telemetry root containing parquet/batches. // Opening only the private copy leaves source schema and staging untouched. @@ -119,9 +146,18 @@ func copyMetricCrosscheckData(tb testing.TB, input string) panelBenchData { tb.Helper() metadata, err := panelReplayMetadata(input) if err != nil { - tb.Fatal("cross-check input needs valid format-5 batch metadata") + tb.Fatal("cross-check input:", err) } data := panelBenchData{Dir: filepath.Join(panelBenchScratch(tb), "state")} + verifier, err := telemetry.NewBatchVerifier(tb.Context(), filepath.Dir(data.Dir)) + if err != nil { + tb.Fatal(err) + } + defer func() { + if err := verifier.Close(); err != nil { + tb.Error(err) + } + }() for _, m := range metadata { names := []string{"metadata.json"} if m.Spans > 0 { @@ -133,6 +169,7 @@ func copyMetricCrosscheckData(tb testing.TB, input string) panelBenchData { if m.Metrics > 0 { names = append(names, "metrics.parquet") } + names = panelReplaySidecars(tb, input, m.ID, names) dst := filepath.Join(data.Dir, "telemetry", "parquet", "batches", m.ID+telemetry.BatchSuffix) if err := os.MkdirAll(dst, 0o755); err != nil { tb.Fatal("cannot create cross-check batch directory") @@ -157,9 +194,9 @@ func copyMetricCrosscheckData(tb testing.TB, input string) panelBenchData { tb.Fatal("cannot copy cross-check input") } } - data.Spans += int64(m.Spans) - data.Logs += int64(m.Logs) - data.Metrics += int64(m.Metrics) + if err := verifier.Validate(tb.Context(), dst); err != nil { + tb.Fatal("invalid cross-check batch:", err) + } } // Repository metadata derives each signal's bounds from its event footer. repo, err := telemetrystore.Open(filepath.Join(data.Dir, "telemetry")) @@ -168,6 +205,9 @@ func copyMetricCrosscheckData(tb testing.TB, input string) panelBenchData { } low, high := int64(math.MaxInt64), int64(math.MinInt64) for _, m := range repo.Parquet.BatchMetadata() { + data.Spans += int64(m.Spans) + data.Logs += int64(m.Logs) + data.Metrics += int64(m.Metrics) if m.Metrics > 0 && m.MetricTime.Known { low = min(low, m.MetricTime.MinNanos) high = max(high, m.MetricTime.MaxNanos) @@ -228,6 +268,15 @@ func preparePanelReplay(tb testing.TB, input string, volume int) panelBenchData tb.Fatal("invalid replay volume") } root := panelBenchScratch(tb) + verifier, err := telemetry.NewBatchVerifier(tb.Context(), root) + if err != nil { + tb.Fatal(err) + } + defer func() { + if err := verifier.Close(); err != nil { + tb.Error(err) + } + }() copied := filepath.Join(root, "input") hash := sha256.New() for _, m := range metadata { @@ -241,6 +290,7 @@ func preparePanelReplay(tb testing.TB, input string, volume int) panelBenchData if m.Metrics > 0 { files = append(files, "metrics.parquet") } + files = panelReplaySidecars(tb, input, m.ID, files) for _, name := range files { rel := filepath.Join("parquet", "batches", m.ID+telemetry.BatchSuffix, name) if err := os.MkdirAll(filepath.Dir(filepath.Join(copied, rel)), 0o755); err != nil { @@ -267,11 +317,17 @@ func preparePanelReplay(tb testing.TB, input string, volume int) panelBenchData } } } + for _, m := range metadata { + if err := verifier.Validate(tb.Context(), filepath.Join(copied, "parquet", "batches", m.ID+telemetry.BatchSuffix)); err != nil { + tb.Fatal("invalid replay batch:", err) + } + } repo, err := telemetrystore.Open(copied) if err != nil { tb.Fatal(err) } defer repo.Close() + metadata = repo.Parquet.BatchMetadata() low, high := int64(math.MaxInt64), int64(math.MinInt64) for _, m := range repo.Parquet.BatchMetadata() { for _, r := range []telemetry.TimeRange{m.SpanTime, m.LogTime, m.MetricTime} { @@ -301,7 +357,7 @@ func preparePanelReplay(tb testing.TB, input string, volume int) panelBenchData if err := os.MkdirAll(dst, 0o755); err != nil { tb.Fatal(err) } - next := telemetry.BatchMetadata{Version: 4, ID: prefix + m.ID, Generation: m.Generation} + next := telemetry.BatchMetadata{Version: panelBenchFormat, ID: prefix + m.ID, Generation: m.Generation} for _, signal := range []string{"spans", "logs", "metrics"} { count := map[string]int{"spans": m.Spans, "logs": m.Logs, "metrics": m.Metrics}[signal] if count == 0 { @@ -337,6 +393,9 @@ func preparePanelReplay(tb testing.TB, input string, volume int) panelBenchData if err := os.WriteFile(filepath.Join(dst, "metadata.json"), raw, 0o644); err != nil { tb.Fatal(err) } + if err := verifier.Validate(tb.Context(), dst); err != nil { + tb.Fatal("invalid tiled batch:", err) + } data.Spans += int64(next.Spans) data.Logs += int64(next.Logs) data.Metrics += int64(next.Metrics) @@ -407,12 +466,29 @@ func tilePanelParquet(input, output, signal, prefix string, start, end time.Time timeColumns[column] = name } } + // Repeated native timestamps and linked identities follow the replay tile too. + for i, path := range reader.Schema().Columns() { + switch path[0] { + case "event_times", "exemplar_times": + timeColumns[i] = path[0] + } + } identityColumns := map[int]bool{} for _, name := range []string{"trace_id", "span_id", "parent_span_id"} { if column, ok := cols[name]; ok { identityColumns[column] = true } } + spillCounts, err := panelReplaySpills(input, reader.File(), signal, start, end) + if err != nil { + return err + } + for i, path := range reader.Schema().Columns() { + switch path[0] { + case "link_trace_ids", "link_span_ids", "exemplar_trace_ids", "exemplar_span_ids": + identityColumns[i] = true + } + } out, err := os.Create(output) if err != nil { return err @@ -422,6 +498,7 @@ func tilePanelParquet(input, output, signal, prefix string, start, end time.Time type rowWriter interface { WriteRows([]parquet.Row) (int, error) Close() error + SetKeyValueMetadata(string, string) } var writer rowWriter if signal == "spans" { @@ -430,6 +507,9 @@ func tilePanelParquet(input, output, signal, prefix string, start, end time.Time } else { writer = parquet.NewGenericWriter[any](out, options...) } + for key, count := range spillCounts { + writer.SetKeyValueMetadata(key, strconv.FormatUint(count, 10)) + } defer func() { err = errors.Join(err, writer.Close()) }() rows := make([]parquet.Row, 256) for { @@ -454,11 +534,12 @@ func tilePanelParquet(input, output, signal, prefix string, start, end time.Time v := value.Int64() // Zero optional raw timestamps remain absent; native event // timestamps, including signed and zero instants, are shifted. - if v != 0 || name == event || name == "end_time" || name == "start_time" { + optional := name == "time_unix_nano" || name == "observed_time_unix_nano" || name == "start_time_unix_nano" + if v != 0 || !optional { row[i] = parquet.Int64Value(v+int64(shift)).Level(value.RepetitionLevel(), value.DefinitionLevel(), column) } } - if identityColumns[column] && value.String() != "" { + if identityColumns[column] && !value.IsNull() && value.String() != "" { row[i] = parquet.ByteArrayValue([]byte(prefix+value.String())).Level(value.RepetitionLevel(), value.DefinitionLevel(), column) } } @@ -498,6 +579,59 @@ func tilePanelParquet(input, output, signal, prefix string, start, end time.Time return nil } +// Attribute maps and their dedicated values are unchanged by replay. After native +// input verification, a non-null map value with a NULL dedicated column is exactly +// a spill. Count only retained rows; copying the source footer overcounts clips. +func panelReplaySpills(input string, file parquet.FileView, signal string, start, end time.Time) (counts map[string]uint64, err error) { + layout, ok := telemetry.LookupSignal(signal) + if !ok { + return nil, fmt.Errorf("unknown replay signal %q", signal) + } + counts = map[string]uint64{} + var db *sql.DB + defer func() { + if db != nil { + err = errors.Join(err, db.Close()) + } + }() + for _, key := range layout.Keys() { + footer := "pitot.spill." + key.Name + if _, exists := file.Lookup(footer); !exists { + continue + } + if db == nil { + db, err = sql.Open("duckdb", "?threads=2&memory_limit=256MB") + if err != nil { + return nil, err + } + db.SetMaxOpenConns(1) + for _, statement := range []string{"SET TimeZone='UTC'", "SET temp_directory=''"} { + if _, err = db.Exec(statement); err != nil { + return nil, err + } + } + if err = native.ConfigurePolicy(context.Background(), db, filepath.Dir(input)); err != nil { + return nil, err + } + } + source := "attributes" + if key.Scope == semconv.Resource { + source = "resource" + } + var count uint64 + // database/sql binds time.Time at microsecond precision. Text parameters + // cast to TIMESTAMP_NS retain the exact clipping bounds. + query := fmt.Sprintf("SELECT count(*) FROM read_parquet(?, hive_partitioning=false, union_by_name=false) WHERE %s >= ?::TIMESTAMP_NS::TIMESTAMPTZ_NS AND %s < ?::TIMESTAMP_NS::TIMESTAMPTZ_NS AND %s IS NULL AND variant_type(variant_normalize(%s), ?) <> 'VARIANT_NULL'", layout.Time, layout.Time, key.Column, source) + if err = db.QueryRow(query, input, start.UTC().Format("2006-01-02 15:04:05.999999999"), end.UTC().Format("2006-01-02 15:04:05.999999999"), key.Name).Scan(&count); err != nil { + return nil, err + } + if count > 0 { + counts[footer] = count + } + } + return counts, nil +} + func panelRowValue(row parquet.Row, column int) parquet.Value { for _, v := range row { if v.Column() == column { @@ -574,6 +708,34 @@ func panelTraceIndex(dir string) (err error) { return err } +// The executor selects analytical rollups for eligible panels. Populate those +// summaries before steady-state measurement rather than timing empty relations. +func panelBenchWarmDashboard(tb testing.TB, d *engine.Duck, horizon int64) { + tb.Helper() + ctx, cancel := context.WithTimeout(tb.Context(), 5*time.Minute) + defer cancel() + var previous time.Time + for { + if err := ctx.Err(); err != nil { + tb.Fatal("dashboard rollup preparation:", err) + } + if _, err := d.RefreshDashboardRollups(ctx); err != nil { + tb.Fatal("dashboard rollup preparation:", err) + } + stamp, err := d.RollupWatermark(ctx) + if err != nil { + tb.Fatal(err) + } + if !stamp.IsZero() && stamp.UnixNano() >= horizon { + return + } + if stamp.IsZero() || !previous.IsZero() && !stamp.After(previous) { + tb.Fatalf("dashboard rollup preparation made no progress: watermark=%s target=%d", stamp, horizon) + } + previous = stamp + } +} + func panelBenchEngine(tb testing.TB, data panelBenchData) (*engine.Duck, config.Config) { tb.Helper() // Load validates authentication even though this harness never serves HTTP. diff --git a/internal/query/panel_bench_replay_test.go b/internal/query/panel_bench_replay_test.go new file mode 100644 index 000000000..42dc50aff --- /dev/null +++ b/internal/query/panel_bench_replay_test.go @@ -0,0 +1,277 @@ +//go:build readbench + +package query + +import ( + "crypto/sha256" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "reflect" + "testing" + "time" + + "github.com/labstack/pitot/internal/telemetry" + telemetrystore "github.com/labstack/pitot/internal/telemetry/store" + "github.com/parquet-go/parquet-go" +) + +func panelReplayInventory(t *testing.T, root string) map[string][32]byte { + t.Helper() + out := map[string][32]byte{} + if err := filepath.WalkDir(root, func(path string, entry os.DirEntry, err error) error { + if err != nil { + return err + } + if entry.IsDir() { + return nil + } + raw, err := os.ReadFile(path) + if err != nil { + return err + } + relative, err := filepath.Rel(root, path) + if err != nil { + return err + } + out[relative] = sha256.Sum256(raw) + return nil + }); err != nil { + t.Fatal(err) + } + return out +} + +func TestPanelBenchmarkReplayRejectsOldAndFutureFormatsWithoutMutation(t *testing.T) { + for _, version := range []int{0, 2, 4, 6} { + t.Run(fmt.Sprint(version), func(t *testing.T) { + root := t.TempDir() + dir := filepath.Join(root, "parquet", "batches", "bad.batch") + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "metadata.json"), []byte(fmt.Sprintf(`{"version":%d,"id":"bad","spans":1}`, version)), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "spans.parquet"), []byte("untouched unsupported data"), 0o644); err != nil { + t.Fatal(err) + } + before := panelReplayInventory(t, root) + if _, err := panelReplayMetadata(root); !errors.Is(err, telemetry.ErrUnsupportedBatchFormat) { + t.Fatalf("error=%v", err) + } + if !reflect.DeepEqual(before, panelReplayInventory(t, root)) { + t.Fatal("unsupported input changed") + } + }) + } +} + +func TestPanelBenchmarkReplayClippedSpillsEpochAndLists(t *testing.T) { + root := t.TempDir() + repo, err := telemetrystore.Open(root) + if err != nil { + t.Fatal(err) + } + base := int64(time.Hour) + batch := telemetrystore.Batch{ID: "source"} + for i, at := range []int64{base - 1, base, base + 1, base + 2} { + var version any = map[string]any{"unconvertible": true} + var status any = "not an integer" + if i == 2 { + version, status = nil, nil + } + resource := map[string]any{"service.version": version} + attrs := map[string]any{"http.response.status_code": status, "literal.dot": int64(7), "nested": map[string]any{"flag": true, "bytes": []byte{0, 255}}} + batch.Spans = append(batch.Spans, telemetry.Span{ServiceName: "checkout", TraceID: fmt.Sprintf("trace%d", i), SpanID: fmt.Sprintf("span%d", i), ParentSpanID: "parent", StartUnixNanos: at, EndUnixNanos: at, IngestedAt: at, Resource: resource, Attributes: attrs, + Events: []telemetry.Event{{TimeUnixNanos: at, Name: "event", Attributes: map[string]any{"a": int64(9)}}}, + Links: []telemetry.Link{{TraceID: "linked", SpanID: "linkedspan", Attributes: map[string]any{"b": true}}}}) + batch.Logs = append(batch.Logs, telemetry.Log{ServiceName: "checkout", TraceID: fmt.Sprintf("trace%d", i), SpanID: fmt.Sprintf("span%d", i), EventUnixNanos: at, TimeUnixNanos: at, IngestedAt: at, Resource: resource, Attributes: attrs, Body: "complete"}) + batch.Metrics = append(batch.Metrics, telemetry.Metric{ServiceName: "checkout", Name: "histogram", Type: "histogram", Temporality: "cumulative", EventUnixNanos: at, TimeUnixNanos: at, IngestedAt: at, Resource: resource, Attributes: attrs, + Histogram: telemetry.Histogram{Count: 3, Bounds: []float64{1, 2}, Counts: []uint64{1, 1, 1}}, + Exemplars: []telemetry.Exemplar{{TimeUnixNanos: at, TraceID: "trace", SpanID: "span", Value: 2, Attributes: map[string]any{"c": int64(3)}}}}) + } + if err := repo.Commit(t.Context(), batch); err != nil { + t.Fatal(err) + } + if err := repo.Close(); err != nil { + t.Fatal(err) + } + src := filepath.Join(root, "parquet", "batches", "source.batch") + if err := telemetry.ValidatePublishedBatch(src); err != nil { + t.Fatal(err) + } + before := panelReplayInventory(t, root) + dst := filepath.Join(t.TempDir(), "shifted.batch") + if err := os.MkdirAll(dst, 0o755); err != nil { + t.Fatal(err) + } + metadata := telemetry.BatchMetadata{Version: panelBenchFormat, ID: "shifted"} + for _, signal := range []string{"spans", "logs", "metrics"} { + if err := tilePanelParquet(filepath.Join(src, signal+".parquet"), filepath.Join(dst, signal+".parquet"), signal, "copy_", time.Unix(0, base), time.Unix(0, base+2), -time.Hour, &metadata, map[string]bool{}, map[string]bool{}); err != nil { + t.Fatal(err) + } + f, err := os.Open(filepath.Join(dst, signal+".parquet")) + if err != nil { + t.Fatal(err) + } + reader := parquet.NewReader(f) + if got, ok := reader.File().Lookup("pitot.spill.service.version"); !ok || got != "1" { + t.Fatalf("%s clipped resource spills=%q exists=%t", signal, got, ok) + } + if signal == "spans" { + if got, ok := reader.File().Lookup("pitot.spill.http.response.status_code"); !ok || got != "1" { + t.Fatalf("attribute spills=%q exists=%t", got, ok) + } + } + if err := errors.Join(reader.Close(), f.Close()); err != nil { + t.Fatal(err) + } + } + if err := panelTraceIndex(dst); err != nil { + t.Fatal(err) + } + raw, err := json.Marshal(metadata) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dst, "metadata.json"), raw, 0o644); err != nil { + t.Fatal(err) + } + if metadata.Spans != 2 || metadata.Logs != 2 || metadata.Metrics != 2 || metadata.MinSpanStartNanos != 0 || metadata.MaxSpanStartNanos != 1 { + t.Fatalf("metadata=%+v", metadata) + } + // Native verification checks every physical column, dedicated coercion, list + // lengths, complete sort tuples and exact trace-index ranges after rehashing. + if err := telemetry.ValidatePublishedBatch(dst); err != nil { + t.Fatal(err) + } + // Starting a second tile at epoch zero catches required raw timestamps + // incorrectly treated as absent optional zero values. + signed := filepath.Join(t.TempDir(), "negative.batch") + if err := os.MkdirAll(signed, 0o755); err != nil { + t.Fatal(err) + } + negative := telemetry.BatchMetadata{Version: panelBenchFormat, ID: "negative"} + for _, signal := range []string{"spans", "logs", "metrics"} { + if err := tilePanelParquet(filepath.Join(dst, signal+".parquet"), filepath.Join(signed, signal+".parquet"), signal, "again_", time.Unix(0, 0), time.Unix(0, 2), -time.Hour, &negative, map[string]bool{}, map[string]bool{}); err != nil { + t.Fatal(err) + } + } + if err := panelTraceIndex(signed); err != nil { + t.Fatal(err) + } + negativeJSON, err := json.Marshal(negative) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(signed, "metadata.json"), negativeJSON, 0o644); err != nil { + t.Fatal(err) + } + if negative.MinSpanStartNanos != -base || negative.MaxSpanStartNanos != -base+1 { + t.Fatalf("negative metadata=%+v", negative) + } + if err := telemetry.ValidatePublishedBatch(signed); err != nil { + t.Fatal(err) + } + // Use a private engine directory to inspect payloads and exact epoch boundaries. + state := t.TempDir() + target := filepath.Join(state, "telemetry", "parquet", "batches", "shifted.batch") + if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Rename(dst, target); err != nil { + t.Fatal(err) + } + negativeTarget := filepath.Join(state, "telemetry", "parquet", "batches", "negative.batch") + if err := os.Rename(signed, negativeTarget); err != nil { + t.Fatal(err) + } + engine, _ := panelBenchEngine(t, panelBenchData{Dir: state}) + var count int + if err := engine.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM read_parquet(?,hive_partitioning=false) WHERE epoch_ns(start_time)=start_unix_nano AND epoch_ns(end_time)=end_unix_nano AND start_unix_nano IN (0,1) AND event_times[1]=start_time AND link_trace_ids[1]='copy_linked' AND link_span_ids[1]='copy_linkedspan' AND attributes['literal.dot']::BIGINT=7 AND attributes['nested']['flag']::BOOLEAN AND attributes['nested']['bytes']::BLOB=from_hex('00ff') AND link_attributes[1]['b']::BOOLEAN AND event_attributes[1]['a']::BIGINT=9`, filepath.Join(target, "spans.parquet")).Scan(&count); err != nil || count != 2 { + t.Fatalf("span payload count=%d err=%v", count, err) + } + if err := engine.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM read_parquet(?,hive_partitioning=false) WHERE epoch_ns(exemplar_times[1])=epoch_ns(metric_time) AND exemplar_trace_ids[1]='copy_trace' AND exemplar_span_ids[1]='copy_span' AND hist_counts=[1,1,1]::UBIGINT[] AND hist_bounds=[1,2]::DOUBLE[] AND start_time_unix_nano=0 AND exemplar_attributes[1]['c']::BIGINT=3`, filepath.Join(target, "metrics.parquet")).Scan(&count); err != nil || count != 2 { + t.Fatalf("metric payload count=%d err=%v", count, err) + } + if err := engine.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM read_parquet(?,hive_partitioning=false) WHERE epoch_ns(start_time)=start_unix_nano AND epoch_ns(end_time)=end_unix_nano AND start_unix_nano IN (?,?) AND epoch_ns(event_times[1])=start_unix_nano`, filepath.Join(negativeTarget, "spans.parquet"), -base, -base+1).Scan(&count); err != nil || count != 2 { + t.Fatalf("negative required timestamps count=%d err=%v", count, err) + } + if err := engine.DB.QueryRowContext(t.Context(), `SELECT count(*) FROM read_parquet(?,hive_partitioning=false) WHERE start_time_unix_nano=0 AND epoch_ns(exemplar_times[1])=epoch_ns(metric_time)`, filepath.Join(negativeTarget, "metrics.parquet")).Scan(&count); err != nil || count != 2 { + t.Fatalf("optional zero starts count=%d err=%v", count, err) + } + if !reflect.DeepEqual(before, panelReplayInventory(t, root)) { + t.Fatal("source changed") + } +} + +func TestPanelBenchmarkReplayRejectsLifecycleMarkersWithoutMutation(t *testing.T) { + for _, marker := range []string{"COMPACTION.json"} { + t.Run(filepath.Base(marker), func(t *testing.T) { + root := t.TempDir() + dir := filepath.Join(root, "parquet", "batches", "old.batch") + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "metadata.json"), []byte(`{"version":5,"id":"old","spans":1}`), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "spans.parquet"), []byte("preserve retired input"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, marker), []byte(`["old"]`), 0o644); err != nil { + t.Fatal(err) + } + before := panelReplayInventory(t, root) + if _, err := panelReplayMetadata(root); err == nil { + t.Fatal("marked capture accepted; retired data could be duplicated") + } + if !reflect.DeepEqual(before, panelReplayInventory(t, root)) { + t.Fatal("marked source changed") + } + }) + } +} + +func TestPanelBenchmarkReplayCopiesRetirementMarkersWithoutDuplicates(t *testing.T) { + root := t.TempDir() + repo, err := telemetrystore.Open(root) + if err != nil { + t.Fatal(err) + } + at := fixtureStart.Add(time.Minute).UnixNano() + for _, id := range []string{"old", "replacement", "pruned"} { + batch := telemetrystore.Batch{ID: id, + Spans: []telemetry.Span{{TraceID: "trace", SpanID: "span", ServiceName: "checkout", StartUnixNanos: at, EndUnixNanos: at + 1}}, + Logs: []telemetry.Log{{TraceID: "trace", ServiceName: "checkout", EventUnixNanos: at, TimeUnixNanos: at, Body: "retained"}}, + Metrics: []telemetry.Metric{{ServiceName: "checkout", Name: "gauge", Type: "gauge", EventUnixNanos: at, TimeUnixNanos: at, Value: 1}}, + } + if err := repo.Commit(t.Context(), batch); err != nil { + t.Fatal(err) + } + } + if err := repo.Close(); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "parquet", "batches", "replacement.batch", "replaces.json"), []byte(`["old"]`), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "parquet", "batches", "pruned.batch", "pruned"), nil, 0o644); err != nil { + t.Fatal(err) + } + before := panelReplayInventory(t, root) + copied := copyMetricCrosscheckData(t, root) + if copied.Spans != 1 || copied.Logs != 1 || copied.Metrics != 1 { + t.Fatalf("copied retired rows: %+v", copied) + } + replayed := preparePanelReplay(t, root, 1) + if replayed.Spans != 24 || replayed.Logs != 24 || replayed.Metrics != 24 { + t.Fatalf("replayed retired rows: %+v", replayed) + } + if !reflect.DeepEqual(before, panelReplayInventory(t, root)) { + t.Fatal("source lifecycle files changed") + } +} diff --git a/internal/query/panel_bench_test.go b/internal/query/panel_bench_test.go index 93134c3c8..9fb119215 100644 --- a/internal/query/panel_bench_test.go +++ b/internal/query/panel_bench_test.go @@ -77,7 +77,7 @@ func TestMetricReplayCopyPreservesSourceAndStartTime(t *testing.T) { } // All derived Parquet stays under the engine's configured query directory. raw := filepath.Join(cfg.QueryDir(), "shifted.parquet") - metadata := telemetry.BatchMetadata{Version: 4, ID: "shifted"} + metadata := telemetry.BatchMetadata{Version: panelBenchFormat, ID: "shifted"} if err := tilePanelParquet(filepath.Join(input, "parquet", "batches", "metric.batch", "metrics.parquet"), raw, "metrics", "", fixtureStart, fixtureStart.Add(time.Minute), time.Hour, &metadata, map[string]bool{}, map[string]bool{}); err != nil { t.Fatal("cannot shift private metric fixture") } @@ -116,12 +116,28 @@ func BenchmarkPanelQueries24Hours(b *testing.B) { data = preparePanelRegression(b) } engine, cfg := panelBenchEngine(b, data) + metadata, err := panelReplayMetadata(cfg.TelemetryDir()) + if err != nil { + b.Fatal(err) + } + horizon := metadata[0].MaxIngestedNanos + for _, m := range metadata { + horizon = max(horizon, m.MaxIngestedNanos) + } + panelBenchWarmDashboard(b, engine, horizon) + var rolledSpans, rolledLogs int64 + if err := engine.DB.QueryRowContext(b.Context(), "SELECT coalesce(sum(spans),0),coalesce(sum(log_count),0) FROM service_rollup").Scan(&rolledSpans, &rolledLogs); err != nil { + b.Fatal(err) + } + if rolledSpans != data.Spans || rolledLogs != data.Logs { + b.Fatalf("unprepared analytical summaries: spans=%d/%d logs=%d/%d", rolledSpans, data.Spans, rolledLogs, data.Logs) + } var version, memory string var threads int if err := engine.DB.QueryRowContext(b.Context(), "SELECT version(),current_setting('memory_limit'),current_setting('threads')").Scan(&version, &memory, &threads); err != nil { b.Fatal(err) } - b.Logf("representative=%t platform=%s/%s engine=%s resolved_memory=%s configured_threads=%d effective_threads=%d connections=%d executor_parallel=4", input != "", runtime.GOOS, runtime.GOARCH, version, memory, cfg.DuckDBThreads, threads, cfg.DuckDBMaxConns) + b.Logf("representative=%t platform=%s/%s engine=%s resolved_memory=%s configured_threads=%d effective_threads=%d connections=%d executor_parallel=4 result_cache_ttl=0 analytical_summaries=warm", input != "", runtime.GOOS, runtime.GOARCH, version, memory, cfg.DuckDBThreads, threads, cfg.DuckDBMaxConns) raw, err := json.Marshal(data) if err != nil { b.Fatal(err) @@ -211,6 +227,8 @@ func panelBenchmarkMetricShapes(tb testing.TB, names map[string]string) []Panel func measurePanelBenchmark(b *testing.B, engine Engine, spec Dashboard, end time.Time) float64 { b.StopTimer() e := NewExecutor(engine, 30) + e.SetCacheTTL(0) // Measure executor/engine work on each sample. + b.Cleanup(e.Close) e.now = func() time.Time { return end } run := func() { results, err := e.Run(b.Context(), RunRequest{Dashboard: spec}) @@ -320,7 +338,7 @@ func TestPanelBenchmarkReplayTilesSignedTimesAndDistinctIdentities(t *testing.T) if err := os.MkdirAll(dst, 0o755); err != nil { t.Fatal(err) } - m := telemetry.BatchMetadata{Version: 4, ID: "signed"} + m := telemetry.BatchMetadata{Version: panelBenchFormat, ID: "signed"} for _, signal := range []string{"spans", "logs", "metrics"} { err := tilePanelParquet(filepath.Join(seed, "parquet", "batches", "signed.batch", signal+".parquet"), filepath.Join(dst, signal+".parquet"), signal, "source_", start, start.Add(time.Hour), -3*time.Hour, &m, map[string]bool{}, map[string]bool{}) if err != nil { diff --git a/internal/query/panel_work_bench_test.go b/internal/query/panel_work_bench_test.go new file mode 100644 index 000000000..db8687a53 --- /dev/null +++ b/internal/query/panel_work_bench_test.go @@ -0,0 +1,99 @@ +//go:build readbench + +package query + +import ( + "context" + "crypto/sha256" + "fmt" + "sort" + "sync" + "testing" + "time" + + "github.com/labstack/pitot/internal/engine" + "github.com/labstack/pitot/internal/queryrows" +) + +// Recording includes result consumption and preserves the concrete engine's +// optional interfaces. It is diagnostic instrumentation, not a latency gate. +type panelWorkEngine struct { + *engine.Duck + mu sync.Mutex + work map[string]*panelWorkStatement +} + +type panelWorkStatement struct { + sql string + count int + elapsed time.Duration +} + +func (e *panelWorkEngine) QueryContext(ctx context.Context, sql string, args ...any) (queryrows.Rows, error) { + started := time.Now() + rows, err := e.Duck.QueryContext(ctx, sql, args...) + if err != nil { + return nil, err + } + return &panelWorkRows{Rows: rows, done: func() { + e.mu.Lock() + defer e.mu.Unlock() + key := fmt.Sprintf("%x", sha256.Sum256([]byte(sql)))[:12] + statement := e.work[key] + if statement == nil { + statement = &panelWorkStatement{sql: sql} + e.work[key] = statement + } + statement.count++ + statement.elapsed += time.Since(started) + }}, nil +} + +type panelWorkRows struct { + queryrows.Rows + once sync.Once + done func() +} + +func (r *panelWorkRows) Close() error { + err := r.Rows.Close() + r.once.Do(r.done) + return err +} + +// Run with a small fixed count to locate repeated statements before choosing +// an optimization. Use the ordinary panel benchmark for acceptance timings. +func BenchmarkPanelReadWork(b *testing.B) { + data := preparePanelRegression(b) + d, cfg := panelBenchEngine(b, data) + metadata, err := panelReplayMetadata(cfg.TelemetryDir()) + if err != nil { + b.Fatal(err) + } + horizon := metadata[0].MaxIngestedNanos + for _, batch := range metadata { + horizon = max(horizon, batch.MaxIngestedNanos) + } + panelBenchWarmDashboard(b, d, horizon) + dashboard := panelBenchmarkDashboard(data) + for _, panel := range dashboard.Panels { + b.Run(panel.ID, func(b *testing.B) { + recorder := &panelWorkEngine{Duck: d, work: map[string]*panelWorkStatement{}} + spec := dashboard + spec.Panels = []Panel{panel} + if panel.ID != "traces_selected" { + spec.Variables = nil + } + measurePanelBenchmark(b, recorder, spec, data.End) + keys := make([]string, 0, len(recorder.work)) + for key := range recorder.work { + keys = append(keys, key) + } + sort.Strings(keys) + for _, key := range keys { + statement := recorder.work[key] + b.Logf("statement=%s count=%d total_ms=%.3f sql=%s", key, statement.count, float64(statement.elapsed)/float64(time.Millisecond), statement.sql) + } + }) + } +} diff --git a/internal/query/result_cache.go b/internal/query/result_cache.go index b959f6279..a30eb91f4 100644 --- a/internal/query/result_cache.go +++ b/internal/query/result_cache.go @@ -78,11 +78,16 @@ type cacheEntry struct { // nobody's abandoned work holds the engine. A background refresh has no // callers and runs to completion. type cacheFlight struct { - done chan struct{} - stamp time.Time - value any - err error - waiters int + done chan struct{} + stamp time.Time + // Fixed-window computations may be registered well after their initiating + // caller arrived, following an older flight. Keep that freshness boundary + // separate from the original stamp used for window identity and TTL age. + registeredAt time.Time + fixedWindow bool + value any + err error + waiters int // detached runs outlive their callers once started, that is once they // hold a read slot. One still waiting in line when its last caller // leaves is cancelled, so abandoned requests cannot fill the queue. @@ -160,6 +165,12 @@ func keyOf(v any) (cacheKey, error) { // shared: callers must not mutate it. The outcome is one of the cache* lookup // results. func loadCached[T any](e *Executor, ctx context.Context, key cacheKey, stamp time.Time, fn flightFunc[T]) (T, string, error) { + return loadCachedForWindow(e, ctx, key, stamp, false, fn) +} + +// Only callers with a clock-independent query window opt in. Relative windows +// and other generic cached computations retain their original stamp semantics. +func loadCachedForWindow[T any](e *Executor, ctx context.Context, key cacheKey, stamp time.Time, fixedWindow bool, fn flightFunc[T]) (T, string, error) { var zero T c := &e.cache run := func(ctx context.Context) (any, int64, bool, error) { return fn(ctx) } @@ -182,7 +193,7 @@ retry: c.lru.MoveToFront(element) // Never duplicate work already running or waiting for admission. if c.flights[key] == nil { - e.startFlight(ctx, key, stamp, true, run) + e.startFlight(ctx, key, stamp, true, fixedWindow, run) } c.mu.Unlock() metrics.QueryCacheRequests.WithLabelValues(cacheStale).Inc() @@ -195,11 +206,11 @@ retry: flight := c.flights[key] if flight == nil { outcome = cacheMiss - flight = e.startFlight(ctx, key, stamp, false, run) + flight = e.startFlight(ctx, key, stamp, false, fixedWindow, run) } // A bypass caller must wait for older work to finish before requesting // a fresh run. Keep its place as a waiter without accepting that answer. - accept := e.joinable(flight, stamp, bypass) + accept := e.joinable(flight, stamp, bypass, fixedWindow) flight.waiters++ // A caller that will read the stored result keeps the run going. flight.detached = flight.detached || c.ttl > 0 && !bypass @@ -232,8 +243,9 @@ retry: } type panelCacheLookup struct { - key cacheKey - id string + key cacheKey + id string + fixedWindow bool } // availablePanels pins a whole refresh's existing answers or flights. It adds @@ -265,7 +277,7 @@ func (e *Executor) availablePanels(ctx context.Context, lookups []panelCacheLook } } flight := c.flights[lookup.key] - if flight == nil || !e.joinable(flight, stamp, bypass) { + if flight == nil || !e.joinable(flight, stamp, bypass, lookup.fixedWindow) { c.mu.Unlock() return nil, false, nil } @@ -316,16 +328,19 @@ func (e *Executor) availablePanels(ctx context.Context, lookups []panelCacheLook // joinable reports whether a caller may take flight's answer. Non-bypass // callers always share in-flight work; the TTL applies only to stored results. -// A no-cache caller accepts only a run whose window ends at or after its -// arrival. c.mu is held. -func (e *Executor) joinable(flight *cacheFlight, stamp time.Time, bypass bool) bool { - return !bypass || !flight.stamp.Before(stamp) +// A no-cache caller requires the original window stamp to reach its arrival. +// For fixed windows, registration after arrival also proves freshness: the +// window is identical and every read snapshot is acquired after registration. +// c.mu is held. +func (e *Executor) joinable(flight *cacheFlight, stamp time.Time, bypass, fixedWindow bool) bool { + return !bypass || !flight.stamp.Before(stamp) || + fixedWindow && flight.fixedWindow && !flight.registeredAt.Before(stamp) } // startFlight registers the only run for key and stores its result under // stamp. c.mu is held and no run for key may be in flight. The run keeps the // caller's values but not its cancellation, and ends with the executor. -func (e *Executor) startFlight(caller context.Context, key cacheKey, stamp time.Time, background bool, fn flightFunc[any]) *cacheFlight { +func (e *Executor) startFlight(caller context.Context, key cacheKey, stamp time.Time, background, fixedWindow bool, fn flightFunc[any]) *cacheFlight { releaseBatch := func() {} if batch, ok := caller.Value(panelBatchKey{}).(*panelBatchAdmission); ok { releaseBatch = batch.retainFlight() @@ -333,6 +348,9 @@ func (e *Executor) startFlight(caller context.Context, key cacheKey, stamp time. c := &e.cache run, cancel := context.WithCancel(context.WithoutCancel(caller)) flight := &cacheFlight{done: make(chan struct{}), stamp: stamp, detached: background, background: background, cancel: cancel} + if fixedWindow { + flight.fixedWindow, flight.registeredAt = true, e.now() + } run = context.WithValue(run, flightContextKey{}, flight) c.flights[key] = flight e.mu.Lock() diff --git a/justfile b/justfile index b04763d92..30747d87a 100644 --- a/justfile +++ b/justfile @@ -183,6 +183,10 @@ vet: lint: bash scripts/with-duckdb.sh golangci-lint run +# Compile the opt-in read and file harnesses without heavyweight fixtures. +readbench-check: + bash scripts/with-duckdb.sh go test -tags=readbench,filebench ./internal/query ./internal/engine -run '^$' + test *ARGS='./...': bash scripts/with-duckdb.sh go test {{ARGS}} diff --git a/scripts/dashboard-eval/testdata/server.json b/scripts/dashboard-eval/testdata/server.json index 35d3eea40..93f13d6b1 100644 --- a/scripts/dashboard-eval/testdata/server.json +++ b/scripts/dashboard-eval/testdata/server.json @@ -42,7 +42,7 @@ 0.9958502021602336 ] }, - "sql": "WITH base AS (SELECT * FROM span_rollup WHERE \"bucket\" \u003e= ?::TIMESTAMP_NS AND \"bucket\" \u003c ?::TIMESTAMP_NS AND ((service = 'checkout')) AND ((namespace = 'shop'))), __panel_hist_aggregate AS (SELECT epoch_ms(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS))::BIGINT AS \"_t\",NULL::DOUBLE AS \"p95\",grouping(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)) AS __hist_group,time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,count(*) AS __hist_rows FROM base GROUP BY GROUPING SETS ((time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)),())), __panel_hist_bins AS (SELECT grouping(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)) AS __hist_group,time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,e.key AS __hist_k,sum(e.value)::DOUBLE AS __hist_n FROM base,unnest(map_entries(hist)) entries(e) GROUP BY GROUPING SETS ((time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS),e.key),(e.key))), __panel_hist_cdf AS (SELECT *,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0 ORDER BY __hist_k) AS __hist_cum,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0) AS __hist_total FROM __panel_hist_bins), __panel_hist_quantiles AS (SELECT __hist_group,__hist_key_0,min(CASE WHEN __hist_cum \u003e= 0.95*__hist_total THEN CASE WHEN __hist_k=-2147483648 THEN 0.0 ELSE pow(2.0,(__hist_k-1)/8.0)+(pow(2.0,__hist_k/8.0)-pow(2.0,(__hist_k-1)/8.0))*greatest(0.0,least(1.0,(0.95*__hist_total-(__hist_cum-__hist_n))/__hist_n)) END END) AS \"p95\" FROM __panel_hist_cdf GROUP BY __hist_group,__hist_key_0), __panel_hist AS (SELECT a.\"_t\" AS \"_t\",q.\"p95\" AS \"p95\" FROM __panel_hist_aggregate a LEFT JOIN __panel_hist_quantiles q ON a.__hist_group IS NOT DISTINCT FROM q.__hist_group AND a.__hist_key_0 IS NOT DISTINCT FROM q.__hist_key_0) SELECT \"_t\", \"p95\" FROM __panel_hist ORDER BY \"_t\" NULLS FIRST LIMIT 18000", + "sql": "WITH base AS (SELECT * FROM span_rollup WHERE \"bucket\" \u003e= ?::TIMESTAMP_NS AND \"bucket\" \u003c ?::TIMESTAMP_NS AND ((service = 'checkout')) AND ((namespace = 'shop'))), __panel_hist_aggregate AS (SELECT epoch_ms(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS))::BIGINT AS \"_t\",NULL::DOUBLE AS \"p95\",grouping(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)) AS __hist_group,time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,count(*) AS __hist_rows FROM base GROUP BY GROUPING SETS ((time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)),())), __panel_hist_bins AS (SELECT grouping(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)) AS __hist_group,time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,e.key AS __hist_k,sum(e.value)::DOUBLE AS __hist_n FROM (SELECT *,unnest(map_entries(hist)) AS e FROM base) __hist_entries GROUP BY GROUPING SETS ((time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS),e.key),(e.key))), __panel_hist_cdf AS (SELECT *,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0 ORDER BY __hist_k) AS __hist_cum,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0) AS __hist_total FROM __panel_hist_bins), __panel_hist_quantiles AS (SELECT __hist_group,__hist_key_0,min(CASE WHEN __hist_cum \u003e= 0.95*__hist_total THEN CASE WHEN __hist_k=-2147483648 THEN 0.0 ELSE pow(2.0,(__hist_k-1)/8.0)+(pow(2.0,__hist_k/8.0)-pow(2.0,(__hist_k-1)/8.0))*greatest(0.0,least(1.0,(0.95*__hist_total-(__hist_cum-__hist_n))/__hist_n)) END END) AS \"p95\" FROM __panel_hist_cdf GROUP BY __hist_group,__hist_key_0), __panel_hist AS (SELECT a.\"_t\" AS \"_t\",q.\"p95\" AS \"p95\" FROM __panel_hist_aggregate a LEFT JOIN __panel_hist_quantiles q ON a.__hist_group IS NOT DISTINCT FROM q.__hist_group AND a.__hist_key_0 IS NOT DISTINCT FROM q.__hist_key_0) SELECT \"_t\", \"p95\" FROM __panel_hist ORDER BY \"_t\" NULLS FIRST LIMIT 18000", "interval": "5m", "elapsed_ms": 0, "from_ms": 1790856000000, @@ -379,7 +379,7 @@ 0.9958502021602336 ] }, - "sql": "WITH base AS (SELECT * FROM span_rollup WHERE \"bucket\" \u003e= ?::TIMESTAMP_NS AND \"bucket\" \u003c ?::TIMESTAMP_NS AND ((service = 'checkout')) AND ((namespace = 'shop'))), __panel_hist_aggregate AS (SELECT epoch_ms(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS))::BIGINT AS \"_t\",NULL::DOUBLE AS \"p95\",grouping(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)) AS __hist_group,time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,count(*) AS __hist_rows FROM base GROUP BY GROUPING SETS ((time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)),())), __panel_hist_bins AS (SELECT grouping(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)) AS __hist_group,time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,e.key AS __hist_k,sum(e.value)::DOUBLE AS __hist_n FROM base,unnest(map_entries(hist)) entries(e) GROUP BY GROUPING SETS ((time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS),e.key),(e.key))), __panel_hist_cdf AS (SELECT *,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0 ORDER BY __hist_k) AS __hist_cum,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0) AS __hist_total FROM __panel_hist_bins), __panel_hist_quantiles AS (SELECT __hist_group,__hist_key_0,min(CASE WHEN __hist_cum \u003e= 0.95*__hist_total THEN CASE WHEN __hist_k=-2147483648 THEN 0.0 ELSE pow(2.0,(__hist_k-1)/8.0)+(pow(2.0,__hist_k/8.0)-pow(2.0,(__hist_k-1)/8.0))*greatest(0.0,least(1.0,(0.95*__hist_total-(__hist_cum-__hist_n))/__hist_n)) END END) AS \"p95\" FROM __panel_hist_cdf GROUP BY __hist_group,__hist_key_0), __panel_hist AS (SELECT a.\"_t\" AS \"_t\",q.\"p95\" AS \"p95\" FROM __panel_hist_aggregate a LEFT JOIN __panel_hist_quantiles q ON a.__hist_group IS NOT DISTINCT FROM q.__hist_group AND a.__hist_key_0 IS NOT DISTINCT FROM q.__hist_key_0) SELECT \"_t\", \"p95\" FROM __panel_hist ORDER BY \"_t\" NULLS FIRST LIMIT 18000", + "sql": "WITH base AS (SELECT * FROM span_rollup WHERE \"bucket\" \u003e= ?::TIMESTAMP_NS AND \"bucket\" \u003c ?::TIMESTAMP_NS AND ((service = 'checkout')) AND ((namespace = 'shop'))), __panel_hist_aggregate AS (SELECT epoch_ms(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS))::BIGINT AS \"_t\",NULL::DOUBLE AS \"p95\",grouping(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)) AS __hist_group,time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,count(*) AS __hist_rows FROM base GROUP BY GROUPING SETS ((time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)),())), __panel_hist_bins AS (SELECT grouping(time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS)) AS __hist_group,time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,e.key AS __hist_k,sum(e.value)::DOUBLE AS __hist_n FROM (SELECT *,unnest(map_entries(hist)) AS e FROM base) __hist_entries GROUP BY GROUPING SETS ((time_bucket(INTERVAL '300 seconds', \"bucket\"::TIMESTAMP_NS),e.key),(e.key))), __panel_hist_cdf AS (SELECT *,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0 ORDER BY __hist_k) AS __hist_cum,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0) AS __hist_total FROM __panel_hist_bins), __panel_hist_quantiles AS (SELECT __hist_group,__hist_key_0,min(CASE WHEN __hist_cum \u003e= 0.95*__hist_total THEN CASE WHEN __hist_k=-2147483648 THEN 0.0 ELSE pow(2.0,(__hist_k-1)/8.0)+(pow(2.0,__hist_k/8.0)-pow(2.0,(__hist_k-1)/8.0))*greatest(0.0,least(1.0,(0.95*__hist_total-(__hist_cum-__hist_n))/__hist_n)) END END) AS \"p95\" FROM __panel_hist_cdf GROUP BY __hist_group,__hist_key_0), __panel_hist AS (SELECT a.\"_t\" AS \"_t\",q.\"p95\" AS \"p95\" FROM __panel_hist_aggregate a LEFT JOIN __panel_hist_quantiles q ON a.__hist_group IS NOT DISTINCT FROM q.__hist_group AND a.__hist_key_0 IS NOT DISTINCT FROM q.__hist_key_0) SELECT \"_t\", \"p95\" FROM __panel_hist ORDER BY \"_t\" NULLS FIRST LIMIT 18000", "interval": "5m", "elapsed_ms": 0, "from_ms": 1790856000000, diff --git a/ui/host/tests/go-fragments/performance.json b/ui/host/tests/go-fragments/performance.json index 85eba151b..79f270663 100644 --- a/ui/host/tests/go-fragments/performance.json +++ b/ui/host/tests/go-fragments/performance.json @@ -134,7 +134,7 @@ ], "rows": 1 }, - "sql": "WITH base AS (SELECT * FROM span_rollup WHERE \"bucket\" \u003e= ?::TIMESTAMP_NS AND \"bucket\" \u003c ?::TIMESTAMP_NS AND ((namespace = 'shop'))), __panel_hist_aggregate AS (SELECT epoch_ms(time_bucket(INTERVAL '60 seconds', \"bucket\"::TIMESTAMP_NS))::BIGINT AS \"_t\",NULL::DOUBLE AS \"p95\",0 AS __hist_group,time_bucket(INTERVAL '60 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,count(*) AS __hist_rows FROM base GROUP BY time_bucket(INTERVAL '60 seconds', \"bucket\"::TIMESTAMP_NS)), __panel_hist_bins AS (SELECT 0 AS __hist_group,time_bucket(INTERVAL '60 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,e.key AS __hist_k,sum(e.value)::DOUBLE AS __hist_n FROM base,unnest(map_entries(hist)) entries(e) GROUP BY time_bucket(INTERVAL '60 seconds', \"bucket\"::TIMESTAMP_NS),e.key), __panel_hist_cdf AS (SELECT *,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0 ORDER BY __hist_k) AS __hist_cum,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0) AS __hist_total FROM __panel_hist_bins), …", + "sql": "WITH base AS (SELECT * FROM span_rollup WHERE \"bucket\" \u003e= ?::TIMESTAMP_NS AND \"bucket\" \u003c ?::TIMESTAMP_NS AND ((namespace = 'shop'))), __panel_hist_aggregate AS (SELECT epoch_ms(time_bucket(INTERVAL '60 seconds', \"bucket\"::TIMESTAMP_NS))::BIGINT AS \"_t\",NULL::DOUBLE AS \"p95\",0 AS __hist_group,time_bucket(INTERVAL '60 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,count(*) AS __hist_rows FROM base GROUP BY time_bucket(INTERVAL '60 seconds', \"bucket\"::TIMESTAMP_NS)), __panel_hist_bins AS (SELECT 0 AS __hist_group,time_bucket(INTERVAL '60 seconds', \"bucket\"::TIMESTAMP_NS) AS __hist_key_0,e.key AS __hist_k,sum(e.value)::DOUBLE AS __hist_n FROM (SELECT *,unnest(map_entries(hist)) AS e FROM base) __hist_entries GROUP BY time_bucket(INTERVAL '60 seconds', \"bucket\"::TIMESTAMP_NS),e.key), __panel_hist_cdf AS (SELECT *,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0 ORDER BY __hist_k) AS __hist_cum,sum(__hist_n) OVER (PARTITION BY __hist_group,__hist_key_0) AS __hist_total …", "interval": "1m", "elapsed_ms": 0, "from_ms": 1791398700000,