Skip to content

fix(engine): resolve three P0 engine defects (cancelled flights, rollup memory, metadata snapshot) - #488

Merged
vishr merged 5 commits into
mainfrom
fix/engine-p0
Nov 15, 2026
Merged

vishr merged 5 commits into
mainfrom
fix/engine-p0

Conversation

@vishr

@vishr vishr commented Nov 15, 2026

Copy link
Copy Markdown
Member

Fixes three P0 defects surfaced by the engine audit. Each is a standalone commit with its own regression test; just vet is clean and the internal/engine and internal/query packages pass.

1. Reject cancelled result-cache flights from new joiners (#476)

A reader could join an in-flight result-cache computation whose originator had already cancelled, then block until the shared context expired and receive a cancellation error instead of its own result. The flight carried no cancelled state, so joinable() had no way to exclude it.

Fix: add a cancelled flag on cacheFlight, set under c.mu at both cancel sites, and have joinable() reject any flight already marked cancelled. New joiners start their own flight instead.

Test: TestResultCacheRejectsCancelledFlight.

2. Key-filter service-rollup log and metric aggregates (#475)

The service-rollup pass filtered the span aggregate to the affected keys but left the log and metric aggregates unfiltered, so a single late signal pulled the full log/metric scan into memory. Peak RSS for the regression workload was 305 MiB.

Fix: materialize the log and metric aggregates into per-pass temp tables (service_pass_logs, service_pass_metrics) restricted to the affected keys via forAffectedKeys, and rewrite serviceRollupJoinSQL to join against them. A serviceRollupReferenceJoinSQL oracle backs the differential-equivalence tests. Peak RSS drops to 8 MiB with identical output.

Tests: TestServiceLateSignalLogMetricMemory, plus the existing differential-equivalence suite (TestServiceRollupKeysMatchSemiJoin, TestRollupManyKeysBoundedAndEquivalent).

3. Publish a pre-sorted BatchMetadata snapshot (#474)

BatchMetadata() copied and sorted the batch list on every call under the metadata read lock, so hot read paths paid an allocation and sort each time and contended on the lock.

Fix: maintain a metadataSnapshot atomic.Pointer[[]BatchMetadata], rebuilt once in publishMetadataLocked() whenever the batch set changes. BatchMetadata() returns the shared read-only snapshot with no lock, allocation, or sort. The returned slice is now immutable by contract; the two callers that mutated the result (batch_cache.go, annotations.go) now slices.Clone first, and the remaining callers were verified read-only.

Closes #474
Closes #475
Closes #476

When a flight's last caller leaves, its run is cancelled but the flight
stays registered in c.flights until its function returns. A non-bypass
caller arriving in that window joined the dead flight, waited for it, and
received context.Canceled, which exec.go rendered as a failed panel for a
live caller.

Mark the flight cancelled under c.mu at both cancel sites and have
joinable (used by loadCached and availablePanels) reject a cancelled
flight. The newcomer now waits for the dead run to clear and then runs its
own query, getting a real answer. The existing fixedWindow joinable logic
and the "keep the flight until its function exits" invariant are preserved.
The service rollup derives one event-time hull [min(bucket), max(bucket)+1m)
from the affected service-minutes and every statement scans it. The span
path was already filtered to the affected keys, but log_agg and metric_agg
scanned and grouped that whole hull with no key filter. A single late or
clock-skewed span pulled the hull back by days, so the pass aggregated every
service-minute of dense history, blew memory, held the write gate, and left
the watermark frozen because the failed pass kept re-including the same span.

Materialize the log and metric aggregates into temp tables the same way the
span path does: probe the constant affected-key list over the same bounded
event-time range, so one old bucket can no longer force a full-hull scan and
group. The final join now reads the three materialized aggregates. Aggregates
stay exact (keys are distinct, so each service-minute is in one chunk) and the
serialized-delete, grouped-insert uniqueness invariant is unchanged.

A new late-signal memory test over a dense 20-day hull drops peak RSS growth
from ~305 MiB to ~8 MiB.
BatchMetadata() allocated and sorted the entire catalog under p.mu.RLock()
on every call. It is called up to ~8 times per panel and by compaction
selection, so readers serialized on the lock and a pending writer queued
behind them, all to recompute an order that only changes when the catalog
changes.

Rebuild the canonically sorted slice once in setBatchLocked, the sole catalog
mutation (already under mu), and publish it through an atomic pointer.
BatchMetadata() now returns that shared snapshot with no lock, allocation, or
sort. The order (generation, min ingest time, ID) and pin/capture semantics
are unchanged.

The returned slice is now shared and read-only. Two callers sorted it in
place (batch_cache and annotations); both now clone before sorting, and the
contract is documented on BatchMetadata.
…g test

BatchMetadata now returns a shared, immutable snapshot. The resource
inventory pruning test mutated batches[0].SpanTime in place to exercise
the unknown-bounds path, which corrupted the shared catalog for the
bindSnapshot call later in the test and broke signal-specific pruning.
Clone the slice first so the mutation stays local, matching the
contract the production callers follow.
@vishr
vishr added this pull request to the merge queue Nov 15, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 15, 2026
@vishr
vishr added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit 77fa266 Nov 15, 2026
31 checks passed
@vishr
vishr deleted the fix/engine-p0 branch November 15, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant