Repository navigation
fix(engine): resolve three P0 engine defects (cancelled flights, rollup memory, metadata snapshot) - #488
Merged
Conversation
When a flight's last caller leaves, its run is cancelled but the flight stays registered in c.flights until its function returns. A non-bypass caller arriving in that window joined the dead flight, waited for it, and received context.Canceled, which exec.go rendered as a failed panel for a live caller. Mark the flight cancelled under c.mu at both cancel sites and have joinable (used by loadCached and availablePanels) reject a cancelled flight. The newcomer now waits for the dead run to clear and then runs its own query, getting a real answer. The existing fixedWindow joinable logic and the "keep the flight until its function exits" invariant are preserved.
The service rollup derives one event-time hull [min(bucket), max(bucket)+1m) from the affected service-minutes and every statement scans it. The span path was already filtered to the affected keys, but log_agg and metric_agg scanned and grouped that whole hull with no key filter. A single late or clock-skewed span pulled the hull back by days, so the pass aggregated every service-minute of dense history, blew memory, held the write gate, and left the watermark frozen because the failed pass kept re-including the same span. Materialize the log and metric aggregates into temp tables the same way the span path does: probe the constant affected-key list over the same bounded event-time range, so one old bucket can no longer force a full-hull scan and group. The final join now reads the three materialized aggregates. Aggregates stay exact (keys are distinct, so each service-minute is in one chunk) and the serialized-delete, grouped-insert uniqueness invariant is unchanged. A new late-signal memory test over a dense 20-day hull drops peak RSS growth from ~305 MiB to ~8 MiB.
BatchMetadata() allocated and sorted the entire catalog under p.mu.RLock() on every call. It is called up to ~8 times per panel and by compaction selection, so readers serialized on the lock and a pending writer queued behind them, all to recompute an order that only changes when the catalog changes. Rebuild the canonically sorted slice once in setBatchLocked, the sole catalog mutation (already under mu), and publish it through an atomic pointer. BatchMetadata() now returns that shared snapshot with no lock, allocation, or sort. The order (generation, min ingest time, ID) and pin/capture semantics are unchanged. The returned slice is now shared and read-only. Two callers sorted it in place (batch_cache and annotations); both now clone before sorting, and the contract is documented on BatchMetadata.
…g test BatchMetadata now returns a shared, immutable snapshot. The resource inventory pruning test mutated batches[0].SpanTime in place to exercise the unknown-bounds path, which corrupted the shared catalog for the bindSnapshot call later in the test and broke signal-specific pruning. Clone the slice first so the mutation stays local, matching the contract the production callers follow.
vishr
enabled auto-merge
November 15, 2026 05:11
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes three P0 defects surfaced by the engine audit. Each is a standalone commit with its own regression test;
just vetis clean and theinternal/engineandinternal/querypackages pass.1. Reject cancelled result-cache flights from new joiners (#476)
A reader could join an in-flight result-cache computation whose originator had already cancelled, then block until the shared context expired and receive a cancellation error instead of its own result. The flight carried no cancelled state, so
joinable()had no way to exclude it.Fix: add a
cancelledflag oncacheFlight, set underc.muat both cancel sites, and havejoinable()reject any flight already marked cancelled. New joiners start their own flight instead.Test:
TestResultCacheRejectsCancelledFlight.2. Key-filter service-rollup log and metric aggregates (#475)
The service-rollup pass filtered the span aggregate to the affected keys but left the log and metric aggregates unfiltered, so a single late signal pulled the full log/metric scan into memory. Peak RSS for the regression workload was 305 MiB.
Fix: materialize the log and metric aggregates into per-pass temp tables (
service_pass_logs,service_pass_metrics) restricted to the affected keys viaforAffectedKeys, and rewriteserviceRollupJoinSQLto join against them. AserviceRollupReferenceJoinSQLoracle backs the differential-equivalence tests. Peak RSS drops to 8 MiB with identical output.Tests:
TestServiceLateSignalLogMetricMemory, plus the existing differential-equivalence suite (TestServiceRollupKeysMatchSemiJoin,TestRollupManyKeysBoundedAndEquivalent).3. Publish a pre-sorted BatchMetadata snapshot (#474)
BatchMetadata()copied and sorted the batch list on every call under the metadata read lock, so hot read paths paid an allocation and sort each time and contended on the lock.Fix: maintain a
metadataSnapshot atomic.Pointer[[]BatchMetadata], rebuilt once inpublishMetadataLocked()whenever the batch set changes.BatchMetadata()returns the shared read-only snapshot with no lock, allocation, or sort. The returned slice is now immutable by contract; the two callers that mutated the result (batch_cache.go,annotations.go) nowslices.Clonefirst, and the remaining callers were verified read-only.Closes #474
Closes #475
Closes #476