db,collections: columnar two-pass top-K (ORDER BY <numeric> LIMIT k at MAX speed)

[bbockelm]

Makes ORDER BY <numeric attr> {DESC|ASC} LIMIT k as fast as the equivalent MAX/MIN. Based on main.

Problem

db.TopK fed QueryProject — it built a classad.Value for every matching row (e.g. ~1M for ProcId < 5), heaped them, and kept k. On a production archive ORDER BY ClusterId DESC LIMIT 1 took 9.1s while the logically identical MAX(ClusterId) WHERE ProcId < 5 took 2.5s, because MAX reduces raw column values with no per-row allocation and TopK allocated a million Values to discard.

Fix: two-pass columnar top-K

1. Cutoff (columnar). Collection.TopKOrderThreshold scans the order column the same way the numeric aggregate does — block-zone pruning, MVCC visibility, cold-tail escape handling, active-segment row fallback — but keeps the best k values and returns the k-th (the cutoff). It reads only floats; it never materializes a record. Same availability gate as NumStatsQuery (accelerator on, numeric schema field, conjunction of scalar numeric comparisons).
2. Fetch (narrowed). db.TopK re-runs the projection with orderAttr >= cutoff (desc) / <= cutoff (asc) appended. The columns and zone maps narrow that to ~k rows (plus ties), so only those are projected and heaped.

Falls back to the original full-scan path unchanged when the cutoff is unavailable (accelerator off, non-numeric order key, non-columnar constraint) or when there are ≤k matching rows, so behavior is never worse than before. Capped at k ≤ 65536 (beyond that "top-k" ≈ full sort). Both the archive and mutable-table paths use it.

Expected: ORDER BY ClusterId DESC LIMIT 1 WHERE ProcId < 5 drops from ~9s to ~MAX speed.

Correctness

db.TestArchiveTopKColumnarMatchesBrute compares the two-pass to an independent brute-force top-K across desc/asc, k ∈ {1, 3, 50, >matches}, four filters (including one that excludes the row holding the global max, forcing the cutoff to reflect the filtered max), on both a columnarized archive and a non-columnarized one — they must agree with each other and with brute. It also checks argmax reassembly (selecting a non-order column with LIMIT 1 returns that column from the actual winning row). Mutation-checked: perturbing the cutoff makes the columnar case fail, proving the fast path is exercised, not silently bypassed. collections.TestTopKOrderThreshold pins the cutoff value + match count directly. Full db suite and the columnar collections suite pass.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!