Skip to content

perf: stop computing an HLL estimate on every merge and size() call - #7629

Merged
andygrove merged 1 commit into
apache:mainfrom
comphead:hll-estimated-size-7341
Nov 16, 2026
Merged

andygrove merged 1 commit into
apache:mainfrom
comphead:hll-estimated-size-7341

Conversation

@comphead

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #7341. #7341 stays open for item 2 (a GroupsAccumulator) and for the datasketches bump once a release includes apache/datasketches-rust#317.

Rationale for this change

#7345 fixed the problem in the title of #7341. Item 3 is still real on main: merging a partial sketch that is already dense costs about 270 µs at lgConfigK 18, while the register-wise max it needs takes 5 µs. datasketches 0.6.0 rebuilds its register cache, one f64 division per register, after every dense merge.

apache/datasketches-rust#317 (merged 2026-11-04, not released yet) defers that rebuild until the next query. Comet would not benefit from it. SparkHllUnion::merge calls estimate() on the union and on the incoming sketch before every update, to keep a dense flag for the heap-size inference, and heap_size() calls estimate() on every accumulator size(). Each of those calls would force the deferred rebuild.

datasketches 0.6.0 exposes the size directly. HllSketch::estimated_size() and HllUnion::estimated_size() read the coupon array or register array length in O(1), and the result is exact. The estimate-based inference predates that API, since 0.3.0 kept the layout private.

What changes are included in this PR?

  • SparkHllSketch::heap_size() and SparkHllUnion::heap_size() return the crate's estimated_size().
  • The layout module, both dense flags and the preamble module are removed, so merge and size() no longer compute estimates.
  • heap_size_follows_the_crates_layout now asserts that heap_size() equals the allocation at every step, instead of "never below, at most one size step above".

How are these changes tested?

The existing heap-size tests cover it. heap_size_follows_the_crates_layout now requires an exact match with the allocation the crate serializes at every cardinality, through each SET resize and the promotion. high_lg_config_k_is_charged_for_what_it_holds, union_heap_size_follows_the_crates_layout and singleton_groups_at_high_lg_config_k_fit_a_small_pool are unchanged. Sketch results do not change: only the size accounting is touched, and the probe below gave identical output bytes in every build.

The numbers come from a throwaway example (not in this PR) that runs a Final AggregateExec over partial states at lgConfigK 18, with rows interleaved one partial per group per batch, the way a reducer reads map outputs. Release build on an Apple M-series laptop, hll_sketch_agg, ns per partial row, median of 5 runs over two interleaved rounds. hll_union_agg gives the same picture.

With datasketches 0.6.0, as shipped by this PR:

Case main This PR
Tiny: 200K groups x 8 partials, 0 to 8 values 211 169
SET: 2K groups x 100 partials, 50 values 1,453 1,450
Small partials into dense groups: 20 x 400, 200 values 3,431 3,445
Dense partials: 8 groups x 32, 30,000 values 359,000 361,000

With datasketches at apache/datasketches-rust@5f080f98e5 (#317, still version 0.6.0, through [patch.crates-io], not part of this PR):

Case main This PR
Dense partials, per row 351,000 ns 24,200 ns
One SparkHllUnion::merge of a dense sketch 266 µs 5.5 µs

A plain 2^18 register-wise max takes 5.2 µs, so with #317 a dense merge costs about what the merge itself has to do. Differences under 5% are within run-to-run noise here.

@github-actions github-actions Bot added enhancement New feature or request performance area:aggregation Hash aggregates, aggregate expressions labels Nov 14, 2026
@comphead
comphead requested review from andygrove and viirya November 14, 2026 22:54
`SparkHllSketch` and `SparkHllUnion` inferred their heap size from
`estimate()`, and `SparkHllUnion::merge` called `estimate()` on both
sides before every update to keep a `dense` flag for that inference.
`datasketches` 0.6.0 exposes the size directly: `estimated_size()`
reads the coupon or register array length in O(1) and is exact. Use
it, and drop the `layout` inference, both `dense` flags and the
`preamble` module.

On 0.6.0 this makes Final-mode `hll_sketch_agg` and `hll_union_agg`
over many tiny groups about 20% faster (211 to 169 ns per partial row
at lgConfigK 18). It matters more once Comet moves to a datasketches
release with apache/datasketches-rust#317, which defers the register
cache rebuild after a dense merge until the next query. The
per-merge `estimate()` forced that rebuild on every merge. Built
against apache#317, merging an already dense sketch drops from 266 us to
5.5 us, close to the 5.2 us a plain register-wise max takes.

Part of apache#7341.
@comphead
comphead force-pushed the hll-estimated-size-7341 branch from ef10fc2 to ffa0694 Compare November 15, 2026 19:22

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @comphead. Using the crate's estimated_size() is a nice simplification, and the exact-match assertion in heap_size_follows_the_crates_layout is a stronger test than the old bound. I checked 0.6.0's implementation. It reads the boxed coupon slice or the register array length, and the Estimator holds no heap, so nothing is missed.

I ran the hll_agg bench on x86 against the merge base, alternating over three rounds. hll_sketch_agg drops from about 1.95 ms to 1.71 ms. hll_union_agg and hll_sketch_agg_final_dense are unchanged within noise, which matches your note that the dense path waits on the next datasketches release with apache/datasketches-rust#317. LGTM.

@andygrove
andygrove added this pull request to the merge queue Nov 16, 2026
Merged via the queue into apache:main with commit 34e5dcf Nov 16, 2026
24 checks passed

@viirya viirya left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice cleanup. Two small follow-ups inline, both fine to fold into the PR that bumps datasketches to a release with apache/datasketches-rust#317.

Comment on lines +123 to +124
/// Merge a sketch into the union without querying it. Querying after each dense merge would
/// rebuild the register cache, which apache/datasketches-rust#317 defers until it is needed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On 0.6.0 the rebuild happens inside update (merge_array_same_lgk calls rebuild_estimator_from_registers), and estimate() only reads the cache, so this reads as if querying is what triggers it today. Maybe:

/// Merge a sketch into the union without querying it. With apache/datasketches-rust#317 the
/// register cache rebuild after a dense merge waits for the next query, and an `estimate()`
/// here would force it on every merge.

Comment on lines +353 to +354
/// register array promotes it however little that sketch has seen, so the charge has to follow
/// the gadget's mode, not its estimate.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hand-edited preamble below existed to defeat the estimate-based mode inference, which is gone. Since heap_size() now reads the container, merging any dense sketch and asserting 1 << 16 covers the same promotion, without tying the test to the HLL preamble offsets. The "not its estimate" here can go with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:aggregation Hash aggregates, aggregate expressions enhancement New feature or request performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants