Skip to content

perf: keep an idle shared broadcast decode for the next wave of tasks - #7655

Open
comphead wants to merge 2 commits into
apache:mainfrom
comphead:broadcast-decode-idle-timeout
Open

comphead wants to merge 2 commits into
apache:mainfrom
comphead:broadcast-decode-idle-timeout

Conversation

@comphead

@comphead comphead commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7487.

Part of #6013.

Rationale for this change

#7016 lets the tasks of an executor that read a Comet broadcast at the same time share one decoded copy of it, and
frees the copy when the last of them completes. The tasks of a wave finish at about the same time, so the copy is gone
before the next wave starts, and every wave decodes the broadcast again. Every task of a wave also waits for that decode.

Counters added temporarily to SharedDecode show this on main, with the bhj mode of a hash-join task-time
benchmark: a 1M-row broadcast probed by 32 tasks on local[8], so 4 waves, lz4, 7 runs. Every run had 32 reads, 4
decodes and 4 frees. 28 tasks waited about 14 ms each, which is the time of the decode they waited for. Each new decode
started 0.6 to 1.3 ms after the copy before it was freed.

What changes are included in this PR?

  • A new config, spark.comet.exec.broadcastExchange.shareDecoded.idleTimeout, 1 second by default. When the last task
    holding the decoded copy completes, the executor keeps the copy for that long, and a task that starts by then reads
    it instead of decoding the broadcast again. A timeout of 0 frees the copy at once, as before.
  • SharedDecode.release keeps an entry that no task holds, and schedules its expiry on one daemon thread. Each entry
    counts its idle periods, so an expiry scheduled before a task took the entry again frees nothing, and an entry is
    never freed while a task holds it. An entry whose decode failed is still dropped at once.
  • A second config, spark.comet.exec.broadcastExchange.shareDecoded.maxIdleSize, 128 MiB by default, caps the bytes
    of the idle copies of all broadcasts on an executor. A copy larger than the cap is freed as soon as no task holds it,
    and keeping a copy frees the copies that have been idle longest until the rest fit. SharedDecode.IdleDecodes keeps
    the idle copies oldest first. It takes no other lock, and evicted copies are freed outside every lock.
  • An expiry frees an entry only while it is still the one that the broadcast hands out, so an expiry scheduled before
    the cap or a read with a timeout of 0 freed the entry frees nothing.
  • CometBroadcastExchangeExec reads both configs on the driver, as it does shareDecoded.enabled, and passes them
    through CometBatchRDD as a SharedDecode.Retention. batches has no default for it.
  • CometTestBase sets the timeout to 0. Some suites compare CometArrowAllocator's memory before and after a step,
    and the idle copy of an earlier test's broadcast could be freed in between.
  • The config docs, the broadcast join section of the tuning guide, and the memory management guide describe the idle
    copy, which is JVM-side Arrow memory that neither Spark nor Comet's pools account for.

Mean time per probe task in the same benchmark, median of 7 runs with the arms interleaved:

codec Spark Comet, this PR Comet, idleTimeout=0 (as main) decodes per run
lz4 7.1 ms 9.0 ms 18.8 ms 4 to 1
zstd, level 5 7.9 ms 12.5 ms 31.8 ms 4 to 1

The first wave still waits for its decode. A cheaper decode would shrink that wait (#2216, #3762). The idle copy in
this benchmark is 31.8 MB. The expiry freed it about 1 second after the last probe task, and the allocator's memory did
not grow across runs.

The default of 1 second is a trade-off. The waves were about 1 ms apart here, and a cluster adds scheduling delay, but a
longer timeout keeps memory that no pool accounts for after the stage that read it.

The idle copies of broadcasts read in consecutive stages stack. A query with four broadcast hash joins in consecutive
stages (4M-row fact in 32 files, four 1.5M-row dims of 24.4 MB decoded each, local[8], AQE off, Spark 4.0, median of
7 runs with the arms interleaved), sampling CometArrowAllocator every millisecond:

arm live shared copies peak shared bytes decodes query time
idleTimeout=0 (as main) 1 24.4 MB 13 657 ms
idleTimeout=1s, no cap 4 97.5 MB 4 556 ms
idleTimeout=1s, cap 128 MiB 4 97.5 MB 4 564 ms
idleTimeout=1s, cap 50 MiB 3 73.1 MB 4 569 ms

The cap bounds what the timeout adds to an executor's peak, the idle copies, while the copy that running tasks hold
is as before. With the 50 MiB cap, keeping a stage's copy frees the oldest idle one, and each broadcast is still
decoded once.

How are these changes tested?

  • A new test in CometBroadcastValueSuite replaces the scheduler with one that the test runs by hand. It checks that a
    task that comes before the timeout reads the copy of earlier tasks without decoding it again, that an expiry
    scheduled before that task came frees nothing, while the task runs or after it completes, that the expiry of the
    last release frees the copy, and that a timeout of 0 frees it at once.
  • A new test runs two tasks one after the other with the real expiry thread and a 500 ms timeout, checks that the
    broadcast is decoded once, and waits for the thread to free the copy.
  • A new test of the cap, with its own IdleDecodes and a hand-run scheduler, checks that keeping a copy frees the one
    idle longest, that a copy over the cap is freed at once, that a task taking an idle copy takes it out of the cap,
    and that the expiries of evicted copies free nothing more.
  • The existing test of broadcast joins over nested columns runs with a 100 ms timeout and waits for the expiry thread
    to free the copy.
  • The benchmark numbers above. The benchmark is not in the tree yet and will come as its own PR.

The tasks of an executor that read a Comet broadcast at the same time
share one decoded copy of it, but the copy was freed as soon as the last
of them completed. The tasks of a wave finish at about the same time, so
every wave of a stage decoded the broadcast again, and every task of a
wave waited for that decode.

Keep the copy for spark.comet.exec.broadcastExchange.shareDecoded.idleTimeout,
1 second by default, after its last task completes, so that a task that
starts by then reads it. One daemon thread frees the copy once the timeout
passes with no task taking it. A timeout of 0 frees it at once, as before.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @comphead. Keeping the decode for the next wave is a good idea, and the benchmark shows a real gain. I read through SharedDecode and could not find a race. acquire and release run under the lock, and expire re-checks holders and idlePeriods under it. A failed decode is forgotten at once, so its other holders close it in release.

My main concern is memory. The idle copy is the whole decoded broadcast in CometArrowAllocator, which no Spark or Comet pool accounts for. 1.2 already has the shared decode, so this adds up to a second of retained container RSS for every 1.2 user by default. The 31.8 MB single-broadcast run does not show whether copies stack. Stage N's copy can still be resident while stage N+1 decodes its own, and a query with several joins in consecutive stages could hold several. Spark's broadcast threshold can also be raised to GB scale. Could you share the peak CometArrowAllocator.getAllocatedMemory for a query with several broadcast joins in consecutive stages, with the timeout at 0 and at 1s? If the copies stack, would a cap on idle bytes per executor, as #7487 suggests, be small enough to include here? If the cap stays out, could the description say Part of #7487 and link a follow-up issue for it? As written, Closes #7487 closes the issue without the cap.

On tests, CometTestBase now sets the timeout to 0 for every suite, so no suite runs the 1s default. The nested-columns test uses the real scheduler at 100ms, but it only checks that the allocator returns to its baseline. It does not check that a later task reused the decode. Could you add a test with the real scheduler and a timeout of a few hundred milliseconds, where two tasks run one after the other and the injected decode function counts decodes? It should assert 1, so the reuse this PR is about is checked end to end. The manual-scheduler test covers the entry bookkeeping well, but it does not cover the thread.

Finally, batches(context, source, idleTimeoutMs: Long = 0) has a default on both CometBroadcastValue and SharedDecode. The only production caller passes the timeout, but a new caller that forgets it silently gets no retention. Would you drop the defaults and have the tests pass the value explicitly?

Address review: the idle copies of broadcasts read in consecutive stages
stack. Add `spark.comet.exec.broadcastExchange.shareDecoded.maxIdleSize`
(128 MiB by default), an executor-wide cap on the bytes of idle decodes.
A decode over the cap is freed at once, and keeping a decode frees the
decodes idle longest until the rest fit.

An expiry now frees an entry only while it is still `current`, so an
expiry scheduled before another path freed the entry frees nothing.

`batches` takes a `SharedDecode.Retention` with no default, so a caller
cannot silently get no retention. Add a test of reuse through the real
expiry thread and a test of the cap.
@comphead

Copy link
Copy Markdown
Contributor Author

Thanks @andygrove, all three points held up when I checked them against the code. Addressed in 51ef4bc.

Memory. The copies do stack. Each broadcast has its own SharedDecode, so stage N's idle copy is still alive while stage N+1 decodes and holds its own. I measured a query with four broadcast hash joins in consecutive stages: a 4M-row fact in 32 files, four 1.5M-row dims of 24.4 MB decoded each, local[8], AQE off, Spark 4.0. I sampled CometArrowAllocator every millisecond and took the median of 7 runs with the arms interleaved:

arm live shared copies peak shared bytes decodes query time
idleTimeout=0 (as main) 1 24.4 MB 13 657 ms
idleTimeout=1s, no cap 4 97.5 MB 4 556 ms
idleTimeout=1s, cap 128 MiB 4 97.5 MB 4 564 ms
idleTimeout=1s, cap 50 MiB 3 73.1 MB 4 569 ms

So without a cap, the timeout adds every broadcast that finished in the last second to the executor's peak. The cap was small enough to include here, so the PR still closes #7487:

  • spark.comet.exec.broadcastExchange.shareDecoded.maxIdleSize, 128 MiB by default, caps the idle copies of all broadcasts on an executor.
  • A copy larger than the cap is freed as soon as no task holds it, so a GB-scale broadcast is never kept idle.
  • Keeping a copy frees the copies that have been idle longest until the rest fit. The newest copy is the one the next wave is about to read.
  • The 50 MiB row shows the cap binding: two idle copies plus the held one, and each broadcast is still decoded once.

The cap bounds only the idle copies. The copy that running tasks hold is unchanged from main.

IdleDecodes never takes a broadcast's lock. The lock order is always the broadcast's lock and then the registry's, and evicted copies are freed outside both. While writing the cap I also found that an idle entry freed by another path, such as a read with a timeout of 0, could later be closed again by its old expiry. That is a double close. An expiry now frees an entry only while it is still current.

Tests. Added "a task that comes after another completes reads its decode, which the timer then frees". It runs two tasks one after the other with the real expiry thread and a 500 ms timeout, asserts one decode, and waits for the thread to free the copy. The cap has its own test with an injected IdleDecodes and a hand-run scheduler.

Defaults. batches on both CometBroadcastValue and SharedDecode now takes a SharedDecode.Retention(idleTimeoutMs, maxIdleBytes) with no default. The tests pass NoRetention or an explicit Retention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: probe tasks wait for the shared broadcast decode, and every wave of tasks decodes again

2 participants