Repository navigation
Conversation
The tasks of an executor that read a Comet broadcast at the same time share one decoded copy of it, but the copy was freed as soon as the last of them completed. The tasks of a wave finish at about the same time, so every wave of a stage decoded the broadcast again, and every task of a wave waited for that decode. Keep the copy for spark.comet.exec.broadcastExchange.shareDecoded.idleTimeout, 1 second by default, after its last task completes, so that a task that starts by then reads it. One daemon thread frees the copy once the timeout passes with no task taking it. A timeout of 0 frees it at once, as before.
andygrove
left a comment
There was a problem hiding this comment.
Thanks @comphead. Keeping the decode for the next wave is a good idea, and the benchmark shows a real gain. I read through SharedDecode and could not find a race. acquire and release run under the lock, and expire re-checks holders and idlePeriods under it. A failed decode is forgotten at once, so its other holders close it in release.
My main concern is memory. The idle copy is the whole decoded broadcast in CometArrowAllocator, which no Spark or Comet pool accounts for. 1.2 already has the shared decode, so this adds up to a second of retained container RSS for every 1.2 user by default. The 31.8 MB single-broadcast run does not show whether copies stack. Stage N's copy can still be resident while stage N+1 decodes its own, and a query with several joins in consecutive stages could hold several. Spark's broadcast threshold can also be raised to GB scale. Could you share the peak CometArrowAllocator.getAllocatedMemory for a query with several broadcast joins in consecutive stages, with the timeout at 0 and at 1s? If the copies stack, would a cap on idle bytes per executor, as #7487 suggests, be small enough to include here? If the cap stays out, could the description say Part of #7487 and link a follow-up issue for it? As written, Closes #7487 closes the issue without the cap.
On tests, CometTestBase now sets the timeout to 0 for every suite, so no suite runs the 1s default. The nested-columns test uses the real scheduler at 100ms, but it only checks that the allocator returns to its baseline. It does not check that a later task reused the decode. Could you add a test with the real scheduler and a timeout of a few hundred milliseconds, where two tasks run one after the other and the injected decode function counts decodes? It should assert 1, so the reuse this PR is about is checked end to end. The manual-scheduler test covers the entry bookkeeping well, but it does not cover the thread.
Finally, batches(context, source, idleTimeoutMs: Long = 0) has a default on both CometBroadcastValue and SharedDecode. The only production caller passes the timeout, but a new caller that forgets it silently gets no retention. Would you drop the defaults and have the tests pass the value explicitly?
Address review: the idle copies of broadcasts read in consecutive stages stack. Add `spark.comet.exec.broadcastExchange.shareDecoded.maxIdleSize` (128 MiB by default), an executor-wide cap on the bytes of idle decodes. A decode over the cap is freed at once, and keeping a decode frees the decodes idle longest until the rest fit. An expiry now frees an entry only while it is still `current`, so an expiry scheduled before another path freed the entry frees nothing. `batches` takes a `SharedDecode.Retention` with no default, so a caller cannot silently get no retention. Add a test of reuse through the real expiry thread and a test of the cap.
|
Thanks @andygrove, all three points held up when I checked them against the code. Addressed in 51ef4bc. Memory. The copies do stack. Each broadcast has its own
So without a cap, the timeout adds every broadcast that finished in the last second to the executor's peak. The cap was small enough to include here, so the PR still closes #7487:
The cap bounds only the idle copies. The copy that running tasks hold is unchanged from
Tests. Added "a task that comes after another completes reads its decode, which the timer then frees". It runs two tasks one after the other with the real expiry thread and a 500 ms timeout, asserts one decode, and waits for the thread to free the copy. The cap has its own test with an injected Defaults. |
Which issue does this PR close?
Closes #7487.
Part of #6013.
Rationale for this change
#7016 lets the tasks of an executor that read a Comet broadcast at the same time share one decoded copy of it, and
frees the copy when the last of them completes. The tasks of a wave finish at about the same time, so the copy is gone
before the next wave starts, and every wave decodes the broadcast again. Every task of a wave also waits for that decode.
Counters added temporarily to
SharedDecodeshow this onmain, with thebhjmode of a hash-join task-timebenchmark: a 1M-row broadcast probed by 32 tasks on
local[8], so 4 waves, lz4, 7 runs. Every run had 32 reads, 4decodes and 4 frees. 28 tasks waited about 14 ms each, which is the time of the decode they waited for. Each new decode
started 0.6 to 1.3 ms after the copy before it was freed.
What changes are included in this PR?
spark.comet.exec.broadcastExchange.shareDecoded.idleTimeout, 1 second by default. When the last taskholding the decoded copy completes, the executor keeps the copy for that long, and a task that starts by then reads
it instead of decoding the broadcast again. A timeout of 0 frees the copy at once, as before.
SharedDecode.releasekeeps an entry that no task holds, and schedules its expiry on one daemon thread. Each entrycounts its idle periods, so an expiry scheduled before a task took the entry again frees nothing, and an entry is
never freed while a task holds it. An entry whose decode failed is still dropped at once.
spark.comet.exec.broadcastExchange.shareDecoded.maxIdleSize, 128 MiB by default, caps the bytesof the idle copies of all broadcasts on an executor. A copy larger than the cap is freed as soon as no task holds it,
and keeping a copy frees the copies that have been idle longest until the rest fit.
SharedDecode.IdleDecodeskeepsthe idle copies oldest first. It takes no other lock, and evicted copies are freed outside every lock.
the cap or a read with a timeout of 0 freed the entry frees nothing.
CometBroadcastExchangeExecreads both configs on the driver, as it doesshareDecoded.enabled, and passes themthrough
CometBatchRDDas aSharedDecode.Retention.batcheshas no default for it.CometTestBasesets the timeout to 0. Some suites compareCometArrowAllocator's memory before and after a step,and the idle copy of an earlier test's broadcast could be freed in between.
copy, which is JVM-side Arrow memory that neither Spark nor Comet's pools account for.
Mean time per probe task in the same benchmark, median of 7 runs with the arms interleaved:
idleTimeout=0(asmain)The first wave still waits for its decode. A cheaper decode would shrink that wait (#2216, #3762). The idle copy in
this benchmark is 31.8 MB. The expiry freed it about 1 second after the last probe task, and the allocator's memory did
not grow across runs.
The default of 1 second is a trade-off. The waves were about 1 ms apart here, and a cluster adds scheduling delay, but a
longer timeout keeps memory that no pool accounts for after the stage that read it.
The idle copies of broadcasts read in consecutive stages stack. A query with four broadcast hash joins in consecutive
stages (4M-row fact in 32 files, four 1.5M-row dims of 24.4 MB decoded each,
local[8], AQE off, Spark 4.0, median of7 runs with the arms interleaved), sampling
CometArrowAllocatorevery millisecond:idleTimeout=0(asmain)idleTimeout=1s, no capidleTimeout=1s, cap 128 MiBidleTimeout=1s, cap 50 MiBThe cap bounds what the timeout adds to an executor's peak, the idle copies, while the copy that running tasks hold
is as before. With the 50 MiB cap, keeping a stage's copy frees the oldest idle one, and each broadcast is still
decoded once.
How are these changes tested?
CometBroadcastValueSuitereplaces the scheduler with one that the test runs by hand. It checks that atask that comes before the timeout reads the copy of earlier tasks without decoding it again, that an expiry
scheduled before that task came frees nothing, while the task runs or after it completes, that the expiry of the
last release frees the copy, and that a timeout of 0 frees it at once.
broadcast is decoded once, and waits for the thread to free the copy.
IdleDecodesand a hand-run scheduler, checks that keeping a copy frees the oneidle longest, that a copy over the cap is freed at once, that a task taking an idle copy takes it out of the cap,
and that the expiries of evicted copies free nothing more.
to free the copy.