Repository navigation
Clarify what @spawn orders, for users and for backends - #833
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #833 +/- ##
=======================================
Coverage 79.86% 79.86%
=======================================
Files 26 26
Lines 2146 2146
=======================================
Hits 1714 1714
Misses 432 432 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Benchmark ResultsShow table
Benchmark PlotsA plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR. |
maleadt
force-pushed
the
tb/spawn-host-access
branch
from
October 4, 2026 08:26
5571d1f to
f3c60b0
Compare
A sentence about the device of a new task was cut off, and the note on exceptions lacked punctuation and ran into the next paragraph.
The spawned task's work is ordered after its parent's on the device, but code in the task that passes the parent's results to something outside the backend, e.g., an MPI call on a GPU buffer, has to synchronize first. Some backends currently do so implicitly when the buffer's pointer is taken, which portable code can't rely on. Also say that the child isn't ordered against the parent's later work, rather than that it can't rely on that work's data.
maleadt
force-pushed
the
tb/spawn-host-access
branch
from
October 7, 2026 15:59
f3c60b0 to
b63709f
Compare
Backends that track which queue last used an array make another queue wait before using it. If that wait ignores the order set up by wait_event, or by a completed synchronize, the spawned task's first use of an array shared with its parent waits for everything the parent has queued, including work queued after @Spawn (#817). CUDA.jl, AMDGPU.jl, Metal.jl and OpenCL.jl all track arrays this way. Recommend respecting that order, and likewise for what synchronize waits for. Like task-local queues, this is about overlap rather than correctness, so it is a recommendation. Waits for memory accessibility or lifetime are not affected.
maleadt
force-pushed
the
tb/spawn-host-access
branch
from
October 7, 2026 16:04
b63709f to
9ade488
Compare
Device selection was explained three times and the implementation requirements of record_event and wait_event were repeated on the notes page. Keep each fact in one place: the user-facing behavior in the @Spawn docstring, the correctness requirements in the KernelInterface docstrings, and the protocol and the performance recommendations on the notes page. Also drop the note on exceptions, which the guarantees at the top already cover, and stop claiming that switching devices leaves the new queue unordered, since it may already be ordered by other means.
vchuravy
marked this pull request as ready for review
October 7, 2026 20:13
vchuravy
reviewed
Oct 7, 2026
vchuravy
reviewed
Oct 7, 2026
vchuravy
reviewed
Oct 7, 2026
vchuravy
reviewed
Oct 7, 2026
A synchronize that also waits for work on other queues, e.g. work another task queued, costs overlap without making any program more correct. All GPU backends except OpenCL.jl already wait only for the current queue (OpenCL.jl waits for other queues to check its shared exception mailbox, JuliaGPU/OpenCL.jl#549), so make this a requirement in the synchronize docstring rather than a recommendation. Also state the purpose of the ownership recommendation as a goal instead of describing what happens without it.
CUDA.jl and AMDGPU.jl make device! set the default device for new tasks as well, while oneAPI.jl and OpenCL.jl keep it task-local, so a task started with Threads.@Spawn may or may not run on its parent's device. Also say which work record_event captures without describing what it doesn't.
vchuravy
approved these changes
Oct 9, 2026
Member
Author
|
CI failures unrelated, this is a doc-only PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On CUDA.jl,
@spawnloses most of its overlap (#817). CUDA.jl records which stream last used each array, and synchronizes that stream before another stream uses the array. That waits for everything the parent has queued, including work it queued after@spawn, even thoughwait_eventalready ordered the child after the work that matters:This PR documents what
@spawnpromises:@spawnorders work on the device, but the host doesn't wait for it. Before passing the parent's results to something outside the backend, like MPI, callsynchronize(backend)in the task, as above.synchronize. Ignoring this is correct but reduces overlap, so it's a recommendation, like task-local queues.synchronizemust not wait for work on other queues that the current queue isn't ordered after. That is also only about overlap, but every GPU backend except OpenCL.jl already meets it, so it's a requirement.CUDA.jl implements this in JuliaGPU/CUDA.jl#3328 (on top of #3329). AMDGPU.jl, Metal.jl and OpenCL.jl track arrays the same way and keep the extra wait for now, which is correct, just with less overlap. OpenCL.jl's
synchronizealso waits for other queues to check its shared exception mailbox, which JuliaGPU/OpenCL.jl#549 will fix. oneAPI.jl and the CPU backend don't track arrays, so the recommendation doesn't apply to them.The PR also tightens the
@spawnand event docs, so that each fact is stated in one place. User-facing behavior goes in the@spawndocstring, requirements in the KernelInterface docstrings, and the protocol and recommendations on the backend notes page. It also fixes two broken sentences, and stops claiming which device a task started with plainThreads.@spawnuses, since that differs between backends.Closes #817.