[maleadt]

CUDA.jl remembers which stream last used each array. When the array is used on another stream, e.g., after passing it to another task, the previous stream is first synchronized from the host. That makes it safe to share arrays between tasks, but the task that takes over the array blocks until everything queued on the other stream has finished:

a = CUDA.rand(N)
long_running_kernel!(b)     # queued after the initialization of `a`

Threads.@spawn begin
    a .+= 1                 # blocks until `long_running_kernel!` is done
end

With this PR, operations that CUDA.jl submits itself (kernel launches, which includes broadcasts and KernelAbstractions kernels, graph launches, copies and fill!) make their stream wait for the previous one on the GPU instead, with cuStreamWaitEvent and an event that's cached per stream. The work still runs in the same order, but the host doesn't block.

Pointers that are passed to other code, like a library, MPI, or a ccall, may be used from the host or from other streams. Those conversions still synchronize, as before. That includes memory that was already handed off to the current stream: CUDA.jl remembers that the stream is waiting for another one, and synchronizes it when a pointer is taken. Pointers taken inside CUDA.with_managed are the exception: they're assumed to be used by an operation on the stream passed to with_managed, which is technically breaking for code that hands them to something else (noted in NEWS).

For this to be safe, an operation needs to hold the lock of the memory it uses until it has been submitted, so that another task can't see the new owner and wait for its stream before the work is there. Kernel launches already did that; the first commit makes copies and fill! do the same.

Some cases still synchronize on the host: unified and host memory, memory used by captures that CUDA.jl doesn't know about (i.e., not started with capture), the default, legacy and per-thread streams, and memory moving between devices. Library calls like cuBLAS could be handed off on the device too, but only after checking that each library queues all its work on the task's stream, so that's left for later.

Captures don't take ownership of the memory they use; launching the graph does. A graph launch is therefore handed off like any other operation: it waits on the device for the streams that last used its memory, wherever it was captured. A stream that is being captured with capture(; stream=s) needs care, because recording an event on it would only add a node to the graph. capture now records an event on s right before it begins capturing, under the same lock that hand-offs hold while recording on s, and other tasks wait for that event instead. That also lifts the documented limitation that other tasks couldn't use memory last used on s until the capture ended. The capturing task itself still gets a CaptureError when it waits for such memory, since the event doesn't cover what it captured.

On an RTX 5080, taking over an array while the other stream is busy for 0.7 s now returns after 0.1 ms instead of 0.7 s. Moving an array back and forth between two streams costs 2.4 µs per operation instead of 5.0 µs. Kernel launches cost the same as before. Copies and fill! take about 50–70 ns longer (~1.9 µs and ~0.9 µs before), because the arrays now stay locked until the operation has been submitted, like kernel launches already did.

The new tests keep a stream busy with a gate kernel. They check that kernels and copies in other tasks don't block but still execute in order, that synchronize(a) waits for the stream the array came from, that a pointer conversion after a hand-off still waits for the original stream, that unified memory still synchronizes, and that a graph captured right after a hand-off and launched on another stream doesn't block and sees the right data. The graph tests check that other tasks can use memory last used on a stream while it's being captured on explicitly, and that the capture itself can't. Tested with and without compute-sanitizer, and on Julia 1.11 for the launch-allocation test.

[codecov]

Codecov Report

❌ Patch coverage is 78.46154% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.07%. Comparing base (b4d35bc) to head (00a2218).

Files with missing lines	Patch %	Lines
CUDACore/src/resources/managed.jl	59.32%	24 Missing ⚠️
CUDACore/src/array.jl	93.10%	2 Missing ⚠️
CUDACore/lib/cudadrv/state.jl	0.00%	1 Missing ⚠️
CUDACore/src/compiler/execution.jl	93.33%	1 Missing ⚠️

Additional details and impacted files

@@                  Coverage Diff                  @@
##           tb/unified_attach    #3329      +/-   ##
=====================================================
- Coverage              85.13%   85.07%   -0.07%     
=====================================================
  Files                    194      194              
  Lines                  19794    19858      +64     
=====================================================
+ Hits                   16852    16894      +42     
- Misses                  2942     2964      +22     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

CUDA.jl Benchmarks

Details

Benchmark suite	Current: 00a2218	Previous: ae637e5	Ratio
array/accumulate/Float32/1d	105156 ns	105001 ns	1.00
array/accumulate/Float32/dims=1	77425 ns	77262 ns	1.00
array/accumulate/Float32/dims=1L	1682503 ns	1684940 ns	1.00
array/accumulate/Float32/dims=2	148260 ns	148043 ns	1.00
array/accumulate/Float32/dims=2L	685632 ns	684338 ns	1.00
array/accumulate/Int64/1d	123753 ns	123503 ns	1.00
array/accumulate/Int64/dims=1	82206 ns	81902 ns	1.00
array/accumulate/Int64/dims=1L	1805482 ns	1803636 ns	1.00
array/accumulate/Int64/dims=2	163315 ns	163105 ns	1.00
array/accumulate/Int64/dims=2L	988439 ns	988275 ns	1.00
array/broadcast	17207 ns	16981 ns	1.01
array/broadcast launch	7558 ns	7497.25 ns	1.01
array/construct	923.8695652173913 ns	886.76 ns	1.04
array/copy	16740 ns	16325 ns	1.03
array/copyto!/cpu_to_gpu	206511 ns	206249 ns	1.00
array/copyto!/gpu_to_cpu	240617 ns	240560 ns	1.00
array/copyto!/gpu_to_gpu	8773 ns	8967 ns	0.98
array/iteration/findall/bool	140306 ns	135815 ns	1.03
array/iteration/findall/int	151058 ns	145400 ns	1.04
array/iteration/findfirst/bool	75029 ns	71077 ns	1.06
array/iteration/findfirst/int	77269 ns	73309 ns	1.05
array/iteration/findmin/1d	62602 ns	59845 ns	1.05
array/iteration/findmin/2d	103580 ns	103127 ns	1.00
array/iteration/logical	199644 ns	187405 ns	1.07
array/iteration/scalar	58016 ns	59115 ns	0.98
array/permutedims/2d	50100 ns	49364 ns	1.01
array/permutedims/3d	50514 ns	50795 ns	0.99
array/permutedims/4d	52783 ns	52271 ns	1.01
array/random/rand/Float32	11051 ns	11109 ns	0.99
array/random/rand/Int64	18544 ns	18411 ns	1.01
array/random/rand!/Float32	8292 ns	8183 ns	1.01
array/random/rand!/Int64	16088 ns	15765 ns	1.02
array/random/randn/Float32	34541 ns	34501 ns	1.00
array/random/randn!/Float32	24587 ns	24403 ns	1.01
array/reductions/mapreduce/Float32/1d	33401 ns	30573 ns	1.09
array/reductions/mapreduce/Float32/dims=1	40208 ns	40213 ns	1.00
array/reductions/mapreduce/Float32/dims=1L	53773 ns	53356 ns	1.01
array/reductions/mapreduce/Float32/dims=2	59264 ns	59358 ns	1.00
array/reductions/mapreduce/Float32/dims=2L	71564 ns	71210 ns	1.00
array/reductions/mapreduce/Int64/1d	40030 ns	37469 ns	1.07
array/reductions/mapreduce/Int64/dims=1	43210 ns	42915 ns	1.01
array/reductions/mapreduce/Int64/dims=1L	89338 ns	89249 ns	1.00
array/reductions/mapreduce/Int64/dims=2	60628 ns	61370 ns	0.99
array/reductions/mapreduce/Int64/dims=2L	87707 ns	87543 ns	1.00
array/reductions/reduce/Float32/1d	33449 ns	30615 ns	1.09
array/reductions/reduce/Float32/dims=1	40150 ns	40264 ns	1.00
array/reductions/reduce/Float32/dims=1L	53090 ns	52907 ns	1.00
array/reductions/reduce/Float32/dims=2	59304 ns	59462 ns	1.00
array/reductions/reduce/Float32/dims=2L	71895 ns	71728 ns	1.00
array/reductions/reduce/Int64/1d	40154 ns	37570 ns	1.07
array/reductions/reduce/Int64/dims=1	43157 ns	42862 ns	1.01
array/reductions/reduce/Int64/dims=1L	89517 ns	89319 ns	1.00
array/reductions/reduce/Int64/dims=2	60391 ns	61437 ns	0.98
array/reductions/reduce/Int64/dims=2L	87455 ns	86880 ns	1.01
array/reverse/1d	16591 ns	16128 ns	1.03
array/reverse/1dL	70335 ns	70115 ns	1.00
array/reverse/1dL_inplace	67730 ns	67602 ns	1.00
array/reverse/1d_inplace	9483 ns	9293 ns	1.02
array/reverse/2d	20784 ns	21118 ns	0.98
array/reverse/2dL	74806 ns	74528 ns	1.00
array/reverse/2dL_inplace	67871 ns	67778 ns	1.00
array/reverse/2d_inplace	10950 ns	10676 ns	1.03
array/sorting/1d	2848402 ns	2846981 ns	1.00
array/sorting/2d	1091139 ns	1088944 ns	1.00
array/sorting/by	3400863 ns	3400388 ns	1.00
cuda/graph/capture	43700 ns
cuda/graph/eager	40217 ns
cuda/graph/launch	14326 ns
cuda/synchronization/context/auto	8005.25 ns	6766.6 ns	1.18
cuda/synchronization/context/blocking	890.1632653061224 ns	793.7692307692307 ns	1.12
cuda/synchronization/context/nonblocking	7591 ns	6805.4 ns	1.12
cuda/synchronization/stream/auto	721.6985294117648 ns	691.0939597315436 ns	1.04
cuda/synchronization/stream/blocking	942.8846153846154 ns	868.875 ns	1.09
cuda/synchronization/stream/nonblocking	8064.333333333333 ns	7338.5 ns	1.10
integration/byval/reference	148055 ns	147882 ns	1.00
integration/byval/slices=1	149260 ns	149033 ns	1.00
integration/byval/slices=2	291313 ns	291277 ns	1.00
integration/byval/slices=3	433765 ns	434160 ns	1.00
integration/cudadevrt	105525 ns	105116 ns	1.00
integration/volumerhs	9207317 ns	9206594 ns	1.00
kernel/indexing	11690 ns	11855 ns	0.99
kernel/indexing_checked	12275 ns	12209 ns	1.01
kernel/launch	2493.6666666666665 ns	2433 ns	1.02
kernel/occupancy	914.75 ns	1008.8235294117648 ns	0.91
kernel/rand	15121 ns	14899 ns	1.01
latency/import	4317006422 ns	4305717656 ns	1.00
latency/precompile	5090821373 ns	5095420673 ns	1.00
latency/ttfp	4798072833 ns	4793974717 ns	1.00

This comment was automatically generated by workflow using github-action-benchmark.