[maleadt]

When memory moves between streams, CUDA.jl makes the new stream wait for everything the previous stream had queued. That's more than needed when the program already ordered the work with an event. KernelAbstractions.@spawn is the typical example (JuliaGPU/KernelAbstractions.jl#817): the parent records an event and the child's stream waits for it. Still, the child's first use of an array waited for all the work the parent queued after the event, so the two tasks didn't overlap:

a .= 1
e = CuEvent(); record(e)
more_work!(b)                   # the parent keeps going

Threads.@spawn begin            # what KernelAbstractions.@spawn does
    CUDA.wait(e)
    a .+= 1                     # doesn't wait for `more_work!` anymore
    MPI.Send(a, comm; dest=1)   # waits for `e`, but not for `more_work!`
end

To know what an event covers, each stream's work is split into epochs. Memory is stamped with the epoch of its last use, and recording an event or synchronizing a stream closes the current epoch. That gives two facts that make waiting unnecessary:

• Completed: the host saw the work finish, by synchronizing the stream or the event, or because isdone(event) returned true.
• Ordered: a stream waited for an event with CUDA.wait(event, stream). Work submitted to it afterwards runs after everything the event covers.

Memory covered by either fact moves to the new stream without waiting. A pointer conversion for code outside CUDA.jl, like the MPI call above, then synchronizes the new stream instead of the parent's, which only waits for what the event covered. Oceananigans can therefore drop its workaround of making the task adopt the parent's stream (CliMA/Oceananigans.jl#5897).

Operations that CUDA.jl submits (kernel launches, copies, fill!, graph launches) stamp memory when they take its pointer and again once the operation has been submitted. Between the two, an epoch closed by another task could appear to cover the operation, so an epoch only counts if the task that submits work to the stream closed it. Events recorded on another task's stream, or on a stream that several live tasks use, cover nothing. A stream whose task has finished can be taken over by another task, which recycling streams relies on. Graph capture doesn't produce either fact: recording an event on a stream that is being captured only adds a node to the graph, a wait on a capturing stream only orders the graph's operations, and captured operations don't stamp memory (launching the graph does).

A pointer taken outside of a CUDA.jl operation, e.g. p = pointer(a) passed to a kernel or library later, or an array passed to a library call, is only stamped once and can be used after the epoch it was stamped with has been closed: p = pointer(a); synchronize(); @cuda f(p) isn't covered by that synchronization. Such memory doesn't use either shortcut until it moves to another stream, which waits for everything submitted to the old one, as before. Work submitted through the old pointer after the array was used by another task isn't ordered with that task's work; that was already the case, and is now documented. Waiting for an event doesn't hold its lock, so other tasks can record or query it meanwhile; a wait only counts if the event wasn't recorded again in the meantime.

On an RTX 5080, record and CUDA.wait take about 150 and 210 ns, up from about 100 ns each, because of a lock and a table of the streams that were waited for. Kernel launches and copies cost the same as before.

The tests use pointer conversions and stream synchronization to check that events synchronized on the host, events waited for by a stream, and streams taken over from finished tasks avoid waiting. They also check that memory used after an event, events recorded by another task, events re-recorded elsewhere (also while another task waits for them), and launches through a pointer taken before the event or synchronization still wait.

[codecov]

Codecov Report

❌ Patch coverage is 90.90909% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.98%. Comparing base (00a2218) to head (38d4a78).

Files with missing lines	Patch %	Lines
CUDACore/lib/cudadrv/events.jl	83.87%	5 Missing ⚠️
CUDACore/src/resources/managed.jl	88.00%	3 Missing ⚠️
CUDACore/lib/cudadrv/stream.jl	93.33%	2 Missing ⚠️

Additional details and impacted files

@@                  Coverage Diff                  @@
##           tb/device-handoff    #3328      +/-   ##
=====================================================
+ Coverage              84.94%   84.98%   +0.04%     
=====================================================
  Files                    194      194              
  Lines                  19858    19960     +102     
=====================================================
+ Hits                   16869    16964      +95     
- Misses                  2989     2996       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

CUDA.jl Benchmarks

Details

Benchmark suite	Current: 38d4a78	Previous: ae637e5	Ratio
array/accumulate/Float32/1d	104846 ns	105001 ns	1.00
array/accumulate/Float32/dims=1	77193 ns	77262 ns	1.00
array/accumulate/Float32/dims=1L	1682465 ns	1684940 ns	1.00
array/accumulate/Float32/dims=2	148206 ns	148043 ns	1.00
array/accumulate/Float32/dims=2L	685134 ns	684338 ns	1.00
array/accumulate/Int64/1d	124001 ns	123503 ns	1.00
array/accumulate/Int64/dims=1	81498 ns	81902 ns	1.00
array/accumulate/Int64/dims=1L	1805414 ns	1803636 ns	1.00
array/accumulate/Int64/dims=2	163133 ns	163105 ns	1.00
array/accumulate/Int64/dims=2L	988121 ns	988275 ns	1.00
array/broadcast	17146 ns	16981 ns	1.01
array/broadcast launch	7845 ns	7497.25 ns	1.05
array/construct	976.8333333333334 ns	886.76 ns	1.10
array/copy	16891 ns	16325 ns	1.03
array/copyto!/cpu_to_gpu	206567 ns	206249 ns	1.00
array/copyto!/gpu_to_cpu	240789 ns	240560 ns	1.00
array/copyto!/gpu_to_gpu	8914 ns	8967 ns	0.99
array/iteration/findall/bool	140537 ns	135815 ns	1.03
array/iteration/findall/int	151930 ns	145400 ns	1.04
array/iteration/findfirst/bool	75027 ns	71077 ns	1.06
array/iteration/findfirst/int	77391 ns	73309 ns	1.06
array/iteration/findmin/1d	68571 ns	59845 ns	1.15
array/iteration/findmin/2d	103858 ns	103127 ns	1.01
array/iteration/logical	201703 ns	187405 ns	1.08
array/iteration/scalar	59057 ns	59115 ns	1.00
array/permutedims/2d	50030 ns	49364 ns	1.01
array/permutedims/3d	50870 ns	50795 ns	1.00
array/permutedims/4d	52559 ns	52271 ns	1.01
array/random/rand/Float32	11095 ns	11109 ns	1.00
array/random/rand/Int64	18320 ns	18411 ns	1.00
array/random/rand!/Float32	8217.333333333334 ns	8183 ns	1.00
array/random/rand!/Int64	15946 ns	15765 ns	1.01
array/random/randn/Float32	34741 ns	34501 ns	1.01
array/random/randn!/Float32	24694 ns	24403 ns	1.01
array/reductions/mapreduce/Float32/1d	33765 ns	30573 ns	1.10
array/reductions/mapreduce/Float32/dims=1	40266 ns	40213 ns	1.00
array/reductions/mapreduce/Float32/dims=1L	53760 ns	53356 ns	1.01
array/reductions/mapreduce/Float32/dims=2	59312 ns	59358 ns	1.00
array/reductions/mapreduce/Float32/dims=2L	71764 ns	71210 ns	1.01
array/reductions/mapreduce/Int64/1d	40226 ns	37469 ns	1.07
array/reductions/mapreduce/Int64/dims=1	43366 ns	42915 ns	1.01
array/reductions/mapreduce/Int64/dims=1L	89520 ns	89249 ns	1.00
array/reductions/mapreduce/Int64/dims=2	60502 ns	61370 ns	0.99
array/reductions/mapreduce/Int64/dims=2L	87674 ns	87543 ns	1.00
array/reductions/reduce/Float32/1d	33965 ns	30615 ns	1.11
array/reductions/reduce/Float32/dims=1	40493 ns	40264 ns	1.01
array/reductions/reduce/Float32/dims=1L	53102 ns	52907 ns	1.00
array/reductions/reduce/Float32/dims=2	59480 ns	59462 ns	1.00
array/reductions/reduce/Float32/dims=2L	71759 ns	71728 ns	1.00
array/reductions/reduce/Int64/1d	40062 ns	37570 ns	1.07
array/reductions/reduce/Int64/dims=1	43210 ns	42862 ns	1.01
array/reductions/reduce/Int64/dims=1L	89633 ns	89319 ns	1.00
array/reductions/reduce/Int64/dims=2	60783 ns	61437 ns	0.99
array/reductions/reduce/Int64/dims=2L	87754 ns	86880 ns	1.01
array/reverse/1d	16318 ns	16128 ns	1.01
array/reverse/1dL	70218 ns	70115 ns	1.00
array/reverse/1dL_inplace	67758 ns	67602 ns	1.00
array/reverse/1d_inplace	9399 ns	9293 ns	1.01
array/reverse/2d	21113 ns	21118 ns	1.00
array/reverse/2dL	75048 ns	74528 ns	1.01
array/reverse/2dL_inplace	67876 ns	67778 ns	1.00
array/reverse/2d_inplace	11001 ns	10676 ns	1.03
array/sorting/1d	2846860 ns	2846981 ns	1.00
array/sorting/2d	1089809 ns	1088944 ns	1.00
array/sorting/by	3399552 ns	3400388 ns	1.00
cuda/graph/capture	43762 ns
cuda/graph/eager	41538 ns
cuda/graph/launch	14344 ns
cuda/synchronization/context/auto	7914 ns	6766.6 ns	1.17
cuda/synchronization/context/blocking	882.952380952381 ns	793.7692307692307 ns	1.11
cuda/synchronization/context/nonblocking	7927.75 ns	6805.4 ns	1.16
cuda/synchronization/stream/auto	750.9736842105264 ns	691.0939597315436 ns	1.09
cuda/synchronization/stream/blocking	965 ns	868.875 ns	1.11
cuda/synchronization/stream/nonblocking	8469.333333333334 ns	7338.5 ns	1.15
integration/byval/reference	148072 ns	147882 ns	1.00
integration/byval/slices=1	149120 ns	149033 ns	1.00
integration/byval/slices=2	291458 ns	291277 ns	1.00
integration/byval/slices=3	433874 ns	434160 ns	1.00
integration/cudadevrt	105357 ns	105116 ns	1.00
integration/volumerhs	9201860 ns	9206594 ns	1.00
kernel/indexing	11664 ns	11855 ns	0.98
kernel/indexing_checked	12311 ns	12209 ns	1.01
kernel/launch	2565.1111111111113 ns	2433 ns	1.05
kernel/occupancy	1058.8 ns	1008.8235294117648 ns	1.05
kernel/rand	15340 ns	14899 ns	1.03
latency/import	4311120727 ns	4305717656 ns	1.00
latency/precompile	5095894836 ns	5095420673 ns	1.00
latency/ttfp	4793981580 ns	4793974717 ns	1.00

This comment was automatically generated by workflow using github-action-benchmark.