[maleadt]

Launching an executable graph cost 55–70 ns of CPU time for every array the graph uses (reported in #3326 (comment)), and kernel launches paid part of the same overhead. Most of it wasn't the ownership protocol itself:

• Graphs and kernel launches keep their memory in a Vector{Managed}. Because of its abstract element type, every lock, take_ownership! and restamp! call dispatched dynamically. These loops now split on the kind of memory.
• acquire_managed (Reduce the host code compiled for every new kernel #3359) returned a NamedTuple whose type was only known at run time. That cost an allocation and ~130 ns on every launch. It now returns a concrete struct.
• Ordering the locks of a kernel's memory hashed each lock with objectid. Locks are now ordered by the address of their Managed.
• Every @cuda queried the device's compute capability (two driver calls) to check whether the cubin loads. capability(dev) is now cached per device.

CPU time on an RTX 5080 (Julia 1.13), compared with main:

	main	PR
exec(), 10 arrays (raw cuGraphLaunch 1.2 µs)	2.2 µs	1.6 µs
exec(), 1000 arrays (raw 1.8 µs)	56–67 µs	17 µs
exec(), 3000 arrays (raw 3.6 µs)	203–216 µs	63–73 µs
acquiring + releasing one memory	178 ns, 96 bytes	44 ns, 0 bytes
@cuda with 4 array arguments	2260 ns	1790 ns

What remains is mostly the uncontended lock of each memory (~15 ns per array), which has to be held until the operation is submitted.

[github-actions]

CUDA.jl Benchmarks

Details

Benchmark suite	Current: 3bd0526	Previous: 22792e1	Ratio
array/accumulate/Float32/1d	104651 ns	106095 ns	0.99
array/accumulate/Float32/dims=1	77469 ns	78498 ns	0.99
array/accumulate/Float32/dims=1L	1675221 ns	1675714 ns	1.00
array/accumulate/Float32/dims=2	147548 ns	149000 ns	0.99
array/accumulate/Float32/dims=2L	683194 ns	684023 ns	1.00
array/accumulate/Int64/1d	122975 ns	124181 ns	0.99
array/accumulate/Int64/dims=1	81889 ns	83033 ns	0.99
array/accumulate/Int64/dims=1L	1797168 ns	1799278 ns	1.00
array/accumulate/Int64/dims=2	162919 ns	163922 ns	0.99
array/accumulate/Int64/dims=2L	988030 ns	989810 ns	1.00
array/broadcast	17152 ns	17842 ns	0.96
array/broadcast launch	7563.75 ns	8366 ns	0.90
array/construct	904.78125 ns	969.6071428571429 ns	0.93
array/copy	17069 ns	17836 ns	0.96
array/copyto!/cpu_to_gpu	206874 ns	209551 ns	0.99
array/copyto!/gpu_to_cpu	239194 ns	240020 ns	1.00
array/copyto!/gpu_to_gpu	9075 ns	9722 ns	0.93
array/iteration/findall/bool	138536 ns	141337 ns	0.98
array/iteration/findall/int	149150 ns	151794 ns	0.98
array/iteration/findfirst/bool	73375 ns	75295 ns	0.97
array/iteration/findfirst/int	76033 ns	78048 ns	0.97
array/iteration/findmin/1d	59459 ns	67907 ns	0.88
array/iteration/findmin/2d	103278 ns	105015 ns	0.98
array/iteration/logical	191961 ns	201995 ns	0.95
array/iteration/scalar	53669 ns	53072 ns	1.01
array/permutedims/2d	49602 ns	50619 ns	0.98
array/permutedims/3d	50574 ns	51692 ns	0.98
array/permutedims/4d	51749 ns	53336 ns	0.97
array/random/rand/Float32	11099 ns	10916 ns	1.02
array/random/rand/Int64	18350 ns	19660 ns	0.93
array/random/rand!/Float32	8400 ns	8353.333333333334 ns	1.01
array/random/rand!/Int64	15989 ns	16803 ns	0.95
array/random/randn/Float32	34732 ns	34841 ns	1.00
array/random/randn!/Float32	25156 ns	25209 ns	1.00
array/reductions/mapreduce/Float32/1d	31382 ns	33817 ns	0.93
array/reductions/mapreduce/Float32/dims=1	39853 ns	41131 ns	0.97
array/reductions/mapreduce/Float32/dims=1L	53487 ns	54519 ns	0.98
array/reductions/mapreduce/Float32/dims=2	59019 ns	60020 ns	0.98
array/reductions/mapreduce/Float32/dims=2L	71023 ns	72757 ns	0.98
array/reductions/mapreduce/Int64/1d	38501 ns	40588 ns	0.95
array/reductions/mapreduce/Int64/dims=1	42735 ns	43755 ns	0.98
array/reductions/mapreduce/Int64/dims=1L	89253 ns	90628 ns	0.98
array/reductions/mapreduce/Int64/dims=2	60153 ns	61480 ns	0.98
array/reductions/mapreduce/Int64/dims=2L	86722 ns	88145 ns	0.98
array/reductions/reduce/Float32/1d	31890 ns	34303 ns	0.93
array/reductions/reduce/Float32/dims=1	39831 ns	41212 ns	0.97
array/reductions/reduce/Float32/dims=1L	52685 ns	54164 ns	0.97
array/reductions/reduce/Float32/dims=2	58826 ns	60037 ns	0.98
array/reductions/reduce/Float32/dims=2L	71228 ns	72958 ns	0.98
array/reductions/reduce/Int64/1d	38533 ns	40426 ns	0.95
array/reductions/reduce/Int64/dims=1	42455 ns	44109 ns	0.96
array/reductions/reduce/Int64/dims=1L	89140 ns	90365 ns	0.99
array/reductions/reduce/Int64/dims=2	60110 ns	61481 ns	0.98
array/reductions/reduce/Int64/dims=2L	87105 ns	88431 ns	0.99
array/reverse/1d	16036 ns	17240 ns	0.93
array/reverse/1dL	69843 ns	71234 ns	0.98
array/reverse/1dL_inplace	67594 ns	68291 ns	0.99
array/reverse/1d_inplace	9326 ns	9977 ns	0.93
array/reverse/2d	20352 ns	21607 ns	0.94
array/reverse/2dL	74308 ns	75460 ns	0.98
array/reverse/2dL_inplace	67648 ns	68544 ns	0.99
array/reverse/2d_inplace	10608 ns	11624 ns	0.91
array/sorting/1d	2840826 ns	2849893 ns	1.00
array/sorting/2d	1087875 ns	1092693 ns	1.00
array/sorting/by	3393861 ns	3402196 ns	1.00
cuda/graph/capture	41003 ns	47685 ns	0.86
cuda/graph/eager	38814 ns	47445 ns	0.82
cuda/graph/launch	14147 ns	14947 ns	0.95
cuda/synchronization/context/auto	7981 ns	7898.25 ns	1.01
cuda/synchronization/context/blocking	886.5833333333334 ns	880.1555555555556 ns	1.01
cuda/synchronization/context/nonblocking	8006.75 ns	7864 ns	1.02
cuda/synchronization/stream/auto	343.875 ns	341.6743119266055 ns	1.01
cuda/synchronization/stream/blocking	816.85 ns	772.1238095238095 ns	1.06
cuda/synchronization/stream/nonblocking	7848.75 ns	7666.5 ns	1.02
integration/byval/reference	147765 ns	148913 ns	0.99
integration/byval/slices=1	148869 ns	150048 ns	0.99
integration/byval/slices=2	290196 ns	292290 ns	0.99
integration/byval/slices=3	432175 ns	434991 ns	0.99
integration/cudadevrt	105134 ns	106261 ns	0.99
integration/volumerhs	9203413 ns	9208448 ns	1.00
kernel/indexing	11518 ns	12631 ns	0.91
kernel/indexing_checked	12059 ns	13009 ns	0.93
kernel/launch	2461.1111111111113 ns	2606.4444444444443 ns	0.94
kernel/occupancy	871.6842105263158 ns	921.2647058823529 ns	0.95
kernel/rand	15087 ns	15746 ns	0.96
latency/import	4335659256 ns	4332428870 ns	1.00
latency/precompile	5108893065 ns	5115422046 ns	1.00
latency/ttfp	4850083359 ns	4823579341 ns	1.01

This comment was automatically generated by workflow using github-action-benchmark.

[codecov]

Codecov Report

❌ Patch coverage is 90.90909% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.10%. Comparing base (22792e1) to head (3bd0526).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
CUDACore/src/compiler/execution.jl	89.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3362      +/-   ##
==========================================
- Coverage   87.17%   85.10%   -2.07%     
==========================================
  Files         195      194       -1     
  Lines       20066    19983      -83     
==========================================
- Hits        17492    17007     -485     
- Misses       2574     2976     +402     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.