[maleadt]

Since #3318, finalizers only retire resources, and a regular task releases them later. If the resource's context is destroyed in between, e.g. by leaving a CuContext(dev) do ... end block, the release activates the dead context, and the driver segfaults (cuCtxGetDevice right after cuCtxSetCurrent on the dangling handle). cuTile.jl's test suite hits this. On main, streams, events, graphs, pinned memory, and arrays or kernels used on a stream of a temporary context all crash or hang this way:

CuContext(device()) do ctx
    context!(ctx) do
        finalize(CuStream())
    end
end
CUDA.reclaim()      # segfault

Destroying a context releases what it owns, and the handles may be reused afterwards, so release paths now check isvalid(ctx) before doing context-dependent driver work, and drop the resource if the context is gone. Before #3178 removed it, skip_destroyed did the same thing. Only the driver calls are skipped: host-side bookkeeping (memory accounting, graph leases, pin counts) still runs. If a held resource's gate event belongs to a dead context, the resource counts as done, and disposal_streams forgets the streams of destroyed contexts.

Some resources outlive their context, so dropping them would leak them:

• Allocations from a memory pool. I verified that the pool's used memory stays up after cuCtxDestroy, and that cuMemFreeAsync from the primary context frees it. They are now freed in a live context: the allocating context if it still exists, otherwise the primary context of the device.
• Graphs and executable graphs. These are destroyed in the current context.
• Memory allocated by executable graphs. This leaks for now, because GraphAllocations doesn't record its device. There's an XXX comment for it.

Memory can also be last used in a context other than the one it was allocated in. If that context is destroyed first, reusing cached host or unified memory (without a memory pool) no longer queries its idle event, which segfaults after destruction, and CPU access to globally attached unified memory attaches it in the context the memory lives in.

A second commit fixes a related crash that has existed since #3178: every synchronization reads the exception flag of every context a kernel was loaded in, and for a destroyed context that host memory has already been freed. check_exceptions skips such contexts again.

The validity check is one cuCtxGetId call, about 21 ns, per released resource. Other per-context state stays stale after a context is destroyed (stream pools, library handle caches, the kernel cache). Using kernels from a non-primary context with context! is also still broken (CUDA Stream must be associated with the same CUDA Context that loaded CUFunction), on 6.4.2 as well; that's a separate issue.

Tested on an RTX 5080: the new tests in core/resources and core/exceptions crash their worker on main and pass with this PR. A set of reproducers (stream, event, graph, pinned memory, an array and a kernel launch on the context's own stream, a stream that was released before the context was destroyed, cached host memory last used in a destroyed context) crash, abort or hang on main and now run without errors. Pool memory allocated in the temporary context is freed.

Fixes #3403

[github-actions]

CUDA.jl Benchmarks

Details

Benchmark suite	Current: 1a207bb	Previous: 2ea18f9	Ratio
array/accumulate/Float32/1d	54599 ns	54772 ns	1.00
array/accumulate/Float32/dims=1	39268 ns	39700 ns	0.99
array/accumulate/Float32/dims=1L	81766 ns	82439 ns	0.99
array/accumulate/Float32/dims=2	49004 ns	49526 ns	0.99
array/accumulate/Float32/dims=2L	398181 ns	399506 ns	1.00
array/accumulate/Int64/1d	88059 ns	89102 ns	0.99
array/accumulate/Int64/dims=1	59504 ns	60313 ns	0.99
array/accumulate/Int64/dims=1L	139997 ns	140583 ns	1.00
array/accumulate/Int64/dims=2	68671 ns	68534 ns	1.00
array/accumulate/Int64/dims=2L	797179 ns	797122 ns	1.00
array/broadcast	10106 ns	10356 ns	0.98
array/broadcast launch	6508 ns	6753 ns	0.96
array/construct	952.6363636363636 ns	960.1785714285714 ns	0.99
array/copy	17077 ns	17189 ns	0.99
array/copyto!/cpu_to_gpu	206656 ns	210329 ns	0.98
array/copyto!/gpu_to_cpu	240761 ns	242774 ns	0.99
array/copyto!/gpu_to_gpu	8721 ns	8873 ns	0.98
array/iteration/findall/bool	48700 ns	49920 ns	0.98
array/iteration/findall/int	65096 ns	66610 ns	0.98
array/iteration/findfirst/bool	34316 ns	34868 ns	0.98
array/iteration/findfirst/int	34344 ns	34998 ns	0.98
array/iteration/findmin/1d	50638 ns	53031 ns	0.95
array/iteration/findmin/2d	51889 ns	53598 ns	0.97
array/iteration/logical	50714 ns	50904 ns	1.00
array/iteration/scalar	60442 ns	60322 ns	1.00
array/permutedims/2d	47030 ns	47093 ns	1.00
array/permutedims/3d	44476 ns	44539 ns	1.00
array/permutedims/4d	47616 ns	47896 ns	0.99
array/random/rand/Float32	10999 ns	11145 ns	0.99
array/random/rand/Int64	17569 ns	17796 ns	0.99
array/random/rand!/Float32	8134.333333333333 ns	8344.666666666666 ns	0.97
array/random/rand!/Int64	15268 ns	15558 ns	0.98
array/random/randn/Float32	33805 ns	34128 ns	0.99
array/random/randn!/Float32	25084 ns	25132 ns	1.00
array/reductions/mapreduce/Float32/1d	35542 ns	34362 ns	1.03
array/reductions/mapreduce/Float32/dims=1	19668 ns	19450 ns	1.01
array/reductions/mapreduce/Float32/dims=1L	57408 ns	57133 ns	1.00
array/reductions/mapreduce/Float32/dims=2	27919 ns	27326 ns	1.02
array/reductions/mapreduce/Float32/dims=2L	58036 ns	57650 ns	1.01
array/reductions/mapreduce/Int64/1d	35512 ns	35313 ns	1.01
array/reductions/mapreduce/Int64/dims=1	22027 ns	21761 ns	1.01
array/reductions/mapreduce/Int64/dims=1L	96564 ns	96707 ns	1.00
array/reductions/mapreduce/Int64/dims=2	29614 ns	28019 ns	1.06
array/reductions/mapreduce/Int64/dims=2L	83283 ns	83491 ns	1.00
array/reductions/reduce/Float32/1d	36003 ns	34934 ns	1.03
array/reductions/reduce/Float32/dims=1	20158 ns	19330 ns	1.04
array/reductions/reduce/Float32/dims=1L	57580 ns	57162 ns	1.01
array/reductions/reduce/Float32/dims=2	28503 ns	27376 ns	1.04
array/reductions/reduce/Float32/dims=2L	58017 ns	57509 ns	1.01
array/reductions/reduce/Int64/1d	36015 ns	35215 ns	1.02
array/reductions/reduce/Int64/dims=1	22279 ns	21685 ns	1.03
array/reductions/reduce/Int64/dims=1L	97037 ns	96542 ns	1.01
array/reductions/reduce/Int64/dims=2	29489 ns	28245 ns	1.04
array/reductions/reduce/Int64/dims=2L	83568 ns	83062 ns	1.01
array/reverse/1d	19414 ns	19915 ns	0.97
array/reverse/1dL	73418 ns	73665 ns	1.00
array/reverse/1dL_inplace	70288 ns	70464 ns	1.00
array/reverse/1d_inplace	12057 ns	12146 ns	0.99
array/reverse/2d	22293 ns	22477 ns	0.99
array/reverse/2dL	80750 ns	80858 ns	1.00
array/reverse/2dL_inplace	70672 ns	70947 ns	1.00
array/reverse/2d_inplace	14566 ns	14822 ns	0.98
array/sorting/1d	748926 ns	750787 ns	1.00
array/sorting/2d	194229 ns	194334 ns	1.00
array/sorting/by	926710 ns	928178 ns	1.00
cuda/graph/capture	36795 ns	37276 ns	0.99
cuda/graph/eager	32989 ns	34127 ns	0.97
cuda/graph/launch	13940 ns	13916 ns	1.00
cuda/synchronization/context/auto	2329.5555555555557 ns	2443.6666666666665 ns	0.95
cuda/synchronization/context/blocking	903.170731707317 ns	906.8809523809524 ns	1.00
cuda/synchronization/context/nonblocking	10638 ns	8349 ns	1.27
cuda/synchronization/stream/auto	900.4772727272727 ns	871.8412698412699 ns	1.03
cuda/synchronization/stream/blocking	803 ns	789.8627450980392 ns	1.02
cuda/synchronization/stream/nonblocking	8429.333333333334 ns	8034 ns	1.05
integration/byval/reference	147586 ns	147627 ns	1.00
integration/byval/slices=1	148729 ns	148709 ns	1.00
integration/byval/slices=2	290127 ns	290270 ns	1.00
integration/byval/slices=3	432418 ns	432303 ns	1.00
integration/cudadevrt	104995 ns	104902 ns	1.00
integration/volumerhs	9606839 ns	9610172 ns	1.00
kernel/indexing	11516 ns	11573 ns	1.00
kernel/indexing_checked	12153 ns	12125 ns	1.00
kernel/launch	2435 ns	2307.777777777778 ns	1.06
kernel/occupancy	680.7467532467532 ns	682.8 ns	1.00
kernel/rand	14839 ns	14981 ns	0.99
latency/import	4052539618 ns	4211043895 ns	0.96
latency/precompile	4975778669 ns	5207029807 ns	0.96
latency/ttfp	4869574074 ns	5033274915 ns	0.97

This comment was automatically generated by workflow using github-action-benchmark.

[codecov]

Codecov Report

❌ Patch coverage is 83.75000% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.14%. Comparing base (f533f03) to head (1a207bb).
⚠️ Report is 22 commits behind head on main.

Files with missing lines	Patch %	Lines
CUDACore/src/resources/managed.jl	0.00%	6 Missing ⚠️
CUDACore/src/resources/backends.jl	82.60%	4 Missing ⚠️
CUDACore/src/resources/streams.jl	75.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3404      +/-   ##
==========================================
- Coverage   87.08%   85.14%   -1.94%     
==========================================
  Files         192      191       -1     
  Lines       19543    19307     -236     
==========================================
- Hits        17019    16439     -580     
- Misses       2524     2868     +344     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.