Segfault in zeCommandListHostSynchronize when reusing memory last used by a finished task #736

The array tests intermittently segfault in the "scans" testset, inside zeCommandListHostSynchronize, while an upload waits for the stream that last used a sub-allocated chunk. It happens on main (7020223) as well as on #735, on an Intel Arc A770 with NEO 26.35.39758. This is the trace from the reproducer below on main, with Julia 1.10 (in the test suite, the trace is the same, starting at test/array.jl:595):

[244547] signal (11.1): Segmentation fault
in expression starting at repro.jl:16
_ZN2L028zeCommandListHostSynchronizeEP25_ze_command_list_handle_tm.cold at libze_intel_gpu.so.1 (unknown line)
macro expansion at GPUToolbox/src/ccalls.jl:161 [inlined]
unchecked_zeCommandListHostSynchronize at lib/level-zero/libze.jl:2515 [inlined]
isdone at lib/level-zero/synchronization.jl:25 [inlined]
spin_until at GPUToolbox/src/synchronization.jl:34
cooperative_wait at GPUToolbox/src/synchronization.jl:439 [inlined]
nonblocking_synchronize at lib/level-zero/synchronization.jl:48
drain at src/memory.jl:92
synchronize_owner! at src/memory.jl:106
take_ownership! at src/memory.jl:118
with_managed_locks at src/memory.jl:141
with_ownership at src/memory.jl:152 [inlined]
unsafe_copyto! at src/array.jl:482
copyto! at src/array.jl:431
oneArray at src/array.jl:348

The kernel log shows a NULL dereference each time: julia[...]: segfault at 0 ip ... error 4 in libze_intel_gpu.so.1.17.39758.

How often:

• Julia 1.10: in almost every run of the array test (with Pkg.test(test_args=["array"]), it crashed on the first attempt and the retry in 3 of 3 runs on main). Running test/array.jl outside the test harness also crashed, with one thread (one run with two threads didn't).
• Julia 1.13: rarely, in 2 of 12 runs of the array test; the retry passed each time.

Another process (an OpenCL.jl test suite) was using the same GPU during some of these runs, which may explain why the frequency varied.

Reproducer

This crashes every time for me, on 1.10 and 1.13 (3 of 3 runs each), with the same stack trace:

using oneAPI

# use sub-allocated arrays from another task, and drop them together with that task,
# and thus with its stream
function use_elsewhere(n)
    fetch(Threads.@spawn begin
        as = [oneArray{Float32}(undef, 1024) for _ in 1:n]
        foreach(a -> a .= 1, as)
        synchronize()
        nothing
    end)
    return
end

for i in 1:20
    use_elsewhere(1000)
    GC.gc(true)
    # once there's no fresh memory left, this reuses chunks last used by the other
    # task's stream, and first waits for that stream
    bs = [oneArray(rand(Float32, 1024)) for _ in 1:1000]
    @assert all(b -> length(Array(b)) == 1024, bs)
end

Within test/array.jl, running only the "aliasing", "shared buffers & unsafe_wrap", "uploads" and "scans" testsets still crashed, about half of the time. "uploads" uploads from several Threads.@spawn tasks, like the reproducer does.

What's going on

Confirmed: the stream being drained has a destroyed immediate command list. With a check added to drain (src/memory.jl:87), the crash is preceded by s.immediate.handle == C_NULL, which the list's finalizer sets after destroying it (lib/level-zero/cmdlist.jl:94-98). The stream's recorder is nothing and it's idle.

Hypothesis (not verified beyond the above): the stream is resurrected by the pool after its list was finalized.

1. A task's stream is only referenced from its task-local storage (src/context.jl:516), from Managed.owner of the memory it used, and from freed chunks in the pool. When the task is done and its arrays are garbage too, the stream, its ZeImmediateCommandList and the arrays all become unreachable in the same collection, and their finalizers run in no particular order.
2. If the list's finalizer runs first, it synchronizes, destroys the list and sets its handle to C_NULL.
3. The arrays' finalizers then free their memory through pool_free (src/pool.jl:261-275), which keeps the chunk's owner and dirty flag in a FreeChunk (src/pool.jl:272). That makes the stream reachable again, with a destroyed list, and dirty still set.
4. Once a size class has no fresh chunks left, take_chunk! hands out any freed chunk (src/pool.jl:223-227), and pool_alloc passes its owner and dirty flag to the new Managed (src/pool.jl:257).
5. The first operation from another stream then calls take_ownership! → synchronize_owner! → drain on the old stream (src/memory.jl:106, 118), which synchronizes the C_NULL handle.

The per-allocation ownership comes from #686 and keeping it in freed chunks from #692. A stream whose list was finalized has no outstanding work, so drain could skip it, or pool_free could drop the owner (or clear dirty) when the owner's list is gone.