You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
synchronize waits for other queues because of the shared exception mailbox #549
OpenCL.synchronize() can wait for kernels that other tasks submitted to their own queues. Device-side exceptions are reported through one mailbox per (context, device), shared by every queue on that device. Before reading and resetting it, check_exceptions waits for every queue that launched a kernel since the last check (src/compiler/exceptions.jl), since the mailbox can't be mapped or reset while another kernel may still write to it.
This has two consequences:
A task's synchronize waits for unrelated work on other tasks' queues, so tasks lose overlap. With KernelAbstractions.@spawn, the child's synchronize also waits for whatever the parent queued after spawning.
An exception thrown by one task's kernel can be reported by another task's synchronize. Because the mailbox is then reset, the task that launched the kernel never sees it.
KernelAbstractions is going to require that synchronize not wait for work on other queues that the current queue isn't ordered after (JuliaGPU/KernelAbstractions.jl#833). Of the GPU back-ends, OpenCL.jl is the only one that does. CUDA.jl and AMDGPU.jl also keep one exception flag per device, but they read it without waiting for other streams.
Proposed fix: one mailbox per command queue, with its address passed in KernelState as today. Once wait_idle(queue) returns, nothing else can be writing to that queue's mailbox. That removes the pending set and the waits on other queues, and exceptions are reported by the task whose kernel threw them. Mailboxes that need mapping (coarse-grained SVM, buffers) can keep using a side queue. The cost is a small host-accessible allocation per queue, released along with the queue.
Activity
Require that synchronize not wait for unrelated queues