CUDA: Allow concurrent streams per split for multi-GPU

[tannerbruhn]

Overview

Previous guard caused multi-GPU to skip the graph optimization. The
graph is already split per device and the optimization doesnt run
over the whole model but once per split, and thus should be allowed.
However, the CUDA event ggml_cuda_concurrent_event belongs to
whichever GPU was "current" when created. If the pass ran while
GPU 0 was current, it would stick and during event creation for the
second GPU it would land on GPU 0.

The fix: set the device explicitly ggml_cuda_set_device(cuda_ctx->device);
Default behaviour remains unchanged, only active for GGML_CUDA_GRAPH_OPT=1.
Explicit device setting pattern re-used from ggml_backend_cuda_graph_compute.

Additional information

Verified with two GPUs

Test setup

Hardware: RTX 5070 Ti 16GB (sm_120, PCIe 3.0 x16) + RTX 3080 Ti 12GB (sm_86, PCIe 3.0 x4, chipset slot), no NVLink, no P2P
Software: Ubuntu 26.04, driver 595.84, CUDA 13.3, GCC 15.2.0
Build: upstream master + this patch, -DCMAKE_CUDA_ARCHITECTURES="86;120"
Models: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL (MoE), Qwen3.8-27B-UD-Q6_K (dense)

Model split across both devices:

load_tensors: CUDA0 model buffer size = 6147.11 MiB
load_tensors: CUDA1 model buffer size = 7426.81 MiB

Concurrent-stream launches observed (Launching N streams at ...):

GGML_CUDA_GRAPH_OPT	launches
unset	0
=1	50

check	result
greedy generation (--temp 0 -s 42), both arms	byte-identical output
perplexity, wikitext-2, 8 chunks, both arms	PPL = 625.5026 +/- 35.78178

Perplexity matched at every individual chunk

Performance — MoE

gemma-4-26B-A4B-it-qat-UD-Q4_K_XL, -ngl 99 -fa 1 -r 3:

test	OPT unset	OPT=1	delta
pp512	5437.44 ± 120.83	5447.48 ± 140.22	+0.2%
tg32	190.24 ± 1.81	197.27 ± 2.24	+3.7%
tg64	191.25 ± 0.65	197.83 ± 0.96	+3.4%
tg128	188.36 ± 0.30	195.38 ± 0.47	+3.7%

Performance — dense

Qwen3.8-27B-UD-Q6_K, -ngl 99 -fa 1 -r 3:

test	OPT unset	OPT=1	delta
pp512	1350.49 ± 14.85	1352.47 ± 13.05	+0.1%
tg32	35.08 ± 0.05	35.00 ± 0.03	-0.2%
tg128	35.31 ± 0.01	35.26 ± 0.01	-0.1%

Prior art — the author of the original stream-concurrency PR on multi-GPU:
#16991 (comment)

as long as the Q, K, V stuff per layer lives on the same device, this
should also work for the multi-GPU case too, but that is not addressed
in this PR

Overlapping work: #21897 touches the same function, happy to rebase if it lands first

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: AI used to discover, patch and benchmark. Verification done on real hardware. Review, commit and pr done by myself.

[tannerbruhn]

Hi @tannerbruhn, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

PR and commit text is mine as disclosed in the AI disclosure. Data and formatting by AI, hope thats fine