[DRAFT]: Formalizing scheduler and async backend behavior through tests

[aendk]

Overview

This draft PR takes a first stab at formalizing the behavior of the scheduler and the async backends through test coverage.
There is lots to improve here, and I look forward to do so with your input. The main contribution is test-backend-sched.cpp. To create more complex test-cases, I also added a new operator OP_SLEEP (see below for details).

I've implemented several classes of tests for this:

1. Test for formalizing behavior: Testing memcpys of user inputs and activations from and to every present backend, and also tests for specific patterns (e.g. Y-shaped graphs)
2. Tests reproducing current bugs and confirming proposed fixes (ggml: fix backend split scheduler race condition #26040). The failing test-cases here (using vulkan) are a feature; the PR by @0cc4m lets all current tests (as of August 17th) succeed. Also, ggml: asynchronous scheduler memory copies #25319 fails, too (@JohannesGaessler).
3. Stress tests which test scheduling of simple (linked-list) and more complex (directed acyclic graph) schedules.

New OP_SLEEP

• to create a reliable bug repro for ggml: fix backend split scheduler race condition #26040, I needed a way to consistently delay device processing. I landed on OP_SLEEP as opposed to some big MatMul for the following reasons:
  • OP_SLEEP was easy to implement in this scope (CPU, CUDA, VK)
  • sleep is not machine specific, so correlation between devices in CI and on local developer machines (=better GPUs) is easier
  • reproduction of errors is easier as sleep can be in-/decreased very granularly
  • Will be run run millions of times in CI going forward, so reduces resource waste a bit.
• If specific platforms do not support it, a proxy for it could still be a large enough MatMul / other expensive ops on the specific devices. Need input from the respective backend maintainers if this op can be implemented in their backend.

TODO

• tests are currently very CPU/CUDA-centric, since these are the platforms I have easy access to, and to keep it simple. Will increase scope to vulkan and will integrate opportunistic testing. Whatever backend is present will be tested.
• Test improvements (pseudo-random small sleeps inside the stress tests to trip up scheduling)
• More tests for weird shapes (eg "X"-shaped) graphs

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, I leveraged AI to create much of this PR. Validity checking was done via nvtx-ranges and NSight Systems, the tests work as designed on CUDA, CPU and Vulkan. I steered the AI to keep changes to a minimum, but I need expert feedback on parts of the codebase I am unfamiliar with, such as the CPU and Vulkan backends (@0cc4m)

cc @ggerganov @am17an @ORippler @gaugarg-nv

[Anbeeld]

I opened a multi-GPU tensor-split issue that may be a useful real-world target for this scheduler/backend test work:

#27750

The failure is strongly timing-sensitive and so far I have not been able to reduce it to a deterministic backend test.

Setup is Windows + 2x RTX 3090 + Qwen 3.8 27B tensor split. I also reproduced a related failure on official upstream b10595 with MTP disabled.

A few things make it interesting for this PR:

• slower synchronization/single-stream variants often make the model workload pass;
• those variants reduce prefill from ~1349 tok/s to roughly 795-806 tok/s, so they appear to be timing perturbations rather than fixes;
• an exact frozen 14,425-token fresh request passed three times even though similar-size runs had previously failed;
• CUDA graphs off did not eliminate the problem;
• graph reuse disabled did not eliminate it either;
• failures happen with ample VRAM;
• the exact projected MUL_MAT observed near one failure works correctly in isolation.

This is exactly the kind of bug where turning the timing relationship into a small deterministic scheduler/backend RED would be much more useful than continuing to reproduce it with 20k+ token model workloads.

I can run specific synthetic tests if there is a scheduler/meta pattern you want exercised.

[aendk]

@Anbeeld interesting, thanks for linking this.
I am currently working on expanding this PR to also cover multi-GPU scheduling more comprehensively.
Could be as early as sometime next week.

If you are curious and want to debug further in the mean time, I suggest disabling PDL (GGML_CUDA_PDL=0), CUDA graphs, and blocking execution(CUDA_LAUNCH_BLOCKING=1) to narrow it down further. Note however that all of this slows down CUDA execution, and the timing error might not appear due to this.