Repository navigation
feat: add an XPU graph backend - #33
Merged
Merged
Conversation
Contributor
|
pls fix dco and rebase |
The backend records the module once during tuning and replays that recording for inference. Replay needs static tensor shapes. A dynamic batch rejects this backend. It sits next to Torch Inductor in the default strategies. Timing waits for the device, so a replay is measured after the GPU finishes. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
PatKamin
reviewed
Oct 22, 2026
PatKamin
reviewed
Oct 22, 2026
|
|
||
| def _build(self, module: nn.Module, graph_spec: GraphSpec, data: list[Sample], cache_dir: Path) -> Backend: | ||
| """Records the XPU graph once and checks it against eager.""" | ||
| if graph_spec.has_dynamic_axis(): |
Contributor
There was a problem hiding this comment.
It's worth considering to exclude only dim axes and build and record batch axes, each batch size is profilied independently anyway
Contributor
Author
There was a problem hiding this comment.
reported SPJT-490 for this
PatKamin
reviewed
Oct 22, 2026
TorchInductorBackend and TorchAOBackend saved every recorded sample. A sample holding a transformers DynamicCache, as recorded during generate, made save fail. These samples only compile the module ahead of the first call, so they are left out and the first call compiles instead. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
wrap() named each Module after the last segment of its object path. Siblings such as layers.0.mlp and layers.1.mlp shared the name "mlp". MODULE_REGISTRY is keyed by name, so only the last one was registered. The others were never tuned and kept recording samples on every call. The cache directory, keyed by the same name, collided as well. The wrapper name is now the object path, which is unique within one inspection. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
It requires minimum Python 3.11 version. Signed-off-by: Lukasz Stolarczuk <lukasz.stolarczuk@intel.com>
Drop the Simple line from every backend in the skill. Move the synchronize comment to the sync after the model call. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
Inputs the recording cannot take run on the original module. A larger batch after tuning reaches the backend, so it no longer raises. FirstWinsStrategy tries the XPU graph before Torch Inductor. Inductor almost always builds, so the graph never ran in second place. Tests cover deactivate then activate, deploy after a checkpoint load, and a larger batch. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
The fallback output equals eager, so comparing to eager cannot see it. An autouse fixture records every fallback call. A test that does not take eager_calls fails when the fallback runs. The fallback tests check which inputs fell back. The tune tests compare against a deep copy made before tuning. After tune, calls of the original module go through the selected backend. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
_live_leaves checks the inputs and returns their flattened leaves. _copy_into copies those leaves, so infer no longer flattens args and kwargs twice. _load_inputs copies nothing when an input does not fit. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
The scale=3.0 call always falls back. len(eager_calls) == 1 already shows the first call replayed. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
test_tune_module calls a larger batch, then the tuned batch. Only the larger batch runs the original module. The skill lists the new FirstWinsStrategy order. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
The default FirstWinsStrategy tries XPUGraphBackend first. Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
Contributor
|
Please, rebase |
PatKamin
approved these changes
Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an XPU graph backend
The backend records the module once during tuning and replays that recording for inference.
Replay needs static tensor shapes.
A dynamic batch rejects this backend.
It sits next to Torch Inductor in the default strategies.
Timing waits for the device, so a replay is measured after the GPU finishes.