Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,17 @@ NOTE: This file has been modified by Intel Corporation.
[![PyTorch](https://img.shields.io/badge/PyTorch-2.13+-blue.svg)](https://pytorch.org/)
[![Tests](https://github.com/intel/torch-tweak/actions/workflows/tests.yml/badge.svg)](https://github.com/intel/torch-tweak/actions/workflows/tests.yml)
[![Nightly Tests](https://github.com/intel/torch-tweak/actions/workflows/nightly.yml/badge.svg?event=schedule)](https://github.com/intel/torch-tweak/actions/workflows/nightly.yml)
[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/intel/torch-tweak/badge)](https://securityscorecards.dev/viewer/?uri=github.com/intel/torch-tweak)

**Torch Tweak** is an inference toolkit designed for tuning and deploying Deep Learning models with a focus on Intel XPU (GPU) accelerators.
It provides model tuning capabilities through compilation and conversion paths that can significantly improve inference speed and efficiency
across various AI workloads including Computer Vision, Natural Language Processing, Speech Recognition, and Generative AI.

The toolkit tunes PyTorch models and pipelines through one Python API. Four backends ship with the package:
The toolkit tunes PyTorch models and pipelines through one Python API. Five backends ship with the package:

* **TorchAO** quantizes weights through PyTorch AO
* **Torch Inductor** compiles with `torch.compile`
* **XPU Graph** records the module once on an Intel GPU during tuning and replays it for inference; input tensor shapes must remain static
* **OpenVINO** converts a module to an OpenVINO model and runs it on an Intel CPU or GPU
* **Torch Eager** runs the module with no compilation, as the baseline other backends are compared against

Expand Down
4 changes: 3 additions & 1 deletion docs/guides/advanced/tuning_workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,10 +159,11 @@ The tuning strategy determines which backend(s) to try and how to select the bes

#### FirstWinsStrategy

Tries backends in priority order and returns the first one that builds and validates successfully. Not every backend can handle every model (e.g., OpenVINO may fail during model conversion, Torch Inductor may hit graph breaks), so this strategy provides automatic fallback instead of aborting.
Tries backends in priority order and returns the first one that builds and validates successfully. Not every backend can handle every model (e.g., OpenVINO may fail during model conversion, Torch Inductor may hit graph breaks, or XPU Graph will reject inputs with dynamic tensor shapes), so this strategy provides automatic fallback instead of aborting.

```python
strategy = tt.FirstWinsStrategy([
tt.backend.XPUGraphBackend(),
tt.backend.TorchInductorBackend(),
tt.backend.OpenVINOBackend(),
tt.backend.TorchEagerBackend(),
Expand Down Expand Up @@ -203,6 +204,7 @@ Before actual tuning, this strategy tries to estimate `max_batch_size`. It does
strategy = tt.HighestThroughputStrategy([
tt.backend.TorchAOBackend(),
tt.backend.TorchInductorBackend(),
tt.backend.XPUGraphBackend(),
tt.backend.OpenVINOBackend(),
tt.backend.TorchEagerBackend(),
])
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/backends/xpu_graph_backend.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ strategy = HighestThroughputStrategy(backends=[XPUGraphBackend()])
The backend is first in the default `FirstWinsStrategy` list, before Torch Inductor.
It is also in the default `HighestThroughputStrategy` list.

PyTorch records a native SYCL graph when it was built with a SYCL compiler of 2026.1 or newer, except on PVC.
PyTorch records a native SYCL graph when it was built with a SYCL compiler of 2026.1 or newer, except on Intel Data Center GPU Max Series.
An older build records the legacy SYCL command graph.
This backend uses the recording that build already performs.

Expand Down
7 changes: 6 additions & 1 deletion docs/guides/tune_strategies/tune_strategies.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ Not every backend can successfully tune every model. Each backend relies on a di

- **OpenVINO** requires converting the model to an OpenVINO representation. Models with unsupported operators, complex dynamic control flow, or symbolic shape constraints may fail during conversion or model compilation. Memory constraints can also prevent the model from being compiled.
- **Torch Inductor** uses `torch.compile`, which may encounter *graph breaks* on unsupported Python constructs or operations, causing partial or failed compilation.
- **XPU Graph** records the module for replay, so it requires static tensor shapes. A failed recording or a mismatch against eager output rejects the backend for that module and input.
- **TorchAO** applies quantization transformations that may not support all layer types or model architectures.
- **Torch Eager** runs the module unchanged, so it always succeeds, which makes it a useful last-resort fallback.

Expand Down Expand Up @@ -87,6 +88,7 @@ Tries backends in priority order and returns the first one that successfully bui

```python
from torch_tweak.torch.backend import (
XPUGraphBackend,
TorchAOBackend,
TorchAOBackendConfig,
TorchInductorBackend,
Expand All @@ -95,6 +97,7 @@ import torch_tweak.torch as tt

# List backends in priority order (fastest → most compatible)
backends = [
XPUGraphBackend(),
TorchAOBackend(config=TorchAOBackendConfig(quantization="fp8wo")), # Quantization: fast inference
TorchInductorBackend(), # Good performance, broader compatibility
]
Expand Down Expand Up @@ -147,12 +150,14 @@ from torch_tweak.torch.backend import (
TorchInductorBackend,
TorchAOBackend,
TorchAOBackendConfig,
XPUGraphBackend,
)
import torch_tweak.torch as tt

# List all candidate backends
backends = [
TorchInductorBackend(config=TorchInductorBackendConfig(mode="default")),
XPUGraphBackend(),
TorchAOBackend(config=TorchAOBackendConfig(quantization="fp8wo")),
]

Expand Down Expand Up @@ -201,6 +206,6 @@ tt.tune(model, input_data)

## Next Steps

- Learn about specific backends: [TorchAO](../backends/torchao_backend.md), [Inductor](../backends/torch_inductor_backend.md)
- Learn about specific backends: [TorchAO](../backends/torchao_backend.md), [Inductor](../backends/torch_inductor_backend.md), [XPU Graph](../backends/xpu_graph_backend.md)
- Explore [Deployment Guide](../deployment/deployment.md)
- Review the [Tuning Guide](../tuning.md)
2 changes: 1 addition & 1 deletion docs/learn/quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ Tuning gives you control over the process:
* it detects the batch axis and dynamic axes (axes that change shape independently of batch size, e.g., sequence length in LLMs)
* allows picking modules to tune
* you can pick a tuning strategy (e.g., best throughput) for the whole process or per-module
* you can pick tuning backends (for example, TorchAO, Torch Inductor, OpenVINO, and eager) which will be used by the strategy
* you can pick tuning backends (for example, XPU graph, TorchAO, Torch Inductor, OpenVINO, and eager) which will be used by the strategy
* you can mix different backends in the same model/pipeline
* you can manually verify the tuning process (note: Torch Tweak performs basic checks for NaNs and errors)
* you can save the resulting artifact and later read it from disk
3 changes: 1 addition & 2 deletions examples/E5Large/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,5 +69,4 @@ uv run inference --prompt "query: What is the capital city of France?"

## Model Details

Can be found in following pages:
* https://huggingface.co/intfloat/e5-large-v2
* [Hugging Face Model](https://huggingface.co/intfloat/e5-large-v2)
5 changes: 2 additions & 3 deletions examples/ParakeetCTC/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,5 @@ uv run inference

## Model Details

Can be found in following pages:
* https://huggingface.co/nvidia/parakeet-ctc-0.6b
* https://docs.nvidia.com/nemo-framework/user-guide/24.09/nemotoolkit/asr/models.html
* [Hugging Face Model](https://huggingface.co/nvidia/parakeet-ctc-0.6b)
* [Nemo Documentation](https://docs.nvidia.com/nemo-framework/user-guide/24.09/nemotoolkit/asr/models.html)
5 changes: 2 additions & 3 deletions examples/ParakeetRNNT/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,5 @@ The service uses dynamic batching - requests are grouped and processed together

## Model Details

Can be found in following pages:
* https://huggingface.co/nvidia/parakeet-rnnt-1.1b
* https://docs.nvidia.com/nemo-framework/user-guide/24.09/nemotoolkit/asr/models.html
* [Hugging Face Model](https://huggingface.co/nvidia/parakeet-rnnt-1.1b)
* [Nemo Documentation](https://docs.nvidia.com/nemo-framework/user-guide/24.09/nemotoolkit/asr/models.html)
5 changes: 2 additions & 3 deletions examples/ResNet/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,5 @@ uv run inference --model-name resnet50 --image-path your_image

## Model Details

Can be found in following pages:
* https://pytorch.org/vision/stable/models.html#classification
* https://huggingface.co/timm
* [Hugging Face Model](https://huggingface.co/timm)
* [Pytorch Documentation](https://pytorch.org/vision/stable/models.html#classification)
Loading