Product: Universal AI Inference Runtime (UAII Runtime)
Document: Vision Statement
Version: 0.1
Status: Draft
Universal AI Inference Runtime (UAII Runtime) aims to become the universal execution platform for AI inference.
Instead of building another model-specific runtime (llama.cpp, Kimi runtime, MLC, vLLM-style engines), UAII Runtime provides modular infrastructure capable of executing any AI model on any hardware through a common intermediate representation and execution engine.
UAII Runtime should become the equivalent of LLVM for inference—not a compiler in the traditional sense, but the foundational execution infrastructure upon which future AI systems are built.
Tagline: Any Model → UAII Runtime → Any Hardware
Current AI inference ecosystems are fragmented.
Every new architecture typically requires:
| Fragmented concern | Today’s cost |
|---|---|
| New runtime | Rewrite execution loops and session APIs |
| New kernels | Reimplement MatMul, Attention, norms, MoE routing |
| New loaders | Ad-hoc GGUF / Safetensors / ONNX / MLX paths |
| New optimizations | Fusion, quantization, memory reuse reinvented |
| New hardware ports | CUDA, Metal, ROCm, Vulkan each siloed |
This duplication slows research, fragments the open-source ecosystem, and forces hardware vendors and model authors into brittle, one-off integrations.
UAII Runtime eliminates this fragmentation by introducing a universal execution architecture centered on a single intermediate representation (UAII IR) and a plugin-driven execution operating system.
UAII Runtime is not “another inference engine.”
UAII Runtime is an execution operating system for AI inference.
Everything is modular. Nothing in the core depends on a specific model family, tokenizer, or accelerator vendor.
| Concept | In UAII Runtime |
|---|---|
| Models | Plugins (loaders + IR graphs) |
| Hardware | Plugins (backends) |
| Operators | Plugins (kernels + registry entries) |
| Schedulers | Plugins |
| Storage | Plugins (RAM, NVMe, cloud, remote memory) |
| Quantization formats | Plugins |
| Profilers / visualizers | Plugins |
The long-term outcome: researchers ship new architectures as operator and loader plugins; hardware vendors ship backends; application developers target one stable runtime API.
| LLVM world | UAII Runtime world |
|---|---|
| Frontends (Clang, Rustc, …) | Loaders (GGUF, Safetensors, ONNX, …) |
| LLVM IR | UAII IR |
| Optimization passes | Graph optimizer, planner, fusion |
| Backends (x86, ARM, GPU ISAs) | Hardware backends (CPU, CUDA, Metal, …) |
| Runtime / codegen | Scheduler + kernel dispatch |
Important distinction: UAII Runtime is execution infrastructure, not a full ML compiler stack. It may host or cooperate with compilers, but its primary job is portable, deterministic, high-performance graph execution.
Support arbitrary model architectures without modifying the runtime core. New families (Transformer, MoE, MLA, Mamba, RWKV, Hyena, and future research models) appear as operators and loaders, not forks of the engine.
First-class target surfaces:
- CPU (x86_64, aarch64; SIMD-first)
- CUDA
- Metal
- ROCm
- Vulkan
- WebGPU
- Future NPUs and ASICs
Backends implement a shared capability and dispatch interface; the core never hardcodes vendor APIs.
Inference is increasingly limited by storage and memory bandwidth, not peak FLOPs.
Treat storage as a scheduling resource, equal in importance to compute:
- RAM
- NVMe / local disk
- Cloud / object storage
- Remote memory
- Compression and streaming
- Future distributed caches
Tensors are addressed via handles, not raw pointers alone.
Every subsystem exposes stable interfaces. No hardcoded model assumptions in uaii-core. Plugin discovery and versioning are first-class.
Produce bit-identical outputs whenever backend precision and reduction order permit. Document and test determinism modes explicitly (strict vs. fast).
- Zero-cost abstractions where possible
- Minimal allocations on hot paths
- Cache-aware execution
- SIMD-first CPU kernels
- Explicit memory budgets and reuse
- Avoid abstraction tax that cannot be optimized away
| Audience | Why UAII Runtime matters |
|---|---|
| Researchers | Run novel architectures without forking a runtime |
| ML infrastructure engineers | One substrate across formats and devices |
| Runtime developers | Clean IR, planner, and backend contracts |
| Model developers | Ship loaders / operators as plugins |
| Edge AI developers | Storage-aware, multi-backend execution |
| Hardware vendors | Integrate via backend plugins |
| Open-source contributors | Modular libraries/modules with clear boundaries |
UAII Runtime deliberately does not aim to be:
- A training framework
- A fine-tuning or RLHF stack
- A dataset management platform
- A model hosting / model hub product
- A full cloud serving platform (autoscaling, multi-tenant SaaS)
- An experiment tracking system (Weights & Biases–style)
Serving layers, UIs, and orchestration systems may build on UAII Runtime; they are not the runtime itself.
When the vision succeeds:
- Any Model — Multiple formats and architectures execute through one IR and one core.
- Any Hardware — New accelerators land as backends without core rewrites.
- Plugin velocity — Operators, loaders, and schedulers ship independently.
- Storage-aware inference — Large models stream and page efficiently across memory tiers.
- Observable by default — Profiling and graph visualization are built-in, not bolt-ons.
- Stable core — The execution OS remains small, versioned, and long-lived while the ecosystem grows around it.
- A contributor adds a new attention variant by registering an operator plugin—no changes to
uaii-runtimeinternals. - A hardware vendor ships a Metal or NPU backend that runs the same UAII IR graphs used on CPU/CUDA.
- A researcher converts a Safetensors checkpoint once, inspects the IR, profiles execution, and ports to edge hardware without rewriting kernels for each stack.
- The community treats UAII IR as a lingua franca between loaders, optimizers, and backends—the way LLVM IR unified compiler frontends and targets.
| Preferred name | Usage |
|---|---|
| Universal AI Inference Runtime | Full product name |
| UAII Runtime | Short form in docs and UI |
| UAII IR | Intermediate representation |
| uaii-* | Repository / library / module prefix |
Do not use alternate product codenames in public documentation.
- Architecture — System design, modules, IR, plugins
- Features — Capability catalog and interfaces
- Plan — Phased roadmap, tech stack, milestones