This project traces a model graph, applies compiler transformations, lowers the result to a Load/Store/Compute instruction set, and evaluates execution with functional simulation and a hardware cost model. Its structure follows the compiler stages used by ML accelerator software, including graph rewriting, code generation, buffer management, and performance analysis.
PyTorch nn.Module
| torch.fx.symbolic_trace
v
mlacc IR (Graph of Ops)
| fusion pass
v
Fused IR (elementwise chains merged)
| codegen
v
Toy ISA (Load / Store / Compute)
| simulator
v
Cycle count + final memory state
Each stage is a standalone module, so any stage can be tested and reasoned about independently of the ones around it.
passes/fusion.py walks the graph and merges consecutive elementwise ops
(add, relu, mul) into a single fused_elementwise instruction whenever
two conditions hold:
- every op in the chain is elementwise
- the tensor connecting two ops in the chain has exactly one consumer
The second condition is what makes this a correctness-preserving
transformation rather than just a memory optimization. If a tensor feeds two
different ops, folding it into one of them would silently drop the other
op's input. matmul is excluded from the elementwise set entirely, since it
has a different cost profile (compute-bound rather than memory-bound) and
fusing it would break the fusability analysis.
The ISA has three instructions: Load, Store, and Compute. The
simulated hardware has one compute core, one on-chip buffer, and one
off-chip memory. This isolates the specific thing the project measures:
the off-chip memory traffic that fusion removes.
Cost model:
Load/Store:size_bytes / memory_bandwidth_bytes_per_cycle + fixed_memory_latency_cyclesCompute:flops / compute_flops_per_cycle
The default HardwareConfig is 16 FLOP/cycle, 1 byte/cycle of memory
bandwidth, 10 cycles of fixed memory latency, and a 4096-byte on-chip
buffer. On the traced 2-layer MLP, this produces 536,048 unfused cycles and
372,168 fused cycles, a 30.6% reduction.
Fusion's effect on this model is direct: a fused chain of N elementwise ops replaces N separate Store+Load round trips to off-chip memory with a single Store and a single Load around the whole chain. Compute cost is unchanged, since fusion doesn't change how much arithmetic happens, only how often intermediate results cross the memory boundary.
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txtlower_to_isa uses the simplest possible strategy: every op's output is
stored to memory, and every op's input is loaded from memory. This keeps the
unfused case's Store+Load round trips visible and simple to reason about,
which is what makes the fused-vs-unfused comparison clean. A buffer-aware
codegen that kept values on-chip between dependent ops would collapse that
distinction and is left as a stretch goal.
Fill in after running main.py on the example workload:
| Cycles | Off-chip Loads/Stores | |
|---|---|---|
| Unfused | ||
| Fused |
The IR/fusion/ISA/codegen split mirrors the structure of TVM's Relay
(graph-level IR) and TE/TIR (schedule and low-level IR) split. The fusion
pass here implements one rule — merging linear chains of elementwise ops —
which is a special case of what TVM's FuseOps computes generally via
post-dominator tree analysis across multiple op categories.
This project's motivating idea — that fusion's benefit comes from cutting memory traffic, not from doing less compute — is also the core idea behind FlashAttention's kernel fusion and the fused kernels used in inference engines like vLLM and SGLang.
mlacc/
ir.py Op and Graph dataclasses
passes/fusion.py elementwise fusion pass
isa.py Load / Store / Compute instruction definitions
cost_model.py HardwareConfig and per-instruction cost functions
simulator.py executes instructions, tracks memory + cycles
codegen.py lowers a Graph to a list of instructions
trace.py traces a PyTorch nn.Module into a Graph
main.py end-to-end demo: fused vs. unfused comparison
examples/tiny_mlp.py
tests/
From the project root:
python tests/test_ir.py
python tests/test_fusion.py
python tests/test_simulator.py
python tests/test_codegen.py
python main.py
The simulator and code generator are NumPy-only, while the end-to-end CLI uses the PyTorch FX front end:
- A memory tiling pass, applying the same traffic-reduction idea within a single op rather than across a chain of ops
- Matmul + bias + relu epilogue fusion, a pattern used in real inference kernels
- A configurable
HardwareConfigsweep showing how fusion's benefit changes as memory bandwidth increases
The command prints the unfused and fused cycle counts, verifies that both schedules produce the same output, and sweeps sustained memory bandwidth and buffer capacity. The current buffer model spills an output when the resident tensors exceed capacity; tiled execution is not part of this simulator.
The measured bandwidth sweep for the traced MLP was:
| Bandwidth | Unfused cycles | Fused cycles | Speedup |
|---|---|---|---|
| 0.5 B/cycle | 1,044,800 | 717,080 | 1.46x |
| 1.0 B/cycle | 536,048 | 372,168 | 1.44x |
| 2.0 B/cycle | 281,672 | 199,712 | 1.41x |
| 4.0 B/cycle | 154,484 | 113,484 | 1.36x |
The 64-, 192-, and 4096-byte buffer settings produced the same values for this workload because its intermediate tensors exceed those capacities.
The cycle estimator also has a C++/pybind11 implementation. Build it with:
python setup.py build_ext --inplacemlacc.native.estimate_cycles accepts the same ISA instructions and hardware
configuration as the Python simulator. The Python implementation remains the
functional reference; the native backend accelerates cycle estimation.
The native matmul ladder can be benchmarked with:
python -m mlacc.matmul_benchmark --m 256 --n 256 --k 256It checks each implementation against NumPy before reporting timings for naive, tiled, and NEON matmul.
Generate the roofline plot with:
python -m mlacc.roofline --output roofline.png \
--benchmark-output matmul_benchmark.pngThe command generates two presentation plots from the same benchmark:
The roofline shows modeled add, relu, and matmul points, the compute and
bandwidth ceilings, the ridge point, and measured native matmul points.
The benchmark chart shows execution time and achieved GFLOP/s for the C++ matmul ladder. Results are local measurements on Apple Silicon; NumPy uses the platform's optimized Accelerate/BLAS implementation and is intentionally not included in the C++-only comparison chart.
The standalone CUDA benchmark is in
benchmarks/flash_attention_bench.py.
It is designed to run in Kaggle or Colab and does not affect the local
simulator demo in mlacc/main.py.
In a GPU notebook:
git clone https://github.com/hectar-glitches/compiler.git
cd compiler
pip install -r requirements.txt
python benchmarks/flash_attention_bench.py \
--output benchmarks/results/flash_attention.csvThe script compares explicitly materialized attention against
torch.nn.functional.scaled_dot_product_attention, runs both non-causal and
causal modes by default, and reports median latency, peak allocated GPU
memory, and achieved TFLOP/s using 4 * batch * heads * sequence_length^2 * head_dim FLOPs. Peak allocated GPU memory means the maximum memory allocated
by one attention call, including the Q/K/V and output tensors; the peak
counter is reset before each measured case. It verifies the outputs and
writes the GPU, Git commit, allocator configuration, PyTorch/CUDA versions,
tensor dimensions, dtype, warmup count, repeat count, and causal mode into the
CSV. Sequence lengths that exceed the available GPU memory are
recorded as oom instead of terminating the whole sweep, with the CUDA
exception text saved in the error column. The benchmark releases comparison
outputs and calls torch.cuda.empty_cache() between implementations and
sequence-length/mode cases so cached allocator blocks are available for the
next measurement.
The checked-in T4 results use batch size 1, 8 heads, head dimension 64, FP16, 10 warmups, and 30 timed repetitions. At sequence length 4096 in non-causal mode:
| Implementation | Median latency | Peak memory | TFLOP/s |
|---|---|---|---|
| Materialized | 9.162 ms | 544.125 MiB | 3.750 |
| SDPA | 2.591 ms | 32.125 MiB | 13.261 |
That is 3.54x lower latency and 16.94x lower peak allocated memory for SDPA.
At 8192, both implementations still complete: materialized attention uses
2104.125 MiB and SDPA uses 56.125 MiB in non-causal mode. At 16384,
non-causal materialized attention completes at 8296.125 MiB, while causal
materialized attention records oom; SDPA completes in both modes. The
causal materialized failure tried to allocate another 4.00 GiB on a T4 with
2.11 GiB free, while 3.98 GiB was reserved but unallocated by PyTorch. This
is consistent with allocator fragmentation while roughly 12.3 GiB of live
tensors were already present; it is not evidence that the live tensors alone
exceeded the T4's 14.56 GiB capacity.
The checked-in CSV is the v2 Kaggle export, including the error column. It
predates the git_sha and pytorch_cuda_alloc_conf provenance columns; new
runs include both. A clean-allocator rerun with
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True would distinguish
fragmentation from a hard capacity limit. A 32768 run would provide a separate
capacity-wall experiment; it has not been run from this machine.
The CSV uses the full-matrix FLOP formula for every row. Causal TFLOP/s values should therefore be described as effective TFLOP/s; they are not a claim that the causal kernel performed the full non-causal amount of useful work.
The unit tests build Graph/instruction objects directly. The FX front end
and end-to-end CLI require PyTorch.
- PyTorch FX tracing for
LinearandReLUmodels. - Elementwise fusion with multi-consumer and non-elementwise safety checks.
- Load/Store/Compute code generation with buffer-aware spilling.
- Functional NumPy simulation and configurable cycle modeling.
- A C++ cycle estimator exposed through pybind11.
- Naive, tiled, and NEON matmul kernels with NumPy correctness checks.
- A roofline model and benchmark visualizations in
roofline.pngandmatmul_benchmark.png.
- The simulator test suite has one pre-existing semantic mismatch: one test
describes
relu(x - 10)but emitsrelu(x). - Buffer-aware spilling is implemented; tiled execution is not.
- The modeled roofline uses explicit assumptions from
HardwareConfig; it is not a measurement of the host CPU's peak performance.

