several types of attention modules written in PyTorch for learning purposes
Collection of different types of transformers for learning purposes
Transformer attention, worked through numerically; from self-attention and RoPE to KV cache, MQA, and GQA.
Examine cost-effective methods for optimizing GQA configurations, comparing the performance with its counterparts like Multi-Head Attention (MHA) and Multi-Query Attention (MQA).
CUDA implementation of Multi-Query Attention achieving 97% KV-cache memory reduction for LLM inference, enabling 32x larger batch sizes. Educational project demonstrating CUDA kernel development with PyTorch integration and Llama model benchmarks.
This repository shows how to build a DeepSeek language model from scratch using PyTorch. It includes clean, well-structured implementations of advanced attention techniques such as key–value caching for fast decoding, multi-query attention, grouped-query attention, and multi-head latent attention.
Empirical profiling of MHA, GQA, and MQA attention variants measuring KV-cache memory, decode throughput, and output fidelity. GQA-g2 halves KV-cache while preserving ~70% cosine similarity; MQA reduces cache by 92% but drops to ~27% similarity.
Add a description, image, and links to the multi-query-attention topic page so that developers can more easily learn about it.
To associate your repository with the multi-query-attention topic, visit your repo's landing page and select "manage topics."