Train decoder-only language models from scratch with PyTorch, featuring RoPE, RMSNorm, SwiGLU, and SDPA

A complete implementation of a Decoder-Only Transformer (GPT-style) built using PyTorch, without relying on high-level abstractions. This implementation includes all core components: token embeddings, positional embeddings, multi-head self-attention, feedforward networks, causal masking, and output logits generation.

A hands-on implementation of the technologies behind modern Large Language Models, covering gradient descent, backpropagation, tokenization, transformers, self-attention, and GPT architecture. The project emphasizes understanding AI systems from first principles rather than relying solely on high-level frameworks.

Decoder-only GPT-style Transformer for autoregressive language modeling with BPE tokenization, supporting greedy, temperature, top-k, and nucleus sampling