llm-from-scratch

Build an LLM from scratch with transformers, attention, tokenization, dataset engineering, training, alignment, inference, multimodality, agents, and a runnable PyTorch mini language model.

Building a complete LLM from scratch. No libraries. No shortcuts. Just NumPy, math, and first principles.

Implementations, assignments, and experiment notes for Stanford CS336: Language Modeling from Scratch.

DeepSeek-style MoE + MLA from scratch in PyTorch, with a router-specialization probe (mutual information) and a dense/+MoE/+MLA ablation. Educational, nano-scale, measured.

LLM Which Was Trained On A Consumer GPU

Modular, step-by-step implementation of GPT & Modern LLMs from scratch in PyTorch. Featuring RoPE, RMSNorm, SwiGLU, GQA, KV-Cache, Unit Tests, and an Interactive Web Playground.

From-scratch PyTorch: frontier LLM techniques as of 2026-Q1 — the Muon optimizer and Multi-Token Prediction, plus a base BPE tokenizer. Self-contained, self-checking modules.

An industry-grade, production-ready Multi-Modal AGI Vision-Language LLM & Agentic Reasoning Framework built from scratch in PyTorch. Features ViT patch encoding, RoPE, SDPA Flash-Attention, KV-Cache, autonomous Plan-Act-Reflect agentic loop with tool dispatch & Google Colab T4 GPU support.

Modern decoder-only Transformer lab with GQA, RoPE, attention bridges, verified KV cache, and cloud experiments.

A large language model built from scratch using PyTorch. Currently, it is in the initial stage of the development process using PyTorch.🔧

A GPT-style language model trained from scratch on public-domain text using a decoder-only Transformer, built for LLM learning, experimentation, and research.

Educational GPT-style LLM built from scratch in PyTorch, covering tokenization, data pipelines, Transformer architecture, pretraining, evaluation, fine-tuning, and text generation.

From-scratch PyTorch GPT: the GPT-2 skeleton plus the components of current open LLMs — RMSNorm, RoPE, SwiGLU, GQA, no-bias + tied embeddings. Each isolated and self-checking.

Some attempts, based on MicroGPT.

面向只有 Python 基础学习者的中文 LLM 从零学习伴侣，配合 rasbt/LLMs-from-scratch，含课程、实验、CLI、个性化计划与测试。

A focused PyTorch implementation of GPT-2 124M with architecture tests and a portfolio-ready technical report.

A from-scratch implementation of a Large Language Model in Python, built while following freeCodeCamp's Create a Large Language Model from Scratch with Python – Tutorial.

An ~8M parameter LLM that talks like a tiny robot vacuum.