Elelems

Mini decoder-only causal Transformer (GPT-like SLM)

The main goal is to learn how LLMs work and create a very simple model capable of performing function calling in video games. I may be wrong about some things, but I will improve them as I learn lol.

I generated the corpus using gpt-5-mini.

Quick Start

after cloning the repo and assuming you're not a boomer and you use UV

uv sync
uv run python main.py prepare
uv run python main.py train --epochs 30 --batch-size 32
uv run python main.py generate --prompt "el gato" --max-new-tokens 8 --top-k 5

Commands

• prepare: cleans the corpus, builds vocab.json, and creates train/val splits in data/processed.
• train: trains the mini transformer and saves data/checkpoints/mini_llm.pt.
• generate: loads the checkpoint and generates text until <END>.

Structure

• data/raws/animals.txt: base corpus.
• elelems/core/data: cleaning, tokenization, vocabulary, and data preparation.
• elelems/core/model: causal mini transformer.
• elelems/core/train.py: training loop.
• elelems/core/generate.py: autoregressive inference.