🎓 Do you want to build your first LLM?
We wrote a step-by-step tutorial explaining every single equation, line of code, and how to train your own model for free on Kaggle with VS Code:
👉 The Complete Master Guide: Build Your First Mini-LLM from Scratch
This is an end-to-end autoregressive language model. There are no high-level trainer wrappers or imported transformer packages. Every layer, attention score, streaming data buffer, and training loop is built directly from scratch.
Training follows the modern two-stage process:
- Pre-training: Next-token language modeling on 160M tokens of synthetic stories (
roneneldan/TinyStories). - Instruction Tuning (SFT): Alignment into a conversational chatbot using ChatML formatting on filtered instruction data (
tatsu-lab/alpaca).
All experiments, loss curves, and emergent generations are tracked live in Weights & Biases.
| Metric | Pre-trained Base Model | Conversational Chat Model |
|---|---|---|
| Validation Loss | 1.5556 | 3.4893 |
| Perplexity | 4.74 | 32.76 |
| Training Dataset | TinyStories (160M tokens) | Filtered Alpaca (10k instructions) |
| Throughput (NVIDIA T4) | 40,500 tokens/sec | 115 - 130 tokens/sec (Inference) |
| FP16 Checkpoint Size | 88.30 MB | 88.30 MB |
The model is a 12-layer causal decoder-only Transformer :
- Layers: 12 Transformer blocks with PyTorch FlashAttention and Pre-LayerNorm.
- Dimensions: 512 embedding size, 8 attention heads, 2048 MLP expansion.
- Context Length: 256 tokens.
- Custom 8k Tokenizer: Instead of a generic 50k GPT-2 vocabulary that consumes over half of the model's parameters in embeddings alone, we trained an 8,192 Byte-Level BPE tokenizer. This saved 83.7% of embedding memory and reserved 89.7% of the parameter budget (37.76M) directly for the core transformer compute.
- Total Parameters: 42.09 Million.
