Skip to content

About

Simple & Best repo to build your first LLM from scratch.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

23 Commits

Folders and files

Repository files navigation

Building a 42M Generative Transformer from Scratch

W&B Report

Read Full Technical Report on Weights & Biases


Transformer Forward Pass


🎓 Do you want to build your first LLM?
We wrote a step-by-step tutorial explaining every single equation, line of code, and how to train your own model for free on Kaggle with VS Code:
👉 The Complete Master Guide: Build Your First Mini-LLM from Scratch


Overview

This is an end-to-end autoregressive language model. There are no high-level trainer wrappers or imported transformer packages. Every layer, attention score, streaming data buffer, and training loop is built directly from scratch.

Training follows the modern two-stage process:

  1. Pre-training: Next-token language modeling on 160M tokens of synthetic stories (roneneldan/TinyStories).
  2. Instruction Tuning (SFT): Alignment into a conversational chatbot using ChatML formatting on filtered instruction data (tatsu-lab/alpaca).

All experiments, loss curves, and emergent generations are tracked live in Weights & Biases.


Key Results

Metric Pre-trained Base Model Conversational Chat Model
Validation Loss 1.5556 3.4893
Perplexity 4.74 32.76
Training Dataset TinyStories (160M tokens) Filtered Alpaca (10k instructions)
Throughput (NVIDIA T4) 40,500 tokens/sec 115 - 130 tokens/sec (Inference)
FP16 Checkpoint Size 88.30 MB 88.30 MB

Model Architecture

The model is a 12-layer causal decoder-only Transformer :

  • Layers: 12 Transformer blocks with PyTorch FlashAttention and Pre-LayerNorm.
  • Dimensions: 512 embedding size, 8 attention heads, 2048 MLP expansion.
  • Context Length: 256 tokens.
  • Custom 8k Tokenizer: Instead of a generic 50k GPT-2 vocabulary that consumes over half of the model's parameters in embeddings alone, we trained an 8,192 Byte-Level BPE tokenizer. This saved 83.7% of embedding memory and reserved 89.7% of the parameter budget (37.76M) directly for the core transformer compute.
  • Total Parameters: 42.09 Million.

About

Simple & Best repo to build your first LLM from scratch.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Contributors

Languages