This Demo
Type What is 1234 plus 5678? in the browser and press Enter—below is the real streaming output (1234+5678 is not in the SFT training data, but the model learned to extend its capabilities via the calc tool):
👤 What is 1234 plus 5678?
💭 I need to compute 1234 + 5678.
🔧 calc(1234 + 5678)
↳ 6912 // Python eval, not model
🤖 6912.
A 124M model absolutely cannot get 4-digit addition right on its own (the chat-SFT version answers "13"). With the calc tool, it can. This is the essence of agents—model capability = knowledge + tool extension.
Eight-Layer Architecture
Each layer has < 300 lines of core code and runs independently. The entire stack runs from scratch on an RTX 5090 in ~70 seconds.
L1
GPU Fundamentals ·
05_gpu/Hand-written CUDA naive matmul / tiled matmul (demonstrating tiling), Triton fused flash-attention (8.4× speedup vs. PyTorch unfused)
~165 lines
L2
Transformer Architecture + BPE ·
04_transformer/330-line hand-written GPT-2 (embed/MHA/FFN/LN/KV cache) + 230-line hand-written BPE (bit-for-bit equivalent to tiktoken, 7/7 test suite 100% pass)
~560 lines
L3
Pretraining ·
00_train/Train a 7M GPT from random weights. 1000 steps on a 1.1MB Shakespeare corpus, loss drops from 10.815 (= ln(50257)) to 4.55
12 sec
L4
Instruction SFT ·
00b_sft/242 hand-written Q/A pairs teach the base model to answer questions. Path A uses our self-trained 7M base; Path B uses OpenAI gpt2-124M (unlocking world knowledge)
28 sec
L5
Agent SFT ·
00c_agent_sft/258 ReAct traces teach the model to call calc/lookup tools. Loss masking ensures the OBSERVATION prefix is learned but its content is not
33 sec
L6
Inference Serving ·
03_model/FastAPI + self-implemented KV cache. Prefill 1.8 ms / decode 2.6 ms (124M, 5090, batch=1). Zero transformers runtime
~140 lines
L7
App / Web UI ·
01_app/FastAPI + SSE streaming output + ~80 lines of vanilla HTML/JS frontend. The browser receives thought / action / observation token by token
~130 lines
L8
Agent Loop ·
02_agent/Defaults to a chat completion client. AGENT_MODE=1 enables the ReAct loop: generate-stop-parse-execute-inject cycle
~200 lines
Key Numbers (Measured)
RTX 5090, three independent cold-start verifications. See full raw logs at reports/.
70 sec
L3 + L4 + L5 total training time (excluding 124M weight download)
10.815
L3 step 0 loss = ln(50257) theoretical value, verifying weight initialization
7/7 ✓
Hand-written BPE bit-for-bit equivalent to tiktoken (incl. Chinese/Japanese/emoji)
< 1e-6
Hand-written KV cache vs. full forward numerical diff (floating-point precision)
8.4×
Triton flash-attention vs. PyTorch unfused speedup ratio
8 / 10
Agent E2E test accuracy (greedy decoding, 1234+5678→6912 ✓)
The Most Important Demo: 1234 + 5678 → 6912
This record is one of the most important empirical results in the entire project.
The model only saw addition within the 1-99 range during SFT training. Yet at inference time it correctly generates
This validates the essential thesis of agents: the model learns the format contract of tool invocation ("when you see an arithmetic problem, call calc"), not the tool's capability itself (computation). This is the same essence as ChatGPT connecting to web_search, Cursor connecting to grep+edit, and Claude connecting to computer use.
calc(1234 + 5678), passing arbitrary numbers to a real Python tool.This validates the essential thesis of agents: the model learns the format contract of tool invocation ("when you see an arithmetic problem, call calc"), not the tool's capability itself (computation). This is the same essence as ChatGPT connecting to web_search, Cursor connecting to grep+edit, and Claude connecting to computer use.
| Query | Category | Output | |
|---|---|---|---|
| capital of France? | KB lookup, in-data | Paris. | ✓ |
| 23 plus 47? | calc, in-data | 70. | ✓ |
| 1234 plus 5678? | calc, OOD | 6912. | ✓ Key generalization |
| Who wrote Hamlet? | KB lookup, in-data | Shakespeare. | ✓ |
| chemical symbol of gold? | KB lookup, in-data | Au. | ✓ |
| capital of Mongolia? | KB miss | not found. | ✓ Honest |
| How are you today? | OOD conversational | hallucinate lookup | ✗ Known failure |
Starting from Scratch
Complete cold-start commands (CN region, tested three times):
# 1. Clone (direct GitHub access doesn't work in CN, use gh-proxy)
git clone https://gh-proxy.com/https://github.com/fxp/LLM-from-query-to-result.git
cd LLM-from-query-to-result
# 2. Configure pip mirror + install
mkdir -p ~/.pip
echo -e "[global]\nindex-url = https://mirrors.aliyun.com/pypi/simple/\ntrusted-host = mirrors.aliyun.com" > ~/.pip/pip.conf
pip install -r requirements.txt
# 3. Train base + SFT + agent SFT (~70 sec on 5090)
cd 00_train && python prepare.py && python train.py
cd ../00b_sft && python train.py && python train_from_gpt2.py # The latter downloads 124M weights on first run (HF mirror auto-fallback)
cd ../00c_agent_sft && python build_data.py && python train.py
# 4. Start server (agent.pt is the 124M model with tool-calling)
MODEL_PATH=$(pwd)/out/agent.pt python ../03_model/server.py &
AGENT_MODE=1 uvicorn 01_app.backend.main:app --port 8000 &
# 5. Open http://localhost:8000 in browser
# Type "What is 1234 plus 5678?" → tokens pop out one by one: "6912."Deep Reading
| Article | Topic | |
|---|---|---|
| 📚 | Condensed Single-Post Version | The story of the entire project, 4000 words |
| 📊 | Experiment Report (HTML) · Markdown | Formal format: methods, results, discussion, reproducibility |
| 📖 | 11-Chapter Blog Series | One post per layer, ~30K words total |
| 🧪 | Raw Experiment Logs | Complete stdout from three independent cold-starts |
| 💻 | Full Source Code | ~10K lines Python + ~165 lines CUDA |
Differentiating Contributions
There are already many excellent "build GPT from scratch" tutorials in the industry (karpathy/nanoGPT, minbpe, etc.). This project's differentiators:
| Topic | Most Tutorials | This Project |
|---|---|---|
| GPT architecture | ✓ | ✓ |
| Pretraining loop | ✓ | ✓ |
| BPE tokenizer | Some (minbpe) | ✓ Verified bit-for-bit equivalent to tiktoken |
| SFT instruction tuning | Rarely | ✓ |
| Agent SFT + ReAct loop | Almost never | ✓ |
| Inference serving (KV cache + SSE) | Rarely | ✓ |
| Web frontend | Almost never | ✓ |
| GPU kernel internals | Some | ✓ Triton + CUDA comparison |
| End-to-end trace from browser to matmul | Almost never | ✓ |