Skip to content
Sign in
Artwork for My Weird Prompts
My Weird Prompts · October 30 · 28 min

What Does It Take To Train An AI Model From Scratch?

Training a conversational AI from completely random weights requires building an entire statistical education rather than fine-tuning existing layers. Looking closely at open-source pipelines like nanoGPT and nanochat shows how tokenizer choices quietly dictate compute costs across languages, why tiny smoke tests prevent catastrophic compute waste, and how midtraining and supervised fine-tuning turn raw next-token predictors into coherent conversationalists. Key sources: • README, last push 2026-09-07 — https://github.com/karpathy/nanochat • README, last push 2025-11-12 (README updated Nov 2025) — https://github.com/karpathy/nanoGPT • README, last push 2025-06-26 — https://github.com/karpathy/llm.c Episode 6368 — myweirdprompts.com/n/6368 (permanent short link: myweirdprompts.com/911971)

0:00 · Intro-28:18
Play in
Transcript file

transcript

No transcript — this publisher did not publish one.

show notes

Training a conversational AI from completely random weights requires building an entire statistical education rather than fine-tuning existing layers. Looking closely at open-source pipelines like nanoGPT and nanochat shows how tokenizer choices quietly dictate compute costs across languages, why tiny smoke tests prevent catastrophic compute waste, and how midtraining and supervised fine-tuning turn raw next-token predictors into coherent conversationalists.

Key sources:
• README, last push 2026-09-07 — https://github.com/karpathy/nanochat
• README, last push 2025-11-12 (README updated Nov 2025) — https://github.com/karpathy/nanoGPT
• README, last push 2025-06-26 — https://github.com/karpathy/llm.c

Episode 6368 — myweirdprompts.com/n/6368 (permanent short link: myweirdprompts.com/911971)

chapters

7 chapters

more episodes

All episodes