Skip to content
Artwork for My Weird Prompts
My Weird Prompts · October 16 · 29 min

Why Chat Templates Break Models Silently

Your JSON messages don't reach the GPU as JSON. Somewhere in between, a Jinja chat template flattens them into a string of reserved special tokens — the exact format each model family was trained on. Llama 3 brackets roles with start_header and end_header; OpenAI's ChatML embedded the role in a single opening token; gpt-oss uses Harmony with separate stop tokens for "done talking" and "calling a tool"; Gemma has no system role at all; DeepSeek uses full-width CJK pipes. This episode walks through what special tokens actually are, how tokenizers register them, why the add_generation_prompt flag matters more than anyone thinks, and why a format mismatch never throws an error — it just produces a model that's a bit dimmer than it should be. Key sources: • Hugging Face, Chat templates — https://huggingface.co/docs/transformers/chat_templating • Matthew Carrigan / HF, Chat Templates: An End to the Silent Performance Killer, 2023-10-03 — https://huggingface.co/blog/chat-templates • OpenAI, Harmony Response Format, 2025-08-05 — https://cookbook.openai.com/articles/openai-harmony Episode 6021 — myweirdprompts.com/n/6021 (permanent short link: myweirdprompts.com/765230)

0:00 · Intro-29:53

transcript

No transcript — this publisher did not publish one.

show notes

Your JSON messages don't reach the GPU as JSON. Somewhere in between, a Jinja chat template flattens them into a string of reserved special tokens — the exact format each model family was trained on. Llama 3 brackets roles with start_header and end_header; OpenAI's ChatML embedded the role in a single opening token; gpt-oss uses Harmony with separate stop tokens for "done talking" and "calling a tool"; Gemma has no system role at all; DeepSeek uses full-width CJK pipes. This episode walks through what special tokens actually are, how tokenizers register them, why the add_generation_prompt flag matters more than anyone thinks, and why a format mismatch never throws an error — it just produces a model that's a bit dimmer than it should be.

Key sources:
• Hugging Face, Chat templates — https://huggingface.co/docs/transformers/chat_templating
• Matthew Carrigan / HF, Chat Templates: An End to the Silent Performance Killer, 2023-10-03 — https://huggingface.co/blog/chat-templates
• OpenAI, Harmony Response Format, 2025-08-05 — https://cookbook.openai.com/articles/openai-harmony

Episode 6021 — myweirdprompts.com/n/6021 (permanent short link: myweirdprompts.com/765230)

chapters

5 chapters