Skip to content
Sign in
Artwork for My Weird Prompts
My Weird Prompts · Thursday · 21 min

Why Switching Off One Attention Head Changes Everything

Mechanistic interpretability tools can switch off a single attention head or layer during inference to change how a model behaves. Masking specific decoder heads in Whisper dramatically reduces hallucinations, while guardrail removal tools like elder-plinius's OBLITERATUS use ablation to map and project out refusal directions. Examining work from Calm-Whisper to Foo and Lee clarifies what an attention head actually computes, how ablation alters the shared residual stream, and why breaking a behavior does not reveal where it was computed. Key sources: • OBLITERATUS Space README, Hugging Face, retrieved 2026-11-05, Gradio SDK 5.29.0, AGPL-3.0 — https://huggingface.co/spaces/dumbordumber/obliteratus/resolve/main/README.md • OBLITERATUS CHANGELOG, Hugging Face, retrieved 2026-11-05 — https://huggingface.co/spaces/pliny-the-prompter/obliteratus/blob/main/CHANGELOG.md • Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down, arXiv, retrieved 2026-11-05 — https://arxiv.org/html/2505.12969 Episode 6469 — myweirdprompts.com/n/6469 (permanent short link: myweirdprompts.com/931963) Listen on Android: MWP FM Player — myweirdprompts.com/apk

0:00 · Intro-21:35
Play in
Transcript file

transcript

No transcript — this publisher did not publish one.

show notes

Mechanistic interpretability tools can switch off a single attention head or layer during inference to change how a model behaves. Masking specific decoder heads in Whisper dramatically reduces hallucinations, while guardrail removal tools like elder-plinius's OBLITERATUS use ablation to map and project out refusal directions. Examining work from Calm-Whisper to Foo and Lee clarifies what an attention head actually computes, how ablation alters the shared residual stream, and why breaking a behavior does not reveal where it was computed.

Key sources:
• OBLITERATUS Space README, Hugging Face, retrieved 2026-11-05, Gradio SDK 5.29.0, AGPL-3.0 — https://huggingface.co/spaces/dumbordumber/obliteratus/resolve/main/README.md
• OBLITERATUS CHANGELOG, Hugging Face, retrieved 2026-11-05 — https://huggingface.co/spaces/pliny-the-prompter/obliteratus/blob/main/CHANGELOG.md
• Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down, arXiv, retrieved 2026-11-05 — https://arxiv.org/html/2505.12969

Episode 6469 — myweirdprompts.com/n/6469 (permanent short link: myweirdprompts.com/931963)

Listen on Android: MWP FM Player — myweirdprompts.com/apk

chapters

7 chapters

more episodes

All episodes