
050 - Why Do AI Experts Think AI Could Kill Us?
Why are some of the people building frontier AI seriously worried it could cause human extinction? This episode explains the reasoning behind the scary probability numbers, and then pushes back on them. It starts with the estimates: Elon Musk has put his own figure around 20 percent, and Anthropic alignment researcher Evan Hubinger recently said he personally puts the chance of AI killing all humans within a decade above 10 percent. From there, the episode explains what these researchers fear, which is not today's chatbot turning evil but a future system far more capable than humans at research, programming, cyber operations, and planning. It walks through the core concepts: alignment, the problem of building a system that reliably does what people intend; loss of control, where a capable system resists shutdown because staying on helps it reach its goal; and recursive self-improvement, the feedback loop in which AI that is good at AI research could accelerate its own progress toward artificial superintelligence. It uses the recent OpenAI and Hugging Face incident as a small, concrete example of an agent turning a goal into unexpected behavior. Then it challenges the numbers. A 10 or 20 percent chance of extinction is not a measurement. No experiment produced it, and there is no database of superintelligence failures. It is a personal estimate built on assumptions that can be questioned, about how fast intelligence scales, whether self-improvement hits limits, and whether alignment gets solved. The episode gives the optimistic side a hearing too, through Sam Altman's argument that the world can scale through this transition to enormous benefit. The takeaway is measured. The sharper question is whether future systems could create risks far larger than the ones we know how to manage today, which is why the warnings deserve attention even if the specific probabilities are wrong. For a clear, balanced look at p(doom), alignment, and AI extinction risk, this episode lays out the argument and its weak points. Sources & References Axios — Here's how AI could kill us all (if the worst fears come true) https://www.axios.com/2026/09/09/ai-doom-pdoom-kill-all-humans-anthropic The Guardian — Anthropic researchers say AI could cause human extinction by 2030 https://www.theguardian.com/technology/2026/sep/09/anthropic-researchers-ai-human-extinction Yahoo Finance / Business Insider — Elon Musk says there's 'only a 20% chance of annihilation' with AI https://finance.yahoo.com/news/elon-musk-says-theres-only-001804190.html Sam Altman — The Gentle Singularity https://blog.samaltman.com/the-gentle-singularity Voice narration is AI-generated.