The BugBash Podcast

Antithesis

The BugBash podcast is a lively look at all aspects of software reliability, by enthusiasts, for everyone.

Each episode brings leading engineers and researchers together for deep dives on everything from formal methods to testing to observability to human factors. There’s concrete advice on best practices, and nuanced discussion of how these strategies combine to deliver software that works.

And if you’re enjoying these conversations, check out the talks from BugBash 2025 on YouTube, and join us at BugBash 2026 on April 23-24, 2026, in Washington DC!

Play

Open in

22 episodes
weekly
Avg 56 min
English

Episodes 22

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

S3 · E3
Wednesday · 53 min
The People Behind the Hamsters and Other Stories of Downtime
Behind every catastrophic system failure is a room full of stressed-out people trying to fix it. Launched in 2021, the Downtime Project was a podcast run by two seasoned distributed system engineers, who dissected post-mortems published after notable outages. Today, those two engineers, Jamie Turner and Tom Kleinpeter join David Wynn on the Bug Bash podcast to talk about their collaboration and what they learned. The two Dropbox veterans look back at the war rooms of the internal incidents behind their service and others. We dive into the emotional toll of managing outages, why zero-blame cultures are mandatory for survival, and the stories that we only tell over beers years after the fact. And of course, AI worms its way into the conversation as well. It’s a look at the pride, the panic, and the people keeping real world services online, and the lessons we learn from going offline. And if you like conversations like this, you’ll want to join us at the first ever European edition of the Bug Bash conference, happening on September 30 in Copenhagen. Check it out at bugbash.antithesis.com The Downtime Project is still online, and you can find all 8 episodes on Spotify or wherever you get your podcasts. If you’re enjoying the Bug Bash podcast, you’ll probably enjoy the Downtime Project too.
S3 · E2
August 19 · 1 hr 12 min
From 18 months to 6: High-velocity database engineering
In this episode, Carl Sverre and David Wynn sit down with Philip O'Toole, creator of rqlite. We dive into his philosophy of using tests as a core design strategy, his tactical advice for turning AI agents into technical research assistants, and how leveraging AI tools compressed his typical 18-month release cycle down to just six months. If you’re interested in a principled approach to solo projects in the AI era, or where the future of AI swarm management for critical infrastructure might be going, you’ll want to give it a listen.
S3 · E1
August 12 · 1 hr 2 min
A Wild Hypothesis: The Python Library that Kills PhDs
What happens when your open source library changes millions of lives? In this episode, we sit down with David MacIver, Zac Hatfield-Dodds, and Liam DeVoe — three generations of maintainers of Hypothesis, the property-based testing library used by roughly 5% of all Python developers. It’s by orders of magnitude the most-widely used property-based testing tool in the world. Hypothesis has introduced millions of developers to property-based testing, and made instant converts out of many others. All three maintainers quit their PhDs to work on Hypothesis, and today they talk about just why they did that, how pull requests lead to lifelong friendships, and about the factors that led to Hypothesis’ success.
August 5 · 35 min
Dev in the Street
In this episode of the Bug Bash podcast, host David Wynn shares a collection of conversations recorded at Bug Bash 2026. 11 developers share their war stories about the most interesting bugs they've encountered and resolved. They talk about JDK issues on ARM architecture, kernel hangs related to NFS and half-duplex Ethernet, memory fences in Spark workloads, and complex logic bugs in iMessage. Together these conversations become a picture of what reliability-focused developers are thinking about today: the importance of formal verification, the rise of property-based testing, and how the only thing that makes systems work is our effort and care.
S2 · E8
April 8 · 57 min
Why simple workloads find the hardest bugs
You hit 100% test coverage, deploy to production, and the system still falls over. An old story, maybe aside from the 100% coverage percent part. In this episode, David sits down with Marco Primi and Sean Allen from the Antithesis Red Team to talk about a likely missing piece in your testing strategy: workloads. They break down exactly what workloads are, why they are so helpful for testing distributed systems, and the biggest pitfalls engineers hit when trying them out. They talk about why you have to let go of deterministic, "happy path" testing, and how even a workload as "dumb" as a basic `while true` loop can expose silent data loss that your unit tests would never catch.
S2 · E7
April 2 · 49 min
Programming as an Act of Building Vocabulary
Why do LLMs struggle to build complex architecture? According to Unmesh Joshi, Distinguished Engineer at ThoughtWorks, it often comes down to a lack of shared abstractions, of a shared vocabulary. Today he joins Will Wilson and David Wynn to unpack how we learn about, build, and test distributed systems. We cover the value of building miniature versions of systems like Kafka to grasp the basics, why Deterministic Simulation Testing (DST) is a highly effective tool for teaching as well as finding bugs , and why Guy Steele's 1998 keynote holds the secret to successfully collaborating with modern AI tools.
S2 · E6
March 25 · 51 min
The Dollar Bet that Fuzzed Figma: Exploding Laptops and UI Reliability with Jonathan Chan
Can you stress-test a React UI completely hands-free? In this episode, Oskar Wickström and I are joined by Jonathan Chan to discuss the origins of Fuzzmap. From a janitor’s closet in Figma—where end-to-end tests once lived on a "spicy pillow" MacBook—Jonathan shares how he applied coverage-guided fuzzing (inspired by AFL) to the world of web front-ends. We dive into the gnarly hacks required to instrument React, the challenges of visualizing complex state machines, and the future of "full-stack" fuzzing in the age of LLMs and team agents.
S2 · E5
March 18 · 51 min
Semmathesy and the Agentic Era: Learning Systems in 2026
With the "Agentic Era" coming online, teams are onboarding new AI "minions" into our codebases every thirty minutes. So a critical question emerges: Are we actually becoming more productive, or just more busy?Jessica Kerr (Jessitron) joins me to explore symmathesis—the theory of learning systems made of learning parts. We dive into viewing software as a teammate, how the definition of "legacy code" is shifting to anything an AI can't understand, and why our most important human contribution is moving from writing lines of code to orchestrating the flows of learning across the entire system.We dive into the shifting boundaries between the social and technical worlds, and why the "productivity unlock" of AI might actually make software engineering a more human endeavor than ever before.
S2 · E4
March 11 · 1 hr 1 min
From Scale to Rigor: An Engineering Journey at Meta and Oxide
Today we’re talking with Rain from Oxide Computing, tracing their journey from the massive, data-driven scale of Meta to the high-stakes, air-gapped world of shipping a 'cloud in a box'. We talk about moving from an environment of over 10,000 engineers—where a 10% tooling improvement is worth a thousand people—to a culture where you’re shipping hardware that simply cannot be patched once it leaves the building. Rain shares how they realized that technical writing is a senior engineer’s ultimate force multiplier for justifying complex designs. We also dive deep into the 'invisible superpower' of property-based testing and using Oracles to kill bugs in the developer’s inner loop long before they ever have a chance to reach production. Mentioned: https://rfd.shared.oxide.computer/rfd/0576 Further reading: https://oxide-and-friends.transistor.fm/episodes/engineering-rigor-in-the-llm-age
S2 · E3
March 4 · 58 min
Escaping the Spaghetti: How to Test Untestable Codebases
Everyone wants reliable software. But nobody wants to test messy legacy code. Today, Lewis Campbell from Outdata joins the show to share a practical approach for bringing deterministic simulation testing to existing systems. We get into why React components are terrible for business logic, and why front end static typing stops non-determinism at the door. We also take on non-tech aspects, like the politics of updating old code bases, the danger of hiding data conflicts, and how to ease your team into property-based testing. There’s even an Internet-hungry sheep named Angus.
S2 · E2
February 25 · 51 min
How rr Became a Protected Species: A Story of Necessary Hacks
Building the rr replay debugger wasn't about academic purity. It was about survival. Today, creator Rob O'Callahan reveals the "necessary” complexity that makes time-travel debugging possible on Firefox. We dig into the engineering nightmares behind the tool: from patching 2-byte system calls to the constant fear of silent hardware failures. Rob explains how rr became a "protected species" inside the Linux kernel—and why messy, manual hacks often beat six months of theoretical work.
S2 · E1
February 18 · 1 hr 20 min
Re-Designing Data-Intensive Applications: The Shift to Cloud-Native Storage
It’s been nine years since Designing Data-Intensive Applications became the standard text for distributed systems. Today, Martin Kleppmann and Chris Riccomini join the show to pull back the curtain on the upcoming Second Edition. After all, the era of local disks are giving way to cloud-native object storage. So we discuss why modern databases are being rebuilt entirely on top of S3. From there, we revisit the CAP theorem—and why it might be time to retire it in favor of "offline availability." We also get into a surprisingly practical debate on AI, exploring how LLMs might be terrible at creative design but perfect as "test oracles" for verifying complex system migrations. You’ll want to stick around for this one.
S1 · E11
Dec 10, 2025 · 1 hr 18 min
Hypothesis vs. Hallucinations: Property Testing AI-Generated Code
Large Language Models can generate code in a flash, but that code is notoriously unreliable. Traditional unit tests often can’t put enough guardrails in place to ensure correctness… even if they’re written by the LLM itself. This is where property-based testing (PBT) becomes essential. Today, we're joined by David R. MacIver, creator of the PBT library Hypothesis, and now an Antithesis employee! We discuss how to build robust feedback loops that are needed to make AI-generated code trustworthy. We'll cover why standard AI coding benchmarks are flawed, how Hypothesis makes PBT approachable, and the challenge of getting developers to think in "invariants." David also shares his perspective on the future of AI in software engineering. If you want to build a reliability backstop for your code, vibed or otherwise, stick around.
S1 · E10
Nov 26, 2025 · 39 min
From the Lab to Production: Making Cutting-Edge Testing Practical
Software testing research is exploding, but in practice, most companies' testing approaches seem stuck in the past. Where does that gap come from? It often boils down to the distance between academic promises and the practical needs of developers who need usable tools and fast results. In this episode, David talks with Rohan Padhye, head of the PASTA research group at Carnegie Mellon University, who has lived on both sides of that divide. They explore how fuzz testing crossed that chasm—from industry curiosity to academic focus and back again—and what it will take for other techniques to do the same. Rohan shares insights on designing testable software, building a robust testing culture, and what truly makes a "good" property for finding bugs.
S1 · E9
Nov 12, 2025 · 40 min
Ergonomics, reliability, durability
Integrating non-deterministic, non-durable elements like AI agents into our workflows tends to lead to a lot of do-overs. But restarting AI processes can be costly, burning through tokens and losing valuable progress. Wouldn’t it be easier if there was always a clear checkpoint to restart a task from? Today I talk with Qian Li, co-founder of the DBOS durable execution engine, about reliability, ergonomics, and actually understanding your software. We discuss the long history of checkpointing, mental models, and how using durable execution allows systems to resume right where they left off after a crash. It makes your software resilient by default. Learn how this architectural pattern can impact an AI-assisted or any complex system that could use a little improvement in how developers work with it.
S1 · E8
Oct 30, 2025 · 54 min
No actually, you can property test your UI
How do you test for bugs that only appear when a user clicks frantically, or when asynchronous data loads in an unexpected order? Standard UI tests often miss the subtle stuff that happens all the time in the stateful, dynamic applications. In this episode, Paul Ryan and I sit down with Oskar Wickström, creator of the QuickStrom framework, among other things, to explore how to apply generative testing to the complex world of user interfaces. Oskar argues that you don't need to be a formal methods genius to get real value out of the approach. Even simple properties can uncover deep bugs, like ensuring a loading spinner eventually disappears or that the screen never goes blank. If you've been intrigued by property-based testing but intimidated by the thought of writing complex formal models for UIs, stick around.
S1 · E7
Oct 15, 2025 · 52 min
Slow down to go fast: TDD in the age of AI with Clare Sudbery
AI coding assistants promise incredible speed, but what happens when you run straight into a wall of buggy code and technical debt? In this episode, Clare Sudbery, a software engineer with over 25 years of experience, discusses a crucial paradox for modern developers. The secret to harnessing AI's power isn't to move faster, but to slow down. Clare explains why deliberate, rigorous practices like Test-Driven Development (TDD) are the essential "guardrails" needed to guide AI tools toward reliable, high-quality software. You'll learn why "more, smaller steps" is the key to tackling technical debt and how throwing your code away might be the most productive thing you do all week.
S1 · E6
Oct 1, 2025 · 1 hr 10 min
Fixing five "two-year" bugs per day
Some bugs are so rare, they can take years to track down and fix. What if you could find and fix five of them per day? For Joran Dirk Greef, the creator of the TigerBeetle database, that's not a wild dream — it's how his team works every day. While most people think building a new database takes a decade, Joran's team built TigerBeetle in just three and a half years. The key is a unique philosophy for writing software called "Tiger Style". Joran joins the show to share the secrets behind their speed and safety. You'll hear why he thinks picking C would have been a "fatal" mistake , how a strict rule about memory can force you to write better code , and why Zig was the perfect choice for TigerBeetle. The key to it all is a powerful testing method that Joran calls "existential" for any important project. If you want to hear more about how his team turns squashing impossible bugs into their normal day-to-day, stay tuned.
S1 · E5
Sep 18, 2025 · 41 min
No really, some bugs aren’t real
When is a bug not really a bug? In this episode, host David Wynn talks with SRE veteran Dan Slimmon about a radical idea: chasing perfect code might not be the best way to make your service reliable. Dan argues that once your code is "good enough," most outages aren't caused by code defects. They're caused by weird interactions between different parts of a system or by users doing things you would never expect. He shares wild stories from his career, including how a tiny database hiccup created a massive, repeating traffic jam and how a single user crashed servers by uploading a 3.2-gigabyte config file. This conversation will make you rethink what you thought you knew about bugs, quality, and what "reliability" truly means.
S1 · E4
Sep 3, 2025 · 53 min
Every map is wrong, but we made one anyway
Every map of a complex territory is inherently wrong, but without one, we're completely lost. So what happens when the territory is the vast, ever-changing landscape of distributed systems? In this episode, David Wynn sits down with Kyle Kingsbury, the renowned researcher behind Jepsen, to discuss a monumental effort to chart this landscape: the new Distributed Systems Reliability Glossary. Kyle explains why he and T.W. Lim from Antithesis felt the need to "put this all in one place," creating a practical roadmap for testers and engineers navigating the field. They explore the challenge of creating "directionally correct" definitions , the surprising "urban legends" that persist in system design (like yelling at hard drives to increase error rates ), and why even the most rigorous formal models can drift from the code they're meant to describe. Tune in for a deep dive into the subtle bugs that defy simple explanations , the future of reliability in the age of AI-generated code, and the one problem Kyle is still determined to solve in his own work: reproducibility.

Showing 1–20 of 22 episodes