Realtor.com on using your AI as a junior data scientist

Summary What do you do when your biggest experiment win turns out to be a loss? Whitney Perez, Director of Product Management at Realtor.com, joins host Ashley Stirrup to share the checkout bundling test that posted a 300% attach rate and still lost revenue, the 30/30/30 rule she uses to set expectations for a new experimentation team, and how AI is turning an English major into an aspirational data scientist. This episode is for product managers, engineers, and data scientists building experimentation programs from the ground up. Chapters 00:00 Cold open and welcome 01:10 From growth hacker to Realtor.com 02:50 Three foundations for a new experimentation team 04:20 The 30/30/30 rule 05:05 The 300% bundling win that lost revenue 07:10 You don't need a stats degree to experiment 09:20 Cascading North Star metrics 12:50 Do the homework before the experiment 13:50 The wishlist: instrumentation, embedded knowledge, culture 15:50 AI as an aspirational data scientist 18:45 Keeping a human in the loop Takeaways -A winning decision metric is not enough. Realtor.com's bundling test hit a 300% attach rate, but funnel fallout from the extra step made it a net revenue loser. Set secondary metrics and their thresholds before launch. -Expect the 30/30/30 rule: roughly a third of tests win, a third are inconclusive, and a third lose. The math is the math, and the losers carry most of the learning. -Start a new team on foundations: what a clean test and an A/A test look like, which surfaces should not be tested, and a peer review program that lets people graduate to more complex experiments. -You don't need a stats background to run good experiments. Teach the simplest definition of a good test, then let people learn by doing. -AI can make anyone an aspirational data scientist for analyzing results and spotting opportunities, but it can be confidently wrong. Keep a human in the loop and sanity check output the way you'd peek at a freshly launched test. Connect with the Guest LinkedIn: https://www.linkedin.com/in/whitneykperez/ Website: https://www.realtor.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io