How Kargo turns losing experiments into competitive edges

Summary In this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with James Falzone, Director of Product Management at Kargo, to unpack how a high scale ad tech marketplace turns failure into its biggest advantage. James explains how Kargo connects advertisers to publishers through real time auctions that resolve in milliseconds across up to 10 billion ad requests a day, why experimentation is embedded in the company's culture rather than siloed in a team, and what happened when a winning click optimization model failed completely after being copied to a new customer type. The conversation is built for product managers, data scientists, engineers, and growth leaders who want a practical, honest view of running experiments at scale, learning from losses, and keeping AI grounded in solid infrastructure. Chapters 00:00 Welcome and introducing James Falzone 01:45 What Kargo does and how real time ad auctions work 04:45 Why experimentation is embedded in Kargo's culture 07:45 The three things every marketplace has to deliver 10:15 The experiment that failed: click optimization on third party demand 12:15 A bad result versus a bad experiment 13:45 Why different customer types need different signals 15:30 Putting "where did you fail?" on every retro 18:45 How experimentation evolves with AI 21:15 Better not bigger: the closing takeaway Takeaways -A bad result is not a bad experiment. If you're not failing, you're probably not trying anything new. -The same metrics and signals don't apply to every customer type. Bad results often come from a lack of context, not bad tech. -Metrics and signals you test against should always be business driven, not ported from the last thing that worked. -Put failure on the agenda. A biweekly "where did you fail?" retro turns one person's dead end into the whole team's shortcut. -AI's biggest unlock is access. More people can run experiments, but it has to be built on solid ML and infrastructure. Better, not bigger. Connect with the Guest LinkedIn: https://www.linkedin.com/in/jamesafalzone/ Website: https://kargo.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io