The Experimentation Edge

Growthbook

How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale.

Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.

Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.

Play

Open in

39 episodes
Avg 26 min
English

Episodes 39

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

S2 · E5
Thursday · 25 min
Building a Culture of Experimentation in Sports Analytics with New York Red Bulls
Summary Vamsi Krishna Kunapuli, Director of Analytics at New York Red Bulls, leads a small team that supports marketing, ticket sales and partnerships for one of the original MLS clubs and its 25,000 seat stadium. He shares how an experiment across five to six seasons changed the club's North Star from average ticket price to gross margin. Host Ashley Stirrup and Vamsi also discuss why a losing test is not a failure, why segments can never be static, where good test ideas come from, and how identity resolution will open the door to machine learning and personalized fan experiences. For analytics, marketing and product leaders building experimentation into a business where every event is different. Chapters 00:00 Intro 00:20 New York Red Bulls: an original MLS club 01:32 Data-driven fan engagement 04:46 Single game buyers outspend season ticket holders 07:21 Testing frameworks 09:29 Holistic fan value 11:11 Segments can never be static 12:59 A losing test isn't a failure 14:51 Loser or lever: judging tests against the North Star 17:13 Building a culture with clear end goals 20:25 Ideas populate when there is more data 23:10 Identity resolution, AI and personalization 26:06 Empowering more teams to experiment 27:57 Every game is like hosting a wedding Takeaways - We have seen instances where single game buyers at the New York Red Bulls outspend season ticket holders once concessions and the Hype House zone are counted - The club moved its North Star from average ticket price to gross margin, which changed couponing, fan profiles and retargeting - A losing test is not a failure; judged against the North Star, a negative result shows what not to do and that a lever exists - Segments should be dynamic, and the club is building toward a separate segmentation for every event - Ideas populate when there is more data, and the stakeholders who know the business best should drive the questions Connect with the Guest Vamsi Krishna Kunapuli LinkedIn: https://www.linkedin.com/in/vamsikunapuli/ Company Website: https://www.newyorkredbulls.com Vamsi is a panelist at DMWF North America 2026, "Connected Data: The Foundation of Intelligent Marketing" Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
- Transcript
S2 · E4
Tuesday · 38 min
Motive's Shiva Manjunath on CRO as a mindset, not a role
Summary Shiva Manjunath, Lead Web Strategist at Motive and host of the From A to B podcast, joins Ashley Stirrup to explain why CRO works best as a mindset every team carries rather than a role one team owns. He shares how building bridges to sales and customer service turns everyday complaints into test ideas. The conversation covers why customers who ask for price are often asking for value, how emotional context earns a CRO credibility, how to make an impact without an executive sponsor, and why experimentation needs the kind of marketing AI gets. For CROs, product managers and growth leaders building an experimentation culture. Chapters 00:00 Cold open: gifts under $50 01:18 What a true culture of experimentation looks like 04:43 CRO as a mindset across every function 11:27 Become the sales team's best advocate 15:29 Fear, uncertainty and doubt: the window shade story 21:51 Customers ask for price when they want value 24:17 GTA 6 and why button colors cost CROs trust 29:14 Making an impact without an executive sponsor 34:27 Making experimentation as exciting as AI Takeaways - CRO works best as a mindset shared by design, marketing and product, not as one team's job. - Sales and customer service teams hold free qualitative data that makes strong test ideas. - When customers ask for price, test value messaging such as benefits, ROI and time saved. - Understanding the emotional context around a purchase earns a CRO more trust than tactical button tests. - A culture of experimentation needs data-driven decision making first and a safe space to lose. Connect with the Guest Shiva Manjunath LinkedIn: https://www.linkedin.com/in/shiva-manjunath/ From A to B podcast: https://open.spotify.com/show/5xDlubE3GZwUDDen64hefz Newsletter: https://fromatob.beehiiv.com/ Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
- Transcript
S2 · E3
October 22 · 26 min
How Quill turned loyalty points into a 1% conversion lift
Summary Vipul Tyagi, Senior Product Manager at Quill, owns the online experience from add to cart through checkout, with a focus on getting customers to use Quill's loyalty program. Host Ashley Stirrup digs into why a homepage rewards widget drew under 0.5% engagement over 60 days, and how moving the value to the cart with a Lego-inspired points to dollars option lifted conversion by 1% while shrinking the points liability on Quill's books. The conversation covers Vipul's 8 winners out of 11 loyalty initiatives and the BillionToOne advice behind them, how he coaches new PMs on primary and secondary hypotheses and guardrail metrics, a live test on showing the $5.99 shipping saving at checkout, a semantic layer experiment built for AI crawlers, and how AI has cut his research time from two months to seven days. For product managers, growth leaders and e-commerce teams who want a higher win rate from their experiments. Chapters 00:00 Cold open 01:06 Quill, office supplies and three kinds of customers 04:20 Owning add to cart through checkout, and 8 winners out of 11 05:32 Know the direction before you run the test 06:54 The homepage loyalty widget and the icing on the cake 10:19 Points into dollars on the cart page: a 1% conversion lift 12:39 Coaching new PMs: ranked hypotheses and guardrail metrics 15:41 Taking credit for free shipping in the order summary 18:10 Copy testing, strategic bets and a semantic layer for AI crawlers 22:24 AI in the PM workflow: research in seven days instead of two months Takeaways - Rewards work where purchase intent already exists: Quill's homepage loyalty widget drew under 0.5% engagement, while the same value on the cart page lifted conversion. - Converting points into dollars at the cart raised conversion 1%, brought small orders back and reduced the unredeemed points liability on the books. - Know the direction of a test before you run it; the A/B test measures how much, which is how Vipul's loyalty work reached 8 winners out of 11. - Design every experiment with a primary hypothesis, ranked secondary hypotheses and guardrail metrics so a loss still explains itself. - Showing the $5.99 shipping saving at checkout raised conversions without changing order value, a sign that transparency builds trust. Connect with the Guest Vipul Tyagi LinkedIn: https://www.linkedin.com/in/vipul-tyagi Company Website: https://www.quill.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io (00:00) - Cold open (01:06) - Quill, office supplies and three kinds of customers (04:20) - Owning add to cart through checkout, and 8 winners out of 11 (05:32) - Know the direction before you run the test (06:54) - The homepage loyalty widget and the icing on the cake (10:19) - Points into dollars on the cart page: a 1% conversion lift (12:39) - Coaching new PMs: ranked hypotheses and guardrail metrics (15:41) - Taking credit for free shipping in the order summary (18:10) - Copy testing, strategic bets and a semantic layer for AI crawlers (22:24) - AI in the PM workflow: research in seven days instead of two months
- Transcript
- Chapters
S2 · E2
October 20 · 40 min
Engagement with safety: Headspace's guardrails for AI tests
Mary Alfheim, VP Data and Analytics at Headspace, joins host Ashley Stirrup to explain why more engagement is not always better when your members come to you with real symptoms. She walks through how Headspace experiments on Ebb, its AI companion, by defi
- Transcript
S2 · E1
October 15 · 19 min
How Walmart Connect learned to align cross-sell with shopper intent
Summary Arv Ramchandani, Senior Product Manager for Experimentation & Analytics at Walmart Connect, runs ad experiments on pages owned by other teams, from search to the conversational AI shopping agents, each with its own primary metric. He explains
- Transcript
S1 · E50
October 14 · 19 min
Why more info lifted VidaXL's account creation nearly 40%
Summary Sapna Kashyap, CRO Lead at VidaXL, runs the experimentation program for an e-commerce company operating in more than 30 countries, where product, development and business teams all validate their decisions with an A/B test first. She explains the
- Transcript
S1 · E49
October 13 · 40 min
The auto researcher: How FanDuel finds a winning model in 300 attempts
Michael Mathews, Director of Data and AI Product at FanDuel, runs the teams behind the auto researcher, an agent that takes a data set, an objective function and the production model, then proposes, writes and scores about 300 strategies in an hour for ro
- Transcript
S1 · E48
October 8 · 21 min
What Grubhub tests for big product bets before they ship
Summary Grubhub does not run much classic A/B testing. Michal Lenik, Associate Director of Product Design at Grubhub, explains what the company does instead to make sure its big product bets win before they reach millions of users. She joins Ashley Stirrup to walk through Grubhub's playbook: small cohort rollouts that protect core order metrics, merchant round tables, AI built clickable prototypes, and a concept test in the merchant promotions space that sent the team toward a much simpler dashboard than their research predicted. Michal also explains why she follows losing experiments with user interviews, why "I didn't even notice it" is valuable feedback, and why every company should decide deliberately whether it de-risks upstream or in a fast follow after launch. Chapters 00:00 Where design is heading in the AI era 00:53 Meet Michal Lenik of Grubhub 02:21 Design as an upstream business lever 03:48 Small cohorts instead of classic A/B tests 05:27 The merchant concept test that changed direction 08:18 Prototyping with AI and merchant round tables 10:16 Success metrics for product, design and engineering 11:56 Pairing experiment data with user interviews 14:26 AI, personalization and the designer's new role 17:45 De-risking upstream versus shipping fast Takeaways - Grubhub tests big bets by releasing to internal users or small cohorts first, so it can read business impact without disrupting core revenue and order metrics. - A concept test with merchants overturned what fifteen research interviews suggested, steering Grubhub to a simpler dashboard and away from a launch that likely would not have won. - AI lets Grubhub's design teams put clickable prototypes in front of users early, turning each iteration into a micro experiment. - Pairing quantitative results with five to ten user interviews after a test explains why it underperformed, including whether users even saw the change. - Both upstream de-risking and fast iteration work. The right choice depends on how quickly engineering can ship, and it should be made on purpose. Connect with the Guest Michal Lenik LinkedIn: https://www.linkedin.com/in/michallenik/ About the guest: https://www.growthbook.io/podcast/guests/michal-lenik Episode page: https://www.growthbook.io/podcast/episode/1-48 Company Website: https://www.grubhub.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io (00:00) - Where design is heading in the AI era (00:53) - Meet Michal Lenik of Grubhub (02:21) - Design as an upstream business lever (03:48) - Small cohorts instead of classic A/B tests (05:27) - The merchant concept test that changed direction (08:18) - Prototyping with AI and merchant round tables (10:16) - Success metrics for product, design and engineering (11:56) - Pairing experiment data with user interviews (14:26) - AI, personalization and the designer's new role (17:45) - De-risking upstream versus shipping fast
- Transcript
- Chapters
S1 · E47
October 6 · 20 min
How MilliporeSigma lifted add to cart with a smarter search
Summary Dorothy Crepin runs the A/B testing program at MilliporeSigma, the life sciences business of Merck KGaA, where one global catalog has to serve three very different visitors: scientists hunting for a specific instrument, procurement buyers working off a list someone else wrote, and students doing research with no purchasing power at all. Add a regulated industry where what you can sell changes country by country, and personas stop being a marketing exercise. She walks Ashley Stirrup through the test that changed how her team thinks about discovery. MilliporeSigma rebuilt its on-site type ahead so it suggests a search term instead of pushing a matching product, and saw double digit increases in feature usage and add to cart. From there Dorothy makes the case for measuring the action a page is actually responsible for rather than revenue alone, for pairing A/B tests with a user testing ambassador panel to get at the why, and for sending product managers back to one question before anything gets built: what problem are you solving for the customer? Chapters 00:00 Revenue is an outcome, not the metric 01:26 Welcome and what MilliporeSigma does 02:22 The three audiences on a B2B science site 03:53 What an A/B testing analyst coordinates 04:20 Running experiments across product pods 05:31 Building leadership buy in for testing 08:08 Where A/B testing ends and user testing begins 08:52 The type ahead search test that lifted add to cart 11:30 How to frame a test with a product manager 13:34 Why testing the opposite pays off 15:26 Personas, personalization and what AI changes 18:57 Building trust across competing teams Takeaways - MilliporeSigma changed its on site search to suggest search terms instead of products and saw double digit gains in feature usage and add to cart. - Revenue is an outcome, so Dorothy's team measures the next action each page is responsible for, like add to cart on a product detail page. - A/B testing tells you what happened. MilliporeSigma's user testing ambassador panel supplies the why behind the click. - Every test starts with one question to the product manager: what problem are you solving for the customer, not what is your hypothesis. - Dorothy is building personas into agents so browsers, not just buyers, can be tested for a directional read before an experiment ships. Connect with the Guest Dorothy Crepin LinkedIn: https://www.linkedin.com/in/dorothycrepin/ About the guest: https://www.growthbook.io/podcast/guests/dorothy-crepin Episode page: https://www.growthbook.io/podcast/episode/1-47 Company Website: https://www.sigmaaldrich.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
- Transcript
S1 · E46
September 30 · 31 min
PayPal's $180 million experimentation win
Summary Gaurav Sethi joins Ashley Stirrup on The Experimentation Edge to explain how PayPal's experimentation program went from 700 to 800 tests a year with four week readouts to 2,587 experiments a year and roughly $180 million in measured impact. Gaurav inherited a reported win rate of 55% to 60%, four times the industry average, and traced it to experiments logging assignment data instead of exposure data, carrying 25% to 30% dilution. The conversation covers the exposure event his team introduced, the instrumentation and metric standards that cut readouts to 24 hours, a carousel test that a multi armed bandit resolved in 51 days instead of a projected 700, and why cost avoidance from losing experiments belongs in the ROI number. It closes on what changes when the thing you are testing is an AI agent rather than a button. Useful for product managers, engineers, data scientists and growth leaders building or defending an experimentation program at scale. Chapters 00:00 Cold open 00:54 Welcome Gaurav Sethi of PayPal 01:30 Elmo, PayPal's homegrown experimentation platform 03:53 $180 million in revenue impact and cost avoidance 04:28 The win rate that was too good to be true 06:29 Instrumentation standards and the exposure event 08:56 The carousel test: 700 days down to 51 11:29 Designing experiments so every result teaches you 13:45 Building the platform is only half the job 15:07 Education, office hours and executive support 18:28 Exposure events and joining transactional data 24:14 AI for experimentation, and experimentation for AI Takeaways - A win rate of 55% to 60% against an industry average of 11% to 14% was a tracking problem, not a performance one: assignment data carried 25% to 30% dilution and reached significance on users who never saw the test. - The exposure event fixed it. Fire an event as close to render as possible so the platform knows exactly when a user entered the experiment, instead of logging on page load. - Standardized instrumentation and canonical metric definitions are what let the analysis pipeline run without human cleanup, taking experiment readouts from four weeks to 24 hours. - A six variant carousel test projected at 700 days was resolved by a multi armed bandit in 51 days, with a winner worth almost five basis points. - About half of PayPal's $180 million impact in 2025 was cost avoidance from features that tested badly and never shipped, which is why Gaurav argues there are no losing experiments. Connect with the Guest Gaurav Sethi LinkedIn: https://www.linkedin.com/in/gauravsethi22 About the guest: https://www.growthbook.io/podcast/guests/gaurav-sethi Episode page: https://www.growthbook.io/podcast/episode/1-46 Company Website: https://www.paypal.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
- Transcript
S1 · E45
September 29 · 27 min
ServiceNow's Customer Zero approach to AI experimentation
Summary ServiceNow runs its own platform on itself. Ashraf Karim, Senior Vice President, Connected Customer Experience, owns everything that happens after a customer buys, and her team is Customer Zero, testing every product and capability internally before it ever reaches a customer. She joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline lesson is not a technical one. ServiceNow built agentic skills that worked, and users abandoned them anyway, because a task that ran five to fifteen minutes broke every expectation people had about what a machine should feel like. The fix was not a faster model. It was telling the user what the AI was doing while it did it. Ashraf also gets into the harder measurement problem underneath all of it: how do you evaluate a nondeterministic system? ServiceNow's answer is an internal LLM judge scored against a golden data set, plus a customer effort score that catches what CSAT reports too late. Chapters 00:00 What customers actually care about 01:14 Meet Ashraf Karim of ServiceNow 01:39 Owning the entire post-sale experience 02:08 Why ServiceNow runs as its own Customer Zero 03:02 Lessons from Google, PayPal and Verizon 07:11 The agentic skills users kept abandoning 09:29 Latency, expectations and the feedback loop fix 12:29 Users wanted the outcome, not the process 14:49 Designing experiments that can afford to lose 18:13 Using an LLM judge on nondeterministic models 21:48 Customer effort score as a guardrail metric 25:03 Where experimentation at ServiceNow goes next Takeaways - ServiceNow is its own Customer Zero, running the platform on itself and testing every feature internally before customers see it. - The agentic skills worked. Users still abandoned them, because a five to fifteen minute wait broke their expectation of what AI should feel like. - The fix was a feedback loop that narrates what the AI is doing in the background, which buys the patience a long task needs. - Users did not want the agentified version of the human process. They wanted the outcome and the next best action, not the steps. - Nondeterministic output needs its own evaluation layer, so ServiceNow built an LLM judge that scores against a golden data set before anything reaches a customer. Connect with the Guest Ashraf Karim LinkedIn: Ashraf Karim on LinkedIn About the guest: https://www.growthbook.io/podcast/guests/ashraf-karim Episode page: https://www.growthbook.io/podcast/episode/1-45 Company Website: ServiceNow Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
- Transcript
S1 · E44
September 24 · 24 min
Why Twilio ships on signals instead of significance
Summary Wanli Lau spent eleven years at Expedia Group, most recently running consumer product analytics on a program that shipped more than 1,500 A/B tests a year. Five months ago she moved to Twilio to lead global R&D analytics. In this episode she walks Ashley Stirrup through what carried over and what did not. The centerpiece is a test that looked like a win. At Expedia, an account sign-up takeover on the front page drove conversion sharply up, and quietly harmed return rate and engagement across every geography. That result is where Wanli's insistence on guardrail metrics and a trade-off decision framework comes from. She also explains the parts of B2B experimentation that have no B2C equivalent: choosing between user-level and account-level randomization, the interference risk when two colleagues at the same company see different pricing, and the many-to-many problem of one developer belonging to several accounts. And because Twilio's customer counts are lower than a consumer business, her teams often ship on signals and team conviction rather than waiting for statistical significance. Chapters 00:00 Cold open 01:02 Welcome Wanli Lau of Twilio 01:23 Inside Wanli's role at Twilio 04:03 Eleven years and 1,500 tests a year at Expedia 04:58 Why B2B experimentation differs from B2C 05:33 Cross-functional collaboration and good hypotheses 06:39 Teaching teams to experiment rigorously 07:48 The Expedia sign-up takeover that won on conversion 11:09 Guardrail metrics and the trade-off framework 14:52 Designing experiments to maximize learning 17:46 Diagnosing where a feature failed 19:10 North Star metrics and account-level randomization at Twilio 21:40 Learning from signals instead of significance 22:11 Where experimentation goes next at Twilio Takeaways - A test that wins on the primary metric can still be a loss. Expedia's sign-up takeover raised conversion and damaged return rate and engagement at the same time. - Name the primary, secondary and guardrail metrics before the test runs, not at readout. Deciding afterwards turns a result into a debate. - Conversion should be a do-no-harm guardrail for teams that do not own it, even when it is not their primary metric. - In B2B the randomization unit is a design decision. User-level bucketing risks two colleagues at one account seeing different prices, and account-level avoids that but costs sample size. - When customer counts are low, significance is often out of reach. Learning from signals and shipping on team conviction beats waiting for a number that will never arrive. Connect with the Guest Wanli Lau LinkedIn: https://www.linkedin.com/in/wanlilau About the guest: https://www.growthbook.io/podcast/guests/wanli-lau Episode page: https://www.growthbook.io/podcast/episode/1-44 Company Website: https://www.twilio.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
- Transcript
S1 · E43
September 22 · 26 min
Why GoPro stopped judging A/B tests by win rate
Will Guyeskey is Director of Digital Product at GoPro, where his team owns the e-commerce side of gopro.com. Before GoPro he ran personalization at Gap and cut his teeth at Brooks Bell, testing for brands like Barnes & Noble, Under Armour and Ralph Lauren. In this episode of The Experimentation Edge, he tells Ashley Stirrup why knowing your customer is the through line of every good test program. Will shares the Barnes & Noble order confirmation test he was sure would lose, and why the same idea never worked for any other client. He walks through a recent GoPro Mission launch test that asked whether a step-by-step configurator adds too much friction, and what a flat result revealed about high consideration buyers. He also explains why GoPro shares interim readouts across the company, why win rate makes a poor North Star for an experimentation program, and how his team plans to use AI for speed without outrunning its own learnings. Chapters 00:00 Intro 00:52 Will's role running e-commerce at GoPro 01:43 Learning A/B testing across retail at Brooks Bell 05:24 How GoPro runs one to three tests a month 06:35 Sharing learnings and interim readouts across teams 09:20 The Barnes & Noble order confirmation win 13:02 Why the win did not transfer to other clients 14:41 Testing friction on the GoPro Mission configurator 18:51 Why win rate is the wrong North Star 22:19 How AI will shape experimentation at GoPro Takeaways - A winning idea rarely travels. The Barnes & Noble recommendation module worked because of that audience's low order values and reading habits, and it failed for every other client that tried it. - Design every test so it teaches you something whether it wins, loses or ends flat. Losing tests are jet fuel when the learning is built in. - A flat result is still an answer. GoPro's configurator test showed that buyers of high consideration products accept extra steps when each choice adds value. - Share interim readouts across the company, and use them to show how volatile results are before a test reaches statistical significance. - AI can speed up building and running experiments, but a team that runs more tests than it can learn from is not getting better. Connect with the Guest Will Guyeskey LinkedIn: https://www.linkedin.com/in/willguyeskey/ About the guest: https://www.growthbook.io/podcast/guests/will-guyeskey Episode page: https://www.growthbook.io/podcast/episode/1-43 Company Website: https://gopro.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
- Transcript
S1 · E42
September 17 · 30 min
What Samsung learned bringing B2C rigor to B2B
Summary What changes when you take a mature B2C experimentation practice and apply it to a B2B storefront? Anuradha Tempe, Lead Product Manager at Samsung Electronics America, joins host Ashley Stirrup to share what she learned managing both sides of Samsung.com. She explains why B2B is not a sidekick to B2C, how a single bulk order can push a test into a false positive unless you normalize the data, and why not everything deserves an A/B test. She walks through the add-on experiment that nearly doubled attach sales once her team realized business buyers decide on mobile and purchase on desktop, and the Buy Now test that lost because B2B buyers value clarity over faster conversions. The conversation closes with her approach to North Star and guardrail metrics and the AI copilot she built to draft A/B test plans with a human still in the loop. A practical episode for product managers, engineers, data scientists, and growth leaders running experimentation across different customer types. Chapters 00:45 Meet Anuradha Tempe: from chip design to leading Samsung e-commerce 02:15 Why B2B is not a sidekick to B2C 03:30 Not everything needs an A/B test 05:15 Spreading experimentation practice across a global conglomerate 07:00 Bringing B2C rigor to a fast-paced B2B team 08:45 The add-on experiment that nearly doubled attach sales 11:15 Buyers decide on mobile and purchase on desktop 16:30 The Buy Now test that lost 21:30 North Star metrics, guardrails, and the EPP discount fix 26:05 An AI copilot for A/B test planning Takeaways - Treat B2B as its own customer base with its own testing discipline; a single bulk order on one day can inflate a B2B test into a false positive unless order data is normalized before results are read. - Not everything needs an A/B test; route lower-risk changes through UAT feedback or pre/post comparisons and reserve full experiments for features where being wrong is expensive. - Map where the decision happens, not just where the purchase happens; Samsung's business buyers decide on mobile and buy on desktop, and surfacing add-ons on mobile nearly doubled attach sales. - B2B buyers value clarity over faster conversions; a Buy Now button earlier in the flow confused bulk purchasers because returns and cancellations on large orders are costly. - Align on the North Star before building, whether it is revenue, engagement, NPS, or fewer support tickets, and set guardrails so an engagement feature can never quietly drag sales down. Connect with the Guest Anuradha Tempe LinkedIn: https://www.linkedin.com/in/anuradha-tempe/ About the guest: https://www.growthbook.io/podcast/guests/anuradha-tempe Episode page: https://www.growthbook.io/podcast/episode/1-42 Company Website: https://www.samsung.com/us/business/ Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io
- Transcript
S1 · E41
September 15 · 30 min
Even a loss is a win: Charlie Health's approach to experiments
Summary On this episode of The Experimentation Edge, Ashley Stirrup talks with Joe Yevoli, Director of Growth at Charlie Health, a virtual intensive outpatient program that sits between weekly therapy and hospitalization. Joe explains how removing a page from Charlie Health's intake form produced a winning test that created a new bottleneck further down the funnel, why Teachers Pay Teachers cut about 80% of its market for one feature and roughly quadrupled retention, and how premortems that ask "what went wrong?" before launch make dissent safe and surface safeguards that avoid catastrophe. He also shares the two questions he asks before any experiment, why more top-of-funnel traffic always dents conversion rate, and how AI is flattening the product pod in ways that speed teams up and put them at risk. It's for growth leaders, product managers, and experimentation teams who want to learn as much from a loss as from a win. Chapters 00:00 Intro 01:00 About Charlie Health 02:05 Joe's role across performance, lifecycle, and experimentation 03:30 Building experimentation rigor across teams 04:40 Even a loss is a win 05:15 The form page test that created a new bottleneck 08:50 Teachers Pay Teachers and the Easel lesson 16:30 Designing experiments that teach you something when they lose 21:00 Premortems for high-risk experiments 24:30 AI is flattening the org Takeaways - A winning test is a data point, not a finish line; Charlie Health's form page removal won on top-of-funnel metrics and still exposed a downstream bottleneck that became the next experiment. - More traffic entering the funnel means conversion rate goes down, even when more people reach the bottom; treat it as a law of physics and plan for it before you read the results. - Build it for everyone and you build it for nobody; Teachers Pay Teachers cut roughly 80% of the market for Easel, repositioned it around one job, and roughly quadrupled retention. - Run a premortem before any large, risky experiment: ask the room "it was a massive failure, what went wrong?", have everyone write and share, and build the safeguards while there is still time. - Before launching, check the quantitative and qualitative data behind the hypothesis, map the full funnel, and measure toward the real business outcome, so a loss still leaves you with a next step. Connect with the Guest Joe Yevoli LinkedIn: https://www.linkedin.com/in/joeyevoli/ About the guest: https://www.growthbook.io/podcast/guests/joe-yevoli Episode page: https://www.growthbook.io/podcast/episode/1-41 Company Website: https://www.charliehealth.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io
- Transcript
S1 · E40
September 10 · 20 min
Realtor.com on using your AI as a junior data scientist
Summary What do you do when your biggest experiment win turns out to be a loss? Whitney Perez, Director of Product Management at Realtor.com, joins host Ashley Stirrup to share the checkout bundling test that posted a 300% attach rate and still lost revenue, the 30/30/30 rule she uses to set expectations for a new experimentation team, and how AI is turning an English major into an aspirational data scientist. This episode is for product managers, engineers, and data scientists building experimentation programs from the ground up. Chapters 00:00 Cold open and welcome 01:10 From growth hacker to Realtor.com 02:50 Three foundations for a new experimentation team 04:20 The 30/30/30 rule 05:05 The 300% bundling win that lost revenue 07:10 You don't need a stats degree to experiment 09:20 Cascading North Star metrics 12:50 Do the homework before the experiment 13:50 The wishlist: instrumentation, embedded knowledge, culture 15:50 AI as an aspirational data scientist 18:45 Keeping a human in the loop Takeaways - A winning decision metric is not enough. Realtor.com's bundling test hit a 300% attach rate, but funnel fallout from the extra step made it a net revenue loser. Set secondary metrics and their thresholds before launch. - Expect the 30/30/30 rule: roughly a third of tests win, a third are inconclusive, and a third lose. The math is the math, and the losers carry most of the learning. - Start a new team on foundations: what a clean test and an A/A test look like, which surfaces should not be tested, and a peer review program that lets people graduate to more complex experiments. - You don't need a stats background to run good experiments. Teach the simplest definition of a good test, then let people learn by doing. - AI can make anyone an aspirational data scientist for analyzing results and spotting opportunities, but it can be confidently wrong. Keep a human in the loop and sanity check output the way you'd peek at a freshly launched test. Connect with the Guest Whitney Perez LinkedIn: https://www.linkedin.com/in/whitneykperez/ About the guest: https://www.growthbook.io/podcast/guests/whitney-perez Episode page: https://www.growthbook.io/podcast/episode/1-40 Company Website: https://www.realtor.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io
- Transcript
S1 · E39
September 9 · 20 min
How Zalando connects every experiment to its North Star
Summary How do you keep 1,000 experiments a year pointed at one North Star? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Mi Tian, Head of Applied Science at Zalando, about running experimentation inside a central economics org that reports to the CFO. Mi shares how Zalando balances safe confirmatory tests with game-changing bets, how the team measured the discovery feeds homepage launch when success had no established metric, and how a KPI tree cascades the company North Star down to the controllable inputs teams ship every day. She also looks ahead to LLM-based agents as a simulation layer for screening hypotheses. A practical conversation for anyone building an experimentation program that wants both rigor and ambition. Chapters 00:45 About Zalando and its marketplace model 02:00 Mi's path from engineering to experimentation 03:10 Economists and data scientists in one decision-making org 04:15 Running over 1,000 experiments a year 05:55 What makes an experiment high risk 07:10 Sharing learnings through standardization and champions 09:10 The discovery feeds launch and its measurement plan 11:15 Balancing the experimentation portfolio 12:35 Growing a KPI tree from the North Star 16:05 LLM agents and the future of experimentation at Zalando 18:35 New missions for a longstanding business Takeaways - Treat experimentation as a portfolio: balance confirmatory tests that protect the business with game-changing bets that can win big. - Assess risk tiers when building the roadmap so measurement rigor scales with the stakes instead of slowing every decision down. - When a launch is too new to have a success metric, pair short term A/B tests with long term holdouts from day one. - Connect every experiment to the company North Star by cascading it down to sensitive proxy metrics and controllable inputs. - Use LLM based agents as a cheap simulation layer to screen hypotheses, not as a replacement for real A/B tests. Connect with the Guest Mi Tian LinkedIn: https://www.linkedin.com/in/mi-tian-941b4767/ About the guest: https://www.growthbook.io/podcast/guests/mi-tian Episode page: https://www.growthbook.io/podcast/episode/1-39 Company Website: https://www.zalando.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io
- Transcript
S1 · E38
September 8 · 21 min
Learneo on testing the opposite of every hypothesis
Summary Rich Liebling, senior director of engineering at Learneo, joins host Ashley Stirrup to explain the practice that came out of growing Shop It To Me from 50,000 subscribers to one million in nine months: test your hypothesis, and test its opposite. Rich covers the page where cutting text lost and adding text won, why the inverse wins more often than teams expect, and why small focused tests are the only ones where "the opposite" means anything. He also walks through translating a million subscriber goal into a target of 300 A/B tests, making a new engineer's second merge request their own experiment, and the different constraints at Course Hero, where competing team metrics were resolved with an early lifetime value model and three-day SQL analyses quietly capped testing velocity. This episode is for product managers, engineers, data scientists, and growth leaders building or scaling an experimentation program. Chapters 00:00 Cold open and introduction 01:50 Shop It To Me and the first engineering hire 02:55 300 A/B tests as the path to one million subscribers 04:00 Testing as a core value and the second merge request 07:20 Learneo, Course Hero, and two ways to get access 08:50 Modeling lifetime value to stop teams competing 11:10 Testing the opposite of the hypothesis 14:35 A portfolio of small, medium, and large tests 17:15 Multi armed bandits and seasonal traffic 20:35 The flywheel that keeps copycats behind Takeaways - Test the opposite of every hypothesis. At Shop It To Me the inverse won surprisingly often, and even when it lost it proved the variable mattered. - Keep tests small and focused. Redesign a whole page and lose, and you learn that version failed but not what to change next. - Translate a growth goal into an execution count. One million subscribers is not actionable. 300 A/B tests by year end is, and everyone can influence it. - Make experimentation part of hiring and onboarding. Every new engineer's second merge request was their own test idea. - Analysis friction sets the ceiling on testing velocity. Three days of ad hoc SQL per test quietly discourages teams from running more. Connect with the Guest Rich Liebling LinkedIn: https://www.linkedin.com/in/richliebling/ About the guest: https://www.growthbook.io/podcast/guests/rich-liebling Episode page: https://www.growthbook.io/podcast/episode/1-38 Company Website: https://www.learneo.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io
- Transcript
S1 · E37
September 3 · 21 min
The four questions Early Warning asks before any A/B test
Summary What separates a valid A/B test from an expensive guess? Priya Singhee, VP of Enterprise Analytics & Data Science at Early Warning — the bank-owned consortium that fights payment fraud and operates Zelle, which processed a trillion dollars last year — joins host Ashley Stirrup to share the experimentation playbook she built leading storefront analytics at Wayfair. She walks through the four questions to ask before launching any A/B test, why 85 to 90% of tests are supposed to fail, how pre-registration and kill criteria stop p-hacking before it starts, the pitfalls that fake wins (novelty effects, hidden heterogeneity, multiple comparisons), and how to roll out winners with gradual ramps and long-running holdouts. A practical episode for product managers, engineers, data scientists, and growth leaders building rigorous experimentation programs. Chapters 00:45 Meet Early Warning: fraud detection, Zelle, and a trillion dollars in payments 02:00 Wayfair and optimizing every step of the storefront funnel 03:05 The four questions to ask before any A/B test 05:15 Test setup best practices: hypotheses, guardrails, power, and pre-registration 07:40 Why 85 to 90% of tests fail and why that's a learning agenda 09:25 Novelty effects, hidden heterogeneity, and the multiple comparisons problem 12:55 Pre-registration, kill criteria, and stopping p-hacking 15:10 Rolling out winners: gradual ramps and long-running holdouts 17:15 Causal inference when you can't A/B test 20:05 The case for more A/B testing, not less Takeaways - Run the four-question framework before any test: clean randomization, a plausible effect size for your traffic, a reversible and cheap change, and a falsifiable hypothesis. - Treat A/B testing as a learning agenda: 85 to 90% of tests are supposed to fail, and a suspiciously high win rate is a red flag, not a trophy. - Pre-register the full analysis plan, including hypothesis, mechanism, primary metric, exact statistical test, and subgroups, so p-hacking can't creep in when a test goes sideways. - Define kill criteria and success, failure, and guardrail-dip actions before launch, with leadership sign-off, so nobody chases a loss into a fake win. - Log every test and its learnings where the whole organization can see them; that reinforcement loop is what separates world-class experimentation programs. Connect with the Guest Priya Singhee LinkedIn: https://www.linkedin.com/in/priya-singhee/ About the guest: https://www.growthbook.io/podcast/guests/priya-singhee Episode page: https://www.growthbook.io/podcast/episode/1-37 Company Website: https://www.earlywarning.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io
- Transcript
S1 · E36
September 1 · 23 min
How Supercell A/B tests 300 million players without breaking trust
Summary On this episode of The Experimentation Edge, Ashley Stirrup talks with Shan Huang, data scientist on the central experimentation team at Supercell, the Helsinki mobile game company behind Clash of Clans, Clash Royale, Brawl Stars, Hay Day, and Boom Beach. Shan explains how a famously decentralized, creative-first company with 300 million monthly active users runs fewer than 100 A/B tests a quarter and why that restraint is deliberate, how Supercell announces experiments to players in advance and promises make-up events to keep testing fair, and how importable AI skills now let anyone at the company analyze their own experiments, making quality consistency the next big challenge. It's for product managers, data scientists, and growth leaders balancing creative conviction with experimental rigor. Chapters 00:00 Intro 01:10 About Supercell and 300 million players 02:30 The central team and a decentralized culture 04:25 Fewer than 100 tests a quarter 05:40 Sharing learnings across independent game teams 07:35 Retention as the North Star 08:35 Onboarding experiments with gems and tutorials 12:45 Telling players about A/B tests 15:15 Hypotheses and proxy metrics 20:45 AI and the future of experiment analysis Takeaways - Supercell runs fewer than 100 A/B tests a quarter for 300 million monthly players, because the goal is to become more hypothesis driven while staying creative, not to maximize volume. - In a decentralized company, a central experimentation team earns its impact by providing the platform, partnering on rigor, and making sure learnings travel across independent game teams. - Announce experiments to users before they run; Supercell's community managers tell players what is being tested and why, which turns a skeptical community into a research partner. - Promise fairness, not just transparency; players who get the worse variant always receive a make-up event later, because game players come to have fun, not to be disadvantaged. - AI-powered self-serve analysis means everyone can now run and analyze experiments, so the next challenge is making the quality of AI analysis consistent across the whole company. Connect with the Guest Shan Huang LinkedIn: https://www.linkedin.com/in/cnshanhuang/ About the guest: https://www.growthbook.io/podcast/guests/shan-huang Episode page: https://www.growthbook.io/podcast/episode/1-36 Company Website: https://supercell.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io
- Transcript

Showing 1–20 of 39 episodes