Skip to content
Sign in
Artwork for The Experimentation Edge
BusinessTechnology

The Experimentation Edge

Growthbook

How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale.

Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.

Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.

Play
Open in
  • 42 episodes
  • Avg 27 min
  • English

Episodes 42

NewestOldestLongest
All20Unplayed20In progress0Played0
Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.
  • S2 · E6
    Thursday · 32 min

    How RealPage finds its biggest A/B test wins, one funnel at a time

    Summary Andrew Miller, Director of Product Management, Growth and E-commerce at RealPage, joins host Ashley Stirrup to share how experimentation works inside Loft, RealPage's end-to-end renter app. Andrew has stood up experimentation programs at 3 companies, and today he runs a portfolio of moving services where a winning A/B test ties directly to revenue. Andrew explains how a small, centralized optimization team ships tests in Optimizely faster than engineering release cycles allow, and how he builds a culture of experimentation with funnels, group ideation, the Opportunity Solution Tree and simple test templates. He also breaks down 2 electricity experiments: one that cut choices from 8 to 3 and lifted conversion 12%, and a price display test that lost. For product and growth leaders who want more ideas, faster launches and tests that actually teach them something. Chapters 00:00 RealPage, Loft and a B2B2C business 01:17 Why every A/B test in Loft focuses on the renter 03:02 The moving marketplace and how it makes money 04:53 Standing up experimentation programs at 3 companies 07:10 The T-shaped PM: depth in data and experimentation 08:57 A centralized optimization team for all of Loft 11:05 Why tests run in Optimizely instead of engineering 12:42 Building a culture of experimentation with funnels 15:34 The Opportunity Solution Tree and test templates 17:06 How good is your gut at predicting winners? 19:01 Cutting electricity options from 8 to 3 for a 12% lift 21:13 The price display test that lost 24:26 Minimizing variables when traffic is limited 26:04 Why diversity of ideas matters 27:38 PMs submitting PRs, not just hypotheses 30:45 Prototyping tests with AI: from 2 weeks to 1 day Takeaways - Start with telemetry and funnels; funnel drop-offs get stakeholders to the insight before you have to say it - Run the first ideation sessions as a group and organize ideas with an Opportunity Solution Tree; one session can fill a year of backlog - Require a hypothesis, a metric and a definition of success for every test idea so learning is the baseline outcome - Fewer choices can lift conversion: cutting electricity plans from 8 to 3 raised conversion 12% - An estimate shown with too much precision can backfire; round estimated prices so they read as estimates Connect with the Guest Andrew Miller LinkedIn: https://www.linkedin.com/in/findandrew/ Company Website: https://www.realpage.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S1 · E51
    Wednesday · 31 min

    Why you're already wrong 75% of the time

    Summary In this special episode of The Experimentation Edge, host Ashley Stirrup looks back at lessons from more than 50 guests with two people from GrowthBook: co-founder and CEO Graham McNicoll and Head of Customer Success & Strategy Cris Dizon, who ran experimentation programs at The Home Depot, Walmart and Peacock. The starting point is a humbling number: across companies, only about 25% of experiments win, so even your best ideas are wrong most of the time. Graham shares the untested redesign that quietly cut new-user sign-ups by about 20% for six months, and the bug that set a paywall to zero and overturned years of tribal knowledge. Cris explains how a streaming onboarding flow that failed came back and won once personalization caught up, and why heavy promo codes train customers to wait. Along the way they revisit stories from UPS, JPMorgan Chase, Fanatics, Chess.com, Groupon, Signet Jewelers, Box, Disney, Twitch, Learneo and Charlie Health, and close with five ways executives can build a culture that measures and learns from being wrong. Chapters 00:00 Welcome to a special episode with Graham McNicoll and Cris Dizon 01:24 Why only about 25% of experiments win 03:28 Building a culture where it's okay to fail 04:31 From Fyxer to UPS: guests who admit their gut is wrong 05:05 The untested redesign that cost millions 06:17 A big-bang homepage redesign that took months to fix 07:48 Spot the million-dollar change in the JPMorgan Chase funnel 08:48 Twitch pricing and the risk of anti-knowledge 09:28 The bug that set the paywall to zero 11:05 The onboarding test that failed, then won 12:59 Capturing learnings so they don't get lost 13:54 UPS, JPMorgan Chase and Fanatics: the size of the wins 14:18 Reframing wins and losses as gains and savings 15:45 Measure your learning rate, not your win rate 16:49 Chess.com: review your wins, not your losses 18:09 Groupon swaps flash sales for scarcity 18:52 How promo codes train customers to wait 20:18 Signet Jewelers and friction that helps buyers 21:11 Box and the wine effect 22:34 The checkbox with more text that lifted conversions 23:28 Test the opposite of your hypothesis 24:15 Disney's painted door test for video support 25:55 Five ways executives can build an experimentation culture 27:58 Could we have learned this faster? 28:32 Share the design and let people pick the winner 29:35 Buddy programs and pre-mortems 30:27 You're already wrong 75% of the time Takeaways - Expect to be wrong: even mature programs see only about 1 in 4 experiments win - Never ship a big-bang redesign untested; a hidden loss can run for months - Track savings from losses alongside gains from wins - Measure your learning rate, not your win rate - Revisit old losers when your product or capabilities change Connect with the Guests Graham McNicoll LinkedIn: https://www.linkedin.com/in/grahammcnicoll/ Company Website: https://www.growthbook.io Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E5
    Tuesday · 32 min

    How Preply runs close to 1,000 experiments a year

    Summary Sebastien Polis, Senior Experimentation Manager at Preply, joins host Ashley Stirrup to explain how a learning marketplace with 25 to 30 product teams runs close to 1,000 experiments a year. At Preply, no change reaches customers without a test, OKRs can only be claimed through experiments with an approved target metric, and a public Slack channel puts every launch, result and kill decision in front of the exec team. Sebastien shares why roughly 80% of experiments lose and 30 to 40% are iterations of an earlier idea, how a global control group shows double-digit growth, and what Preply learned when a control group went negative and a simple wording win turned out to be Twyman's law in action. He also covers why marketplaces randomize tutors instead of learners, the clustered analysis that requires, and his plan to build a learning system that gives AI the context it needs. For product, data and growth leaders building an experimentation culture at scale. Chapters 00:00 Preply's learning marketplace and Sebastien's path into experimentation 04:52 Tying OKRs to experiments across 25 to 30 product teams 06:39 Why 80% of experiments lose and 30 to 40% are iterations 08:35 Double-digit growth from a global control group 09:46 The control group that went negative and Twyman's law 15:03 How a new PM designs an experiment at Preply 17:57 A public Slack channel where execs question every test 20:48 When clicks go up but learners don't buy 23:23 Building a learning system before feeding AI 25:20 An experiment run entirely by AI, with a human approver 28:13 Randomizing tutors, not learners, in a 2-sided marketplace Takeaways - Preply ties OKRs to experimentation: teams can only claim results through experiments that use an approved target metric - Losing is the norm; about 80% of Preply's tests fail and 30 to 40% of each year's experiments are iterations of an earlier idea - A surprising win from a simple change deserves a rerun; Preply unscaled a wording test and the lift disappeared - A public Slack channel where execs question PMs about launches and kills keeps rigor and transparency high across 1,000 tests a year - In a marketplace, randomize the tutor and measure learner outcomes with a clustered analysis, and ship bug-free code because sample size is scarce Connect with the Guest Sebastien Polis LinkedIn: https://www.linkedin.com/in/sebastien-polis-b5aa8256/ Company Website: https://preply.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E4
    October 29 · 25 min

    Building a Culture of Experimentation in Sports Analytics with New York Red Bulls

    Summary Vamsi Krishna Kunapuli, Director of Analytics at New York Red Bulls, leads a small team that supports marketing, ticket sales and partnerships for one of the original MLS clubs and its 25,000 seat stadium. He shares how an experiment across five to six seasons changed the club's North Star from average ticket price to gross margin. Host Ashley Stirrup and Vamsi also discuss why a losing test is not a failure, why segments can never be static, where good test ideas come from, and how identity resolution will open the door to machine learning and personalized fan experiences. For analytics, marketing and product leaders building experimentation into a business where every event is different. Chapters 00:00 Intro 00:20 New York Red Bulls: an original MLS club 01:32 Data-driven fan engagement 04:46 Single game buyers outspend season ticket holders 07:21 Testing frameworks 09:29 Holistic fan value 11:11 Segments can never be static 12:59 A losing test isn't a failure 14:51 Loser or lever: judging tests against the North Star 17:13 Building a culture with clear end goals 20:25 Ideas populate when there is more data 23:10 Identity resolution, AI and personalization 26:06 Empowering more teams to experiment 27:57 Every game is like hosting a wedding Takeaways - We have seen instances where single game buyers at the New York Red Bulls outspend season ticket holders once concessions and the Hype House zone are counted - The club moved its North Star from average ticket price to gross margin, which changed couponing, fan profiles and retargeting - A losing test is not a failure; judged against the North Star, a negative result shows what not to do and that a lever exists - Segments should be dynamic, and the club is building toward a separate segmentation for every event - Ideas populate when there is more data, and the stakeholders who know the business best should drive the questions Connect with the Guest Vamsi Krishna Kunapuli LinkedIn: https://www.linkedin.com/in/vamsikunapuli/ Company Website: https://www.newyorkredbulls.com Vamsi is a panelist at DMWF North America 2026, "Connected Data: The Foundation of Intelligent Marketing" Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E3
    October 27 · 38 min

    Motive's Shiva Manjunath on CRO as a mindset, not a role

    Summary Shiva Manjunath, Lead Web Strategist at Motive and host of the From A to B podcast, joins Ashley Stirrup to explain why CRO works best as a mindset every team carries rather than a role one team owns. He shares how building bridges to sales and customer service turns everyday complaints into test ideas. The conversation covers why customers who ask for price are often asking for value, how emotional context earns a CRO credibility, how to make an impact without an executive sponsor, and why experimentation needs the kind of marketing AI gets. For CROs, product managers and growth leaders building an experimentation culture. Chapters 00:00 Cold open: gifts under $50 01:18 What a true culture of experimentation looks like 04:43 CRO as a mindset across every function 11:27 Become the sales team's best advocate 15:29 Fear, uncertainty and doubt: the window shade story 21:51 Customers ask for price when they want value 24:17 GTA 6 and why button colors cost CROs trust 29:14 Making an impact without an executive sponsor 34:27 Making experimentation as exciting as AI Takeaways - CRO works best as a mindset shared by design, marketing and product, not as one team's job. - Sales and customer service teams hold free qualitative data that makes strong test ideas. - When customers ask for price, test value messaging such as benefits, ROI and time saved. - Understanding the emotional context around a purchase earns a CRO more trust than tactical button tests. - A culture of experimentation needs data-driven decision making first and a safe space to lose. Connect with the Guest Shiva Manjunath LinkedIn: https://www.linkedin.com/in/shiva-manjunath/ From A to B podcast: https://open.spotify.com/show/5xDlubE3GZwUDDen64hefz Newsletter: https://fromatob.beehiiv.com/ Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S2 · E3
    October 22 · 26 min

    How Quill turned loyalty points into a 1% conversion lift

    Summary Vipul Tyagi, Senior Product Manager at Quill, owns the online experience from add to cart through checkout, with a focus on getting customers to use Quill's loyalty program. Host Ashley Stirrup digs into why a homepage rewards widget drew under 0.5% engagement over 60 days, and how moving the value to the cart with a Lego-inspired points to dollars option lifted conversion by 1% while shrinking the points liability on Quill's books. The conversation covers Vipul's 8 winners out of 11 loyalty initiatives and the BillionToOne advice behind them, how he coaches new PMs on primary and secondary hypotheses and guardrail metrics, a live test on showing the $5.99 shipping saving at checkout, a semantic layer experiment built for AI crawlers, and how AI has cut his research time from two months to seven days. For product managers, growth leaders and e-commerce teams who want a higher win rate from their experiments. Chapters 00:00 Cold open 01:06 Quill, office supplies and three kinds of customers 04:20 Owning add to cart through checkout, and 8 winners out of 11 05:32 Know the direction before you run the test 06:54 The homepage loyalty widget and the icing on the cake 10:19 Points into dollars on the cart page: a 1% conversion lift 12:39 Coaching new PMs: ranked hypotheses and guardrail metrics 15:41 Taking credit for free shipping in the order summary 18:10 Copy testing, strategic bets and a semantic layer for AI crawlers 22:24 AI in the PM workflow: research in seven days instead of two months Takeaways - Rewards work where purchase intent already exists: Quill's homepage loyalty widget drew under 0.5% engagement, while the same value on the cart page lifted conversion. - Converting points into dollars at the cart raised conversion 1%, brought small orders back and reduced the unredeemed points liability on the books. - Know the direction of a test before you run it; the A/B test measures how much, which is how Vipul's loyalty work reached 8 winners out of 11. - Design every experiment with a primary hypothesis, ranked secondary hypotheses and guardrail metrics so a loss still explains itself. - Showing the $5.99 shipping saving at checkout raised conversions without changing order value, a sign that transparency builds trust. Connect with the Guest Vipul Tyagi LinkedIn: https://www.linkedin.com/in/vipul-tyagi Company Website: https://www.quill.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io (00:00) - Cold open (01:06) - Quill, office supplies and three kinds of customers (04:20) - Owning add to cart through checkout, and 8 winners out of 11 (05:32) - Know the direction before you run the test (06:54) - The homepage loyalty widget and the icing on the cake (10:19) - Points into dollars on the cart page: a 1% conversion lift (12:39) - Coaching new PMs: ranked hypotheses and guardrail metrics (15:41) - Taking credit for free shipping in the order summary (18:10) - Copy testing, strategic bets and a semantic layer for AI crawlers (22:24) - AI in the PM workflow: research in seven days instead of two months

    • Transcript
    • Chapters
  • S2 · E2
    October 20 · 40 min

    Engagement with safety: Headspace's guardrails for AI tests

    Mary Alfheim, VP Data and Analytics at Headspace, joins host Ashley Stirrup to explain why more engagement is not always better when your members come to you with real symptoms. She walks through how Headspace experiments on Ebb, its AI companion, by defi

    • Transcript
  • S2 · E1
    October 15 · 19 min

    How Walmart Connect learned to align cross-sell with shopper intent

    Summary Arv Ramchandani, Senior Product Manager for Experimentation & Analytics at Walmart Connect, runs ad experiments on pages owned by other teams, from search to the conversational AI shopping agents, each with its own primary metric. He explains

    • Transcript
  • S1 · E50
    October 14 · 19 min

    Why more info lifted VidaXL's account creation nearly 40%

    Summary Sapna Kashyap, CRO Lead at VidaXL, runs the experimentation program for an e-commerce company operating in more than 30 countries, where product, development and business teams all validate their decisions with an A/B test first. She explains the

    • Transcript
  • S1 · E49
    October 13 · 40 min

    The auto researcher: How FanDuel finds a winning model in 300 attempts

    Michael Mathews, Director of Data and AI Product at FanDuel, runs the teams behind the auto researcher, an agent that takes a data set, an objective function and the production model, then proposes, writes and scores about 300 strategies in an hour for ro

    • Transcript
  • S1 · E48
    October 8 · 21 min

    What Grubhub tests for big product bets before they ship

    Summary Grubhub does not run much classic A/B testing. Michal Lenik, Associate Director of Product Design at Grubhub, explains what the company does instead to make sure its big product bets win before they reach millions of users. She joins Ashley Stirrup to walk through Grubhub's playbook: small cohort rollouts that protect core order metrics, merchant round tables, AI built clickable prototypes, and a concept test in the merchant promotions space that sent the team toward a much simpler dashboard than their research predicted. Michal also explains why she follows losing experiments with user interviews, why "I didn't even notice it" is valuable feedback, and why every company should decide deliberately whether it de-risks upstream or in a fast follow after launch. Chapters 00:00 Where design is heading in the AI era 00:53 Meet Michal Lenik of Grubhub 02:21 Design as an upstream business lever 03:48 Small cohorts instead of classic A/B tests 05:27 The merchant concept test that changed direction 08:18 Prototyping with AI and merchant round tables 10:16 Success metrics for product, design and engineering 11:56 Pairing experiment data with user interviews 14:26 AI, personalization and the designer's new role 17:45 De-risking upstream versus shipping fast Takeaways - Grubhub tests big bets by releasing to internal users or small cohorts first, so it can read business impact without disrupting core revenue and order metrics. - A concept test with merchants overturned what fifteen research interviews suggested, steering Grubhub to a simpler dashboard and away from a launch that likely would not have won. - AI lets Grubhub's design teams put clickable prototypes in front of users early, turning each iteration into a micro experiment. - Pairing quantitative results with five to ten user interviews after a test explains why it underperformed, including whether users even saw the change. - Both upstream de-risking and fast iteration work. The right choice depends on how quickly engineering can ship, and it should be made on purpose. Connect with the Guest Michal Lenik LinkedIn: https://www.linkedin.com/in/michallenik/ About the guest: https://www.growthbook.io/podcast/guests/michal-lenik Episode page: https://www.growthbook.io/podcast/episode/1-48 Company Website: https://www.grubhub.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io (00:00) - Where design is heading in the AI era (00:53) - Meet Michal Lenik of Grubhub (02:21) - Design as an upstream business lever (03:48) - Small cohorts instead of classic A/B tests (05:27) - The merchant concept test that changed direction (08:18) - Prototyping with AI and merchant round tables (10:16) - Success metrics for product, design and engineering (11:56) - Pairing experiment data with user interviews (14:26) - AI, personalization and the designer's new role (17:45) - De-risking upstream versus shipping fast

    • Transcript
    • Chapters
  • S1 · E47
    October 6 · 20 min

    How MilliporeSigma lifted add to cart with a smarter search

    Summary Dorothy Crepin runs the A/B testing program at MilliporeSigma, the life sciences business of Merck KGaA, where one global catalog has to serve three very different visitors: scientists hunting for a specific instrument, procurement buyers working off a list someone else wrote, and students doing research with no purchasing power at all. Add a regulated industry where what you can sell changes country by country, and personas stop being a marketing exercise. She walks Ashley Stirrup through the test that changed how her team thinks about discovery. MilliporeSigma rebuilt its on-site type ahead so it suggests a search term instead of pushing a matching product, and saw double digit increases in feature usage and add to cart. From there Dorothy makes the case for measuring the action a page is actually responsible for rather than revenue alone, for pairing A/B tests with a user testing ambassador panel to get at the why, and for sending product managers back to one question before anything gets built: what problem are you solving for the customer? Chapters 00:00 Revenue is an outcome, not the metric 01:26 Welcome and what MilliporeSigma does 02:22 The three audiences on a B2B science site 03:53 What an A/B testing analyst coordinates 04:20 Running experiments across product pods 05:31 Building leadership buy in for testing 08:08 Where A/B testing ends and user testing begins 08:52 The type ahead search test that lifted add to cart 11:30 How to frame a test with a product manager 13:34 Why testing the opposite pays off 15:26 Personas, personalization and what AI changes 18:57 Building trust across competing teams Takeaways - MilliporeSigma changed its on site search to suggest search terms instead of products and saw double digit gains in feature usage and add to cart. - Revenue is an outcome, so Dorothy's team measures the next action each page is responsible for, like add to cart on a product detail page. - A/B testing tells you what happened. MilliporeSigma's user testing ambassador panel supplies the why behind the click. - Every test starts with one question to the product manager: what problem are you solving for the customer, not what is your hypothesis. - Dorothy is building personas into agents so browsers, not just buyers, can be tested for a directional read before an experiment ships. Connect with the Guest Dorothy Crepin LinkedIn: https://www.linkedin.com/in/dorothycrepin/ About the guest: https://www.growthbook.io/podcast/guests/dorothy-crepin Episode page: https://www.growthbook.io/podcast/episode/1-47 Company Website: https://www.sigmaaldrich.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S1 · E46
    September 30 · 31 min

    PayPal's $180 million experimentation win

    Summary Gaurav Sethi joins Ashley Stirrup on The Experimentation Edge to explain how PayPal's experimentation program went from 700 to 800 tests a year with four week readouts to 2,587 experiments a year and roughly $180 million in measured impact. Gaurav inherited a reported win rate of 55% to 60%, four times the industry average, and traced it to experiments logging assignment data instead of exposure data, carrying 25% to 30% dilution. The conversation covers the exposure event his team introduced, the instrumentation and metric standards that cut readouts to 24 hours, a carousel test that a multi armed bandit resolved in 51 days instead of a projected 700, and why cost avoidance from losing experiments belongs in the ROI number. It closes on what changes when the thing you are testing is an AI agent rather than a button. Useful for product managers, engineers, data scientists and growth leaders building or defending an experimentation program at scale. Chapters 00:00 Cold open 00:54 Welcome Gaurav Sethi of PayPal 01:30 Elmo, PayPal's homegrown experimentation platform 03:53 $180 million in revenue impact and cost avoidance 04:28 The win rate that was too good to be true 06:29 Instrumentation standards and the exposure event 08:56 The carousel test: 700 days down to 51 11:29 Designing experiments so every result teaches you 13:45 Building the platform is only half the job 15:07 Education, office hours and executive support 18:28 Exposure events and joining transactional data 24:14 AI for experimentation, and experimentation for AI Takeaways - A win rate of 55% to 60% against an industry average of 11% to 14% was a tracking problem, not a performance one: assignment data carried 25% to 30% dilution and reached significance on users who never saw the test. - The exposure event fixed it. Fire an event as close to render as possible so the platform knows exactly when a user entered the experiment, instead of logging on page load. - Standardized instrumentation and canonical metric definitions are what let the analysis pipeline run without human cleanup, taking experiment readouts from four weeks to 24 hours. - A six variant carousel test projected at 700 days was resolved by a multi armed bandit in 51 days, with a winner worth almost five basis points. - About half of PayPal's $180 million impact in 2025 was cost avoidance from features that tested badly and never shipped, which is why Gaurav argues there are no losing experiments. Connect with the Guest Gaurav Sethi LinkedIn: https://www.linkedin.com/in/gauravsethi22 About the guest: https://www.growthbook.io/podcast/guests/gaurav-sethi Episode page: https://www.growthbook.io/podcast/episode/1-46 Company Website: https://www.paypal.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S1 · E45
    September 29 · 27 min

    ServiceNow's Customer Zero approach to AI experimentation

    Summary ServiceNow runs its own platform on itself. Ashraf Karim, Senior Vice President, Connected Customer Experience, owns everything that happens after a customer buys, and her team is Customer Zero, testing every product and capability internally before it ever reaches a customer. She joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline lesson is not a technical one. ServiceNow built agentic skills that worked, and users abandoned them anyway, because a task that ran five to fifteen minutes broke every expectation people had about what a machine should feel like. The fix was not a faster model. It was telling the user what the AI was doing while it did it. Ashraf also gets into the harder measurement problem underneath all of it: how do you evaluate a nondeterministic system? ServiceNow's answer is an internal LLM judge scored against a golden data set, plus a customer effort score that catches what CSAT reports too late. Chapters 00:00 What customers actually care about 01:14 Meet Ashraf Karim of ServiceNow 01:39 Owning the entire post-sale experience 02:08 Why ServiceNow runs as its own Customer Zero 03:02 Lessons from Google, PayPal and Verizon 07:11 The agentic skills users kept abandoning 09:29 Latency, expectations and the feedback loop fix 12:29 Users wanted the outcome, not the process 14:49 Designing experiments that can afford to lose 18:13 Using an LLM judge on nondeterministic models 21:48 Customer effort score as a guardrail metric 25:03 Where experimentation at ServiceNow goes next Takeaways - ServiceNow is its own Customer Zero, running the platform on itself and testing every feature internally before customers see it. - The agentic skills worked. Users still abandoned them, because a five to fifteen minute wait broke their expectation of what AI should feel like. - The fix was a feedback loop that narrates what the AI is doing in the background, which buys the patience a long task needs. - Users did not want the agentified version of the human process. They wanted the outcome and the next best action, not the steps. - Nondeterministic output needs its own evaluation layer, so ServiceNow built an LLM judge that scores against a golden data set before anything reaches a customer. Connect with the Guest Ashraf Karim LinkedIn: Ashraf Karim on LinkedIn About the guest: https://www.growthbook.io/podcast/guests/ashraf-karim Episode page: https://www.growthbook.io/podcast/episode/1-45 Company Website: ServiceNow Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S1 · E44
    September 24 · 24 min

    Why Twilio ships on signals instead of significance

    Summary Wanli Lau spent eleven years at Expedia Group, most recently running consumer product analytics on a program that shipped more than 1,500 A/B tests a year. Five months ago she moved to Twilio to lead global R&D analytics. In this episode she walks Ashley Stirrup through what carried over and what did not. The centerpiece is a test that looked like a win. At Expedia, an account sign-up takeover on the front page drove conversion sharply up, and quietly harmed return rate and engagement across every geography. That result is where Wanli's insistence on guardrail metrics and a trade-off decision framework comes from. She also explains the parts of B2B experimentation that have no B2C equivalent: choosing between user-level and account-level randomization, the interference risk when two colleagues at the same company see different pricing, and the many-to-many problem of one developer belonging to several accounts. And because Twilio's customer counts are lower than a consumer business, her teams often ship on signals and team conviction rather than waiting for statistical significance. Chapters 00:00 Cold open 01:02 Welcome Wanli Lau of Twilio 01:23 Inside Wanli's role at Twilio 04:03 Eleven years and 1,500 tests a year at Expedia 04:58 Why B2B experimentation differs from B2C 05:33 Cross-functional collaboration and good hypotheses 06:39 Teaching teams to experiment rigorously 07:48 The Expedia sign-up takeover that won on conversion 11:09 Guardrail metrics and the trade-off framework 14:52 Designing experiments to maximize learning 17:46 Diagnosing where a feature failed 19:10 North Star metrics and account-level randomization at Twilio 21:40 Learning from signals instead of significance 22:11 Where experimentation goes next at Twilio Takeaways - A test that wins on the primary metric can still be a loss. Expedia's sign-up takeover raised conversion and damaged return rate and engagement at the same time. - Name the primary, secondary and guardrail metrics before the test runs, not at readout. Deciding afterwards turns a result into a debate. - Conversion should be a do-no-harm guardrail for teams that do not own it, even when it is not their primary metric. - In B2B the randomization unit is a design decision. User-level bucketing risks two colleagues at one account seeing different prices, and account-level avoids that but costs sample size. - When customer counts are low, significance is often out of reach. Learning from signals and shipping on team conviction beats waiting for a number that will never arrive. Connect with the Guest Wanli Lau LinkedIn: https://www.linkedin.com/in/wanlilau About the guest: https://www.growthbook.io/podcast/guests/wanli-lau Episode page: https://www.growthbook.io/podcast/episode/1-44 Company Website: https://www.twilio.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S1 · E43
    September 22 · 26 min

    Why GoPro stopped judging A/B tests by win rate

    Will Guyeskey is Director of Digital Product at GoPro, where his team owns the e-commerce side of gopro.com. Before GoPro he ran personalization at Gap and cut his teeth at Brooks Bell, testing for brands like Barnes & Noble, Under Armour and Ralph Lauren. In this episode of The Experimentation Edge, he tells Ashley Stirrup why knowing your customer is the through line of every good test program. Will shares the Barnes & Noble order confirmation test he was sure would lose, and why the same idea never worked for any other client. He walks through a recent GoPro Mission launch test that asked whether a step-by-step configurator adds too much friction, and what a flat result revealed about high consideration buyers. He also explains why GoPro shares interim readouts across the company, why win rate makes a poor North Star for an experimentation program, and how his team plans to use AI for speed without outrunning its own learnings. Chapters 00:00 Intro 00:52 Will's role running e-commerce at GoPro 01:43 Learning A/B testing across retail at Brooks Bell 05:24 How GoPro runs one to three tests a month 06:35 Sharing learnings and interim readouts across teams 09:20 The Barnes & Noble order confirmation win 13:02 Why the win did not transfer to other clients 14:41 Testing friction on the GoPro Mission configurator 18:51 Why win rate is the wrong North Star 22:19 How AI will shape experimentation at GoPro Takeaways - A winning idea rarely travels. The Barnes & Noble recommendation module worked because of that audience's low order values and reading habits, and it failed for every other client that tried it. - Design every test so it teaches you something whether it wins, loses or ends flat. Losing tests are jet fuel when the learning is built in. - A flat result is still an answer. GoPro's configurator test showed that buyers of high consideration products accept extra steps when each choice adds value. - Share interim readouts across the company, and use them to show how volatile results are before a test reaches statistical significance. - AI can speed up building and running experiments, but a team that runs more tests than it can learn from is not getting better. Connect with the Guest Will Guyeskey LinkedIn: https://www.linkedin.com/in/willguyeskey/ About the guest: https://www.growthbook.io/podcast/guests/will-guyeskey Episode page: https://www.growthbook.io/podcast/episode/1-43 Company Website: https://gopro.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    • Transcript
  • S1 · E42
    September 17 · 30 min

    What Samsung learned bringing B2C rigor to B2B

    Summary What changes when you take a mature B2C experimentation practice and apply it to a B2B storefront? Anuradha Tempe, Lead Product Manager at Samsung Electronics America, joins host Ashley Stirrup to share what she learned managing both sides of Samsung.com. She explains why B2B is not a sidekick to B2C, how a single bulk order can push a test into a false positive unless you normalize the data, and why not everything deserves an A/B test. She walks through the add-on experiment that nearly doubled attach sales once her team realized business buyers decide on mobile and purchase on desktop, and the Buy Now test that lost because B2B buyers value clarity over faster conversions. The conversation closes with her approach to North Star and guardrail metrics and the AI copilot she built to draft A/B test plans with a human still in the loop. A practical episode for product managers, engineers, data scientists, and growth leaders running experimentation across different customer types. Chapters 00:45 Meet Anuradha Tempe: from chip design to leading Samsung e-commerce 02:15 Why B2B is not a sidekick to B2C 03:30 Not everything needs an A/B test 05:15 Spreading experimentation practice across a global conglomerate 07:00 Bringing B2C rigor to a fast-paced B2B team 08:45 The add-on experiment that nearly doubled attach sales 11:15 Buyers decide on mobile and purchase on desktop 16:30 The Buy Now test that lost 21:30 North Star metrics, guardrails, and the EPP discount fix 26:05 An AI copilot for A/B test planning Takeaways - Treat B2B as its own customer base with its own testing discipline; a single bulk order on one day can inflate a B2B test into a false positive unless order data is normalized before results are read. - Not everything needs an A/B test; route lower-risk changes through UAT feedback or pre/post comparisons and reserve full experiments for features where being wrong is expensive. - Map where the decision happens, not just where the purchase happens; Samsung's business buyers decide on mobile and buy on desktop, and surfacing add-ons on mobile nearly doubled attach sales. - B2B buyers value clarity over faster conversions; a Buy Now button earlier in the flow confused bulk purchasers because returns and cancellations on large orders are costly. - Align on the North Star before building, whether it is revenue, engagement, NPS, or fewer support tickets, and set guardrails so an engagement feature can never quietly drag sales down. Connect with the Guest Anuradha Tempe LinkedIn: https://www.linkedin.com/in/anuradha-tempe/ About the guest: https://www.growthbook.io/podcast/guests/anuradha-tempe Episode page: https://www.growthbook.io/podcast/episode/1-42 Company Website: https://www.samsung.com/us/business/ Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io

    • Transcript
  • S1 · E41
    September 15 · 30 min

    Even a loss is a win: Charlie Health's approach to experiments

    Summary On this episode of The Experimentation Edge, Ashley Stirrup talks with Joe Yevoli, Director of Growth at Charlie Health, a virtual intensive outpatient program that sits between weekly therapy and hospitalization. Joe explains how removing a page from Charlie Health's intake form produced a winning test that created a new bottleneck further down the funnel, why Teachers Pay Teachers cut about 80% of its market for one feature and roughly quadrupled retention, and how premortems that ask "what went wrong?" before launch make dissent safe and surface safeguards that avoid catastrophe. He also shares the two questions he asks before any experiment, why more top-of-funnel traffic always dents conversion rate, and how AI is flattening the product pod in ways that speed teams up and put them at risk. It's for growth leaders, product managers, and experimentation teams who want to learn as much from a loss as from a win. Chapters 00:00 Intro 01:00 About Charlie Health 02:05 Joe's role across performance, lifecycle, and experimentation 03:30 Building experimentation rigor across teams 04:40 Even a loss is a win 05:15 The form page test that created a new bottleneck 08:50 Teachers Pay Teachers and the Easel lesson 16:30 Designing experiments that teach you something when they lose 21:00 Premortems for high-risk experiments 24:30 AI is flattening the org Takeaways - A winning test is a data point, not a finish line; Charlie Health's form page removal won on top-of-funnel metrics and still exposed a downstream bottleneck that became the next experiment. - More traffic entering the funnel means conversion rate goes down, even when more people reach the bottom; treat it as a law of physics and plan for it before you read the results. - Build it for everyone and you build it for nobody; Teachers Pay Teachers cut roughly 80% of the market for Easel, repositioned it around one job, and roughly quadrupled retention. - Run a premortem before any large, risky experiment: ask the room "it was a massive failure, what went wrong?", have everyone write and share, and build the safeguards while there is still time. - Before launching, check the quantitative and qualitative data behind the hypothesis, map the full funnel, and measure toward the real business outcome, so a loss still leaves you with a next step. Connect with the Guest Joe Yevoli LinkedIn: https://www.linkedin.com/in/joeyevoli/ About the guest: https://www.growthbook.io/podcast/guests/joe-yevoli Episode page: https://www.growthbook.io/podcast/episode/1-41 Company Website: https://www.charliehealth.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io

    • Transcript
  • S1 · E40
    September 10 · 20 min

    Realtor.com on using your AI as a junior data scientist

    Summary What do you do when your biggest experiment win turns out to be a loss? Whitney Perez, Director of Product Management at Realtor.com, joins host Ashley Stirrup to share the checkout bundling test that posted a 300% attach rate and still lost revenue, the 30/30/30 rule she uses to set expectations for a new experimentation team, and how AI is turning an English major into an aspirational data scientist. This episode is for product managers, engineers, and data scientists building experimentation programs from the ground up. Chapters 00:00 Cold open and welcome 01:10 From growth hacker to Realtor.com 02:50 Three foundations for a new experimentation team 04:20 The 30/30/30 rule 05:05 The 300% bundling win that lost revenue 07:10 You don't need a stats degree to experiment 09:20 Cascading North Star metrics 12:50 Do the homework before the experiment 13:50 The wishlist: instrumentation, embedded knowledge, culture 15:50 AI as an aspirational data scientist 18:45 Keeping a human in the loop Takeaways - A winning decision metric is not enough. Realtor.com's bundling test hit a 300% attach rate, but funnel fallout from the extra step made it a net revenue loser. Set secondary metrics and their thresholds before launch. - Expect the 30/30/30 rule: roughly a third of tests win, a third are inconclusive, and a third lose. The math is the math, and the losers carry most of the learning. - Start a new team on foundations: what a clean test and an A/A test look like, which surfaces should not be tested, and a peer review program that lets people graduate to more complex experiments. - You don't need a stats background to run good experiments. Teach the simplest definition of a good test, then let people learn by doing. - AI can make anyone an aspirational data scientist for analyzing results and spotting opportunities, but it can be confidently wrong. Keep a human in the loop and sanity check output the way you'd peek at a freshly launched test. Connect with the Guest Whitney Perez LinkedIn: https://www.linkedin.com/in/whitneykperez/ About the guest: https://www.growthbook.io/podcast/guests/whitney-perez Episode page: https://www.growthbook.io/podcast/episode/1-40 Company Website: https://www.realtor.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io

    • Transcript
  • S1 · E39
    September 9 · 20 min

    How Zalando connects every experiment to its North Star

    Summary How do you keep 1,000 experiments a year pointed at one North Star? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Mi Tian, Head of Applied Science at Zalando, about running experimentation inside a central economics org that reports to the CFO. Mi shares how Zalando balances safe confirmatory tests with game-changing bets, how the team measured the discovery feeds homepage launch when success had no established metric, and how a KPI tree cascades the company North Star down to the controllable inputs teams ship every day. She also looks ahead to LLM-based agents as a simulation layer for screening hypotheses. A practical conversation for anyone building an experimentation program that wants both rigor and ambition. Chapters 00:45 About Zalando and its marketplace model 02:00 Mi's path from engineering to experimentation 03:10 Economists and data scientists in one decision-making org 04:15 Running over 1,000 experiments a year 05:55 What makes an experiment high risk 07:10 Sharing learnings through standardization and champions 09:10 The discovery feeds launch and its measurement plan 11:15 Balancing the experimentation portfolio 12:35 Growing a KPI tree from the North Star 16:05 LLM agents and the future of experimentation at Zalando 18:35 New missions for a longstanding business Takeaways - Treat experimentation as a portfolio: balance confirmatory tests that protect the business with game-changing bets that can win big. - Assess risk tiers when building the roadmap so measurement rigor scales with the stakes instead of slowing every decision down. - When a launch is too new to have a success metric, pair short term A/B tests with long term holdouts from day one. - Connect every experiment to the company North Star by cascading it down to sensitive proxy metrics and controllable inputs. - Use LLM based agents as a cheap simulation layer to screen hypotheses, not as a replacement for real A/B tests. Connect with the Guest Mi Tian LinkedIn: https://www.linkedin.com/in/mi-tian-941b4767/ About the guest: https://www.growthbook.io/podcast/guests/mi-tian Episode page: https://www.growthbook.io/podcast/episode/1-39 Company Website: https://www.zalando.com Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io

    • Transcript
Showing 1–20 of 42 episodes