
ServiceNow's Customer Zero approach to AI experimentation
transcript
show notes
Summary
ServiceNow runs its own platform on itself. Ashraf Karim, Senior Vice President, Connected Customer Experience, owns everything that happens after a customer buys, and her team is Customer Zero, testing every product and capability internally before it ever reaches a customer.
She joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline lesson is not a technical one. ServiceNow built agentic skills that worked, and users abandoned them anyway, because a task that ran five to fifteen minutes broke every expectation people had about what a machine should feel like. The fix was not a faster model. It was telling the user what the AI was doing while it did it.
Ashraf also gets into the harder measurement problem underneath all of it: how do you evaluate a nondeterministic system? ServiceNow's answer is an internal LLM judge scored against a golden data set, plus a customer effort score that catches what CSAT reports too late.
Chapters
00:00 What customers actually care about
01:14 Meet Ashraf Karim of ServiceNow
01:39 Owning the entire post-sale experience
02:08 Why ServiceNow runs as its own Customer Zero
03:02 Lessons from Google, PayPal and Verizon
07:11 The agentic skills users kept abandoning
09:29 Latency, expectations and the feedback loop fix
12:29 Users wanted the outcome, not the process
14:49 Designing experiments that can afford to lose
18:13 Using an LLM judge on nondeterministic models
21:48 Customer effort score as a guardrail metric
25:03 Where experimentation at ServiceNow goes next
Takeaways
- ServiceNow is its own Customer Zero, running the platform on itself and testing every feature internally before customers see it.
- The agentic skills worked. Users still abandoned them, because a five to fifteen minute wait broke their expectation of what AI should feel like.
- The fix was a feedback loop that narrates what the AI is doing in the background, which buys the patience a long task needs.
- Users did not want the agentified version of the human process. They wanted the outcome and the next best action, not the steps.
- Nondeterministic output needs its own evaluation layer, so ServiceNow built an LLM judge that scores against a golden data set before anything reaches a customer.
Connect with the Guest
Ashraf Karim LinkedIn: Ashraf Karim on LinkedIn
About the guest: https://www.growthbook.io/podcast/guests/ashraf-karim
Episode page: https://www.growthbook.io/podcast/episode/1-45
Company Website: ServiceNow
Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.
Go to growthbook.io





