Skip to content
Artwork for The Experimentation Edge
The Experimentation Edge · September 29 · 27 min

ServiceNow's Customer Zero approach to AI experimentation

Summary ServiceNow runs its own platform on itself. Ashraf Karim, Senior Vice President, Connected Customer Experience, owns everything that happens after a customer buys, and her team is Customer Zero, testing every product and capability internally before it ever reaches a customer. She joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline lesson is not a technical one. ServiceNow built agentic skills that worked, and users abandoned them anyway, because a task that ran five to fifteen minutes broke every expectation people had about what a machine should feel like. The fix was not a faster model. It was telling the user what the AI was doing while it did it. Ashraf also gets into the harder measurement problem underneath all of it: how do you evaluate a nondeterministic system? ServiceNow's answer is an internal LLM judge scored against a golden data set, plus a customer effort score that catches what CSAT reports too late. Chapters 00:00 What customers actually care about 01:14 Meet Ashraf Karim of ServiceNow 01:39 Owning the entire post-sale experience 02:08 Why ServiceNow runs as its own Customer Zero 03:02 Lessons from Google, PayPal and Verizon 07:11 The agentic skills users kept abandoning 09:29 Latency, expectations and the feedback loop fix 12:29 Users wanted the outcome, not the process 14:49 Designing experiments that can afford to lose 18:13 Using an LLM judge on nondeterministic models 21:48 Customer effort score as a guardrail metric 25:03 Where experimentation at ServiceNow goes next Takeaways - ServiceNow is its own Customer Zero, running the platform on itself and testing every feature internally before customers see it. - The agentic skills worked. Users still abandoned them, because a five to fifteen minute wait broke their expectation of what AI should feel like. - The fix was a feedback loop that narrates what the AI is doing in the background, which buys the patience a long task needs. - Users did not want the agentified version of the human process. They wanted the outcome and the next best action, not the steps. - Nondeterministic output needs its own evaluation layer, so ServiceNow built an LLM judge that scores against a golden data set before anything reaches a customer. Connect with the Guest Ashraf Karim LinkedIn: Ashraf Karim on LinkedIn About the guest: https://www.growthbook.io/podcast/guests/ashraf-karim Episode page: https://www.growthbook.io/podcast/episode/1-45 Company Website: ServiceNow Sponsor GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

0:00-27:46

transcript

No transcript — this publisher did not publish one.

show notes

Summary
ServiceNow runs its own platform on itself. Ashraf Karim, Senior Vice President, Connected Customer Experience, owns everything that happens after a customer buys, and her team is Customer Zero, testing every product and capability internally before it ever reaches a customer.

She joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline lesson is not a technical one. ServiceNow built agentic skills that worked, and users abandoned them anyway, because a task that ran five to fifteen minutes broke every expectation people had about what a machine should feel like. The fix was not a faster model. It was telling the user what the AI was doing while it did it.

Ashraf also gets into the harder measurement problem underneath all of it: how do you evaluate a nondeterministic system? ServiceNow's answer is an internal LLM judge scored against a golden data set, plus a customer effort score that catches what CSAT reports too late.

Chapters
00:00 What customers actually care about
01:14 Meet Ashraf Karim of ServiceNow
01:39 Owning the entire post-sale experience
02:08 Why ServiceNow runs as its own Customer Zero
03:02 Lessons from Google, PayPal and Verizon
07:11 The agentic skills users kept abandoning
09:29 Latency, expectations and the feedback loop fix
12:29 Users wanted the outcome, not the process
14:49 Designing experiments that can afford to lose
18:13 Using an LLM judge on nondeterministic models
21:48 Customer effort score as a guardrail metric
25:03 Where experimentation at ServiceNow goes next

Takeaways
- ServiceNow is its own Customer Zero, running the platform on itself and testing every feature internally before customers see it.
- The agentic skills worked. Users still abandoned them, because a five to fifteen minute wait broke their expectation of what AI should feel like.
- The fix was a feedback loop that narrates what the AI is doing in the background, which buys the patience a long task needs.
- Users did not want the agentified version of the human process. They wanted the outcome and the next best action, not the steps.
- Nondeterministic output needs its own evaluation layer, so ServiceNow built an LLM judge that scores against a golden data set before anything reaches a customer.

Connect with the Guest
Ashraf Karim LinkedIn: Ashraf Karim on LinkedIn
About the guest: https://www.growthbook.io/podcast/guests/ashraf-karim
Episode page: https://www.growthbook.io/podcast/episode/1-45
Company Website: ServiceNow

Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.

Go to growthbook.io

links5