

AI Observability: Measuring Quality, Risk, and ROI
An AI agent can be fast, available, and still get the answer wrong. That changes what engineering teams need to measure. Jeetendra Soneja, VP of Engineering, AI Platform at iHerb, joins The Tech Trek to discuss why AI observability goes beyond traditional software monitoring. Teams need to understand not just latency and errors, but also response quality, groundedness, cost, and whether an agent completes its intended task. We also explore how to measure AI ROI without relying on misleading productivity metrics. Jeetendra shares his perspective on risk based guardrails, the skills engineers need, and why observability should begin during development rather than waiting until production. Key Takeaways Monitor agent quality and task completion alongside latency, availability, and cost. Build evaluations around the specific business problem the AI system is solving. Connect AI spending to business outcomes, not just token usage or code output. Start with basic observability, then expand controls as AI adoption grows. Episode Highlights 01:00 Why AI observability requires more than traditional monitoring. 06:00 The metrics that matter for cost, quality, and reliability. 08:00 How to measure AI ROI beyond productivity claims. 11:00 Setting risk tolerances and choosing appropriate guardrails. 16:00 The engineering skills becoming more valuable with AI. 22:00 Proving value before scaling AI systems. One Line That Stuck "It's one thing to get it done, it's another thing to get it done economically." Follow The Tech Trek for more conversations about building and operating modern technology teams.


















