
Measuring AI: What Policymakers Can Learn from Benchmarks
AI benchmarks are becoming increasingly central to policy debates. But much of the value of these benchmarks depends on how they are built: what they measure, who develops and runs them, and how quickly they evolve alongside the models they evaluate. In this episode, Matt Perault speaks with Rayan Krishnan, cofounder and CEO of Vals, and Glenn Parham, head of public sector, about their work building AI evaluation tools and partnerships for industry and government. They discuss why static tests quickly become obsolete, why credible evaluations require deep domain expertise and independent testing, and how a growing market of evaluators is moving the field beyond “evaluations by vibes” toward more rigorous, data-driven evidence for policymakers. Topics covered: 00:00: Intro 02:00: What is Vals? 05:00: Keeping benchmarks current 08:00: New strides in AI evaluation 11:00: Measuring AI’s cyber capabilities 14:00: Why government needs independent evaluation 21:00: The growing market for AI benchmarks 26:00: What benchmarks can—and cannot—measure 29:00: Building trust in AI benchmarks 35:00: Why AI performance varies across domains and tasks 37:00: Bringing AI evaluation to government 41:00: From “evaluations by vibes” to better evidence Resources: Subscribe to the a16z AI Policy Brief: https://a16zpolicy.substack.com/ Follow Matt Perault on X: https://x.com/MattPerault Follow Rayan Krishnan on X: https://x.com/RayanKrishnan Follow Glenn Parham on X: https://x.com/glenn__parham Follow Vals.ai on X: https://x.com/ValsAI The content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. Please note that a16z and its affiliates may maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com/disclosures. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit a16zpolicy.substack.com


















