transcript
show notes
Vincent Chen, co-founder at Snorkel AI, announces an expansion of Snorkel's Open Benchmarks Grants program to $30 million to keep evaluation infrastructure in line with accelerating frontier model capabilities. He breaks down the program's three core pillars—funding new benchmarks, launching the Open Benchmarks Red Team, and establishing the Snorkel Research Fellowship—while addressing continuous benchmark maintenance and AI alignment. Check out our sponsor Lovable: https://lovable.dev. Turn ideas into software people love.
MTS is a live news and interview show covering technology, business, politics, and culture as it happens.
Follow MTS:
X | https://twitter.com/MTSlive
YouTube | https://www.youtube.com/@mtsituation
Spotify | https://open.spotify.com/show/4HUNNmV1pjV7CW6Lphm1rM
Apple | https://podcasts.apple.com/us/podcast/mts/id1891088763
Substack | https://mtslive.substack.com/
Interested in sponsoring the show? sponsors@mts.now
--
Brought to you by:
Mongo DB - Anyone can prototype an agent. Learn how to get them into production on September 30th in New York - register at https://www.mongodb.com/events/mongodb-local/nyc
Lovable - Turn ideas into software people love with Lovable. Start today at https://lovable.dev/
Larridin - Finally prove what your AI investment is worth. Head to https://larridin.com/ today to start measuring your AI’s bottom line.
Neon (by Databricks) - Get up to $100K in credits for your startup at https://neon.com/mts
ElevenLabs - AI that communicates at human level across every channel and modality. https://elevenlabs.io/mts
Arena - Measuring AI performance in the real world. https://arena.ai/
Kong - The AI connectivity platform. Connect APIs, LLMs, agents, and systems with serious security and governance. https://konghq.com
Today’s stream is sponsored by VCX, the public ticker for private tech. Head over to http://getvcx.com to learn more
Blitzy - Autonomous software development for enterprise codebases. Ship 5x faster. https://blitzy.com
AdQuick - Make your brand a billboard. Out-of-home advertising as easy to scale as digital. https://adquick.com
Chargebee - Billing and monetization infrastructure for the AI era. https://www.chargebee.com/
--
Timestamps:
(00:00) Introduction
(02:09) Snorkel Research Fellowship
(03:54) Open Benchmarks Red Team
(05:02) Auditing Broken Benchmarks
(07:32) Benchmarks in AI Alignment
(08:17) Science of Empirical Benchmarking
(09:48) Closing the Capability Gap
--
Note: This podcast is not investment, legal, or tax advice, and is intended for informational and entertainment purposes only. Hosts and guests may hold positions in the companies and securities discussed; do your own research before acting on anything you hear.
