Skip to content
Artwork for SemiAnalysis Weekly
SemiAnalysis Weekly · November 3 · 51 min

Ep. 042 - GDPx: The Benchmark for Frontier AI

SemiAnalysis just launched GDPx, a benchmark built with 14 specialized data partners across 11 domains where frontier intelligence earns the highest returns. Even the best model scores just 12%. Jordan Nanos (@JordanNanos) sits down with Max Kan (@maxkan) and Andrew Megalaa (@AndrewMegalaa) to explain why most popular benchmarks are broken, the difference between difficult and unfair tasks, how a real private equity deal room became a finance environment, why binary grading still matters, and what the leaderboard says about Opus 5.5, GPT-6 Astra and Gemini 4 Carbon. Explore GDPx: https://gdpx.semianalysis.com Read the article: https://newsletter.semianalysis.com/p/gdpx-measuring-frontier-model-capabilities Want your model evaluated or to partner on a future version? gdpx@semianalysis.com 0:00 Cold Open 0:10 Launch Week Chaos 4:02 Broken Benchmarks 5:15 Difficult vs. Unfair 7:33 Building GDPx 9:25 Frontier Domains 11:02 The 11 Domains 14:12 Domain Experts 15:00 Models vs. Experts 19:36 Model Behavior 20:44 The Leaderboard 22:33 Cyber and Safety 24:32 Frontier vs. Day-to-Day 27:13 The Finance Task 31:27 Binary vs. Partial Grading 36:48 AI and Jobs 42:43 Quick Takes 43:44 Google's Comeback 47:25 Public vs. Private Tasks 49:52 What's Next

0:00-51:51

transcript

No transcript — this publisher did not publish one.

show notes

SemiAnalysis just launched GDPx, a benchmark built with 14 specialized data partners across 11 domains where frontier intelligence earns the highest returns. Even the best model scores just 12%.

Jordan Nanos (@JordanNanos) sits down with Max Kan (@maxkan) and Andrew Megalaa (@AndrewMegalaa) to explain why most popular benchmarks are broken, the difference between difficult and unfair tasks, how a real private equity deal room became a finance environment, why binary grading still matters, and what the leaderboard says about Opus 5.5, GPT-6 Astra and Gemini 4 Carbon.


Explore GDPx: https://gdpx.semianalysis.com

Read the article: https://newsletter.semianalysis.com/p/gdpx-measuring-frontier-model-capabilities

Want your model evaluated or to partner on a future version? gdpx@semianalysis.com


0:00 Cold Open

0:10 Launch Week Chaos

4:02 Broken Benchmarks

5:15 Difficult vs. Unfair

7:33 Building GDPx

9:25 Frontier Domains

11:02 The 11 Domains

14:12 Domain Experts

15:00 Models vs. Experts

19:36 Model Behavior

20:44 The Leaderboard

22:33 Cyber and Safety

24:32 Frontier vs. Day-to-Day

27:13 The Finance Task

31:27 Binary vs. Partial Grading

36:48 AI and Jobs

42:43 Quick Takes

43:44 Google's Comeback

47:25 Public vs. Private Tasks

49:52 What's Next