Skip to content
Artwork for The Didi & Lital Show
The Didi & Lital Show · June 5 · 30 min

AI Tokens Problem - Solvable!

Didi and Lital open with casual sports talk (NBA, NHL, World Cup traffic) before returning to a cliffhanger about soaring AI token costs. They cite Marc Benioff saying Salesforce will pay Anthropic $300M in tokens for coding, and discuss Microsoft pausing Anthropic use after exhausting its AI budget. They argue costs are driven by energy/physics and by developers running many parallel agents, making outcome measurement crucial so token spend aligns with valuable work rather than performative usage. They describe organizational resistance ("Luddites") and poor prompting/problem decomposition ("one-liners"), stressing the need for core computer science thinking. They predict new solutions: on-prem/local models, company-built data centers, and software “broker” layers to route tasks to the right models and enforce policy. They describe building such a layer for security SOC triage to reduce alerts to ~2% needing humans, and discuss tradeoffs between general models for innovation and cheaper specialized models for repetitive tasks. They close noting rapid startup exits in this space. Topics 00:24 Sports Rants 02:43 World Cup Traffic 03:43 AI Token Shock 07:59 Measuring AI Productivity 08:45 Luddites And One Liners 13:17 Family Travel Chaos 15:02 Token Optimization Layers 19:14 Generalists Vs Specialists 23:45 Efficiency War Stories 26:57 Future AI Cost Tools 28:55 Startup Exits Teaser 29:43 Wrap Up And Outro

0:00-30:23

transcript

No transcript — this publisher did not publish one.

show notes

Didi and Lital open with casual sports talk (NBA, NHL, World Cup traffic) before returning to a cliffhanger about soaring AI token costs. They cite Marc Benioff saying Salesforce will pay Anthropic $300M in tokens for coding, and discuss Microsoft pausing Anthropic use after exhausting its AI budget. They argue costs are driven by energy/physics and by developers running many parallel agents, making outcome measurement crucial so token spend aligns with valuable work rather than performative usage. They describe organizational resistance ("Luddites") and poor prompting/problem decomposition ("one-liners"), stressing the need for core computer science thinking. They predict new solutions: on-prem/local models, company-built data centers, and software “broker” layers to route tasks to the right models and enforce policy. They describe building such a layer for security SOC triage to reduce alerts to ~2% needing humans, and discuss tradeoffs between general models for innovation and cheaper specialized models for repetitive tasks. They close noting rapid startup exits in this space.

 

Topics

00:24 Sports Rants

02:43 World Cup Traffic

03:43 AI Token Shock

07:59 Measuring AI Productivity

08:45 Luddites And One Liners

13:17 Family Travel Chaos

15:02 Token Optimization Layers

19:14 Generalists Vs Specialists

23:45 Efficiency War Stories

26:57 Future AI Cost Tools

28:55 Startup Exits Teaser

29:43 Wrap Up And Outro