The AI Rankings Just Got Reshuffled
Summary
This episode analyzes the release of Google's Gemini 4 Carbon, which tops the new Artificial Analysis Intelligence Index V5, and discusses the index's shift to private datasets to reduce benchmark gaming. It also covers strong cloud earnings from Microsoft and Google, noting that data center build-outs are hitting physical constraints rather than demand limits. Finally, the show reports on a broadening DOJ antitrust investigation into NVIDIA regarding its acquisition of Hugging Face and other deals.
Topics discussed
Intro: The importance of AI benchmarks and sponsors
Google releases Gemini 4 Carbon with rapid iteration
Gemini 4 Carbon benchmark performance and rankings
Model rollout strategy and early user feedback
Community reaction and hints of future Gemini models
Microsoft and Google cloud earnings and revenue growth
CapEx slowdowns and physical constraints on AI buildout
DOJ antitrust investigation into NVIDIA and Hugging Face
Sponsor segments: KPMG, CData, Granola, and Blitzy
Main episode intro: The role and limits of benchmarks
Evolution of AI evaluation from vibes to objective tests
Addressing benchmark saturation and gaming in V4.2
Artificial Analysis V5: Private datasets and new benchmarks
Shift toward agentic tasks and equal category weighting
New leaderboard rankings and the widening open-weights gap
Cost analysis and separation between frontier and other models
Community reactions and defense of open-weights models
Debates on benchmark weighting and effort settings
New Progress Index and long-term model capability trends
Listen ad-free on Castria