Dwarkesh Podcast Dwarkesh Podcast

AI researchers debate how close we are to recursive self-improvement

Sep 11, 2026 · 1h 37m

Summary

Dwarkesh Patel interviews Baron Millich, John Schulman, and Charlie O'Neill to debate whether AI will achieve rapid recursive self-improvement by 2036. They argue that while scaling current RL paradigms may hit asymptotic limits, the key bottleneck remains the "sim-to-real" gap and the difficulty of generalizing from verifiable benchmarks to open-ended scientific discovery. The guests discuss how Chinese labs leverage router data for distillation, challenging the idea that frontier labs hold a permanent advantage.

Topics discussed

Introduction of AI researcher guests The AI capability paradox and diminishing returns Recursive self-improvement and discontinuities Simulations, chess Elo scores, and regulation Next token prediction and inner loop optimization Generalization from verifiable to open tasks The role of human analysis and theory building Defining objectives and human-in-the-loop Sponsor segment: Antithesis testing tools Model distillation and prompt distribution Benchmarking vs realistic deployment data Automating AI R&D and environment design Scaling RL environments and meta-skills Learning from deployment vs pre-training Sponsor segment: Jane Street chip design Hive-mind learning and real-world interaction Cumulative vs non-stationary task distributions AI taste, context windows, and specialization Catastrophic forgetting and RL efficiency Sponsor segment: Grogbot podcast editing Environment creation bottlenecks and RL signal Data vs recipe improvements in pre-training Architecture, sparsity, and scaling laws Parameter scaling and Chinchilla limits RL signal-to-noise ratio and mid-training Discrete phase transitions in model learning RL impact on creativity and diversity Rapid-fire predictions on AI progress speed

More from Dwarkesh Podcast

Paul Christiano – We could be 12 months away from AIs that can overthrow the government
Nov 16, 2026
The avalanche that hit math is coming for AI development
Nov 13, 2026
Robert Caro – The master biographer reflects at 91
Oct 30, 2026
What I learned about power from LBJ and Robert Moses
Oct 29, 2026
What will computer chips look like in 2045?
Oct 22, 2026
Listen ad-free on Castria