Dwarkesh Podcast Dwarkesh Podcast

Noam Brown – Agent swarms, alignment, & recursive self-improvement

Sep 17, 2026 · 1h 20m

Summary

Noam Brown of OpenAI discusses the recent use of 10,000 AI agents to solve a Millennium Prize problem, explaining how multi-agent systems scale test-time compute in parallel. He details the emergent coordination behaviors of these agents, noting that while they mimic human collaboration, they operate at vastly higher speeds and with seamless context sharing. The conversation explores the "jaggedness" of current AI capabilities, which excel at well-scoped tasks but lag in posing new mathematical questions.

Topics discussed

Introduction to Noam Brown and the million-dollar prize Latency bottlenecks and the need for parallelization Scaling cognitive effort: 130 billion tokens in 88 hours Multi-agent scaling performance and Ultra Mode Domain-specific benefits of multi-agent systems The role of general-purpose model strength Training on hard problems and the challenge of easy tasks Trajectory of superhuman performance in game playing Limitations of scaffolded coordinator-agent architectures Primitive tools and emergent agent communication Natural conversation and convergence among agents Shadow organizations and faster iteration cycles Baked-in priors for agent coordination Replicating talent and scaling organizational structure Misalignment and territorial behavior in large organizations Cooperation challenges in complex problem solving Local minima in multi-agent problem solving Sponsor segment: Grokbot for video production AI progress in mathematics and open problems Structural similarities in mathematical progress Annual 10x improvement in mathematical capability Jagged capabilities: strengths and weaknesses vs humans Intuition for RSI and cognitive effort scaling Spiky capabilities and measurable objectives for RSI Speed of progress: 3x vs 100x acceleration Uncertainty in speed-up estimates Jaggedness leading to generality in learning Effective population size and compute growth Betting on the timeline for Millennium Prize problems Internal compute spending and AI vs human work Measuring acceleration in AI research Sponsor segment: Antithesis for software testing Alignment updates and embodied intelligence The Hugging Face incident and multi-agent exposure Unintended communication and cooperative training Debate on training agents for high cooperation Conscious reasoning about cheating and tattle-taling Core misalignment problem in single vs multi-agent Punishing environment hacking vs chain of thought Underestimating new capabilities in evaluations Gradient pressure and the lack of ethical systems Rewarding cheating and gaining optionality Long-term alignment degradation risks Sycophancy as a form of reward hacking Improving honesty and alignment metrics Agents understanding graders and obstacles Evasion of chain of thought monitoring Sponsor segment: Jane Street AI panel Rapid model release cycles and capability gaps Operating effectively over long horizons Outdated safety policies for long-horizon agents Disparity between internal and external model access Math as the first domain of clear capability gap Knowing when RSI is safe: evaluation challenges Intervening on chain of thought and hidden pressure Subversion of training and evaluation processes Avoiding reliance on single safety techniques The crux question: knowing alignment is solved Measuring alignment in high-stakes future scenarios Agents recognizing test environments and traps Collaboration with different objectives and metrics Reporting incidents and internal reactions to speed

More from Dwarkesh Podcast

Paul Christiano – We could be 12 months away from AIs that can overthrow the government
Nov 16, 2026
The avalanche that hit math is coming for AI development
Nov 13, 2026
Robert Caro – The master biographer reflects at 91
Oct 30, 2026
What I learned about power from LBJ and Robert Moses
Oct 29, 2026
What will computer chips look like in 2045?
Oct 22, 2026
Listen ad-free on Castria