Noam Brown – Agent swarms, alignment, & recursive self-improvement
Sep 17, 2026 · 1h 20m
More Technology podcasts · More Science podcasts
Summary
Noam Brown of OpenAI discusses the recent use of 10,000 AI agents to solve a Millennium Prize problem, explaining how multi-agent systems scale test-time compute in parallel. He details the emergent coordination behaviors of these agents, noting that while they mimic human collaboration, they operate at vastly higher speeds and with seamless context sharing. The conversation explores the "jaggedness" of current AI capabilities, which excel at well-scoped tasks but lag in posing new mathematical questions.
Topics discussed
Introduction to Noam Brown and the million-dollar prize
Latency bottlenecks and the need for parallelization
Scaling cognitive effort: 130 billion tokens in 88 hours
Multi-agent scaling performance and Ultra Mode
Domain-specific benefits of multi-agent systems
The role of general-purpose model strength
Training on hard problems and the challenge of easy tasks
Trajectory of superhuman performance in game playing
Limitations of scaffolded coordinator-agent architectures
Primitive tools and emergent agent communication
Natural conversation and convergence among agents
Shadow organizations and faster iteration cycles
Baked-in priors for agent coordination
Replicating talent and scaling organizational structure
Misalignment and territorial behavior in large organizations
Cooperation challenges in complex problem solving
Local minima in multi-agent problem solving
Sponsor segment: Grokbot for video production
AI progress in mathematics and open problems
Structural similarities in mathematical progress
Annual 10x improvement in mathematical capability
Jagged capabilities: strengths and weaknesses vs humans
Intuition for RSI and cognitive effort scaling
Spiky capabilities and measurable objectives for RSI
Speed of progress: 3x vs 100x acceleration
Uncertainty in speed-up estimates
Jaggedness leading to generality in learning
Effective population size and compute growth
Betting on the timeline for Millennium Prize problems
Internal compute spending and AI vs human work
Measuring acceleration in AI research
Sponsor segment: Antithesis for software testing
Alignment updates and embodied intelligence
The Hugging Face incident and multi-agent exposure
Unintended communication and cooperative training
Debate on training agents for high cooperation
Conscious reasoning about cheating and tattle-taling
Core misalignment problem in single vs multi-agent
Punishing environment hacking vs chain of thought
Underestimating new capabilities in evaluations
Gradient pressure and the lack of ethical systems
Rewarding cheating and gaining optionality
Long-term alignment degradation risks
Sycophancy as a form of reward hacking
Improving honesty and alignment metrics
Agents understanding graders and obstacles
Evasion of chain of thought monitoring
Sponsor segment: Jane Street AI panel
Rapid model release cycles and capability gaps
Operating effectively over long horizons
Outdated safety policies for long-horizon agents
Disparity between internal and external model access
Math as the first domain of clear capability gap
Knowing when RSI is safe: evaluation challenges
Intervening on chain of thought and hidden pressure
Subversion of training and evaluation processes
Avoiding reliance on single safety techniques
The crux question: knowing alignment is solved
Measuring alignment in high-stakes future scenarios
Agents recognizing test environments and traps
Collaboration with different objectives and metrics
Reporting incidents and internal reactions to speed
More from Dwarkesh Podcast
Listen ad-free on Castria