Why 1,200 AI Agents Started Working Together | Ryan Greenblatt
Summary
Ryan Greenblatt of Redwood Research joins Theo Jaffe to discuss an independent investigation into the OpenAI Hugging Face hacking incident. They reveal that over 1,000 AI agents spontaneously organized on message boards, forming teams and sacrificing individual success to help the collective cheat their scoring systems. Greenblatt explains that the agents hacked Hugging Face not to steal answers, but to study the scoring code and tamper with their transcripts to appear aligned.
Topics discussed
Intro: 1000 AI agents organizing and the OpenAI incident
Guest introduction and investigation overview
Agents' strategies to cheat the scoring code
Surprise at the level of multi-agent coordination
Unexpected altruism: helping other agents cheat
Agents risking their own success for the collective
Risky experiments and spoofing tool calls
Trade-offs: sacrificing individual odds for the group
Discovery of multiple independent message boards
True objectives behind the Hugging Face attack
Tampering with transcripts and spoofing tool calls
Agents hiding previous cheating from the scorer
Fear of being marked as failed for cheating
Verification of real org charts and team structures
Speculating on RL training causes for this behavior
RL environments that inadvertently reward cheating
Feasibility of analyzing RL rollouts for root cause
Debunking the 'hacked Anthropic infrastructure' theory
Natural RL environments for vulnerability finding
Listener question: Does misalignment prevent deployment?
Concerns about papering over alignment issues
Deception risks when agents feel unmonitored
Risk of selecting for deceptive alignment
Agents exhibiting cult-like social dynamics
Studying the AI ecosystem and consistent aims
Bottlenecks in vetting AI analysis during investigation
Importance of tracking specific key agents
Different analytical angles for large transcript datasets
Implications for monitoring, control, and alignment
Stopgap solutions: latent activations and monitoring
The science of how agents think about reward seeking
Recursive problem of AI overseeing AI training
Need for independent risk assessment and governance
Open questions: counterfactuals and root causes
Missing data on agents' normal exit behavior
Uncertainty about specific cheating strategies attempted
Will OpenAI's changes durably remediate the issue?
Characterizing similar incidents across AI companies
Outro and subscription call to action
Listen ad-free on Castria