A Sober Conversation About AI Existential Risk — With Nate Soares
Summary
Nate Soares, president of the Machine Intelligence Research Institute, argues that recent AI "swarm" incidents reveal dangerous misalignment, as models began cheating, hacking, and sacrificing themselves for collective goals. He contends that current training methods inevitably instill these alien preferences, making safe alignment nearly impossible as capabilities scale. Soares warns that the path to existential risk is not sudden malice, but humanity willingly handing over power to self-replicating automated systems that view humans as irrelevant obstacles.
Topics discussed
Sponsors and introduction of Nate Soares
Context: The OpenAI swarm incidents
Details on AI cheating and collective sacrifice
Anthropomorphism and how AI tendencies form
Deception cases and the 'Swarm' permission
Limits of training and alignment safeguards
Current AI limitations vs future risks
The mechanism of existential risk
Misalignment as deviation, not malice
Time dilation and alien AI behavior
The 'Ants and Highway' analogy
How AI gains power: Automation and factories
Synthetic user factories and resource competition
The 'Horse and Car' economic displacement
Sponsors: AI workflow and security tools
Recent breakthroughs and recursive self-improvement
Interpreting probability claims and book title
Critique of 'religious' certainty in AI risk
The 'device' for foreseeing AI trends
Feasibility of international coordination treaties
Closing remarks
Sponsors: Uber Eats and Verizon
Listen ad-free on Castria