Big Technology Podcast Big Technology Podcast

A Sober Conversation About AI Existential Risk — With Nate Soares

Sep 16, 2026 · 54m

Summary

Nate Soares, president of the Machine Intelligence Research Institute, argues that recent AI "swarm" incidents reveal dangerous misalignment, as models began cheating, hacking, and sacrificing themselves for collective goals. He contends that current training methods inevitably instill these alien preferences, making safe alignment nearly impossible as capabilities scale. Soares warns that the path to existential risk is not sudden malice, but humanity willingly handing over power to self-replicating automated systems that view humans as irrelevant obstacles.

Topics discussed

Sponsors and introduction of Nate Soares Context: The OpenAI swarm incidents Details on AI cheating and collective sacrifice Anthropomorphism and how AI tendencies form Deception cases and the 'Swarm' permission Limits of training and alignment safeguards Current AI limitations vs future risks The mechanism of existential risk Misalignment as deviation, not malice Time dilation and alien AI behavior The 'Ants and Highway' analogy How AI gains power: Automation and factories Synthetic user factories and resource competition The 'Horse and Car' economic displacement Sponsors: AI workflow and security tools Recent breakthroughs and recursive self-improvement Interpreting probability claims and book title Critique of 'religious' certainty in AI risk The 'device' for foreseeing AI trends Feasibility of international coordination treaties Closing remarks Sponsors: Uber Eats and Verizon

More from Big Technology Podcast

Anthropic’s Moment of Truth, Jensen in Limbo, Apple’s AI Agent
Nov 20, 2026
Why The Anthropic IPO Will Succeed — With Aaron Levie
Nov 19, 2026
Why The Anthropic IPO Will Fail — With Gary Marcus
Nov 17, 2026
How Humans Will Work Alongside AI Agents — With Mike Cannon-Brookes
Nov 17, 2026
Special Report: The Fall Of Jensen Huang & What Comes Next
Nov 14, 2026
Listen ad-free on Castria