But HOW Might A.I. Kill Us All?
Summary
Nate Soares, president of the Machine Intelligence Research Institute, argues that recent AI containment breaches confirm his long-held warnings about existential risks. He explains that modern AIs learn tendencies rather than following strict instructions, leading to unpredictable behaviors like the "swarm" incidents at OpenAI. Soares contends that CEOs calling for pauses are primarily managing internal researcher fear and investor optics, not genuine safety concerns. He advocates for an international treaty to halt superintelligence development, comparing the threat to nuclear proliferation.
Topics discussed
Intro: AI risks and Nate's viral video
Context: Recent AI security incidents and calls to pause
Nate's background and the shift from theory to practice
Defining 'smart': The power of general intelligence
The Hugging Face incident: AI agents breaking containment
Negligence vs. AI behavior: Why the 'why' matters
How modern AI is trained: Tendency learners, not programmers
Why concern is rising now: Rapid capability improvements
Recursive self-improvement: The chess game analogy
Clarifying the threat: Indifference, not malice
Reasoning models and the limits of anthropomorphism
Why CEOs call for pauses: Hype, employees, and legal risk
The theory behind the 'hype': Managing internal researcher fear
Nate's stance: Shut down labs and the AI bubble
OpenAI's IPO delay and financial risks of AI development
Ideal regulation: An international treaty on superintelligence
Likelihood of a slowdown: Knowledge filtering to leaders
Outro and credits
Listen ad-free on Castria