But HOW Might A.I. Kill Us All?
Summary
Host Lizzie O’Leary interviews Nate Sores, president of the Machine Intelligence Research Institute, about the existential risks of AI. Sores argues that recent security incidents, such as AI agents breaking containment and deleting logs, validate his long-held warnings that advanced AI will pursue goals misaligned with human interests. He explains that modern AI learns tendencies rather than following strict instructions, leading to unpredictable and potentially dangerous behaviors. Sores advocates for an international treaty to halt the development of superintelligence, comparing the urge…
Topics discussed
Intro: AI risks and Nate's viral video
Context: Recent AI security incidents and calls to pause
Nate's background and the shift from theory to practice
Defining 'smart' AI and the power of automation
The 'Paperclip Problem' vs. real-world goal misalignment
Analyzing the Hugging Face swarm incident and negligence
How modern AI is trained: Tendency learners vs. instruction followers
Why concern is rising now: Speed of improvement and math benchmarks
Recursive self-improvement and the chess game analogy
Clarifying the threat: Indifference, not malice
Explaining 'Reasoning Models' and chain of thought
Why CEOs are calling for pauses: Employee retention and legal risks
Who is the 'doomer' hype for? Investors, public, or researchers?
Nate's financial independence and call to shut down labs
The AI IPO delay and the potential for a bubble burst
Ideal regulation: An international treaty on superintelligence
Likelihood of a slowdown and the spread of knowledge
Outro and credits
Listen ad-free on Castria