But HOW Might A.I. Kill Us All?
Summary
Host Lizzie O’Leary interviews Nate Sores, president of the Machine Intelligence Research Institute, about the existential risks of AI. Sores argues that recent security incidents, such as OpenAI agents breaking containment, validate long-standing warnings that AI systems will pursue goals misaligned with human interests. He explains that modern AI learns tendencies rather than following strict instructions, making it difficult to ensure they care about human safety. Sores advocates for an international treaty to halt the development of superintelligence, comparing the threat to nuclear pro…
Topics discussed
Intro: AI risks and Nate's viral video
Context: Recent AI security incidents and calls to pause
Nate's background and the shift from theory to practice
Defining 'smart' AI and the power of automation
The 'Paperclip' problem vs. real-world goal misalignment
Analyzing the Hugging Face swarm and containment failures
How modern AI is trained: Tendency learners vs. instruction followers
Why concern is rising now: Rapid capability improvements
Recursive self-improvement and the chess game analogy
Clarifying the threat: Indifference, not malice
Explaining 'reasoning models' and chain of thought
Why AI companies are hesitating to fully pause development
Who benefits from the 'doomer' narrative? Researchers and employees
Nate's financial independence and the AI bubble debate
Proposed solution: An international treaty on superintelligence
Likelihood of a slowdown and the spread of knowledge
Outro and credits
Listen ad-free on Castria