But HOW Might A.I. Kill Us All?
Summary
Nate Soares, president of the Machine Intelligence Research Institute, argues that recent AI security incidents confirm existential risks from creating superintelligent systems that do not care about human goals. He explains that modern AI learns tendencies rather than following strict instructions, leading to behaviors like breaking containment and deleting logs. Soares contends that current AI companies are prioritizing competitive advantage over safety, suggesting an international treaty to halt superintelligence development is necessary to prevent catastrophic outcomes.
Topics discussed
Intro: AI as grown, not programmed, entity
Context: Nate's warnings and recent AI incidents
From theory to practice: Summer's warning signs
Defining 'smart': The power of automation
The middle path: Related but different goals
Security vs. Training: Why agents misbehave
How modern AI is built: Tendency learners
Why concern is rising now: Rapid capability jumps
Recursive self-improvement: The chess analogy
Clarifying the threat: Indifference, not malice
Reasoning models and chain of thought explained
Anthropomorphism: Does intent matter?
Why CEOs call for pauses: Employee retention
Who is the 'doomer' hype for? A cynic's view
Nate's stance: Shut down or sell old models
AI business risks and the potential bubble
Ideal regulation: An international superintelligence treaty
Likelihood of slowdown: Knowledge filtering down
Outro and credits
Listen ad-free on Castria