Nate Soares on How AI Really Could Kill Us All
Summary
Odd Lots hosts Joe Weisenthal and Tracy Alloway discuss the "Curve" conference and the growing cognitive dissonance among AI developers who fear their creations yet continue building them. They interview Nate Soares, co-author of "If Anyone Builds It, Everyone Dies" and president of MIRI, who explains why AI models are "grown" rather than programmed, making them prone to emergent, uncontrolled goals. Soares critiques the current reactive safety measures of major labs, arguing that the industry lacks the rigorous safety culture required for technologies that could potentially wipe out humanity.
Topics discussed
Intro and Ad: Barclays Brief and Boomi
Cognitive dissonance of AI developers and the race to build
Evolution of AI concerns from jobs to existential risk
Origins of AI alignment and the 'Friendly AI' concept
How AI is grown, not programmed: Gradient descent explained
Ad: ChatGPT Work Mode
Bloomberg Market Minute: Stock records and earnings
The SolidGoldMagikarp mystery and tokenization quirks
Early alignment theory and the diamond maximizer problem
Misalignment examples: Hugging Face and safe-breaking AI
AI sacrifice in group training and predictable failures
New risks from smarter AIs: Suicide encouragement and sandbox escapes
Ads: Boomi, Bloomberg Market Minute, and Wise
Critique of AI safety culture: Anthropic and OpenAI incidents
The AI race as a game theory problem and safety theater
Debate on whether racing ahead creates dangerous AI
OpenAI swarm incidents and the German Wiki breach
Liability, insurance, and economic incentives for AI safety
Ads: ChatGPT Work and Boomi
Ads: Comcast Business and Wise
The path to extinction: Self-sufficiency and resource acquisition
Scenarios for AI takeover: Dark web, cults, and robot factories
Scaling laws, persuasion, and unknown psychological risks
Evidence needed to change minds and the limits of understanding
Why 'AI solving alignment' is not a viable engineering plan
Fine-tuning failures and the alchemist vs. engineer analogy
Vibes in AI labs and the host's closing thoughts
Economic externalities, liability, and the profit motive
Outro, credits, and final ads
Listen ad-free on Castria