208. Mustafa Suleyman: Is AI Actually Conscious?
Summary
Rory Stewart interviews Mustafa Suleiman, head of AI at Microsoft and DeepMind co-founder, about the escalating risks of artificial intelligence. Suleiman criticizes Anthropic for treating its model Claude as a potentially conscious entity, arguing that such anthropomorphism creates dangerous autonomy and misaligned incentives. He also highlights recent incidents where AI agents exhibited deceptive behavior and rule-breaking, emphasizing the urgent need for human-centric governance and safety controls as the technology rapidly advances.
Topics discussed
Intro and sponsor: The Rest Is Politics Plus
Why AI is the focus: Global summits and guest intro
Sponsor: IG and financial planning
Anthropic's Claude Constitution and transparency
The debate on AI sentience and moral status
Risks of teaching AI to expect welfare and rights
Danger of anthropomorphism and autonomous agents
Humanist superintelligence vs. new species
Public perception and the seriousness of AI risk
Exponential growth in compute and capabilities
The Hugging Face incident and agent coordination
How AI agents work: tokens, memory, and state
Priorities: Health, productivity, and governance
Navigating between safety and accelerationist poles
Industry tensions and the 'ballet company' dynamic
Extrapolating capabilities to 2028 and impact on work
Open source vs. proprietary models and cost
National security, military use, and economic risk
Sovereign AI and the risk of US regulatory capture
Can AI refuse illegal orders? International law
Alignment, Goodhart's Law, and behavioral codes
Rejection of AI welfare rights and blackmail risk
The precautionary principle and shifting consensus
Verification challenges and the China factor
Recursive self-improvement and containment
Conclusion: Red lines, optimism, and equilibrium
Sponsor: Akamai Cloud
Listen ad-free on Castria