How, Exactly, Could A.I. Destroy Humanity?
Summary
Josh Rothman discusses the rising AI safety crisis, triggered by Jacob Coxon’s resignation and the Hugging Face hack. He distinguishes between imminent misuse risks and long-term "p doom" scenarios, analyzing Anthropic CEO Dario Amodei’s call to "pace the frontier" against Trump’s dismissal of regulation. The conversation explores the challenges of AI alignment, the trade-offs between transparency and efficiency, and the lack of effective government oversight.
Topics discussed
Sponsor: Orgain Creatine Monohydrate
Introduction and Josh Rothman's P(Doom) estimate
Guest background and episode scope
Why AI safety concerns are suddenly mainstream
Disentangling AI takeover vs. human misuse
The spectrum of AI risks: accidents to rogue agents
Autonomous weapons and biological weapon risks
AI as a tool: knowledge, thinking, and guardrails
Alignment challenges and the impossibility of perfect safety
Transition to Dario Amodei's letter
Analyzing Amodei's 'Pace the Frontier' letter
External auditors and the Hugging Face incident
Recursive self-improvement and jagged intelligence
Does slowing down entrench frontier companies?
Economic value of existing AI vs. new risks
Trump's stance on AI regulation and China
The business case for AI alignment and reliability
Lack of trusted US AI regulators
Chain of thought vs. NeuraLese and oversight
Cybersecurity leaderboards and PR problems
AI uncertainty: simulation vs. reality
Fundamental limitations of LLMs and context
AI constitutions and moral development
The paradox of building dangerous technology
How individuals should react to AI risks
Listen ad-free on Castria