SN 1093: Tokens in the Stream - Why LLMs are inherently insecure and prompt injection will persist
Summary
Steve Gibson and Leo Laporte discuss a new research paper revealing that LLMs are inherently insecure due to "role confusion," which makes prompt injection attacks nearly impossible to prevent. The episode explores the controversy surrounding AI model distillation, where competitors allegedly train new models using outputs from proprietary frontier models, raising legal and ethical questions. Additionally, they cover Anthropic’s expansion of its powerful Mythos 5 model for cybersecurity defense and Bitwarden’s new Secrets Manager, which addresses the risk of AI agents accidentally exfiltrat…
Topics discussed
Intro: LLMs are inherently insecure
Why understanding low-level mechanics matters
AI agents and credential abuse risks
Picture of the week and listener safety
Sponsor: ThreatLocker AI governance controls
Model distillation and data sourcing debates
Anthropic expands Claude Mythos 5 access
Sponsor: Box enterprise AI knowledge tools
Sponsor: 1Password secrets for AI agents
Sponsor: Hawk Hunt phishing simulations
Guest intro: Lauren Kofelder and PKI
Lauren's thesis: LLMs are untrustworthy
Explaining the 'token soup' architecture
Sponsor: GuardSquare mobile app security
Research: Role confusion and prompt injection
Deep dive: How tags fail to enforce roles
Sponsor: Material workspace security
Experiments: Proving implicit role assumption
History of roles and the single-channel limit
Outro: GRC.com and show details
Listen ad-free on Castria