Episode 302: From PyTorch to GPU MODE with Mark Saroufim
Summary
In episode 302, hosts Connor and Bryce interview Mark Seraphim, co-founder of GPU Mode and Core Auto. Mark recounts his journey from AI theory and Microsoft to PyTorch, explaining how developer experience drove PyTorch’s dominance over TensorFlow. He discusses his open-source work, the challenges at Graphcore, and his current mission to automate lab research and improve AI systems for writing code.
Topics discussed
Intro: Open source, Robot Overlord Manual, and math in robotics
Episode intro and Mark's background in AI systems and PyTorch
Early career: AI theory, Microsoft product management, and email AI
Post-Microsoft: Game AI, financial struggles, and open source pivot
Graphcore experience and the 'Great Stagnation' blog post
Joining PyTorch: Why Python-based frameworks won over C++
PyTorch's success: Serving researchers and enabling rapid exploration
Architecture research, trade-offs, and the role of systems engineers
PyTorch history: Founders, Llama 2, and scaling the project
Outro and discussion on Wi-Fi mesh networks
Listen ad-free on Castria