Dwarkesh Podcast Dwarkesh Podcast

Reiner Pope – The math behind how LLMs are trained and served

Apr 29, 2026

Summary

Dwarkesh Patel interviews Rainer Pope, CEO of chip startup Maddox, in a blackboard lecture format to explain the mechanics of AI inference. They use roofline analysis to derive how batch size, memory bandwidth, and compute dictate latency and cost, explaining why "fast mode" APIs are more expensive. The discussion covers the trade-offs of sparse mixture-of-experts models, the physical constraints of GPU rack interconnects, and why hardware limitations have slowed the deployment of larger models.

Topics discussed

Introduction: AI Architecture, Pricing, and Latency Transformer Inference Mechanics: Batch Size and KV Cache Cost Analysis: Memory Bandwidth vs. Compute Limits Optimal Batch Size and Hardware Constraints Mixture of Experts (MoE) and Sparsity Benefits GPU Rack Architecture: NVLink and Scale-Up Limits Training vs. Inference: Pipeline Parallelism Micro-batching and Memory Optimization Strategies Pipelining Trade-offs: Latency and Memory Footprint Scaling Laws: Pre-training vs. Inference Costs Context Length Impact on Compute and Memory Time API Pricing: Pre-fill vs. Decode Costs KV Cache Storage: HBM, DDR, and Disk Tiers Cryptography and Reversible Neural Networks

More from Dwarkesh Podcast

Paul Christiano – We could be 12 months away from AIs that can overthrow the government
Nov 16, 2026
The avalanche that hit math is coming for AI development
Nov 13, 2026
Robert Caro – The master biographer reflects at 91
Oct 30, 2026
What I learned about power from LBJ and Robert Moses
Oct 29, 2026
What will computer chips look like in 2045?
Oct 22, 2026
Listen ad-free on Castria