A System for Testing New AI Models
Summary
This episode guides listeners through building a personal AI benchmark to evaluate new models against their specific workflows, rather than relying on generic published scores. The host demonstrates a five-step process, including task selection, blind testing, and decision-making, using a custom script to compare seven frontier and open-weight models. The discussion highlights how subjective preferences and cost constraints often outweigh raw benchmark performance, encouraging users to tailor their AI stacks for maximum practical utility.
Topics discussed
Intro: The fall glut of new AI models and benchmarks
Why published benchmarks are losing value
The case for building a personal AI benchmark
Step 1: Deciding which models to evaluate
Navigating the noise of new model releases
Addressing common objections to personal benchmarking
Step 2: Selecting concrete use cases for your benchmark
Step 3: Choosing the right number of model candidates
Step 4: Anonymizing results for a blind taste test
Step 5: Scoring methods and using AI as a judge
Ensuring benchmark validity and consistency
Sponsors: Robots and Pencils, Harbor ETFs, Blitzy
Making the final decision: Switch or stay?
Live demo: Setting up the benchmark environment
Generating use cases and prompts with AI assistance
Demo: The local UI for blind scoring
Demo: Evaluating visual and text-based outputs
Demo: Analyzing results, costs, and judge discrepancies
Alternative tools for standardized evaluation
Summary of the five-step process
Q&A: Evolving criteria, effort settings, and costs
Listen ad-free on Castria