When Building Evals Is a Waste of Your Time
About this episode
Does a personal AI assistant built with a coding agent need evals? Not yet, says Hamel Husain, co-author with Shreya Shankar of O'Reilly's "Evals for AI Engineers" and now leading evals at OpenAI: if you are its only user and read every output, you are already doing them. So Angelina Yang turns hers into a product for every podcaster, and he builds the eval system from scratch with no new tools: which logs to read and why a perfect score is a bad sign.
Key moments
- 0:00 The #1 mistake people make with AI evals
- 1:03 Who is Hamel Husain, and what are evals?
- 2:34 Let's build an eval system: a live mini-workshop
- 3:35 Does a personal AI assistant even need evals?
- 4:22 Where to start: read your logs and write what's wrong
- 5:35 Why a generic "3.2" score tells you nothing
- 8:41 What a trace is, and which logs to sample
- 13:22 How many traces do you have to read?
- 14:43 How to read a trace fast
- 19:09 Why you can't just ask AI to find the problems
- 20:25 Can you test your AI on synthetic data?
- 22:56 Which failures should you fix first?
- 25:45 Can you trust an AI judge? Pass/fail vs 1 to 5
- 29:55 Letting your AI improve itself without cheating
- 31:19 What still needs a human
- 33:29 Why a perfect eval score is a bad sign
- 35:49 The one metric worth reporting
- 37:45 Is observability the same as evals?
- 40:15 Who's accountable when an AI agent gets it wrong?
- 41:20 How AI is collapsing engineering roles
- 42:53 One piece of advice for AI engineers
- 43:45 outro
From the TwoSetAI Playbook
What Is Founder Visibility in AI Search?
The definition. Founder visibility in AI search is whether ChatGPT, Perplexity, Google AI Mode, and Claude name and cite a specific founder in buyer-facing answers. How it is measured, and how it differs from PR, SEO, personal branding, and the services sold under the same phrase.
Why Video Content Is Winning AI Search: The GEO Case for Long-Form Video
94% of YouTube citations in a 100-million-citation AI study went to long-form video. See what engines read in a video and how founders can use it.