Evaluating AI when normal tests don't work
By the end of today you can build an evaluation set for an AI feature, choose a grading method that fits the task, and explain why a passing test suite tells you almost nothing about a probabilistic system.
YesterdayOn Day 47 you learned a test earns its place by failing when behaviour breaks. That definition survives here; assertEquals does not.
TomorrowTomorrow you apply it to the two hardest cases: retrieval and agents.
Why this matters
Without evaluation you cannot tell whether a change improved anything. Teams tune prompts for weeks on vibes, ship regressions they never detect, and have no way to compare two models.
- Evaluating AI
- Eval sets
- Grading methods
- Evals as regression tests
Learn it
75 minCopy this into Claude or ChatGPT. It quizzes you before it explains anything, which is deliberate. The resources under it are how you check what it told you.
Today's Master Prompt
Free · sign inA prompt written for this day alone: your level, the exact scope, what to leave out, and an instruction to quiz you before it explains anything. Paste it into Claude or ChatGPT and it teaches you today's material.
Check it against something that is not a model
An assistant can be fluent and wrong, and on a topic you met today you will not catch it. These cover the same ground and were made by people who do this for a living, so they are what you hold the explanation up against. They are other people's work and we only link to them, so judge them for yourself.
3 hand-picked resources
Free · sign inVideos, official docs and articles covering the same ground, each opened and annotated by hand. They are what you check the assistant against on a day you cannot yet catch it being wrong.
Build it
55 minBuild an eval set of at least fifteen real cases for an AI feature you have built, drawn from actual inputs rather than invented ones. Grade programmatically wherever possible and by rubric otherwise. Record a baseline pass rate. Then make a prompt change you believe is an improvement and re-run it, comparing pass rate, cost and latency.
Recall it
20 minAnswer out loud, reveal, then mark honestly whether you had it. That score is the only thing on this page you do not get to choose.
5 recall questions
Free · sign inQuestions you answer from memory, then grade yourself against the real answer. The score is carried into the mastery rating below it, so an honest miss cannot quietly become a tick.
Rate it
Completion and mastery are tracked separately. Be honest, because an inflated rating only means the concept resurfaces sooner.
Mastery tracking
Free · sign inRate yourself against five named criteria per concept. Completion and mastery are tracked separately, and anything you rate shakily comes back automatically on a spaced schedule.
Recap
- 01Output is not deterministic, so assertion equality was never going to work
- 02Real cases from production beat invented ones every time
- 03Grade programmatically wherever the property is checkable in code
- 04The grader needs its own validation, or you are measuring with a broken ruler
Your progress
Free · sign inMark days complete, pick up where you left off across devices, and watch completion and mastery diverge. Free, and the account exists only so ninety days of work cannot vanish with a cleared browser.