Evals over vibes
If you can't say whether the new prompt beats the old one, you're vibing, not iterating. Here's a 30-minute eval that fixes it.
If your team can't tell me whether the new prompt is better than the old one, you're vibing, not iterating. You changed something, it felt better on the two examples you happened to try, and you shipped. Next week someone changes it back for the same reason. Nobody wrote down what the first change was supposed to fix, so the team walks in circles and calls it progress.
The fix is small enough to start today: a rubric, a held-out set of inputs, and the discipline to run it the same way twice.
The 30-minute eval
You don't need an eval framework or a platform for this. You need three things:
- 20 representative inputs: real ones, anonymized, pulled from what your agent receives in production. Invented happy-path examples don't count. Twenty is enough to catch regressions, and spread across input types matters more than adding a 21st.
- A rubric: five to twenty yes/no questions about a single output. Did it answer what was asked? Did every claim have a source? Did it skip the required disclaimer? Keep them binary, because "7 out of 10" means something different to every rater and yes/no doesn't.
- Two raters, scoring blind: you and one other person, neither told which version produced which output. Then count the yeses.
Run your current prompt against the 20 inputs and score the outputs. That gives you a number. Run the new prompt, score it the same way, and "is this better?" now has a numeric answer.
What to do with the scores
Two things around the score tell you more than the score itself.
The first is inter-rater agreement. If you and your co-rater disagree a lot, fix the rubric before you blame the model. A question two careful people score differently is too vague to use. Rewrite it until you agree, and you'll have a sharper definition of "good" than you started with, which is half the value of the exercise.
The second is the trend. Keep the same 20 inputs for every run and track the score over time. A single run tells you little. Plot the runs and you can see whether you're improving, holding, or slipping while everyone insists the new version "feels smarter." When a model update lands, the eval tells you it got worse at the one thing you cared about before a customer does.
The mistakes that make evals lie to you
- Testing on inputs you tuned on: if you optimized the prompt against these 20 examples, of course it scores well on them. Hold a set back. The eval has to see inputs the prompt-writer didn't.
- Moving the rubric to match the output: when a result disappoints, you'll feel the pull to soften the question until it passes. That's Goodhart's law with extra steps. Freeze the rubric before you score.
- n = 3: three examples is an anecdote. Twenty is the floor where a one-point swing starts to mean something.
- Scoring un-blind: a rater who knows which output is the "new" version roots for it, so strip the labels.
Move past the manual version when it starts to chafe. Automate scoring with a model grader once the rubric is stable enough that the model agrees with your human raters. Grow the set when 20 inputs stop catching regressions, and add adversarial inputs once the easy cases all pass. The manual 30-minute version is still the one that changes how a team works, because it replaces "I think it's better" with "it scores four points higher, here's the sheet."