Evals over vibes

If your team can’t tell me whether the new prompt is better than the old one, you’re not iterating. You’re vibing. You changed something, it felt better in the two examples you happened to try, and you shipped. Next week someone changes it back for the same reason. Vibes have no memory, so you never accumulate anything — you just walk in circles that feel like progress.

The fix is small and you can start today. You need a rubric, a held-out set of inputs, and the discipline to run it the same way twice.

The 30-minute eval

You don’t need an eval framework or a platform. You need three things:

1. 20 representative inputs. Real ones, anonymized — pulled from what your agent actually gets, not invented happy-path examples. Twenty is enough to catch regressions; diversity across them matters far more than count.
2. A rubric. Five to twenty yes/no questions about a single output. Did it answer what was asked? Did every claim have a source? Did it skip the required disclaimer? Binary, because “7 out of 10” means something different to every rater and yes/no doesn’t.
3. Two raters, scoring blind. You and one other person, neither of you told which version produced which output. Then count the yeses.

That’s it. Run your current prompt against the 20 inputs, score them, and you have a number. Run the new prompt, score it the same way, and now the question “is this better” has an answer instead of a feeling.

What to do with the scores

The single number matters less than two things around it.

Inter-rater agreement. If you and your co-rater disagree a lot, the problem isn’t the model — it’s your rubric. A question two careful people score differently is too vague to be useful. Rewrite it until you agree, and you’ve sharpened your own definition of “good” in the process. That clarity is half the value of the exercise.

The trajectory. Keep the same 20 inputs for every run and track the score over time. One snapshot tells you little; the line tells you whether you’re improving, holding, or quietly regressing while everyone insists the new version “feels smarter.” When a model update lands, the eval is how you find out it got worse at the one thing you cared about before a customer does.

The mistakes that make evals lie to you

• Testing on inputs you tuned on. If you optimized the prompt against these 20 examples, of course it scores well on them. Hold a set back. The eval has to see inputs the prompt-writer didn’t.
• Moving the rubric to match the output. The moment a result is disappointing, there’s a pull to soften the question so it passes. That’s Goodhart with extra steps. Freeze the rubric before you score.
• n = 3. Three examples is an anecdote. Twenty is the floor where a one-point swing starts to mean something.
• Scoring un-blind. If the rater knows which is the “new” version, the rater roots for it. Strip the labels.

Graduate from here when you feel the friction: automate the scoring with a model-grader once your rubric is stable enough that a model agrees with your humans, expand the set when 20 stops catching things, add adversarial inputs once the easy cases all pass. But the manual 30-minute version is the one that changes how a team works, because it replaces “I think it’s better” with “it scores four points higher, here’s the sheet.”