Evals – systematically testing prompts and models

Evals are tests that measure how well a prompt or model actually solves a task – instead of judging by gut feeling.

Why evals are needed

If you change a prompt and the next answer "feels" better, that still doesn't tell you whether it's really better – or just better on this one example. Without a systematic test, prompt improvement stays guesswork. Evals (short for "evaluations") solve this by objectively checking a change against many examples at once.

The basic setup

An eval consists of a test set (several input examples, often with an expected answer or grading criteria) and a method for scoring the output – exact match, a rule-based check, or a second model acting as a "judge" (LLM-as-judge). After every prompt or model change, the test set runs again, and you compare the before/after results.

Difference from public benchmarks

Benchmarks like MMLU or SWE-bench are external, general-purpose tests. Your own evals are tailored to your specific task – often more meaningful for your product, even though they're smaller and less well known.

A practical starting point

Even 20-30 carefully chosen, realistic examples from your own use case give useful signal. What matters is that the test cases cover real edge cases and typical failures, not just easy standard cases.