To evaluate LLM outputs, write down what a good answer looks like, build a test set of real cases with expected results, and score each output with code checks, a judge model or human review. Re-run the set after every prompt or model change, and check automated scores against human judgement before you trust them.
Why reading a few answers and nodding stops working
Most teams start the same way. Someone reads ten outputs, they look fine, and the feature ships. Then a colleague tweaks the system prompt, or a newer model comes out, and nobody can say whether the answers got better or worse. You're left arguing about examples instead of looking at a score.
An evaluation fixes that. It's a repeatable test: the same set of inputs runs through your prompt or model, and every output is scored against a standard you wrote down in advance. That standard can be an expected label, a code check or a second model acting as a judge. The point is that it doesn't change when your mood does.
Two things make evals useful in practice. First, they're written before you look at the results, so you can't bend the target to fit the output. Second, they run again after every change. Evaluation is a continuing process, not a launch checklist, and treating it as a one-off is how quality slides without anyone noticing.
Here's how I'd lay out the job for a small team or a solo developer, in six steps, with the tools that fit each one.
Six steps from a vague goal to a score you can trust
1. Write the pass rule before you write a single test
Describe what a good answer is for this task in one or two plain sentences, then add a pass or fail rule. For a support assistant that might be: answers the customer's question, uses only facts from the help centre, and never promises a refund. Without a target like this, scores are noise, because every reviewer is grading against a different picture in their head.
Promptfoo is a sensible place to write that rule down, since its CLI lets you define metrics and score outputs against them from the first run.
2. Build the test set from real traffic, not made-up prompts
Collect cases from real usage or from people who know the domain, and give each one an expected answer or label. OpenAI's evaluation guidance warns that datasets which don't reproduce production traffic give biased results. Invented prompts tend to be cleaner and easier than what customers type, so a set built from them flatters your system.
Platforms that keep datasets next to your traces make this easier, because you can pull real cases straight from logged conversations. Arize Phoenix holds datasets and experiments alongside tracing for exactly that reason.
3. Choose your scoring in order: code first, judge second, people last
Start with checks a computer can settle on its own: exact match, a required field, valid JSON, a banned phrase. Quick deterministic checks catch many failures before you need a judge model at all, and they never disagree with themselves.
Bring in a judge model for open-ended quality, such as tone or helpfulness, and keep humans for the cases where judgement matters most. If you're testing a retrieval app, score retrieval and the answer separately. Ragas offers metrics built for that, including Context Precision, Context Recall, Faithfulness and Response Relevancy. DeepEval takes a different angle and lets you write assertions inside ordinary pytest files, so evals sit beside the rest of your tests.
4. Run the whole set on every change
Every new prompt, model or retrieval setting gets the full test set. Ideally that happens in CI/CD, so a drop in score blocks the release the same way a failing unit test would. Both of the code-first frameworks above say they can run in CI/CD, and that's the setup worth aiming for even if you start by running things by hand.
A matrix view that puts prompts and models side by side helps here, because you'll see at a glance which change moved which case. Promptfoo is built around that comparison.
5. Check the judge against human labels
Take a sample of outputs, have people label them, and compare those labels with the judge model's scores. Where they disagree, rewrite the rubric and run the comparison again until they line up. An automated score is only useful if it matches what people think is good. This step doesn't need a special tool, just a spreadsheet and some patience, and it's the one most teams skip.
6. Feed production failures back into the test set
New kinds of failure never show up in your first test set. Each time something goes wrong in production, add that case, with the right answer, so the set grows with your product. LangSmith can attach scores to live traces with online evaluators, and Braintrust pairs evals with production traces, which shortens the trip from a bad answer in the wild to a new test case.
Want AI doing the heavy lifting in your marketing?
I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.
The judge model has blind spots of its own
Using one model to grade another is fast and scales far better than human review, which is why almost everyone does it. But a judge isn't neutral. OpenAI's Evaluation best practices guide lists position bias, where a response is favoured because of the order it appears in, and verbosity bias, where longer responses are preferred, as known challenges of LLM-as-a-judge.
That matters more than it sounds. If your new prompt simply makes answers longer, a biased judge can report an improvement that no customer would notice, or that customers would find worse. If you always show the judge the old answer first, the order can tilt the result before quality comes into it.
The same guide recommends a few defences: use pairwise comparison or pass or fail scoring instead of loose ratings, control for response length, and validate the judge against human annotations before you scale it up. In practice, swapping the order of the two answers and checking the verdict holds is a quick sanity test, and so is comparing scores with answer length to see whether one is driving the other.
If you'd like a starting structure for your own rubric, this rubric template for scoring AI agents gives you criteria you can adapt to your pass rule.
Six eval tools and the steps each one covers
| Tool | Type | What it does | Steps it helps with |
|---|---|---|---|
| Promptfoo | Open-source eval and red-team CLI | Defines metrics, scores outputs and runs in CI/CD, with matrix views comparing prompts and models | 2, 3, 4 |
| DeepEval | Pytest-style LLM evals | Unit testing for LLMs: assertions with metrics inside normal test files, run in CI/CD or as scripts | 3, 4 |
| Ragas | Open-source RAG metrics | Scores retrieval-augmented generation with dedicated measures | 3, 4 |
| Braintrust | Evals and observability platform | Runs evals that compare prompts and models | 2, 4, 6 |
| LangSmith | Tracing, datasets and evaluators | Scores traces, including online evaluators on live traffic | 4, 6 |
| Arize Phoenix | Open-source tracing and evals | Evaluation, datasets and experiments plus tracing | 2, 4, 6 |
The first three are code libraries you run yourself; the last three add tracing, so you can see what happened in production as well as in tests. For a wider comparison, including tools not covered here, see my list of LLM evaluation tools worth shortlisting.
My advice if you only do one thing this week
Write the pass rule and collect twenty or thirty real cases with the right answers next to them. That's it. You don't need a platform to start, and you don't need a judge model on day one. Once you have a set you trust, every tool above becomes useful, and every change you make stops being a guess. When you do add a judge, check it against people before you let it decide anything.
If your app answers from your own documents, scoring retrieval separately is the next thing to get right, and the tools for building retrieval apps are worth a look alongside your eval setup.
Want your tool on this list? Featured spots and reviews here are paid partnerships. Here's how to partner with me.
Frequently asked questions
What is an LLM eval?
A repeatable test that runs a set of inputs through your prompt or model and scores the outputs against a standard, such as an expected label, a code check or a judge model.
What is LLM-as-a-judge?
Using a model to grade another model's output, for example by pairwise comparison, single-answer scoring or grading against a reference answer. OpenAI's guide notes it's faster to run and more scalable than human review but has biases, including position and verbosity bias.
How big should my test set be?
There's no fixed number in the guidance. Start with real cases from your own use case and keep adding failures found in production, so the set grows with the product.
Can I run evals in CI/CD?
Yes. Two of the code-first frameworks in the table above say they can run in CI/CD, so a drop in score can block a release the same way a failing test does.
How do I evaluate a RAG app?
Score retrieval and the answer separately. Ragas lists metrics such as Context Precision, Context Recall, Faithfulness and Response Relevancy for this.
