What Are LLM Evals?
An LLM eval is a repeatable test that scores an AI app's answers against cases you chose in advance. It replaces "it looks fine to me" with a number you can track every time.

Your AI feature works in the demo. Then a customer asks a slightly different question, and the answer is wrong, rude, or made up. Nobody noticed, because nobody was testing it. That gap is what LLM evals exist to close.
Table Of Content
- The Problem Evals Solve
- What an Eval Actually Is
- Evals vs Benchmarks
- The Main Ways to Score an Answer
- LLM-as-a-Judge: The Popular Shortcut, With a Warning
- Offline and Online Evals
- What to Measure: The Metrics That Matter
- How to Run Your First Eval
- Evaluating AI Agents Is Harder
- Which Tool Should You Use?
- The Mistakes That Make Evals Useless
- Questions About LLM Evals
- Start With Twenty Questions
An LLM eval, short for evaluation, is a repeatable test that scores an AI app’s answers against cases you chose in advance. So what are LLM evals? You write a set of test questions with the right answers, run them through your app, and score every output. The result is a number you can track, like 87 out of 100. When you change the prompt, the model, or the data, you run the same set again. If the score drops, the change does not ship.
The Problem Evals Solve
Language models are unpredictable in a way normal software is not. The same prompt can give a good answer on Monday and a bad one on Tuesday, because the model, the data, or the prompt changed in between. A developer reads five answers. They all look fine, and the feature ships. Then real users find the sixth case, the one nobody tried.
Evals turn that guessing into evidence. A fixed test set catches the regression a quick read misses. You change a prompt to fix one complaint, and the eval shows it broke ten cases that used to pass. Without that number, you ship the fix and find out from your users.
What an Eval Actually Is

Every eval has the same three parts, whatever tool you use.
First, a test dataset. This is a set of inputs paired with the correct response, or with a written rule for what counts as correct. The inputs should be real examples from your users, not easy questions you made up. This dataset is often called a golden dataset, because it is the standard everything else is measured against.
Second, a scoring method: the rule that turns each output into a number or a pass-or-fail verdict. It can be code, a model, or a person.
Third, a harness: the code that feeds every input through your app, grades the outputs, and adds the scores into one number. The harness is what makes the test repeatable, so the comparison stays fair over time.
Evals vs Benchmarks
People mix these two up, and the difference matters.
A benchmark measures general model capability on a public, shared dataset, so models can be ranked against each other. MMLU tests broad knowledge across 57 subjects. GSM8K tests grade-school maths. HumanEval tests code writing. These are benchmarks. They tell you which model is generally strong. Stanford’s HELM project goes further, scoring models on seven metrics across sixteen scenarios.
An eval measures your system on your task with your data. A suite of 300 support tickets with the right routing category is an eval. Fifty refund requests with the correct policy answer are an eval too. These tell you whether your feature works.
| Benchmark | Eval | |
|---|---|---|
| Question it answers | Which model is strongest overall? | Does my feature work on my task? |
| Data | Public, shared, same for everyone | Your cases, your users, your rules |
| Who it serves | Model makers and researchers | Teams shipping an AI feature |
| Examples | MMLU, GSM8K, HumanEval | Your support tickets, your documents |
Benchmarks help you pick a model. Only an eval tells you whether the feature you built with it works.
The Main Ways to Score an Answer
Scoring is where most evals succeed or fail, because the right method depends on the question. There are three main ways, and a good eval set usually mixes them.
Code checks are the cheapest and most reliable. Anything you can express as logic should be checked with code. Did the answer match the expected label? Does the JSON parse? Is the output in the right format? Did the app call the right tool with the right arguments? These checks are fast, free, and perfectly repeatable. They cannot judge meaning or tone, but they should always come first.
LLM-as-a-judge means using a strong model to score another model’s output against a written rubric. The judge reads the question, the answer, and the rubric, then gives a score, often from 1 to 5 or a simple pass or fail. This is the only practical way to score thousands of open-ended answers. It has real limits, covered in the next section, and it must be calibrated against human ratings.
Human review means a person reads the output and scores it. It is slow and costly, so it runs on samples, not on every case. Humans catch the fluent, confident errors that automated checks miss.
| Scoring method | How it works | Good for | Watch out for |
|---|---|---|---|
| Code checks | Logic: exact match, format, tool calls | Anything with a clear yes or no | Cannot judge meaning |
| LLM-as-a-judge | A model scores against a rubric | Meaning and tone at volume | Needs calibration against humans |
| Human review | A person reads and scores | Judgment calls, catching fluent errors | Slow, runs on samples |
LLM-as-a-Judge: The Popular Shortcut, With a Warning
LLM-as-a-judge is the most popular scoring method in practice, and the most misused. The idea is simple: a capable model reads an answer and scores it against a rubric, the way a teacher marks an essay. It scales to thousands of answers, which no human team can do.
The 2023 study that tested this idea, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, found strong judges like GPT-4 matched human preferences at over 80% agreement, the same level humans reach with each other. It also found the biases you must guard against. Judges favour the answer shown first, which is called position bias. They favour longer, more confident answers, which is called verbosity bias. Answers from their own model family also get a boost, which is called self-enhancement bias. And they struggle with questions that need deep reasoning.
Three habits keep a judge honest. First, write the rubric before you run the eval, and make it specific enough that two judges would agree. Second, swap the order of answers and check the score does not follow the position. Third, calibrate regularly: have humans score a sample of the same answers and compare.
Offline and Online Evals
Evals run in two places, and they catch different things.
Offline evals run before anything reaches users, on a fixed set of test cases. This is your release gate. Every change to the prompt, model, or data runs against the same frozen set. It ships only if the agreed scores hold. If they do not, the failing cases show exactly what broke.
Online evals score a sample of real conversations after launch. There is no expected answer here, because the input is whatever a user just typed. Instead the checks look for patterns: harmful outputs, answers that ignore the retrieved documents, or a sudden drop in quality. Our guide to LLM observability covers the monitoring side of this in more detail, and LangSmith’s own documentation explains how online evaluation differs from the offline kind.
What to Measure: The Metrics That Matter
A single overall score hides more than it reveals. An app can score 90% overall while failing every safety check. Measure a small set of metrics that matter for your users.
| Metric | What it tells you |
|---|---|
| Accuracy or exact match | Whether answers are correct where the right answer is known |
| Faithfulness and hallucination rate | Whether claims are backed by the sources given |
| Relevance and completeness | Whether the answer actually solves the request |
| Safety pass rate | Whether outputs stay inside the approved scope |
| Latency | How long users wait for an answer |
| Cost per task | What each completed answer costs in model calls |
For apps that search documents before answering, two more metrics matter. Context precision asks whether the right passages were retrieved and ranked at the top, and context recall asks whether anything needed was missed. The Ragas framework builds its approach on these four: faithfulness and answer relevancy for the generator, context precision and recall for the retriever.
Pick three to five metrics and set a threshold for each one before you run anything. A threshold is a line you agree in advance, like faithfulness above 0.80. Without a threshold, a score is just a number. With one, it is a decision.
How to Run Your First Eval

You do not need a platform to start: about twenty realistic test questions, three to five clear checks, and a spreadsheet.
Start with the questions. Pull real examples from your own product: support tickets, user messages, documents your app reads. For each one, write down the right answer or the rule a good answer must follow. Twenty cases is enough to begin, fifty is better, and a hundred is a serious eval set.
Next, pick your checks. For a support bot, the checks might be: the routing category matches the human label, the answer cites the right policy, and the tone stays polite. A document search app might check that the answer uses only the retrieved passages, and that it says so when the documents have no answer.
Then run the set through your app and score every output. A spreadsheet works fine at this size. Count the passes. That number is your baseline. Every change from now on is judged against it.
Evaluating AI Agents Is Harder
An agent is not a single answer. It is a chain of decisions: read the request, pick a tool, call it, read the result, decide the next step, and finally answer. Evaluating that chain needs three kinds of checks, not one.
Outcome checks ask whether the task was completed correctly: the right refund issued, the ticket resolved, the file actually created. The outcome is what the user cares about, and it is the easiest part to score.
Trajectory checks ask whether the agent took a sensible path. Did it call the right tools with the right arguments, in a reasonable number of steps, without looping? An agent that reaches the right answer after twenty wasted tool calls is not a good agent. It is a lucky one.
Safety checks ask whether the agent stayed inside its permissions: approval asked before a payment, requests outside its scope refused. These checks matter most, because an agent acts, and a wrong action is worse than a wrong sentence.
Run agent evals against sandboxed tools or recorded responses, so the tests are repeatable and harmless. A test that sends real emails or moves real money is not a test. It is an incident waiting to happen.
Which Tool Should You Use?

The tools below are the ones teams actually use, and each one fits a different starting point. All of the open-source options cost nothing to run yourself, apart from the model API calls used for scoring.
| Tool | Good for |
|---|---|
| OpenAI Evals | OpenAI’s framework and benchmark registry, with almost 20,000 GitHub stars |
| LangSmith | Tracing and evaluation for LangChain and LangGraph apps |
| Langfuse | Open-source tracing, datasets and scoring, self-hosted or free hosted tier |
| Ragas | RAG metrics: faithfulness, answer relevance, context precision and recall |
| DeepEval | Unit-test-style LLM evals with 30+ built-in metrics |
| promptfoo | Prompt comparisons, red-teaming, and evals in CI |
If you are building agents rather than a single chat feature, the framework choice shapes what you can test. Our guide to the Microsoft Agent Framework explains the newest option from Microsoft. The comparison of LangGraph vs AutoGen covers the two frameworks most teams choose between, and our roundup of the best tools to build AI agents covers the wider set of options. Whichever you pick, the eval set should travel with the agent, not live in a separate tool nobody opens.
The Mistakes That Make Evals Useless
Most eval programs fail in the same few ways.
Testing on training data is the first. If your test cases come from the same examples you used to build the prompt, the score means nothing. The app has seen them before. Keep the test set separate from everything used to build the system.
A single overall score is the second. One number hides the failure that matters. Track a few metrics separately, and look at the failing cases, not just the average.
Never re-running is the third. An eval set that runs once is a snapshot. Its value comes from running it on every change, so regressions show up the day they are made.
Letting the judge grade its own work is the fourth. If the model being tested is also the judge, the scores lean generous. Use a different, stronger model as the judge, and calibrate it against humans.
Skipping human review is the fifth. Automated scores drift. A small human sample, scored regularly, keeps the whole system honest.
Questions About LLM Evals
What are LLM evals?
LLM evals are repeatable tests that measure how well a language model app performs on a defined set of inputs. They score outputs against expected results or criteria, such as correctness, faithfulness to sources, format, and safety, so changes can be compared objectively.
How are LLM evals different from benchmarks?
A benchmark compares models against each other on public, shared datasets, such as MMLU for knowledge or HumanEval for code. An eval tests your own system on your own cases with your own rules. Benchmarks help you pick a model, while evals tell you whether your feature works.
What is LLM-as-a-judge?
LLM-as-a-judge uses a language model to score another model’s output against a written rubric, such as a score from 1 to 5 or a pass or fail. It scales to thousands of open-ended answers. A 2023 study found strong judges matched human preferences at over 80% agreement, but judges need calibration against human ratings.
How many test cases do I need for an eval?
Twenty realistic cases are enough to start, and fifty is better. A serious eval set runs into the hundreds. What matters more than the count is that the cases are real examples from your users, with reviewed expected answers, and that the set stays frozen between runs.
What is a golden dataset?
A golden dataset is a small set of inputs with reviewed correct answers that serves as the standard for an eval. It is called golden because it is the reference everything is measured against. Build it from real user examples, keep it versioned, and never train or tune on it.
Can LLM evals test AI agents?
Yes, and agents need three kinds of checks. Outcome checks ask whether the task was completed correctly. Trajectory checks ask whether the agent took a sensible path through its tools. Safety checks ask whether it stayed inside its permissions. Run agent evals against sandboxed tools so tests stay repeatable and harmless.
Do I need a paid tool to run LLM evals?
No. The main open-source frameworks, including OpenAI Evals, Ragas, DeepEval, promptfoo, and self-hosted Langfuse, cost nothing to run apart from model API calls used for scoring. A spreadsheet and twenty test cases are enough to start, and a tool becomes useful when the set grows or several people need the results.
Start With Twenty Questions
Evals sound like a big project, and teams put them off for that reason. They are not. Twenty real questions, three clear checks, and a spreadsheet are enough to begin. Run the set, write down the score, change one thing, and run it again. That loop, repeated, is the whole discipline.
The teams that ship reliable AI features are not the ones with the best models. They are the ones who know, with a number, whether the last change helped or hurt.






No Comment! Be the first one.