LLM Evaluation Guide: Metrics, Test Sets, and Regression Testing

Overview

LLM evaluation is the process of checking whether a language model produces outputs that meet the requirements of a real task. It applies to a single prompt, a retrieval-augmented generation workflow, an AI classifier, or a tool-using agent. The objective is not to find one universal score. It is to establish a repeatable way to answer a more practical question: did this version perform better, worse, or differently on the work that matters?

A useful evaluation system has four parts:

• A defined task: Describe the input, expected behavior, acceptable variation, and unacceptable failure modes.
• A representative test set: Include ordinary examples, difficult cases, edge cases, and known failures.
• Task-specific measurements: Combine automated checks with human review where meaning, tone, or judgment matters.
• A comparison process: Run the same cases against a baseline before approving a prompt or model change.

This approach supports prompt optimization without encouraging teams to chase isolated examples. A prompt that improves one visible response may still reduce consistency, increase unsupported claims, or break structured output elsewhere. Evaluation makes those tradeoffs visible.

For broader prompt design principles, see Prompt Engineering Best Practices: A Framework for Reliable LLM Outputs. If the system is an agent, evaluation should also be connected to tool calls, memory, and guardrails rather than limited to the final text. The guide to building an AI agent with tools, memory, and guardrails provides useful architectural context.

What to track

1. Task success

Start with the outcome the system is expected to produce. For a classifier, track whether the predicted category matches the approved label. For a support assistant, assess whether the response answers the question, follows the relevant policy, and gives the next useful step. For an extraction prompt, check whether required fields are present and correctly mapped.

Write success criteria before reviewing outputs. A compact evaluation specification might include:

• Input: The type of request or document being processed.
• Required output: The fields, decision, or response elements that must appear.
• Allowed variation: Differences in wording or formatting that do not affect usefulness.
• Blocking errors: Mistakes that make the output unsafe, unusable, or misleading.

2. Accuracy and completeness

Accuracy asks whether the output is correct. Completeness asks whether it covers everything required. These are separate dimensions. A concise answer may be accurate but omit a necessary condition. An extraction may identify the right values but leave several fields blank.

For structured outputs, automated validation can check JSON syntax, required keys, data types, allowed enum values, and basic relationships between fields. Validation does not prove that the content is true, but it quickly identifies outputs that cannot be safely passed to the next workflow step.

3. Grounding and citation quality

When an application uses a knowledge base or retrieval pipeline, evaluate whether the answer is supported by the supplied context. Track unsupported statements, missed evidence, incorrect interpretation of retrieved passages, and answers that should have acknowledged insufficient information.

Keep retrieval quality and generation quality distinguishable. If the necessary passage was never retrieved, changing the prompt may not solve the underlying problem. A RAG workflow may require separate checks for document selection, context ordering, answer grounding, and refusal behavior.

4. Consistency and format compliance

Run the same or equivalent inputs more than once when variation matters. Record whether the model follows the requested tone, schema, language, length, and decision rules consistently. In production, format compliance can be as important as prose quality because downstream code may depend on predictable fields.

5. Safety and failure behavior

Track what happens when the model lacks information, receives ambiguous instructions, encounters malformed input, or is asked to perform an action outside its authority. A good result is not always a direct answer. In some cases, the correct behavior is to ask a clarifying question, return a controlled fallback, or decline to invent a value.

6. Operational variables

Quality should be reviewed alongside latency, token usage, tool-call count, error rate, and cost-related measurements available in your environment. These variables help explain whether a quality improvement is practical. An evaluation record should identify the model version, prompt version, retrieval configuration, tool configuration, and other settings needed to reproduce the result.

Cadence and checkpoints

Evaluation works best as a routine rather than a final launch activity. Create a small golden dataset for every important task. Each case should include the input, expected properties, severity guidance, and—where appropriate—an approved reference answer. Do not treat the reference answer as the only acceptable wording unless the task genuinely requires exact matching.

Use a layered testing workflow:

1. Fast checks on every change: Validate syntax, required fields, prohibited content, tool-call arguments, and a small set of high-risk examples.
2. Full regression tests before release: Run the complete golden dataset whenever the system prompt, model, output schema, retrieval settings, tools, or business rules change.
3. Human review on a sample: Inspect both strong and weak examples. Include cases selected randomly and cases selected because automated checks disagree.
4. Production monitoring: Review sampled interactions, user corrections, escalation rates, fallback behavior, and newly observed failure patterns.

A monthly or quarterly review is useful even when no deployment is planned. Examine whether the task distribution has changed, whether new failure categories have appeared, and whether the test set still represents current usage. Add anonymized production examples to the dataset only after applying the appropriate privacy and access controls.

Maintain a change record for each evaluation run. At minimum, capture the date, system version, test-set version, model identifier as available, evaluator version, aggregate results, notable failures, and release decision. This turns a collection of scores into an audit-friendly development history.

How to interpret changes

Do not rely on one aggregate score. Break results down by task type, difficulty, language, input length, customer segment, retrieval condition, or other dimensions that affect performance. A stable average can conceal a serious regression in a smaller but important category.

Use a failure taxonomy to make results actionable. Common categories include:

• Instruction failure: The model ignored or misunderstood the prompt requirement.
• Knowledge failure: The answer lacked required information or used unsupported information.
• Reasoning failure: The relevant facts were available, but the conclusion was incorrect.
• Extraction failure: The content was present but mapped to the wrong field or format.
• Tool failure: The agent selected the wrong tool, supplied invalid arguments, or misread the result.
• Boundary failure: The system answered when it should have asked, escalated, or declined.

Severity is as important as frequency. A rare error that triggers an incorrect external action may deserve more attention than many minor style issues. Give each failure a severity level and define the required response, such as prompt revision, data improvement, tool validation, guardrail adjustment, or human escalation.

When comparing versions, look for both improvements and regressions. A prompt change that raises completeness but increases unsupported claims requires a deliberate tradeoff, not an automatic approval. If two models perform similarly, latency, operational complexity, output stability, and integration behavior may determine the better choice.

Human evaluation should use a clear scorecard. Reviewers can rate criteria such as correctness, relevance, completeness, groundedness, clarity, and instruction following on a small defined scale. Provide examples for each rating and separate “not applicable” from a poor score. Periodically compare reviewers on the same cases to identify ambiguous criteria or inconsistent interpretation.

For AI classifiers, confidence checks and an explicit uncertain category can make evaluation more useful than forcing every input into a label. See How to Build Reliable AI Classifiers with Prompts and Confidence Checks for a related workflow. For agents, connect evaluation results to traces that show planning steps, tool calls, retrieved context, and final responses. The guide to AI agent observability covers the evidence needed to investigate these failures.

When to revisit

Revisit your LLM evaluation plan on a regular monthly or quarterly cadence, and immediately after any material change. Triggers include a new model, a revised system prompt, changes to few-shot examples, a new output schema, modified retrieval sources, updated tools, altered routing logic, or a change in the users and documents the system handles.

Update the golden dataset when production monitoring reveals a new recurring failure, when a business rule changes, or when an old case no longer reflects real usage. Retire examples only with a documented reason; otherwise, keep historical cases so you can detect regressions over time. Review thresholds whenever the cost of an error, the acceptable response time, or the required level of human oversight changes.

Use this practical review loop:

1. Choose the release or review trigger.
2. Freeze the test-set and evaluator versions for a comparable baseline.
3. Run fast checks and the full regression suite.
4. Compare results by task, severity, and failure category rather than average score alone.
5. Inspect representative wins, regressions, and disagreements between automated and human review.
6. Record the decision: approve, revise, roll back, or release with additional monitoring.
7. Add confirmed new failures to the test set and schedule the next review.

Evaluation is most valuable when it changes day-to-day engineering decisions. Keep the scorecard small enough to maintain, the test cases close to real work, and the failure labels specific enough to guide a fix. With that discipline, prompt optimization becomes a controlled process: each model or prompt update can be tested against known requirements, explained when results move, and revisited as the application evolves.