How do you evaluate an LLM application?

Define what a correct output looks like, build a test set of realistic inputs with expected results, score outputs with a mix of deterministic checks, model-based graders and human review, and rerun the suite whenever prompts, models or retrieval change.

What is LLM-as-a-judge?

LLM-as-a-judge means using a language model with a clear rubric to grade another model’s outputs. It scales evaluation of open-ended answers, but it should be calibrated against human judgements on a sample before its scores are trusted.

How large should an LLM evaluation set be?

Large enough to cover the important task types and failure modes. Many teams start with fifty to a few hundred carefully chosen cases, then grow the set continuously by adding real production failures.

How to evaluate an LLM feature before it reaches customers

Evaluating an LLM feature means measuring, repeatably, how often it produces acceptable outputs on realistic inputs — and noticing when that changes. Without it, every prompt edit or model upgrade is a guess. With it, AI features can be developed with the same discipline as any other production software.

Start with a definition of success

“Good answers” is not a specification. Write down what correct means for this feature:

For extraction: the right fields with the right values.
For classification: the right label, and acceptable confusion between labels.
For answers grounded in documents: correct, supported by sources, and silent when the sources do not cover the question.
For agents: the right tools called with the right parameters, and confirmation requested where required.

Also define what is unacceptable: invented policies, wrong prices, leaked data, or an answer when the system should have escalated.

Build a test set from reality

The best test cases come from real inputs: support tickets, search queries, emails, documents. For each case, record the input and the expected outcome or the criteria for a good one. Include:

common, easy cases (to protect what already works),
difficult and ambiguous cases,
adversarial inputs and instructions hidden in content,
cases where the correct answer is “I don’t know” or “escalate”.

Start small and curated. A hundred well-chosen cases are more useful than ten thousand random ones.

Score with three kinds of graders

Deterministic checks — exact match, JSON schema validation, numeric tolerance, presence of a required citation. Fast, cheap and unambiguous; use them wherever possible.
Model graders — a separate model with a precise rubric scores open-ended outputs on criteria such as accuracy, completeness and tone. Ask for a short justification with each score.
Human review — experts review a sample, especially disagreements and low scores. Their judgements also calibrate the model graders.

Validate model graders before trusting them: compare their scores with human ratings on a sample and refine the rubric until they agree on the cases that matter.

Evaluate the pipeline, not just the model

In retrieval-augmented systems, many failures happen before the model writes anything. Measure stages separately:

Retrieval — did the right passages appear in the top results?
Generation — given the right passages, was the answer correct and supported?

This separation tells you whether to fix chunking and search or prompts and models. The retrieval side is discussed in practical RAG for operations teams and choosing a vector database.

Make it a regression suite

Run the evaluation automatically whenever something changes: prompt, model version, retrieval settings, tools. Track scores over time per category of case. A change that improves the average while breaking a critical category should not ship.

Monitor in production

Offline evaluation never covers everything. In production, log inputs and outputs (with appropriate privacy controls), collect user feedback, sample conversations for review, and add every real failure to the test set. The suite then grows in exactly the places the product is weak.

Watch cost and latency too

Quality is not the only metric. Record tokens, cost per request and latency per case. A model that is marginally better but several times slower may be the wrong choice for an interactive feature.

The takeaway

LLM features become dependable when they are measured like software: clear success criteria, a realistic test set, layered graders and automatic regression runs. The investment is modest, and it is the only reliable way to improve a feature rather than just change it.