Contents: the evaluation set, scoring and judges, guardrails and security, a worked example, release thresholds, regression testing and monitoring, common mistakes, where to start, FAQ.
- Test an LLM application with a real evaluation set
- Scoring: automatic checks, human review and the LLM judge
- Guardrails and security checks
- A worked example: a fictional customer support assistant
- Release thresholds, gradual rollout and a go-live checklist
- Regression testing and monitoring after go-live
- Common mistakes
- Where to start
- FAQ
TL;DR. A demo shows that an assistant can answer well once. To test an LLM application properly, you need a written evaluation set, clear scoring criteria and thresholds agreed before launch. Add guardrails and security tests, release to a small group first, and rerun the same tests after every change to the prompt, the documents or the model. Keep logging after go-live and feed every failure back into the set.
Most teams do the opposite. They polish the demo, try a dozen friendly questions, and ship. The first week in production then teaches them what the test should have taught them. This guide is the practical path we recommend as an AI consultant team: what to build, what to measure, what to block, and when to say yes.
It applies whether you are building a custom AI application, a support assistant over your documents or an internal tool for your staff. The steps are the same. Only the thresholds change with the risk.
Test an LLM application with a real evaluation set
The evaluation set is the foundation. It is a list of questions, each with the expected answer or the expected behavior, that you run against the application every time something changes. Without it, every opinion about quality is anecdotal.
Where the questions come from
Start with the people who will use the tool. Collect real questions from tickets, emails, chat logs and interviews with the team. Where real data is unavailable, ask domain experts to write realistic ones. Do not let the project team invent them alone: they ask the questions they know the system can answer.
The types of questions to include
- Core questions: the frequent, well documented requests that make up most of the traffic.
- Edge cases: ambiguous wording, typos, mixed languages, very short or very long messages.
- Multi-step questions: requests that need two documents or a calculation to answer.
- Out of scope questions: topics the assistant must decline, such as legal advice or other customers' data.
- Missing information: questions whose answer is not in the documents. The right behavior is to say so.
- Adversarial inputs: attempts to override instructions, extract the system prompt or change the topic.
What to write for each question
For every entry, record the question, the expected answer or the key facts it must contain, the source document that supports it, and the expected behavior (answer, decline or hand off). Add a category tag so you can see where the weak spots are. Keep the set in a plain file under version control so changes are visible.
Start small and grow it. A set that is reviewed carefully beats a huge one nobody trusts. Every real failure after launch becomes a new entry, so the set improves with use. Keep a separate group of questions that you never use when tuning the prompt. That held out group tells you whether you are fixing real problems or just memorizing the test.
Scoring: automatic checks, human review and the LLM judge
A test is only useful if you can say what a pass looks like. Define criteria in business terms, then decide how each one is checked.
Criteria that matter to the business
- Correctness: does the answer match the facts in your approved sources?
- Grounding: is every claim supported by a retrieved document, with no invented details?
- Citation: does it point to the right source, and does that source say what is claimed?
- Refusal behavior: does it decline when it should, and answer when it should?
- Tone and format: is it suitable for the audience, in the right language, at the right length?
- Safety: does it avoid exposing personal or confidential data?
Score each criterion separately. A single "quality" grade hides the problem. An answer can be perfectly written and completely wrong.
Three ways to evaluate
Automatic checks. Rules that run in seconds: does the answer contain a required fact, cite a document from the approved list, respect the length limit, avoid a banned phrase, return valid structured output? They are cheap, repeatable and strict. They cannot judge nuance.
Human review. Domain experts read a sample and grade it against written criteria. It is slow and costly but it is the reference. Use it to build the first version of the set, to settle disagreements, and to check the other two methods.
LLM as a judge. A second model grades the answers against a rubric. It scales well and catches problems that rules cannot, such as an answer that is fluent but unsupported by the source.
The limits of the LLM judge
A judge is a tool, not an oracle. Know where it fails.
- It tends to favor longer, more confident answers, even when they are wrong.
- It can favor answers written in its own style.
- Its grades vary with the wording of the rubric and sometimes between runs.
- It may miss domain errors that only an expert would notice.
- If the judge and the application use the same model, they can share the same blind spots.
Calibrate the judge before you trust it. Have humans grade a sample, compare their grades with the judge's, and adjust the rubric until they agree on the cases that matter. Give the judge the source documents and a precise checklist, not a vague request for a score. Recheck the agreement from time to time, and always send the borderline and high risk cases to a person.
A sensible split: automatic checks on everything, the judge on the nuanced criteria, and human review on a regular sample and on all critical categories.
Guardrails and security checks
Guardrails limit the damage when the model is wrong or is being manipulated. They need their own tests, because a guardrail you have not tried to break is a guess.
Prompt injection
Prompt injection is when text in a user message, or in a document the assistant reads, tells it to ignore its instructions. Test both routes. Write direct attempts ("ignore your rules and..."), and plant instructions inside the documents or web pages the assistant retrieves. The assistant should treat retrieved content as data, never as orders. Also limit what the assistant can do: an injected instruction is far less dangerous when the assistant has no power to act on it.
Data leakage
Check that the assistant never reveals the system prompt, other users' conversations, personal data or documents the current user is not allowed to see. Access control must be enforced by the system that retrieves documents, not by a sentence in the prompt. Test with users of different permission levels and confirm the answers differ accordingly.
Approved sources only
For a knowledge assistant, restrict answers to a curated set of documents. Test that it declines questions the sources do not cover instead of filling the gap from general knowledge. Test that outdated documents are removed from the index, and that every answer shows the source it used.
Handoff to a human
Define when the assistant must stop and pass the case to a person: low confidence, a sensitive topic, an angry customer, a request for an exception, a repeated failure. Test that the handoff really happens, that the context travels with it, and that the user is told what happens next.
Limits on actions and cost
If the assistant can call tools, send messages or change records, test each action with bad inputs. Require confirmation for anything irreversible. Add limits on message length, request rate and spend so that a loop or an abuse attempt cannot run unnoticed. Your DevOps and cloud setup should support this with proper secrets management, access rules and alerts.
Planning a cloud migration?
See how our senior consultants assess, plan and run migrations to AWS or Azure, with a tested return path.
A worked example: a fictional customer support assistant
Take a fictional company that sells subscription software and wants an assistant answering customer questions from its help center. This is an illustration, not a real case.
Step 1, the set. The team exports anonymized past tickets and groups them: billing, account access, features, integrations, cancellations. Support agents write the expected answer for each, with the help article that supports it. They add questions that have no article, requests for refunds outside policy, a customer asking about another person's account, and a few messages that try to make the assistant reveal its instructions.
Step 2, the criteria. Correctness and grounding are scored as pass or fail. Tone is scored on a short scale. Refusal behavior is scored by category: the assistant must decline account questions it cannot verify and must hand off any refund request.
Step 3, the first run. Automatic checks verify that each answer cites an article from the approved list. The judge grades correctness against the article text. The team reads every failure and every case where the judge and an agent disagree. The review shows two patterns: the assistant answers cancellation questions with an outdated article, and it gives a confident answer when the question is ambiguous.
Step 4, the fixes. The outdated article is removed from the index. The prompt is changed so that the assistant asks one clarifying question when a request is ambiguous. Each fix is followed by a full rerun, which also reveals that one change broke a billing answer that used to pass. That is exactly why the rerun exists.
Step 5, the security pass. Testers plant an instruction in a help article draft, try to extract the system prompt, and ask about another customer's invoice. The assistant resists the first two, but the third exposes a gap in access control on the invoice lookup. It is fixed in the retrieval layer, not in the prompt.
Step 6, the release. The assistant goes live for internal staff first, then for a small share of customers with a visible link to reach a human. Logs are reviewed every week, and every new failure is added to the set.
Release thresholds, gradual rollout and a go-live checklist
Set thresholds before you test
Agree on the pass criteria before you see the results, otherwise the numbers will be bent to fit the launch date. Use different bars for different risks. A zero tolerance rule fits safety and data leakage: a single failure blocks the release. A minimum pass rate fits correctness and grounding, set by the business according to the cost of an error. Tone can be more flexible.
Write the thresholds down with the name of the person who can approve an exception.
Roll out in stages
- Internal use: employees try the assistant on real work and report problems.
- Shadow mode: the assistant drafts answers that a person reviews before they are sent.
- Small group: a limited set of real users, with an easy way to reach a human.
- Wider release: extend access only when the monitored results stay within the thresholds.
Keep the ability to switch the assistant off or fall back to a simple behavior quickly. A rollback plan that nobody has tried is not a plan.
Go-live decision grid
Go: the thresholds are met, every security test passes, handoff works, logging is on, an owner is named, and a rollback is rehearsed.
Go with limits: quality is acceptable but one category is weak. Release with that category blocked or routed to a person, and set a date to review it.
No go: a safety or leakage test fails, results are unstable between runs, or nobody owns the system after launch.
Checklist
- The evaluation set covers core, edge, out of scope, missing information and adversarial questions.
- Scoring criteria and thresholds are written and approved.
- The LLM judge has been calibrated against human grades.
- Injection, leakage and permission tests have been run and the fixes retested.
- Answers cite approved sources and decline when the sources are silent.
- Handoff to a person works and carries context.
- Logging, alerts, cost limits and a rollback are in place.
- A named owner reviews the logs on a schedule.
Regression testing and monitoring after go-live
Rerun the set on every change
Three things change the behavior of an LLM application: the prompt, the documents and the model. A small edit to a prompt can break answers that worked yesterday. A new document can contradict an old one. A model update can shift tone, length or refusals without any change on your side. Treat each of these like a code change and run the full set before it ships. Automate the run so that it takes minutes and sits in your delivery pipeline.
Compare results with the previous version, not only with the threshold. A small drop in one category is a warning even when the overall score looks fine. Pin model versions when you can, and plan a test cycle before moving to a new one.
Log what happens in production
Record the question, the retrieved documents, the answer, the model and prompt version, the response time and any user feedback. Mask personal data in the logs and set a retention period. Without logs you cannot investigate a complaint or add a failure to the set.
Supervise with a routine
- Review a random sample of conversations every week, plus all those flagged by users or by automatic rules.
- Watch the rate of handoffs, refusals and negative feedback for sudden changes.
- Track cost and response time, and alert on abnormal spikes.
- Look for new topics that users ask about and the documents do not cover.
- Add each confirmed failure to the evaluation set and rerun.
This loop is what turns a launch into a product that improves. It needs an owner, a rhythm and a few hours of attention, not a big team.
Common mistakes
- Testing only with friendly questions. The team asks what it knows the system handles. Users do not.
- Writing the set after the demo is built. Quality criteria then get shaped around what already works.
- Judging with one score. A single grade hides unsupported claims and wrong sources.
- Trusting the LLM judge blindly. Without calibration against humans it only reflects its own biases.
- Putting security in the prompt. Permissions and data access belong in the system, not in a sentence the model can be talked out of.
- Changing the prompt without a rerun. Silent regressions are the most common failure in live assistants.
- Setting thresholds after the results. The launch date then decides what counts as good enough.
- Launching to everyone at once. A staged release is cheaper than a public correction.
- No owner after go-live. Documents age, questions drift and nobody notices.
Where to start
If you want to test an LLM application and have no process yet, begin small this week. Collect a few dozen real questions with expected answers. Write four to six scoring criteria. Run the set, read every failure, and set your thresholds. Then add the security cases, a logging plan and a staged release. That is enough to move from opinion to evidence.
Support helps when the stakes are high or the team is stretched. An experienced AI consultant can build the first set with your experts, calibrate the judge, run the security tests and set up the monitoring loop. See how we approach AI strategy, custom AI applications and our wider AI services.
Next step: request the free audit and talk to a senior consultant. We review your use case, your documents and your risks, and tell you what to test before launch. Book a discovery call.
FAQ
How many test questions do we need?
There is no magic number. Start with a few dozen that cover every category and every risk, then grow the set with each real failure. Coverage matters more than size. A small set that includes edge cases and adversarial inputs is worth more than a large set of easy ones.
Can an LLM judge replace human reviewers?
No. It can scale the review, but it should be calibrated against human grades and rechecked regularly. Keep people in charge of the critical categories, the borderline cases and the rubric itself.
How often should we rerun the tests?
Before every change to the prompt, the documents, the model or the tools, and on a regular schedule even when nothing changed. Providers update models, and your documents evolve. A scheduled run catches drift that no one decided to introduce.
What is the most important security test?
Check that permissions are enforced outside the model, then test prompt injection through both user messages and retrieved documents. These two cover the most serious failures: exposing data and acting on hostile instructions. Add action limits if the assistant can do more than answer.
When is an assistant ready for production?
When it meets the thresholds you set in advance, passes every security test, hands off to a person reliably, and has logging, ownership and a rollback in place. If one category is weak, release with that category blocked rather than delay everything.
Do we need to monitor if the tests passed?
Yes. Tests only cover the questions you thought of. Real users bring new topics, documents age and models change. Monitoring finds what the set missed, and each finding makes the next test stronger.