- Home
- AI development
- LLM evaluation
LLM evaluation
LLM evaluation: test the model on your own cases before you rely on it
Evaluation means running candidate models on the same set of your real past cases and scoring the answers against what you would accept. We build the test set, write down what pass and fail mean, compare Claude, Gemini and an open-weight model, and give you the accuracy and cost per task of each.
- Prices published on the site
- First working version in two weeks
- Support month to month, cancel any time
A model is good for your business only if it passes your cases
Evaluation tests every candidate model on the same cases with the same scoring. We take 50 to 200 of your past cases, such as real invoices, emails or support tickets, and write down what counts as pass and fail for each kind.
Then we run candidate models over the set, usually Claude, Gemini and an open-weight model such as Llama or Qwen, and choose by accuracy and cost per task. A cheaper model that passes can beat a stronger one that costs several times as much.
How a model's answers get checked
A real evaluation mixes them. Code checks what code can check, a second model grades what needs reading, and a person looks at a sample of both.
Spot checks by a person
The first check everyone doesSomeone who knows the work reads a handful of answers and decides whether they look right. It needs no setup and fits a first prototype.
- Catches wrong numbers and formats: Low
- Catches tone and judgment errors: High
- Cost per run: High
Best for: A first prototype, and the final read before launch.
A test set with automatic checks
The backbone of an evaluation50 to 200 past cases with the known right answer. A script compares each answer with it: the amount, the date, the category, the format.
- Catches wrong numbers and formats: High
- Catches tone and judgment errors: Low
- Cost per run: Low
Best for: Anything with a right answer: extraction, classification, routing.
Grading by a second model against a written rubric
For free-text answersA second model reads each answer next to a written rubric, such as "mentions the refund window, promises no date, under 120 words", and scores it. A person checks a sample of its grades.
- Catches wrong numbers and formats: Medium
- Catches tone and judgment errors: Medium
- Cost per run: Medium
Best for: Replies, summaries and explanations where wording matters.
We start with the test set, add model grading for free-text answers, and keep a person on a sample. Ratings compare the three methods with each other.
What an evaluation project hands over
A test set
50 to 200 of your past cases, each with the answer you would accept, stored in your repository so anyone can rerun it.
Pass and fail in writing
One page that says what counts as correct for each kind of case: the amount matches to the cent, the reply promises no delivery date. Your team decides what is acceptable, which questions go to billing staff for example, and signs it off before the first run.
A comparison
Each model scored on the same set, with accuracy, cost per task and response time side by side, and the cases each one misses.
A rerun command
One command rescores any model or prompt version. Run it on every prompt change and every new model version, because providers update and retire models and a model that passed last quarter can answer differently today.
When a test set pays off, and when spot checks will do
Worth a test set when
- The system writes text customers see or changes records, so a wrong answer costs money or trust
- You are choosing between models, or a cheaper model might do the job
- The prompt keeps changing and nobody wants to reread 50 outputs after each edit
- A provider announces a model retirement and you need to know the replacement is safe
Spot checks are enough when
- A pilot used by two people who review every output anyway: they are the test
- A one-off task where you read every result
- You have no past cases yet: collect 50 in a spreadsheet first, and we start from that
Priced by the number of tasks you test
One workflow running end to end, or one internal tool.
Several systems made to work as one: integrations, roles, reports.
An evaluation suite for one task, with the test set, the scoring checks and the comparison report, usually lands in one process. Evaluation across several tasks, or a suite wired into your release process, is connected systems. Running candidate models costs tokens or GPU time, billed by the provider; we estimate it in the written review.
Care $300 to $500 a month. Active $750 to $1,500 a month. AI model usage, GPU time and hardware are paid to the provider, on accounts in your name.
Questions about testing a model
How many test cases do we need?
Between 50 and 200 for most business tasks. With fewer than 50, one odd case swings the score. We take the cases from your real history and include the rare, awkward ones, because those cost money.
Can the model just check its own answers?
A model grading its own answer tends to miss the errors it made. A second model grading against a written rubric works better, which is the third method above, and a person checks a sample of its grades. Numbers, dates and formats are checked by code, which gives the same verdict on every run.
Which models do you compare?
Claude and Gemini by default, an open-weight model such as Llama or Qwen when the data must stay on your server, and OpenAI if your company already uses it. We choose by accuracy and cost per task on your cases, and a cheaper model wins when it passes.
How often do we rerun it?
On every prompt change and every new model version. Providers update and retire models on their own schedule, so a model that passed last quarter can answer differently now. A rerun takes minutes. MLOps covers running the check on a schedule after launch.
What happens when a model fails some cases?
We look at the pattern in the failures. Usually the fix is a clearer prompt, a lookup in your documents through retrieval, or sending that type of case to a person. Fine-tuning comes last, and the same test set shows whether it helped.
Related services
- LLM fine-tuning Train a model on your own examples
- MLOps Run, monitor and update models in production
- AI agents Agents that finish a task, with your sign-off
- Local AI setup AI on hardware you own, set up on-site
- Generative AI Drafts, summaries and documents from your data
- AI development Every AI service we build, on one page
Tell us what's slowing you down.
You get a free written review by email within two business days.