Nov 2026 . 13 min read

my harness works, i promise. why evals?

I have said that sentence out loud. Not in those exact words, but close. We built an agent, wired up the tools, opened the chat, asked it the thing we built it for, and it did the thing. So we told people it worked. And it did work. Once. On a laptop. On the one question somebody happened to type, with the one model that happened to be selected, on a morning when the API was not having a bad day.

Quick word on "harness", since it is in the title. The model is the part that reads and writes text. The harness is everything we build around it: the loop that sends the conversation to the model, runs the tools it asks for, feeds the results back, and decides when to stop. Most of what goes wrong with an agent goes wrong in there, not in the model.

Then the bug reports came in, and they had a very particular flavour. Almost none of them said "the agent is broken." They said "it told me it was done but there was no file." "It worked yesterday." "It works with the other model." Every one of those sentences is about something we had never actually measured. We had tested whether the agent could do the task. We had not tested whether it does do the task, again and again, the way real people use it.

This post is what we learned closing that gap, mostly the hard way. None of it is new to the people who have been doing this for years. It was new to us, and I would have saved a lot of time if someone had written it down for me like this.

what we mean by an eval now

The word gets used for so many things that it stops meaning anything, so here is what we settled on. An eval is a question about the agent's output that has a clear answer, that you can ask automatically, over and over, on lots of inputs.

"Did it call the calendar tool?" is an eval. "Did the date in the invite match the date the user asked for?" is an eval. "Did the reply contain any text at all?" is an eval, and you will see later why that one turned out to matter far more than it sounds like it should. "Helpfulness: 0.82" is not really an eval. It is a number, and I was never once able to say what to change in the agent when it moved from 0.82 to 0.79.

There are two ways to answer the question.

Both should end in pass or fail, not a score out of ten. A scale sounds richer, but nobody can explain the difference between a 6 and a 7, two people rarely agree on one, and everything slowly drifts to the middle. Pass or fail forces you to decide what you actually mean by "good", which turns out to be the hard and useful part.

we started by reading, not measuring

This is the part we skipped at first, and the part I now think matters most. Before you decide what to measure, you go and look at what actually went wrong. Not a dashboard. The raw conversations, one by one.

The process is almost embarrassingly simple:

  1. Write one note per failure. Read a conversation or a bug report. Write down the first thing that went wrong, in your own words. "Said it sent the email, it didn't." Next one.
  2. Group the notes. Once you have a pile, sort it into groups and name each group. The groups will not be the ones you would have guessed.
  3. Count. The biggest group is what you fix first, and what you write your first eval for.
Notes, then groups, then counts. The notes here are made up, but the shape is what we actually got: one pile was much bigger than the rest.

When we did this on a few hundred real bug reports, I expected the biggest group to be the model making things up. It was not. It was the agent saying it was done when it was not. A green tick next to a step, no answer underneath. "File created", no file. A long task that simply stopped halfway with no message. And the reason that group had grown so big without anyone noticing is that our own logs counted every one of those turns as a success, because nothing threw an error. If you only watch error rates, the thing users complain about most is invisible to you.

The other thing reading taught us is that you cannot write down what "good" means in advance. Every time we read another batch, we found a kind of failure we had not thought to define. That is normal. Treat your criteria like code you keep editing, not a document you write once.

it passed once. that is not a test.

This one changed how I talk about agents. Models are not deterministic in practice, and agents are worse, because one small wobble early on, a different tool picked or a slightly different search, sends the rest of the run somewhere else entirely. So "I tried it and it worked" is a sample size of one. Rerun the same request a few times and you will often see one of the runs quietly take a different path.

The honest way to measure this is to run the same task several times and look at two numbers.

The same 20 runs, read two ways. 16 of 20 cells are green, so pass@1 is 80%. Only 2 of the 5 tasks are green every time, so pass^4 is 40%.

You can feel this with some quick arithmetic. If a task succeeds 90% of the time, and runs are independent, the chance of four successes in a row is 0.9 × 0.9 × 0.9 × 0.9, about 66%. At 80% per run it is about 41%. Real runs are not perfectly independent, so take it as a feel rather than a forecast, but the direction never changes. Reliability drops much faster than the single-run number makes you think.

our test set had a margin of error

The next mistake was quieter. We had a test suite of 36 cases. Run model A, run model B, A gets 80%, B gets 85%, switch to B. It felt like a decision backed by data. It mostly was not.

A pass rate on a set of test cases is an estimate, and it comes with a margin. You do not need to remember the formula. The only thing that matters is that the number of cases, n, sits under a square root, so four times as many cases only halves the margin:

range ≈ 1.96 × √( p (1 − p) / n ) p = 0.80, n = 36 → ± 13 points p = 0.85, n = 36 → ± 12 points p = 0.80, n = 1000 → ± 2.5 points

With 36 cases, the honest version of "A scored 80%" is "A scored somewhere between about 67% and 93%". B's range sits almost entirely on top of it. We had not learned which model was better. We had learned that we needed more cases.

The same 5 point gap between two models. With 36 cases the ranges overlap and the gap could be noise. With 1000 they do not.

A few things that helped us once we took this seriously:

Rough rule we use now: a few dozen cases is enough to catch something that broke badly. Picking between two decent models needs hundreds, and telling apart a three or four point difference needs on the order of a thousand.

the judge is a model too

As soon as a model grades another model, you have a new thing that can be wrong. A judge is really just something that sorts answers into pass and fail, so you check it the same way you would check any sorter: against answers where you already know the right pile.

The trap is measuring it with plain agreement. Say 5 out of 100 of your agent's answers are actually bad. A judge that lazily says "pass" to everything agrees with your human labels 95% of the time. It also catches none of the bad answers.

Two judges, 95% and 96% agreement. One is useless and one is useful. You only see it if you count caught-bad and passed-good separately.

So track two numbers instead of one: how many of the truly bad answers the judge catches, and how many of the truly good ones it lets through. To get them, someone who knows the domain labels a set of outputs by hand, a hundred or two per failure type with plenty of both passes and fails, and you check the judge against those labels. A sensible bar is roughly nine out of ten on both before a judge is allowed to block anything. The fastest way to get there is to take the expert's written reasons for each label and feed a few of them back into the judge's prompt.

Judges also have habits worth knowing: they tend to prefer whichever answer they see first, they prefer longer answers, and they go easier on outputs from their own model family. None of that makes them useless. It means you ask narrow yes/no questions, you check them against people, and you use code instead whenever code can answer the question.

the mistakes we kept seeing

This is the list I wish had been pinned above my desk from day one. Every one of these showed up in our own reports, most of them more than once, and every one has a cheap check that would have caught it.

1. "done", with nothing behind it

The run ends cleanly, the UI shows a tick, and there is no answer, no file, or a message claiming an action that never happened. Nothing errored, so it gets logged as a success.

Check: the final message is not empty, and anything the agent claims to have done (a file, an email, a ticket) actually exists afterwards.

2. works on one model, breaks on another

Providers do not accept the same things. Every tool comes with a small description of what inputs it takes, and one of ours used a pattern-matching rule that one provider simply refused to accept. Every message on that model failed before the agent did anything. Different people reported the same error for weeks. It finally went away because those tools were replaced, not because anyone added a test, which means nothing stopped it from coming back. Other versions of the same mistake: a message field one API refuses, or a tool call left in the history without its result.

Check: for every model you offer, send one message with the full tool set attached, and do one short tool round trip. Run it every time the model list or a tool changes.

3. forgetting what happened ten minutes ago

A model can only look at a limited amount of text at once. Long sessions go past that limit, something gets cut or squashed to make room, and the agent asks for the file you already gave it or starts over on work it already finished. Good harnesses design for this on purpose: big tool results get written to files instead of sitting in the prompt, and old history gets summarised when space runs out, with the full version still recoverable.

Check: take a real long conversation, replay it up to a late turn, and test only the next answer. Much more reliable than simulating a whole chat with a fake user.

4. picking the wrong tool

Give an agent a hundred tools and it will sometimes search the web when the answer is sitting on the company drive. Showing only a small, relevant set of tools per turn and letting the model search for more helps, but then the search itself can miss.

Check: for realistic requests, assert the right tool was called, and separately that it was even offered. Those are two different bugs with two different fixes.

5. a tool that quietly returns half the data

One of our tools returned the first 50 results with no hint there were more. Ask about the 51st thing and the agent would tell you, confidently, that it did not exist. The model did nothing wrong. It was lied to.

Check: tool outputs say when they were cut short, and there is a test with more items than fit on one page.

6. dates

The wrong day of the week. "This week" shifted by a week. A timezone applied twice. Models are bad at calendar arithmetic and very sure of themselves while doing it.

Check: give the agent today's date and the user's timezone explicitly, and keep a small set of date questions with exact expected answers. Pure code, no judge needed.

7. sources that do not exist

A report that cites an analyst study nobody ever wrote, or a table of numbers that looks like analysis but was invented on the spot.

Check: every source the answer cites must appear in what the tools actually returned. Code does most of this; a checked judge does the rest.

8. doing the risky thing without asking

Sending the email instead of drafting it. Overwriting a file instead of adding to it. Messaging the wrong person.

Check: in a test environment, assert that every write action went through an approval step, and that the target was the one the user meant.

9. tests nobody runs

We had an accuracy suite that checked whether the agent called tools by their exact names. Months earlier, the tools had been renamed. The suite was still checking the old names. Nobody noticed, because it only ran when someone remembered to run it by hand. If a test does not run by itself, sooner or later nobody runs it at all.

Check: the eval set runs automatically before any release and before any model change. Same cases, a few repeats each, error ranges, and cost.

cost is part of the answer

Most eval reports I come across are about quality only. That leaves out half the question, the half that finance eventually asks you about. A fancier setup that costs several times more for a barely different pass rate is not an upgrade. You only see that if you report both numbers side by side.

The number we care about now is cost per successful task: total spend on a batch of runs divided by the number of runs that actually passed. A cheaper model that fails half the time and needs a retry is not cheaper. A clever routing setup that saves money on paper only counts if the pass rate holds, and it has to hold on your own kind of traffic, not on someone else's benchmark.

A sneaky one we only found by tracking cached tokens. Providers give you a big discount when the start of a request is identical to the previous one, because they can reuse work they already did. That reuse stops at the first thing that changed. So if something that changes every turn (the current time down to the minute, or the list of tools for that turn) sits before the conversation history, the whole history is paid at full price on every turn. It never shows up as an error. It just shows up on the bill.

what the tools gave us, and what they did not

What none of them do is the reading. Every tool will happily hand you a dashboard of generic metrics on day one. None of them could have told us that our biggest problem was the agent saying "done" when it was not. We found that by opening reports and writing notes.

so what do you actually do on monday

  1. Read 50 to 100 real failures. One note each, first thing that broke. Group them. Count them.
  2. Write one pass/fail check for the biggest group. Code if at all possible. Ten to twenty cases taken from those real failures.
  3. Run each case a few times, on every model you offer. Report pass@1 and pass^k, with the error range next to every number.
  4. Log tokens and cost per run. Report cost per successful task, not cost per call.
  5. Every new bug report becomes a test case. That is the whole loop. Read, group, check, repeat.

I still think our harness works. The difference is that now we can say how often, on which models, with how much uncertainty, and at what cost per answer that actually helped someone. That is a much less exciting sentence than "it works, I promise". It is also the only one I would believe from somebody else.

← Back to writing Home