Why evaluation is the hardest problem in AI systems

In traditional software, evaluation is mostly about verification. You define what correct behaviour looks like, and you check that the system adheres to it. There's a clear relationship between input and output. Even when systems are complex, they are still fundamentally deterministic.

AI systems don't give you that, at least not consistently.

Don't get me wrong, there are cases where correctness is well-defined. Like if you're extracting structured fields from an invoice, there's a right answer. If you're classifying whether an image contains a pedestrian, you can label it as true or false.

In these settings, evaluation can look very similar to traditional systems: you measure accuracy, track errors, and iterate. And it is a necessary part.

But a lot of modern AI systems don't operate in that space.

The moment you move into open-ended tasks like summarisation, reasoning, multi-step tool use, or anything involving natural language interaction where the output is largely subjective, the idea of a single correct output starts to break down. There are multiple valid answers, subtle trade-offs, nuances, and failure modes that don't show up as cleanly incorrect.

So instead of verifying behaviour, you start sampling it.

The system doesn't have one correct response, it has a range of possible ones. Some are better, some worse, some are slightly off in ways that are hard to formalise. You can build test sets and benchmarks, but they only ever give you a partial view.

That's not to say that you should not try! However, it still looks structured. Quantified. Measured. But it's no longer the same kind of certainty.

Once a system starts making decisions, like choosing tools, retrieving information, chaining steps together, you're no longer evaluating a response. You're evaluating a sequence of actions unfolding over time. And that changes the nature of failure.

A mistake at the end is easy to see. A mistake at the beginning is harder. It gets absorbed into everything that follows. By the time you notice something is wrong, the cause is buried somewhere upstream, mixed into a chain of otherwise reasonable decisions.

You don't just have incorrect outputs. You have incorrect trajectories.

Two answers can look equally fine on the surface, but one might have taken a clean path while the other wandered through unnecessary steps, made questionable decisions, or relied on fragile assumptions. From the outside, they look the same. From the inside, they are very different systems. That difference is where evaluation becomes difficult.

There's another complication, which is harder to admit. A lot of the time, you don't actually know what “correct” looks like. Not precisely anyway, and often that is the main problem.

Especially in systems where, the data is incomplete, the context matters, multiple answers could be acceptable, or correctness is entangled with safety, usefulness, or tone. This happens in legal, editorial cases.

So you reach for proxies. Accuracy on a dataset. Tool selection metrics. Retrieval quality. They help, but they don't capture the full experience of using the system.

You can have something that scores well and still feels wrong.

This is also where engineering quality starts to show in a different way.

In more traditional systems, you can sometimes get away with ambiguity. A loosely defined interface, an implicit assumption, a bit of inconsistency in how things are structured.

In AI systems, those things don't stay hidden. They turn into behaviour. A vague tool interface becomes incorrect tool usage. Missing constraints or context become hallucinations. Inconsistent structure turns into brittleness that only shows up under certain conditions.

Nothing fails cleanly. It degrades. And because it degrades probabilistically, it's harder to see, harder to reproduce, and easier to dismiss. This problem is big and most new to the new age of AI tools will only feel it once they start getting multiple complaints which their engineering teams simply cannot easily address.

Most teams still try to solve this the same way they would in traditional systems. Evaluate before deployment. Run the test suite. Check the metrics. Then ship.

But the problem is that you can't meaningfully enumerate the behaviour of these systems upfront. You can only sample it. Which means the system you evaluated is, at best, an approximation of the system that exists in production. The real system only fully emerges once people start using it.

This is where runtime analytics stops being a “nice to have” and becomes something else entirely.

Because once the system is live, the most useful signals aren't coming from your carefully constructed datasets. They're coming from what actually happens.

Where people get stuck. Where the system loops. Where it takes too long to arrive somewhere simple. Where it produces something that looks right but isn't.

These aren't edge cases. They're the system expressing itself under conditions you didn't think to or could not really test.

Offline evaluation tells you what the system can do.
Runtime analytics tells you what it actually does.

And the gap between the two is where most of the work is.

At some point, the distinction between evaluation and observability starts to blur.

You're no longer only checking outputs. You're trying to understand behaviour over time. You care about how decisions are made, not just what they produce. You start looking at trajectories, not just endpoints.

And to do that, you need visibility into the system's internals. What it chose, what it saw, how it moved from one step to the next. Without that, you're left guessing. Why is it not doing what it is supposed to do?

You can't fix what you can't see, and you can't evaluate what you don't understand.

The teams that get through this phase don't treat evaluation as something that happens once. They treat it as something that continues. Production isn't the end of evaluation. It's where evaluation becomes real.

The data changes. The usage patterns shift. New failure modes appear. And gradually, the system you thought you had becomes the system you actually have.

In traditional software, evaluation gives you confidence before you ship. In AI systems, evaluation gives you a partial view and sadly sometimes a misleading one.

Confidence is something you build after the fact. Slowly, by observing, adjusting, and feeding what you learn back into the system.

Which makes evaluation feel less like a step in the process. But more like the process itself.