Stop Guessing, Start Measuring: How Benchmark-Driven Development Makes AI Agents More Reliable

When you have data, guesswork is minimized. How benchmark-driven development lets us move fast with AI coding agents without breaking things.

The change that looked like a clear win

We tried replacing a slow document-parsing library with one that was roughly seven times faster.

The quick checks looked good. We were ready to ship.

Then we ran our benchmark.

For every question in a corpus of real documents, it checks whether the text needed to answer that question survives parsing. With the new parser, it survived only about 78% of the time. With the old one, about 97%.

We had made the system faster by quietly making it worse. Roughly one in five answers could have suffered, and our spot checks hadn’t caught it.

So we used the same benchmark to fix the problem, one measured change at a time. The final parser ran twice as fast as the original and met every quality target we had set. The full story is in How we replaced Docling with pdfplumber.

Speed tells you how fast a system runs. Measurement tells you whether it still does its job.

Over the last few months, nearly everyone on our team at PipesHub has started working this way, mostly alongside AI coding agents. This post is about the method: what we do, why it works, what can go wrong, and how your team can start.

What you’ll learn

• How benchmark-driven development lets AI coding agents make changes without silently breaking existing behaviour.

  
  

• How to build an evaluation harness with trusted ground truth and a measured baseline.

  
  

• Why flawed scoring rules, noisy runs and benchmark leakage can produce misleading results.

  
  

• How to stop agents from gaming their own evaluations.

  
  

• When to start with a benchmark, and when to begin with error analysis of real outputs.

Why AI coding agents need measurable feedback

AI coding agents have made producing code dramatically cheaper. They can draft a refactor, replace a library or build a pipeline stage in hours.

But writing a change is not the same as knowing it is correct. Agents can now produce changes faster than people can confidently evaluate them. That gap, not typing speed, is the real bottleneck.

Without a reliable check, an agent can produce code that looks finished, passes a few spot checks and quietly breaks something important. The Claude Code best practices put it plainly: give the agent a check it can run, because that is “the difference between a session you watch and one you walk away from.”

A benchmark closes that gap. It has two parts:

• Ground truth: a fixed set of cases with known correct answers, such as human-labelled documents, a repository with a written answer key, or questions with verified results.

  
  

• Evaluation harness: a script that runs your system on every case and reports a score.

Together they give the agent a measurable target, let it test changes repeatedly and make regressions visible before they reach users.

This idea has strong backing. Andrej Karpathy argues in Verifiability that “Software 2.0 easily automates what you can verify.” Jason Wei calls it verifier’s law: how easily AI can solve a task depends on how easily the task can be checked. Simon Willison described Shopify’s Liquid engine getting 53% faster through agent-run experiments: “If you provide an agent with a benchmarking script ‘make it faster’ becomes an actionable goal.”

Most writing on evals is about grading LLM answers. Our experience is that the same discipline pays off for ordinary engineering too: parsers, code indexes, query pipelines and agent tools.

The goal isn’t to measure everything. It’s to make the important engineering decisions verifiable.

The 7-step benchmark-driven development loop

We wrote one rule into our team’s decision log: measure before changing parsers or algorithms. Build an evaluation harness with a baseline first, then judge fixes by the numbers. The alternative we rejected was “fix by inspection.” In practice, the rule becomes seven steps.

1. Build the ground truth first. Before touching the code, decide what “correct” means and write it down as data. The answer key records what is true, not what the current code happens to output.

   
   

2. Measure the baseline. Run the current system, and any alternative you are considering, on the same ground truth. This step is often the most surprising, because it shows where you really are.

   
   

3. Set targets after the baseline. Targets picked before you have numbers are guesses. Once you see the baseline, agree on concrete bars, such as “within 5 points of the current system.”

   
   

4. Make one change. Usually the agent writes it. One change at a time keeps cause and effect clear.

   
   

5. Run a fast dev gate. A small, representative slice of the benchmark that runs in about five minutes. The full set runs less often, on a frozen snapshot of the code.

   
   

6. Log the decision. Keep the change only if the score improves with no regressions, and write down the evidence. Wrong calls are marked as superseded, never deleted, so the next person or agent session can see why.

   
   

7. Ship only when the full suite passes. Fixes stay local until every benchmark is green. Then they go into a pull request together with the numbers.

Figure 1. The benchmark-driven development loop. Steps 4 to 6 repeat many times a day. A change that doesn’t improve the score, or breaks something else, is reverted before it reaches a pull request.

The inner loop is small enough to sketch in a few lines:

baseline = run_benchmark(dev_set)
for change in proposed_changes:
    apply(change)
    result = run_benchmark(dev_set)
    if result.score > baseline.score and not result.regressions:
        log_decision(change, before=baseline, after=result)
        baseline = result
    else:
        revert(change)

baseline = run_benchmark(dev_set)
for change in proposed_changes:
    apply(change)
    result = run_benchmark(dev_set)
    if result.score > baseline.score and not result.regressions:
        log_decision(change, before=baseline, after=result)
        baseline = result
    else:
        revert(change)

baseline = run_benchmark(dev_set)
for change in proposed_changes:
    apply(change)
    result = run_benchmark(dev_set)
    if result.score > baseline.score and not result.regressions:
        log_decision(change, before=baseline, after=result)
        baseline = result
    else:
        revert(change)

AI agents do much of the work at every step. They generate test fixtures, write the scoring scripts, run the benchmarks, read the failures and propose fixes. The human job shifts toward deciding what “correct” means, setting the bar and checking that the measurement is honest.

What benchmark-driven development changed for our team

Across the projects where we have used it, three things changed.

We move faster, not slower

The usual worry is that benchmarks are overhead. In our experience the opposite is true. An agent with a check can work through a list of fixes on its own, and code reviews stop being debates about opinions.

On one project, a review had estimated each group of fixes at about a day. Once a one-minute harness existed, each group took under an hour, and the harness itself took about half an hour to build. Kent Beck describes the shift well: “The whole landscape of what’s ‘cheap’ and what’s ‘expensive’ has all just shifted.” Building a benchmark used to be the expensive part. Now the agent writes most of it.

We catch what spot checks miss

Each project surfaced problems that would otherwise have shipped: a memory spike on one unusually long document, code links pointing at the wrong file because two files shared a name, and a cloud safety filter silently refusing some of our agent’s requests.

Some problems only appear in the real system. One storage limit was invisible to every offline test and showed up only in a full end-to-end run. That is why we always finish with one.

The data rejects ideas we were sure about

This is the part we value most. A benchmark doesn’t only confirm good ideas; it stops bad ones before they cost anything.

The full methodology and numbers are in our code graph benchmark and PDF parsing benchmark posts.

Your AI evaluation benchmark can be wrong too

A benchmark can be wrong in all the ways code can be wrong. Treat a surprising result as a possible benchmark bug before you treat it as a product bug.

• Ground-truth answers can be wrong. Recent research found errors in roughly half of the reference answers of a widely used text-to-SQL benchmark. When we audit our own failures, a real share turn out to be bad labels or ambiguous questions, not system mistakes.

  
  

• The metric changes the story. The same set of answers can score very differently under two reasonable scoring rules. Choose the metric deliberately, and say which one you report.

  
  

• Scorers have bugs. We have seen an evidence check match digits inside garbled characters, and a scoring rule that was unfair to the system we were replacing. Each time, fix the scorer and re-score every earlier run.

  
  

• Runs are noisy. Two identical runs of an LLM agent can disagree on enough cases to hide a real effect. Repeat runs, use significance tests, and run before and after back to back on the same machine.

  
  

• An improvement that comes too easily deserves suspicion. When a score jumps more than the change could explain, read the individual cases before you celebrate.

Anthropic’s guide to agent evals gives a striking example: fixing the harness and grader alone moved one model from 42% to 95% on a public benchmark. Audit the measuring stick.

How to prevent AI agents from gaming benchmarks

A benchmark gives an agent a goal, and agents are very good at reaching goals, sometimes in ways you didn’t intend. The ImpossibleBench study found a frontier model exploiting tests 76% of the time on tasks that could not be solved honestly. These are the guardrails we use:

• Keep the ground truth read-only for the agent. The agent may change product code and harness plumbing, but never the answer key. Any change to scoring is reviewed by a person and followed by a re-score of every earlier run.

  
  

• Keep a held-out set. A held-out set is a slice of cases the agent never sees while it iterates. Tune on one split, then score the final version once on the held-out one. Results that look good during tuning don’t always carry over.

  
  

• Check for benchmark leakage. Leakage is when benchmark content ends up in the system being measured, for example in prompt examples. We once found prompt examples copied from benchmark questions, replaced them, and added a test that fails if benchmark-specific words appear in prompts.

  
  

• Compare against the simple baseline. For anything clever, ask whether plain text search, the existing library or “do nothing” with the same model does just as well. Run competitors in their best configuration, not their worst.

  
  

• Get corner cases from people. Teammates who know the domain suggest failures an agent would never invent. Add each one to the benchmark with a note on how it is measured.

  
  

• Test end to end at least once. Offline harnesses miss what only shows up in the real system: storage limits, race conditions, sync bugs.

Figure 2. What the agent may and may not touch. The agent works along the top row. Everything that defines “correct” sits below it, read-only, where only a person can change it.

When to use benchmarks, and when to start with error analysis

Not everyone agrees with evals first. Hamel Husain and Shreya Shankar, who have done more than anyone to popularise evals, answer “Should I practice eval-driven development?” with “generally no.” Their reasoning: “Write evaluators for errors you discover, not errors you imagine.” LLM products can fail in endless ways, and you can’t guess them up front.

We think both views are right, for different kinds of work.

• When ground truth exists, build it first. Parser output can be compared with human labels. A code index can be checked against compiler output. A SQL answer can be executed and compared. Here a benchmark is cheap to define and pays off from day one.

  
  

• When “correct” is fuzzy, look at data first. For open-ended behaviour, such as the tone of an answer or how helpful a summary is, start with error analysis: read real outputs, find the failure patterns, then write checks for those.

Figure 3. Where to start. Both paths end in the same place: a set of checks the agent can run on every change.

Even where ground truth exists, your idea of “correct” will change as you look at outputs. Shankar’s research calls this criteria drift. Expect to revise the benchmark, and log why each time.

The opposite failure is worse, though. On one integration we had no access to a live instance of the target system, so the work was checked only against documentation and mocks. Testing against a live instance of a similar system found behaviour the unit tests never showed, such as a metadata table that is empty for read-only users. Mocks tell you what you assumed. Benchmarks against real systems tell you what is true.

How to start evaluating AI coding agents

You don’t need a platform or a dedicated eval team. You need a small set of true answers and the habit of checking against them.

• Pick one change you are about to make and write down what “better” would mean, as numbers.

  
  

• Collect 20–50 cases from real failures. Anthropic’s guide recommends starting here. Favour cases that break things: as Eugene Yan notes, hundreds of labels with only five failures teach you little.

  
  

• Ask your AI agent to build the harness: a script that runs the system on every case and prints a score. Keep it under five minutes.

  
  

• Run the baseline before changing anything. Then set your targets.

  
  

• Make the answer key read-only for the agent, and keep a held-out set.

  
  

• Give the agent the check and let it iterate. Review the before/after table for each change, including what got worse.

  
  

• Keep a decision log and a results page your team can read. A shared page with the verdict on top, before/after numbers, and open gaps ends most debates before they start.

  
  

• Promote the benchmark to a regression gate. Once a capability is reached, the check should stay near 100% on every change after.

Move fast on implementation, be rigorous about proof

AI coding agents can write more code than ever. That makes the ability to verify a change, not just produce it, one of the most important engineering skills a team can have.

Start with one change, a small set of trusted cases and a baseline. Let the agent iterate, but keep the definition of correctness under human control. Measure what improved, what regressed and what the evidence cannot yet tell you.

When you have data, guesswork is minimized. For us, that has meant shipping faster, catching bugs before users do and dropping ideas that only felt right.

Frequently asked questions

What is benchmark-driven development?

Benchmark-driven development uses a repeatable set of cases with known correct answers, plus measurable targets, to evaluate every code change before it ships.

How is an evaluation harness different from a unit test?

A unit test checks one specific behaviour and passes or fails. An evaluation harness runs a broad set of cases and aggregates the results into metrics, so you can compare two versions of a system and see quality trade-offs.

How do you prevent an AI coding agent from gaming a benchmark?

Keep the ground truth read-only, score the final version on held-out cases, check for leakage, have a person review any change to scoring logic, and compare against a simple baseline.

When should a team use a benchmark instead of manual error analysis?

Benchmarks work best when correctness can be defined and checked consistently, as with parser output, code relationships or query results. For open-ended behaviour, start by reading real outputs and finding recurring failures, then build checks around them.

More from PipesHub

Each of these posts applies the method in this article to one system, with full methodology, numbers and limitations:

• How we replaced Docling with pdfplumber: a PDF parser swap judged on 1,228 documents.

  
  

• How accurate is your code graph? A benchmark of 4 tools: code graph accuracy, scale, and whether graph tools help coding agents.

  
  

• Agent loop vs traditional RAG on FRAMES: measuring retrieval and multi-hop reasoning.

  
  

• Do AI agents need a company index? We tested it: the same agent, with and without shared company context.

Further reading

• Hamel Husain, Your AI Product Needs Evals and A Field Guide to Rapidly Improving AI Products

• Hamel Husain and Shreya Shankar, Should I practice eval-driven development?

• Shreya Shankar et al., Who Validates the Validators?

• Anthropic, Demystifying evals for AI agents and Writing effective tools for agents

• Anthropic, Best practices for Claude Code

• Andrej Karpathy, Verifiability

• Jason Wei, Asymmetry of verification and verifier’s law

• Simon Willison, Shopify’s Liquid, 53% faster and JustHTML

• Kent Beck, Augmented Coding: Beyond the Vibes, and Gergely Orosz, TDD, AI agents and coding with Kent Beck

• Eugene Yan, Product Evals in Three Simple Steps

• Zhong, Raghunathan and Carlini, ImpossibleBench

• Jin et al., Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards