# AI Agent Verifier: Four of Five Builds Needed a Fix

> Agents write two things: code your tests run, and claims about it nobody checks. A batch where a verifier caught four bad builds, the prompt and its limits

**Short answer:** a coding agent's tests check its code, not what it says about its code, and that second part is where most of its confident mistakes end up. Give a separate agent, in a fresh context and on a stronger model, the job of tracing every claim back to the code or source behind it, and tell it to fail when unsure. In one batch of five tasks, that verifier sent four builds back for a fix before any of them reached the main branch, and let the fifth through.

An agent hands you two kinds of output. One is code, which your tests execute. The other is everything it says about that code: the page copy, the README, the PR description, the figures in a table, the line that says "tests pass". Nothing executes that second kind. It costs the model nothing to write, and it reads exactly as confident when it's wrong.

Most review setups I've seen, mine included until recently, point the second agent at the first kind. Here's what happened when I pointed it at both.

## A verifier sent four of five builds back, and two had tests for the bug

This evening I ran five independent tasks in parallel on the repo behind my calculators. Each one was built by an agent on Sonnet in its own git worktree, then checked by a separate agent on Opus that could read the branch but not change it. If the verifier found anything serious, a third agent fixed it on the same branch.

| Task | What the verifier caught |
| --- | --- |
| Page publishing an S&P 500 annual returns dataset | The text said three columns were copied "without recalculation". The extractor computes one of them from index levels and divides inflation by 100. Also a JSON-LD licence that would read as covering someone else's data, and a placeholder DOI in the BibTeX. |
| November rate block for the I bond calculator | Semiannual inflation printed 100 times too small, and a 12-month comparison that ignored the three months of interest you lose cashing in early (31 CFR 359.7). |
| Public MCP server for the calculators | The I bond fixed rate shown 100 times too small: a decimal printed with a % sign. |
| Drafts for Show HN and Medium | Wrong posting time and a wrong reading of the Hacker News rule on promotion. |
| A Spanish data page tracking monthly Euríbor and CPI | Nothing serious. It went through without a fix. |

Each of the four has a fix commit on its branch, five to ten minutes after the build. The part that worries me is the tests. The I bond build shipped tests for the 12-month comparison and its break-even rate, the exact area with the cash-in bug. The MCP build shipped a test for the I bond tool, the one with the wrong rate. Both passed. The agent wrote the code and the tests from the same wrong assumption, so they agreed with each other.

The I bond one is the one I'd have hated to ship. A rate shown at a hundredth of its value on a page about savings is the kind of mistake a reader screenshots.

I need to own one more data point. The first drafts of this post got the batch wrong twice: once saying three tasks had passed clean, once saying the fifth had left an empty branch. Both times I'd read the branches while agents were still committing and written down a stale reading as fact. Both times the reviewer I'd set on the draft re-ran `git log` and failed it. The post you're reading is the corrected version.

## The instruction is six lines

The setup above has a lot of plumbing, but the part doing the work is the verifier's brief. Trimmed to what matters, it says:

```text
Your job is to find what is wrong, not to approve.
For every claim the output makes about how something works, find the code
or the cited source that makes it true. A claim nothing supports is a failure.
Recompute at least two worked examples and compare them with the code.
Report what was asked for and is missing, and what was done that nobody asked for.
When unsure, fail.
```

Keep the last line. Without it, doubt resolves to "probably fine", and that's the default you're paying a second agent to override.

## A second model shares blind spots, so treat it as a filter

The obvious objection is that you've asked one language model to mark another's homework. Same family, similar prompts, a real chance they agree on the same wrong answer. That's correct, and it's why I don't treat a pass as proof.

What reduces it, in order of cost: a fresh context, so the verifier never sees the reasoning that produced the work, only the brief and the finished branch; a different and stronger model for the check than for the build; and moving each class of bug the verifier finds into a deterministic test, so next time it's caught without asking anyone's opinion. The I bond fix already did a little of that: it added a render test for the block text.

What it doesn't reduce is the failure you never see. I can tell you the verifier caught four builds and passed one. I can't tell you whether that one, or any of the fixed ones, still has something it waved through, because measuring that means planting known bugs and counting the misses, and I haven't done it.

## I haven't measured the cost, and one batch is not a rate

I haven't measured the tokens. Each task costs at least one build call and one verify call; this batch also paid for research, four fixes and one final review across the batch. Everything was committed within a quarter of an hour of the first build.

The result is one batch on one repo. It shows the approach catches things my tests don't. It doesn't give you a rate to plan around.

## I've published it as two agent skills

The run above used `agency`, a skill I'd been using privately in Claude Code. Today I cleaned it up and published it as [skeptic-agents](https://github.com/garbarok/skeptic-agents), MIT licensed. `skeptic-build` runs a batch through research, build, verify and fix; `skeptic-research` looks at one question from several angles in parallel and writes a sourced plan.

Publishing changed three things. Correctness and the repo's written standards are now reviewed separately and reported side by side, an idea I took from the `code-review` skill in [Matt Pocock's skills](https://github.com/mattpocock/skills), which are the ones I use most. Every task now carries a brief with acceptance criteria and an out-of-scope list, so the verifier has a contract to check against. And the role prompts live inside each skill, because the installer copies only the skill's folder and my private version depended on files that never left my machine.

One caveat before you install it. Everything in this post ran in Claude Code's workflow mode on the old version. The skill also has instructions for agents with subagents and for agents without them; the install works on those, but I haven't watched either run a real batch.

```bash
npx skills@latest add garbarok/skeptic-agents
```

Claude Code, Codex and Copilot CLI can install it as a plugin instead; the README has those commands.

## Start with one document an agent wrote

Take the last thing an agent wrote for you that makes claims (a README, a PR description, a page with numbers) and give it to a fresh session with the six lines above and read access to the code. Fix whatever it can't back up. If it finds something, that's your case for adding the step to how your team reviews agent work.

**Related reading:** the verifier is one layer of the [harness around an agent](/en/blog/harness-engineering-why-your-agent-fails/), and the way I split work across subagents comes from [how I build Rydger](/en/blog/claude-code-subagents-stories-worktrees/).

## Sources

- [garbarok/skeptic-agents](https://github.com/garbarok/skeptic-agents), version 0.1.0, MIT licence.
- [mattpocock/skills](https://github.com/mattpocock/skills), whose `code-review` skill is the source of the two-axis review.
- The batch: five tasks run on 27 October 2026 in the private repo behind my calculators. Evidence from one run, not a measured error rate.
- Interest penalty on early I bond redemption: [31 CFR 359.7](https://www.ecfr.gov/current/title-31/subtitle-B/chapter-II/subchapter-A/part-359/subpart-A/section-359.7).

---

**P.S.** If you've measured how often an LLM reviewer misses planted bugs, I'd like to see the numbers. Tell me on [LinkedIn](https://www.linkedin.com/in/oscargallegoruiz).