What Is Harness Engineering? Why Your AI Agent Fails

Short answer: harness engineering is everything you build around the model to keep an agent honest: how you split the work, where state lives, what checks the critical steps, and what catches the failure before production does. When an agent breaks in prod, look for the missing check before you swap models. The cheapest fix: never let the agent grade its own work. Put a deterministic gate in front of “done”, like astro check && astro build on this blog, so a wrong result fails loudly instead of shipping green.

I renamed a post on this very blog. Mechanical work, the kind you hand to an agent without thinking: change the slug, leave redirects so the old URLs don’t die, check the build passes, deploy. Green everywhere. I called it done.

Weeks later, the post that brought me the most traffic had vanished from Google. Not dropped in ranking: gone from the index. The cause was almost funny. The site runs with trailingSlash: "always", so Google had the URL stored with a trailing slash, and the redirects were written without one. Not a single one matched. Every request fell through to the 404, and the build doesn’t care what Google thinks, so it stayed green the whole time.

Nothing warned me. The system kept going, happily serving a redirect that had been broken since the first commit, and I found out from the traffic drop, which is the worst way to find out. That, exactly that, is what kills AI agents in production: a plan that looks right, an execution that drifts halfway through, and no mechanism that notices. The model wasn’t at fault. Nothing was watching.

Agent = Model + Harness

The line that rearranged my head comes from the LangChain folks, and it’s short enough to sound dumb: an agent is the model plus the harness. Almost everything in your codebase that makes the agent actually work in production, anything that isn’t the model API call itself, is harness.

In the loops post I used “harness” in passing, as the equipment: the tools, the skills, the MCPs you hand an agent. I sold it short. The harness isn’t just what the agent can do. It’s the scaffolding that watches what it does, catches its failures, enforces its limits, and drags it back on course when it drifts. The name is from real harnesses: the reins, the safety tether, the gear you strap onto something that moves.

The analogy that stuck: sending a junior into an important client meeting. The prompt is the agenda (“say hello, pitch the product, ask about requirements”). The context is the dossier (“here’s the client history and the pricing sheet”). The harness is everything else: the checklist they carry, the mid-meeting check-in you force them to do, the recorded transcript, and the brutal standard you’ll hold their report to. No matter how good the briefing, it doesn’t make up for nobody checking whether what they did was actually right.

Three stages, and why most of us are stuck on the second

This matures in three jumps, and most projects I see live on the second one wondering why third-stage problems keep biting them.

Prompt engineering is where you start. You learn the model is exquisitely sensitive to how you talk to it, and a good role with a couple of examples transforms the output. Real and useful. But I hit the ceiling fast: no matter how well I word the instruction, I can’t give it knowledge it lacks, and I can’t stop it inventing a fact when reality disappoints it.

Context engineering is the next jump, and it’s the one people mistake for “having a RAG.” You give the model what it needs to see right when it decides: dynamic state, summarized tool outputs, history truncated on purpose. My agent got far smarter when it stopped guessing. But context doesn’t fix execution drift. The redirect mess is exactly that: all the information in the world, a correct plan, and it still wandered off with everything green.

Harness engineering is the third, and it’s the humbling one if you thought a better model was the answer. The first two make the model think better. The third makes it act reliably. Change how tasks get decomposed, how state is kept, and how critical steps get validated, and the same model with the same prompts goes from stuck to something you can genuinely leave running. The heart of it is one move I’ll keep coming back to: split the part that generates from the part that judges, and never let them be the same call.

The layers that actually changed something

Some people draw six perfect layers in a nice diagram. I’ll spare you the diagram and give you the three that already live in my code, with the commit that put them there.

Bad input should break the build, not render broken. On this blog, until recently, each post’s images were resolved through a slug-keyed registry. Publish a post, forget to register its slug, and the build stayed green while the page rendered an upload/undefined: a broken image, in production, with no warning. It actually happened to me with the Fable posts. The fix wasn’t remembering better, it was moving the URL into frontmatter and dropping a Zod .refine() into src/content.config.ts that says it has to be a valid Cloudinary URL or astro check fails. Now a bad image doesn’t reach production: it blows up the build in my face, which is where I want it to blow up. In the wealth calculator I keep on the side, same thing with money: a Zod schema bounds every input (years between 1 and 50, the rate between -10 and 30) before the simulation touches a single number. The boundary where outside data comes in is where you scream.

The order of steps isn’t the model’s call. Models fail less from not knowing how to do a thing than from not knowing how to string things together. They jump steps, skip validation, ship the output before they have everything. That’s why this blog’s build isn’t plain astro build, it’s astro check && astro build: the check goes first, and if the schema rejects anything, there’s no build. In the calculator the rail is CI: lint, then type-check, then unit tests, then end-to-end, in that order, each one blocking the next. Those rails aren’t decoration. They move the project management out of the part that improvises and into the part that doesn’t.

An agent that grades its own work lies. This is the layer where primitive agents break. They produce output, declare themselves satisfied, and have no way to know whether it’s correct; they’ve got a textbook optimistic bias and they’ll rate broken code as working. The fix is an evaluator the agent can’t sweet-talk. In the calculator I keep a test file with a comment in all caps that I wrote precisely for future-me and for any agent: “DO NOT MODIFY these tests to make them pass”, because they’re designed to fail if anyone reintroduces a mathematically wrong formula. One checks that the real return is computed with the Fisher equation and not the naive inflation subtraction, and it refuses to pass if the two numbers come out equal. It’s a gatekeeper that won’t be talked into it. “I finished generating the text” and “I ran it, the Fisher test failed, I fixed it” are two different sentences, and the harness is what forces the second one.

The honest part: “harness” is also the buzzword for hiding a weak model

Three snags I ran into myself, in case they save you the bruise.

First, almost none of this is new. That financial-integrity test and those Zod schemas are plain old software engineering: validate inputs, separate generation from QA, don’t trust a process’s output without checking it. We’ve done it for decades with services that have no LLM inside. The unpredictable component being a model now doesn’t make the pattern a new discipline. If you already know how to build systems that fail gracefully, you already know harness engineering; the unpredictable junior just speaks English now.

And yet three things genuinely shift when the unpredictable part is a model and not a service, and they’re what make this worth a name at all. One: the error message stops being a dead end and becomes an interface. My .refine("has to be a Cloudinary URL") isn’t only a guard anymore; the agent reads that string and corrects itself on the next pass. Write your validation messages for the model, not just for the log. Two: the cost flips. In normal code writing is expensive and checking is cheap; with a model, generating is nearly free and verifying is the whole bottleneck, which is exactly why all your effort migrates into the harness. Three: the enemy isn’t bad input, it’s non-determinism. The same prompt gives a different answer next run, so the demo that worked was one lucky sample. The honest checks aren’t “reject bad data”, they’re “run it twice and diff”, “make it idempotent”, “feed the failure back and retry”.

Second, the scaffolding has a dark side. The more harness you bolt on, the more you’re building a cage that papers over a model that isn’t good enough for the task. There’s a point where “my agent nails 95%” really means “I’ve constrained the problem so hard it’s not an agent anymore, it’s a workflow with an LLM filling blanks.” Sometimes that’s exactly what you want. But calling it autonomy is kidding yourself. And the 95% number going around I take with tongs: it depends entirely on which tasks you put in the denominator, and almost nobody publishes that denominator.

Third, and this ties straight to Fable’s pricing: every validation layer you add is another billed model call. A harness that fires an independent evaluator on every step doubles or triples your cost per task. At $10 per million input tokens, the scaffolding that saves you from slop also drains your wallet. The token ceiling isn’t a harness extra. It’s part of the harness.

What does “autonomous” actually mean?

The trick behind agents that supposedly ship whole products with no human review for hours isn’t a smarter model. It’s separating two roles that refuse to trust each other: the one that generates and the one that evaluates. And the key is that the evaluator validates against the world, not against the generator’s version of the world.

There’s a trap the moment you build that evaluator: if the evaluator is itself an LLM, it carries the same optimistic bias you’re trying to escape: you’ve just hired a second junior to grade the first one’s homework. The way out is a pecking order. Deterministic checks (types, schemas, a test with the expected number hardcoded) are free and can’t be sweet-talked, so push every check you possibly can down into that layer. Save the expensive, fallible LLM-judge for the things that genuinely can’t be pinned to an assertion. The cheapest evaluator that can’t lie beats the smartest one that can.

I’ve got a silly but perfect example of this in the calculator. A unit test in jsdom swears the chart is rendered: the node exists, the props are right, all good. The real browser sees that same chart is 0 pixels wide. That’s why the test that matters is the end-to-end one that opens actual Chromium and WebKit, locates the boundingBox of the Recharts SVG, and refuses to pass if the width or height is zero. The generator would tell you the chart is there. The evaluator actually looks. That’s the whole line between a demo and a product: “done” stops meaning “I generated the component” and starts meaning “I opened it in a browser and watched it paint.”

Which layer are you missing?

If you want to skip the philosophy: find your agent’s symptom on the left, build the thing on the right.

Three of those four are a layer you haven’t built. The fourth is the reminder that a harness bounds the chaos, it doesn’t delete it.

Where you start, if you only do one thing

You don’t build all three layers on a Friday night. You pick the one that’s bleeding and you bound it. Here’s the order I’d go in, cheapest first, because the cheap one also catches the most.

Start at the boundary. Find the single place where outside data enters your agent: the form field, the API payload, the frontmatter, the tool output you parse before you trust it. Put a schema there that throws instead of coasting. In Zod that’s a few lines:

// two real boundaries from my repos, same move:
const Input = z.object({
  years: z.number().min(1).max(50),            // the calculator
  imageUrl: z.string().refine(                 // the blog
    (u) => u.startsWith("https://res.cloudinary.com/"),
    "has to be a Cloudinary URL",
  ),
})

That .refine() is the whole lesson. A bad value dies right here, loud, instead of three steps downstream as an upload/undefined nobody notices until the traffic drops. Half my production fires would have been a red build with one of these. Fifteen minutes, today.

Then move the check before the build, not after. Whatever your generate step is, take the validation you already have and put it in front, blocking. On this blog it’s literally one &&: astro check && astro build. If you’ve got CI, order the rails so each blocks the next (types, then unit, then end-to-end) and make the slow honest one, the browser or the integration test, impossible to skip. You’re not writing new checks here. You’re moving the ones you have to a place the agent can’t sail past.

Then write one test the agent can’t sweet-talk. Pick the single piece of logic that has a right answer (a formula, an invariant, a thing that has to render) and write one test that validates it against the world, not against the generator’s word for it. Drop a comment in caps so future-you and any agent know not to “fix” it by deleting it. One test is enough to start. You’re not chasing coverage; you’re putting one judge in the room the generator doesn’t control.

If tonight you do only the first one, you’ve already moved your worst failure mode from “silent in production” to “loud in my face.” That one move is the difference.

What stays yours

This is what I took from face-planting into it for a while.

The model’s intelligence sets its ceiling on a benchmark table. The harness’s robustness decides whether that intelligence survives when everything’s green and something’s still broken. The model isn’t your bottleneck. It hasn’t been for a good while. The gap between a demo and a product lives entirely in the scaffolding you build around it: the .refine() that breaks the build, the test that won’t be talked into passing, the browser that actually looks.

So next time your agent fails, before you swap models, look at what was watching it. Odds are the answer is “nothing”, and that’s exactly the problem.

Related reading: the harness is what goes inside each agent; the loop is what orchestrates them. One without the other generates slop faster or generates it slower, but it generates it all the same. The skills I load into mine are in 5 agent skills that run my coding workflow.

Sources

• The “Agent = Model + Harness” framing comes from Vivek Trivedy at LangChain, The Anatomy of an Agent Harness; I picked it up from an essay going around on harness engineering that laid out these three stages and the layers.
• Everything else comes from my own code: this blog (the deleted image registry, the trailing-slash redirects, the Cloudinary .refine()) and the wealth calculator (the Zod schema, the Fisher-equation test, the chart end-to-end). If it itches, it’s in my repos.

P.S. Got a harness that saved you from a lying agent, or one that grew into a cage so big it stopped being an agent? I want both. Tell me on Twitter/X.