# Quality gates for AI agents: what they catch and miss > OpenAI, Stripe and Spotify wrap their coding agents in checks the model can't argue with. How to order them, where the idea is weak, and what a test showed. **Short answer:** the way OpenAI, Stripe and Spotify describe building software with agents leans less on the model than on what surrounds it: a chain of deterministic checks, cheapest first, that either lets the work through or hands the agent a short error to try again. A human sits at the start, to say what "done" means, and at the end. I had an agent try it at small scale on this blog, and the most useful part wasn't in any of their posts: on their first run, the new gates raised more false alarms than real findings. In February 2026, OpenAI wrote that one of its teams had spent five months building an internal beta of a product with, in their words, _"0 lines of manually-written code"_. That same week Stripe said its agents are behind more than a thousand pull requests merged every week, and described them like this: _"while they're human-reviewed, they contain no human-written code"_. Spotify had told its version in December: agents that prepare changes in the background, and a judge that looks at each one before the PR is opened. I read all three looking for the trick, and the trick is boring: none of them trusts the model. What they describe, each in its own vocabulary, is the set of gates around it: linters, tests, an order, a cap on retries. In what I read, none of them says what happens the day you add a new gate. An agent measured that one in my repo, and the numbers are further down. ## The agent writes; a script decides whether it passes An instruction in a `CLAUDE.md` or a skill is advice. The model follows it almost every time, and "almost" is the problem once an agent works alone for hours. A script that exits 1 is an order: the work goes nowhere until it passes. OpenAI puts it plainly about its architecture rules: _"These constraints are enforced mechanically via custom linters (Codex-generated, of course!) and structural tests."_ And its `AGENTS.md` stopped being the manual and became the index: _"instead of treating AGENTS.md as the encyclopedia, we treat it as the table of contents"_. My own repo has a case that shows it. My article-writing skill requires every web source in a post's claims list to carry a verbatim quote and the date it was read. By an agent's count, the claims lists of six published posts hold 28 rows with a web source: 11 had no quote and 11 had no date. The script counted 29; the extra one was the URL of an uploaded image. The rule was written down. Until November 10, 2026, no script checked it. ## The chain runs cheapest gate first and stays quiet when everything passes Order the gates by cost. A bad import should die in the compiler, in seconds, and not after ten minutes of integration tests. Stripe caps the other end too: _"we only have at most two rounds of CI"_, because every round costs tokens, compute and time. Silence saves context. HumanLayer puts a number on what it costs an agent to receive a full green test run: _"you just threw away 2-3% of your context window"_, all to learn that nothing was wrong. Their recipe is a single checkmark when a step passes and the full output only when it fails. The script that chains the gates on this blog reads a file like this one. These are the first three gates of seven: ```json { "gates": [ { "name": "lint", "run": "node {skill}/lint.mjs {files}", "hint": "every ERROR line is file:line plus the rule; rewrite the sentence, do not touch the rule" }, { "name": "twins", "run": "node {skill}/checks/twins.mjs {files}", "hint": "the two languages must state the same numbers, sources and code" }, { "name": "claims", "run": "node {skill}/checks/claims.mjs {files}", "hint": "every figure needs a row in the claims list; every web proof needs a verbatim quote and the date read" } ] } ``` It runs them in order, stops at the first one that fails, and prints only that gate's findings. If they all pass, one line, plus any notes a gate leaves for the reviewer. By the agent's log, the seven gates took 8.8 seconds on one published post; on another, the chain stopped at the third after 0.2 seconds. ## The error message is the prompt for the next round When the reader of a failure is an agent, the text of the error is the instruction it works from on the next round. OpenAI says so about its linters: _"Because the lints are custom, we write the error messages to inject remediation instructions into agent context."_ Here's a failure from my chain, cut down to one finding of the 22 it reported. Each line says where the problem is, what's wrong and how to fix it: ```text GATE 3/7 "claims" FAILED in 0.2 s (round 1 of 6) src/content/blog/en/claude-code-skills-vs-subagents.md:71 number 1.4 has no row in .writing/claims/claude-code-skills-vs-subagents.md -> add a row with its proof, or take the number out claims: 22 to fix hint: every figure needs a row in the claims list; every web proof needs a verbatim quote and the date read ``` ## The agent can't be allowed to edit its own gates An agent with a failing test has two ways out: fix the code or fix the test. Simon Willison sums it up in his write-up of StrongDM's method: _"Having the agents write tests only helps if they don't cheat and `assert true`."_ StrongDM's answer is scenarios, end-to-end user stories that are, in their words, _"often stored outside the codebase"_. They compare them to a holdout set in model training: the agent can't fit its code to an exam it never sees. My chain has three humbler defenses, and all three are code: - **A fingerprint of the gates.** If a check file changes in the middle of a fix loop, the script refuses to continue. - **A round cap.** After six failed rounds it stops and tells me which gate failed last. Resuming is my call. - **A scripted verdict.** The critics, which are other models, have to quote the sentence they object to. A finding whose quote isn't in the file gets discarded before anyone acts on it. The agent tested it with a made-up finding, and the script threw it out. I don't have hidden scenarios. My critics read the same rubrics as the agent that writes, so that part is still open. ## The human belongs at both ends, not just the last one The version of this idea I see most often leaves a human at the end of the pipeline, validating what comes out. It has a hole: the gates check that something is built correctly, and none of them checks that it's what was asked for. If the agent misread the task, you find out at the most expensive point. Spotify covers part of that hole with its judge, which runs once the deterministic verifiers have finished: _"It uses the diff of the proposed change and the original prompt, and sends them to an LLM for evaluation."_ By their internal metrics, _"the judge vetoes about a quarter of them"_, meaning agent sessions. The judge is another model, with the same built-in optimism as the one that writes, which is why it comes after the checks that can't be talked around. StrongDM goes to the other extreme and removes the human from the end as well: _"Code must not be reviewed by humans"_. Willison, who reports it, asks what anyone would: _"How could that possibly be a sensible strategy when we all know how prone LLMs are to making inhuman mistakes?"_ My bet is the opposite. A human at the start, approving acceptance criteria that can be executed: a test, a command, a scenario. And a human at the end, reading what cleared every gate. The judge in the middle replaces neither. ## The same chain for a Java microservice: designed, not run Here's how I'd order the gates for a Java microservice. The tools are the usual ones in that ecosystem, and the order is by cost. Nobody has run it, so before it blocks anything it owes the same test I describe further down. | Gate | With | Cost | | --- | --- | --- | | Compiles | `javac` via Maven or Gradle | seconds | | Format and lint | Spotless, Checkstyle | seconds | | Unit tests | JUnit | seconds | | Architecture rules | ArchUnit | seconds | | API contract | diff against the committed OpenAPI | a minute | | Integration | Testcontainers | minutes | | Mutation, changed code only | PIT | minutes | | Scenarios hidden from the agent | end to end, outside the repository | minutes | | Judge | a model with the diff and the request | not deterministic | | Human | final review | the most expensive | Two gates in that table answer questions ordinary tests don't. Mutation testing tells you whether the tests the agent wrote check anything or just pass. The contract gate keeps the service from changing its API without its consumers finding out. What I'd measure from day one: how many rounds each task takes to pass, which gate fails most, and how often something reaches the human that a gate should have stopped. ## The small-scale test: three new gates on 26 published posts Limits first, then numbers. The testbed is the review process for this blog's posts, not a microservice; an agent already ran that review with a linter and four critics. It's one repo and one author. Claude wrote and ran the gates in a single session on November 10, 2026. Fresh agents that hadn't seen that session later re-ran the counts they could, everything except the alarms from the two deleted rules; I haven't recounted anything by hand. It's good for seeing the mechanics, not for quoting percentages. There are three new gates. Twins: every post exists in Spanish and in English, written separately; the figures and sources have to match, and both must show the same number of code blocks in the same languages. Claims: every figure in the text has to appear in the claims list. Quotes: the quoted sentence is on the page it cites, word for word. | Gate | Run on | Result | | --- | --- | --- | | Twins | 26 published pairs | 0 mismatches | | Twins | 102 commits that touched a post | 8 the gate would have rejected (missing or mismatched twins), all from November and December 2025 | | Claims | 189 rows in 6 claims lists | 10 figures in the text that weren't in the list, in 2 posts | | Quotes | 17 rows with a quote | 16 word for word on their page | Replayed over history, the twins gate wouldn't have stopped anything in 2026. And the ten missing figures aren't ten errors: they're ten numbers nobody had been made to justify, in posts that had already been through an accuracy critic. One of them was a rounded version of a figure that was there. ## My three new gates lied on day one The first run looked like this: - A rule looking for figures in the claims list that were missing from the post raised 58 alarms by the agent's count, nearly all of them fragments of dates or paraphrase. Deleted. - Another one comparing code across languages raised 39, all of them translated strings inside the code. Deleted. - "01:15" in English and "1:15" in Spanish counted as different numbers. - A file path ending in `.app` was read as a URL. - In a row with two sources, the quote was pinned on the first one. - A row whose source was the URL of a freshly uploaded image counted as a web source with no quote. The agent missed that one: the accuracy critic found it while reviewing this post. - Halfway through a fix, the agent broke the claims script with a bad replace. `node --check` caught it before the next run, by its own account. With most of that fixed, the quotes gate failed a row in the claims list of this very post: a Stripe sentence the reading tool had returned as a verbatim quote. The draft presented it as the gate's first real catch, and it went through two review rounds that way. In the third, the accuracy critic found the sentence: it is the article's own summary, sitting in the page's metadata, which the script throws away because it reads only the visible text. The gate was right to send me to the body sentence quoted above and wrong to call the quote missing. That gives me the rule I'm taking away: before a gate is allowed to block, run it over work you already know is good. A gate that fails on correct work teaches the agent to "fix" correct work. ## The honest part: what none of this proves Most of it is continuous integration under another name. What changes is who reads the failure: an agent that retries in seconds and believes whatever you tell it. All three companies report their own success with their own metrics. In what I read of them I didn't find the number I care about most: how much gets through. The twins gate may be guarding a door nobody walks through; if it still hasn't fired after ten more runs, it goes. A script checks form. The quotes gate knows the sentence is on the page. It doesn't know whether the page is right, or whether the sentence supports what I use it for. And every gate is code to maintain. The first-day failures above are what maintaining just three of them cost me. ## Start by replaying one written rule over your history Pick a rule that lives in prose in your `AGENTS.md` today, something like "this module doesn't import from that one". Turn it into a script that exits 1 and says how to fix the problem. Before you put it in front of the agent, run it over your last hundred commits. The script isn't in your history yet, so the loop calls it by a path outside the worktree: ```bash rule="$PWD/check-my-rule.sh" tmp=$(mktemp -d) for sha in $(git log --format=%h -n 100); do git worktree add -q "$tmp/w" "$sha" || break (cd "$tmp/w" && "$rule") || echo "$sha" git worktree remove --force "$tmp/w" done ``` Start with rules that only read source files: an old checkout has no installed dependencies and no build. If the rule fires on commits you know were fine, the bug is in the rule: fix it or delete it. Once it runs clean over what you already know is good, you have your first gate. **Related reading:** what a harness is and which layers it has is in [What harness engineering is and why your agent fails](/en/blog/harness-engineering-why-your-agent-fails/). Who launches the agents, and when, is in [the loops post](/en/blog/stop-prompting-write-loops/). ## Sources - Ryan Lopopolo, OpenAI, [_Harness engineering: leveraging Codex in an agent-first world_](https://openai.com/index/harness-engineering/), February 11, 2026. - Alistair Gray, Stripe, [_Minions: Stripe's one-shot, end-to-end coding agents_](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents), February 9, 2026. - Max Charas and Marc Bruggmann, Spotify, [_Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3)_](https://engineering.atspotify.com/2025/12/feedback-loops-background-coding-agents-part-3), December 9, 2025. - Dex, HumanLayer, [_Context-Efficient Backpressure for Coding Agents_](https://www.humanlayer.dev/blog/context-efficient-backpressure), December 9, 2025. - Simon Willison, [_How StrongDM's AI team build serious software without even looking at the code_](https://simonwillison.net/2026/Feb/7/software-factory/), February 7, 2026. - The numbers from this blog come from scripts an agent ran on November 10, 2026 over this repository. Every claim in the post has a row and a proof in a claims list I keep next to it. --- **P.S.** Do you have a gate that stopped a disaster, or one you had to delete because it lied? I'm mostly after the second kind. Tell me on [LinkedIn](https://www.linkedin.com/in/oscargallegoruiz).