This is Part 1 of a four-part series, "The Specification as Quality Gate." The series develops three hypotheses about executable specifications, AI code review, and what each is actually for. Parts 2, 3, and 4 will follow.

When an AI coding agent generates code and a separate AI reviewer examines it, both agents are reasoning from the same artefact: the code itself. If neither has an external reference (a specification, a contract, a statement of what the system is supposed to do), the reviewer has no ground truth to compare against. It checks the code against the code. Not against intent.

The architecture is circular. And there is now enough empirical evidence to say precisely where and how it fails.

The Mechanism

In classical ensemble learning, stacking multiple estimators improves reliability under one condition: the estimators must fail independently. If two classifiers share the same training distribution and the same blind spots, combining them does not reduce error. It consolidates it. The joint miss rate of two correlated estimators approaches the miss rate of either one alone, not the product of both.

This holds for LLM pipelines for a specific reason: a code-generating model and a code-reviewing model drawn from the same model family share architecture, training corpus, and reward signal. That is the same class of correlation the independence condition prohibits. They are not two independent estimators. They are two samples from the same prior.

A 2025 paper (Vallecillos-Ruiz, Hort, and Moonen, arXiv:2510.21513) studying LLM ensembles for code generation and repair named this the "popularity trap." Models trained on similar distributions converge on the same syntactically plausible but semantically wrong answers. Consensus selection (the default review heuristic in most multi-agent pipelines) filters out the minority correct solutions and amplifies the shared error. Diversity-based selection, by contrast, recovered up to 95% of the gain that a perfectly independent ensemble would achieve. The implication is direct: naive stacking of homogeneous LLMs is counterproductive.

A second paper (Mi et al., arXiv:2412.11014) examined a researcher-then-reviser pipeline for Verilog code generation and found that erroneous information from the first agent was accepted downstream because the agents shared the same training distribution and lacked adversarial diversity. Self-review repeated the original errors rather than correcting them.

A February 2026 paper (Pappu et al., arXiv:2602.01011) went further, showing that even heterogeneous multi-agent teams consistently fail to match their best individual member, incurring performance losses of up to 37.6%, even when explicitly told which member is the expert. The failure mechanism is consensus-seeking over expertise. Homogeneous copies make it worse.

None of these papers tested a pure generator-then-reviewer pipeline without external grounding in the exact configuration described here. That controlled experiment has not been published. But three independent papers from 2025 and 2026, approaching the problem from different angles, document the same failure mode: shared training distributions produce correlated errors, and consensus amplifies rather than corrects them.

A Concrete Example

This example uses a deliberately simple case. Modern frontier models catch classic boundary conditions reliably, as the experiments described in the caveats section confirm this. The point here is the sequence: how a BDD scenario makes a defect detectable before any reviewer is involved.

Consider a pagination function. A developer asks an AI coding agent to implement it. The agent produces:

def paginate(items, page, page_size):
    start = page * page_size
    end = start + page_size
    return items[start:end]

The agent also produces tests: one for page 0 and one for page 1 against a ten-item list. Both pass. The implementation is correct for those cases.

The flaw is in what was not tested. When the total number of items is exactly divisible by the page size and the caller requests the last page by calculating it from the total count, the edge case is invisible to the tests provided.

The review agent, given only the code and the tests, validates the implementation against what is there. The tests pass. The reviewer reports no issues. It has no basis to ask what was not tested, because there is no specification defining the expected behaviour for boundary conditions.

The BDD scenario that would have caught it:

Scenario: Last page when total is exactly divisible by page size
  Given a list of 10 items
  And a page size of 5
  When I request page 1
  Then I receive items 6 through 10
  And the result contains exactly 5 items

This scenario would have failed against the implementation before the reviewer was ever involved. The pipeline stops at the specification, not the review.

There is a second reason this matters beyond the review question. Even where models reliably identify boundary conditions during focused review, general refactoring passes introduce a different risk. An agent refactoring for readability or performance is not focused on correctness. Conditional checks and edge-case handling can be silently removed. The BDD pipeline catches that regression the same way it catches the original defect. The protection is unconditional on what the agent was trying to do.

What the SGCR Paper Actually Says

A December 2025 paper from HiThink Research (arXiv:2512.17540) proposed a Specification-Grounded Code Review framework and reported a 90.9% improvement over a baseline LLM reviewer. This figure circulates widely and requires a correction.

The 90.9% is an improvement in developer adoption rate of review suggestions: 42% versus 22%. Developer adoption reflects whether suggestions are relevant and actionable. It is not a defect detection rate. These are different claims.

The SGCR paper validates the hypothesis partially. Without specification grounding, baseline LLM reviewers produce generic suggestions and hallucinated issues. Grounding in human-authored specifications filters the noise and produces suggestions developers act on. But the paper's architecture, combining an explicit specification-driven path and an implicit heuristic discovery path, does not claim specifications make review redundant. It positions them as making review better.

SGCR's implicit pathway explicitly handles discovery of issues beyond the stated rules. This is the paper's own acknowledgement that specifications alone are insufficient. It is consistent with the argument that an AI review agent has a legitimate residual role, but only after the specification pipeline has done its work.

The Independence Condition

The correlated error claim has a precise boundary. Stacking any two AI reviewers is not always counterproductive. The failure condition is specific: estimators that share a training distribution and lack an external reference exhibit correlated failures. The condition for genuine benefit is diversity plus external grounding.

If the reviewer uses a different model family, a different temperature, or a substantially different prompting strategy, errors may be partially independent. A cross-family pipeline, Grok reviewing Claude-generated code for instance, has more independence than a same-family pipeline. But model diversity and external grounding are two separate conditions. Diversity reduces correlation between estimators. It does not supply ground truth. A cross-family reviewer without an external specification is still checking code against code, not code against intent. It will catch some things the generator missed. It will share the same blind spots on anything not well-represented in either training corpus, and it has no basis to identify what was not specified regardless of how different its architecture is from the generator.

The ensemble literature is clear that diversity is what makes stacking work. The specification is what makes review non-circular. Both conditions matter. The current industry architecture typically satisfies neither.

But What About Documentation?

A reasonable objection: if the problem is that the reviewer lacks ground truth, why not solve it by adding documentation? Write better docstrings. Maintain a spec document. Give the reviewer the context it needs.

Our own experiment accidentally tested this. The first version of the v2 corpus included docstrings that stated the domain convention explicitly. The prorate_premium docstring said "ISDA actual/actual." The schedule_maintenance docstring said "ANY of these thresholds." Claude caught every bug at 100%. Documentation in close proximity to the code works, at least when the agent reads it.

That result is real, but it does not solve the problem. It reframes it.

The first issue is retrieval and compliance. A docstring sitting on the same function is the best case for specification proximity. An external policy document, a requirements file, a Confluence page is a much weaker guarantee. The agent needs to find it, read it, and apply it faithfully. Anyone who has watched an AI agent work at scale knows that is not a safe assumption. Agents read context selectively. They produce plausible output that satisfies surface checks without necessarily following the rule they were given. I have seen this enough times firsthand to know I cannot treat documentation-as-specification as reliable ground truth.

The second issue is drift. Docstrings are not executable. They do not fail when the code diverges from them. A function can be refactored, a business rule can change, an edge case can be added, and the docstring sits there describing what the function used to do, confidently, in present tense. Every codebase accumulates this. The reviewer checking code against a stale docstring is not checking against intent. It is checking against what someone intended when they first wrote it.

A BDD scenario cannot drift silently. When the system stops behaving the way the scenario describes, the scenario fails. The build stops. The divergence is visible immediately. That is not a property of documentation. It is a property of executable specifications. The difference is not one of quality or discipline. It is structural.

The objection "just write better documentation" asks for the same discipline that produced the bad documentation in the first place, and adds an assumption that AI agents will follow it reliably. The BDD pipeline removes both dependencies.

The Implication

An AI reviewer without an external specification is a probability estimate, not a quality gate. It samples from the same distribution as the generator and applies pattern matching against its training data. It will catch some things the generator missed, particularly common vulnerability patterns and well-documented anti-patterns that appear frequently in training data. It will systematically miss whatever the generator systematically missed, because they share the same prior.

A BDD scenario that fails is a falsified claim. The pipeline either passes or it does not. There is no hallucination, no false confidence, no noise to filter.

The architecture that follows is not "no AI review." It is: specifications first, verification pipeline second, AI review only for the residual. The residual is real. It includes architectural properties, structural drift, and defect classes that resist specification, but it is a much smaller target than the current industry framing suggests. The next post in this series maps that residual precisely.

Caveats and Open Questions

This argument rests on three empirical papers and two small contrived experiments. Both experiments are same-family only (Claude reviewing Claude-generated code) and use a planted bug corpus rather than a natural defect sample. They are directional evidence, not a controlled demonstration.

The first experiment used classic boundary-condition bugs. Claude caught all five at 100%, which refined rather than confirmed the hypothesis. Classic boundary conditions are pattern-recognition problems that are dense in training data. The correlated error claim applies where the convention is absent from training data, not where it is ubiquitous.

The second experiment used domain-convention violations with neutral docstrings. Detection ranged from 0% to 100% depending on domain opacity. The log-linear interpolation function, where the convention is market practice in fixed income rather than general programming knowledge, was missed in all five runs. The reviewer flagged a different concern instead and declared the implementation correct. BDD caught it. The full corpus, specifications, scripts, and results are available in the experiments directory.

The independence condition also requires more empirical work. Same-family models fail the independence requirement. How much architectural diversity (different model families, different prompting strategies, different temperatures) is required to provide genuine signal rather than correlated noise remains to be established.

These are open questions, not fatal weaknesses. Without an external reference, circular review is circular. The empirical work so far points in one direction. A controlled demonstration using a natural defect sample and cross-family pipelines would make the argument more precise.

The authors of the papers cited here, along with anyone working on LLM ensemble reliability, multi-agent code pipelines, or specification-grounded review, are welcome to challenge, correct, or build on this argument. If the hypothesis is wrong, knowing precisely where it breaks is more useful than leaving it unchallenged. Comments are open.

The papers cited in this post are: Vallecillos-Ruiz, F., Hort, M., and Moonen, L., "Wisdom and Delusion of LLM Ensembles for Code Generation and Repair," arXiv:2510.21513, October 2025; Mi, Z. et al., "CoopetitiveV: Leveraging LLM-powered Coopetitive Multi-Agent Prompting for High-quality Verilog Generation," arXiv:2412.11014, v2 June 2025; Pappu, A. et al., "Multi-Agent Teams Hold Experts Back," arXiv:2602.01011, February 2026; and Wang, K., Mao, B., Jia, S., Ding, Y., Han, D., Ma, T., and Cao, B., "SGCR: A Specification-Grounded Framework for Trustworthy LLM Code Review," arXiv:2512.17540, December 2025, v2 January 2026. Author lists and findings verified against the original papers.

Series: The Specification as Quality Gate Part 1: The Echo Chamber in Your Pipeline (this post) Part 2: From Complex to Complicated Part 3: What Specifications Cannot Catch Part 4: The Specification as Quality Gate