# Your AI reviewer is judging its own output. That's a blind spot

> Teams drowning in AI-generated code often let an LLM review the LLM's own patches. Amazon's judge-correlation work shows why that misses real defects.

- Published: 2026-09-16
- Canonical: https://aicodereview.io/blog/your-ai-reviewer-is-judging-its-own-output-thats-a-blind-spot/
- Author: aicodereview.io Editorial

---
Teams are drowning in AI-generated code, and the buzziest answer is to hand the review to another LLM. That feels clean, but it has a structural blind spot most how-to checklists never mention: the reviewer and the author often share the same priors, the same blind spots, and sometimes literally the same weights. You are not getting an independent signal, you are getting one model's opinion measured N times.

This is not abstract. Amazon researchers Krishna Balasubramanian and Sasha Podkopaev wrote about exactly this problem, and the paper behind it, "Dependence-aware label aggregation for LLM-as-a-judge via Ising models," is in ICML 2026. Their core finding: when you run a panel of LLM judges, a plain vote count over them is misleading because the judges are correlated. Qwen and DeepSeek and GPT do not reason independently; they inherit the same training distributions and evaluation habits. The researchers show that once you discount for correlated outputs, the panel's confidence drops, and decisions you would have trusted flip. An "agreement" across five models is often, statistically, closer to a single opinion repeated.

The translation to code review is immediate. When your PR is generated by Claude and "reviewed" by Claude, or by an agent whose reviewer is the same family of model, the reviewer will reliably agree with its own reasoning. It will not catch the confident-but-wrong pattern it just emitted, because it does not know it is wrong. Your pipeline reports "LLM review passed," but that sentence should read "the generator voted on itself."

So here is the practical protocol for reviewing AI-generated code at volume, and it is an evaluation-design fix, not a checklist of more eyeballs.

First, engineer the judge to be deliberately un-aligned with the author. Use a different model family for review than for generation, ideally a different paradigm. If a code agent (GPT-family) writes it, have a non-reasoning or deliberately strict small model or, better, a deterministic analyzer flag it. Correlation is the enemy.

Second, treat an AI reviewer's "pass" as a prior, never a verdict. The only independent sources are ones that do not share the author's weights: a single test that runs the code, a dependency scanner, a human who reads the diff under time pressure. Every aggregate "2 LLMs agreed on the patch" claim should be read as one correlated signal plus noise.

Third, and this is the part vendors leave out: measure your review tool's own agreements. If your reviewer flags the defects its own generator makes, count how often those flags overlap with the things the second-pass reviewer catches independently. When overlap is high, you are paying for a mirror.

None of this is a reason to stop using AI code review. The cost economics are real, and a cheap reviewer that catches early mistakes saves more than it costs. But the reason to run a reviewer is independence, and independence is exactly what you do not get when the judge and the author are the same model. Check the family of the model on both sides of your pipeline before you trust the green checkmark.