Don't Let Your AI Grade Its Own Homework

I pay for Claude and I pay for ChatGPT, and people keep asking which one I’d keep if I had to drop one. For about a year I gave a bad answer to that: I’d talk about strengths, one better at long documents, the other at current information, pick whichever fits your work.

That’s not why I have both.

The reason came from a slightly embarrassing evening. I’d spent hours with Claude working through a business case, the kind with a dozen linked assumptions where one wrong number quietly poisons everything downstream. Good session, the sort where you push back and it pushes back and by the end you’ve genuinely thought something through. Then I asked it to review the result before I took it to the team. It told me the logic held.

Out of curiosity I pasted the same thing into ChatGPT, cold, with no history of how it was built. Second paragraph, it asked why I was treating one of my cost inputs as fixed when nothing in the brief said it was. It was right. I’d made that assumption three hours earlier, Claude had helped me build on top of it, and by the time I asked for a review the assumption wasn’t something to check anymore. It was part of the furniture. The case survived, but the floor under it moved, and I’d rather find that out on a Tuesday evening than in front of the team on Thursday.

So the second model is a reader who wasn’t in the room while the thing was made.

The whole habit is four steps, and you can stop reading after this paragraph and still get most of the value. Finish the work in one model. Open the other one in a fresh chat and give it two things: the finished artifact, and a two-line statement of what it was supposed to achieve. Never the conversation. Ask it where the draft fails the brief. Then take what comes back to the first model and make it argue with each point instead of agreeing.

The rest of this is why that works, what it actually catches, when it wastes your time, and how to run it in the chat apps or on the command line.

Why a model can’t mark its own work

There’s actual research under this, and it’s the strongest part of the case.

At NeurIPS 2024, Arjun Panickssery, Samuel Bowman and Shi Feng published LLM Evaluators Recognize and Favor Their Own Generations. Models asked to judge text score their own output higher than other models’ output, on work human annotators rate as equal quality. It’s called self-preference bias. What the paper adds is a cause: models can recognise their own writing, at well above chance, and when the researchers turned that self-recognition up and down by fine-tuning, the size of the bias moved with it. The better a model is at spotting its own work, the more it flatters it.

My own layman’s translation is that familiarity is doing the work, and it’s the same thing that happens to you with your own draft. You reread it and it sounds fine, because you already know what every sentence is trying to say. You’re not evaluating the writing, you’re recognising it.

Once you’ve seen that, “ask Claude to check Claude’s work” stops looking like quality control. The model that wrote the thing, or that sat with you for three hours while you wrote it, is the worst-placed reader you can pick. And a better prompt won’t rescue it. I’ve written before about how hard it is to prompt a model out of agreeing with you. “Be critical” gets steamrolled by the underlying disposition. This is the same problem one level up, and it has a structural fix rather than a wording one: change who’s reading.

That’s also what separates this from the research pipeline I run with Perplexity and Claude. That one splits a job into stages and gives each stage to the tool built for it. Here both models do the same work on the same material, and the whole point is that one of them didn’t write it.

What a cold read actually catches

Prompts are easy to copy and hard to judge, so here’s what one of these rounds looks like.

The brief I gave the cold model was two lines: “This is a business case with a five-year projection. It has to hold up in front of a leadership team who will ask where every number comes from.”

Three of the four things it came back with were noise. A suggestion to add a sensitivity analysis I’d already decided against, a comment about formatting, a generic warning about competitor response. That ratio is normal, and worth saying out loud so you don’t think you’re doing it wrong. The fourth was this:

“You treat the per-unit cost as a fixed input, but the brief doesn’t say it’s fixed, and the margin is most sensitive to exactly that number. If it moves 10% the case stops working. Either state why it’s fixed, or the conclusion doesn’t hold.”

That’s the one. Not an arithmetic error. A decision I’d stopped seeing as a decision.

The part that matters more than the catch is what came next. I took all four points back to Claude and asked it to argue with each rather than accept them. It agreed on the cost assumption and rewrote that section. On the sensitivity analysis it pushed back and explained why the added complexity wouldn’t change the decision the leadership team actually had to make. I agreed, and we changed nothing there.

That’s the loop working: one real catch, one defended refusal, two things dropped. Take every criticism the second model produces and you’ll end up with a worse document than you started with, because a model asked to review will always find something.

The part that hurts

The counter-evidence here is stronger than most people pushing multi-model workflows admit, so let me put it up front.

The comfortable story is that two models from two labs make independent mistakes, so between them they catch nearly everything. That story is mostly wrong. At ICML 2025, a Cornell group published Correlated Errors in Large Language Models, the largest empirical look at this I’ve found. Across more than 350 models, on one of the two leaderboards they tested, two models that both got a question wrong picked the same wrong answer about 60% of the time. Far above chance.

The direction of the trend is the uncomfortable part. Bigger, more accurate models had more correlated errors, and it held across different architectures and different providers. Everyone trains on overlapping slices of the same web, and the post-training recipes have grown to resemble each other. Two frontier models are less like two independent experts and more like two graduates of the same programme who happened to have different professors.

So calibrate the claim. A second read catches a real slice of what the first model missed. It is not an audit, it does not make you safe, and where both models are confidently wrong they will usually be confidently wrong in the same direction.

Ask for it, don’t automate it

The next finding changed how I work, and I’d have bet against it.

Three pre-registered experiments, published in the Proceedings of the ACM on Human-Computer Interaction, looked at what a second opinion does to human decisions. When people were always shown one alongside the AI’s recommendation, their over-reliance on the AI dropped, which is what you’d hope. But their under-reliance rose at the same time. They started overriding the AI when the AI was right. Net result: not obviously better. And it made no difference whether the second opinion came from another AI or from a human peer.

The third condition worked differently. When people could decide for themselves when to go and get a second opinion, over-reliance dropped without the same cost on the other side.

A permanent chorus of disagreement seems to teach you to distrust everything. Choosing to ask means you already noticed something worth checking, so you read the answer with your judgment switched on.

That’s a lab study on human decision-making rather than on review pipelines, so I’m extrapolating. But it makes me sceptical of the setup most people build first: a standing two-model arrangement where everything gets cross-checked automatically before you see it. That produces noise, costs double, and leaves your reliance less calibrated rather than more. Keep the second model one deliberate step away and reach for it when the stakes justify it. Slightly annoying finding, because the automated version is the one that feels sophisticated.

You can do most of this with one tool

If your company blocked the consumer AI tools and handed you Copilot or an internal chatbot, the last four sections probably read as advice for somebody else. They’re not, and this is the part I’d have wanted spelled out.

What you’re buying with a second model is a fresh context. A different vendor is the strongest version of that, and not the only one. Open a brand new chat in whatever tool you’re allowed to use, with memory and project instructions off, and paste in only the artifact and the two-line brief. No history, no reasoning, nothing about how you got there. You’ve removed the conversation that made the model blind, which is most of the effect. What you haven’t removed is the shared training and the shared habits, so it will miss things a genuinely different model would catch.

Call it 60 to 70% of the technique for none of the money, and it works inside a sanctioned corporate stack. Learn the loop there. If you later get access to a second tool, you already know how to use it. The two-track rules still apply: work material goes in the work tool, private learning happens on private tools with private material, and the two never mix. The guide on working when your company blocked ChatGPT covers that split properly.

When it’s worth it, and when it’s theatre

I decide it with one question: what does it cost me if this is wrong and I don’t find out until later?

If the answer is “nothing, I’ll notice and fix it,” use one model and move on. Emails, first drafts, brainstorming, summarising a document you’re about to read anyway, anything where you’re the immediate consumer and wrongness surfaces fast. Almost all of my AI use is this, and it stays single-model.

The second read earns its keep on a specific shape of work — the output leaves your hands, and by the time anyone finds the flaw you’ve already acted on it. A number that goes into a business case. A strategy memo the leadership team will decide from. A client-facing analysis, a migration plan, a contract summary you’re relying on instead of reading. In those cases you check because you stopped being able to see the work about an hour before you finished it.

Three situations where it’s theatre, and I’ve been in all three:

Both models get the same context. Hand model two the whole conversation, including your reasoning and your framing, and you’ve recreated the conditions that made model one blind. It will agree with you. Artifact and brief, never the history.

You can’t tell which answer is better. If the two disagree and you have no way to adjudicate, all you’ve bought is a tie. When that happens, sort the disagreement into one of two boxes. If it’s factual, go and look it up; the models have just told you which fact is load-bearing. If it’s a judgment call, it’s yours, and the disagreement has done its real job by surfacing a decision you were about to make without noticing you were making it. What you must not do is pick the one you liked and call it verified.

You already know which answer you want. Then you’ll keep the model that agreed and quietly discount the other. This is the failure mode I have to watch in myself, and it looks exactly like diligence.

There’s a cost side too. Anthropic reported their multi-agent research setup burning roughly fifteen times the tokens of a normal chat exchange, and framed it as only worth doing when the task is valuable enough to justify that. Cognition’s engineers went further and published Don’t Build Multi-Agents, arguing that splitting work across agents produces incoherent output because each one makes implicit decisions the others can’t see. They’ve since softened the position: parallelise the thinking, keep the writing single-threaded. That’s roughly where I’ve landed. Parallel review, one writer.

Doing it in the chat apps

No technical setup, about four minutes of extra work on a short artifact.

Before anything else, sort out where the material is allowed to go. If it’s company work, use the account your company sanctioned, and check whether that’s a consumer tier or a business one. Consumer tiers have historically defaulted to using conversations for training and business tiers haven’t, and that default is worth verifying rather than assuming. Never paste work material into a personal subscription to save a few francs. If the artifact is genuinely confidential and you only have consumer accounts, this technique isn’t available to you for that document, and no workflow benefit is worth arguing otherwise.

On cost, since I’m recommending a habit and not a purchase: both Claude and ChatGPT have free tiers, and this job is about the lowest-volume thing you can ask of an AI. One artifact, one read, maybe one follow-up. You can test the whole technique this week without paying anyone. What you’ll run into on free is a weaker model and a message cap, which is the wrong place to economise once you’re doing this on work that matters.

The mechanical rule that makes it work: hand over the artifact and the brief, never the conversation.

Claude in the lead is my default for anything written or analytical. I work the problem there, then take the output into ChatGPT in a new chat:

I’m giving you a brief and a draft that was produced from it. You didn’t write the draft and you have no stake in it. Don’t rewrite it and don’t give me an improved version. Tell me three things: where the draft fails the brief, which single claim is weakest and why, and what a well-informed hostile reader would attack first. If the draft is actually fine, say that instead of finding something.

That last sentence matters more than it looks. Without it you get manufactured criticism, which is the same reflex to please wearing different clothes, and worse than praise because it looks like rigour.

ChatGPT in the lead for anything that depends on current, checkable facts. Its search and deep research have been more thorough for me and it’s better at going and finding the thing, so the research and first synthesis happen there and Claude takes the critic’s chair. Same prompt shape, one addition: check every factual claim against the sources given and list any claim those sources don’t support. That catches the specific failure of research output, which is a confident sentence with a citation attached that doesn’t say what the sentence says.

Most people stop there, and the step they skip is where the value lands:

Here’s a review of your draft from a different model. Go point by point. For each one, say whether you agree, partly agree, or disagree, and why. Where you disagree, defend the original and change nothing. Only revise what you actually think is wrong.

Without that framing the first model capitulates to every criticism, including the wrong ones, and you get a worse draft that took twice as long. Models are as agreeable toward each other as they are toward you.

Practical notes from doing this a lot. Markdown survives the trip between apps and formatted text doesn’t, so copy as plain text. Files don’t carry across, you re-upload on both sides. Memory and project instructions don’t carry across either, which here is exactly what you want. And if you keep a standing critic setup, a Claude Project or a Custom GPT holding the review instructions, don’t let it accumulate memory of your projects, or you’ve slowly rebuilt the invested reader you were trying to escape.

If you write code, the command line version is better

Skip this section if you don’t. Nothing below changes the advice above.

The artifact under review is a diff, and both agents can read the repository themselves. Claude Code in the lead has an officially supported path: OpenAI ships and maintains a plugin that wires Codex into Claude Code. Install it from inside Claude Code:

/plugin marketplace add openai/codex-plugin-cc
/plugin install codex@openai-codex
/codex:setup

That gives you /codex:review for a standard review of uncommitted changes, and /codex:adversarial-review, which is the one worth having. It argues with the approach rather than checking whether the code runs. There’s also /codex:rescue to hand a stuck problem over as a delegated task, and --background so Codex works while you keep going. /codex:setup --enable-review-gate makes review automatic on every change, and per the research above I leave that off. The plugin’s own README warns it “can create a long-running Claude/Codex loop and may drain usage limits quickly.”

Worth sitting with for a second: one lab builds and maintains the tooling for its competitor’s agent to consult it. This has stopped being a community hack.

Codex in the lead has no equivalent plugin in the other direction, so it’s a second terminal and a shared brief. Getting the shared brief right is what stops the review from becoming a style argument. Codex and most other agents read AGENTS.md. Claude Code doesn’t. The docs are explicit that Claude Code reads CLAUDE.md, not AGENTS.md, so don’t maintain two files that drift apart. Put the real content in AGENTS.md and make CLAUDE.md one line:

@AGENTS.md

Anything Claude-specific goes underneath. Now both agents work from the same standards, and a review comment means the code is wrong rather than differently opinionated. Point the reviewer at git diff plus the ticket, not the transcript, and give each agent its own git worktree if they’re running at the same time, or they’ll fight over the same files.

Is the second subscription worth it?

This part is opinion. If your work regularly leaves your hands and gets acted on, yes. Twenty a month against a memo that sends a team down the wrong road for three weeks isn’t a close call, and the expensive part was never the mistake itself. It’s everything built on top of it before anyone noticed. But I want to be straight that this is a real recurring cost on a normal salary rather than a rounding error, and that my own best story here is a near-miss I caught, not a disaster I lived through. Test it free first. If two months of that doesn’t produce a catch you’d have paid for, don’t buy the second subscription. (I have no affiliate relationship with either company, and there are no referral links in this guide.)

An honest word on the time, since I’ve said four minutes twice now. Four minutes is the trial: one artifact, one read. A real round on a document that matters, with the defence step and an adjudication you have to think about, runs closer to twenty. Still cheap for something going in front of a leadership team, and still not nothing.

It’s also genuinely not for everyone. Ethan Mollick, who teaches at Wharton and is probably the most-read voice in this field, tells people to pick Claude or ChatGPT, pay the twenty dollars, and give an agent a real task from their life. For most people, he says, just picking the one they like best will do. I think he’s right about most people. Two models won’t fix a weak brief; you’ll get two answers to a bad question. If you’re still building the basic skill of writing a proper brief, get good with one tool first. And if you find yourself running everything past both models out of anxiety rather than judgment, that’s a compulsion wearing a check’s clothes.

Try it on one piece of work this week, using whichever account you’re allowed to use for it. Write down in two lines what the work was supposed to achieve. Open a fresh chat in the model that didn’t write it, paste in those two lines and the finished artifact, and use the cold-read prompt above.

Pay attention to what you do with the answer. If you catch yourself explaining to the second model why it doesn’t understand the context, that’s usually a sign the draft assumes something it never says out loud. Which is worth knowing before someone else reads it.