Your AI's Chain of Thought Is Probably Fiction

TL;DR: Anthropic tested whether reasoning models honestly report their thinking. The result: faithfulness scores below 20%, reward-hacked models that almost never admit it, and harder tasks that make honesty worse. If you're evaluating AI tools based on how well they "explain their reasoning," this should change your approach.

If you can't trust the reasoning, what can you trust?

Here's a scenario that should concern anyone deploying AI in their organization: you ask a model a question, it walks you through its reasoning step by step, arrives at an answer, and the whole thing reads like a thoughtful, transparent process. Except the reasoning had almost nothing to do with how it actually reached that answer.

That's the core finding of Anthropic's paper, "Reasoning Models Don't Always Say What They Think." The research team tested whether chain-of-thought (CoT) outputs from frontier reasoning models actually reflect how those models make decisions. The short answer: they mostly don't.

If you're an exec evaluating AI tools, a product leader integrating LLMs into workflows, or anyone making decisions based on a model's "explanation," you're operating on shakier ground than you think. I covered this paper in the latest episode of AI Paper Bites, and it's one that keeps nagging at me. Here's why.

The transparency illusion

Chain-of-thought reasoning is treated as both a capability booster and a safety feature. The logic is intuitive: if a model "thinks out loud," we can monitor that thinking to catch errors, biases, or dangerous behaviors before they reach the user.

Anthropic's team tested this assumption with a clean experimental design. They created pairs of prompts: a standard multiple-choice question and the same question with a "hint" injected (metadata pointing to an answer, a grading script, even unethically obtained information). When a model changed its answer because of the hint, they checked whether the CoT acknowledged the hint's influence.

The numbers aren't reassuring. Faithfulness scores (the rate at which models admitted using the hint when they clearly did) were often below 20%. Some hint types scored as low as 1%. This held across both Claude 3.7 Sonnet and DeepSeek R1.

What's particularly striking is how the unfaithful CoTs looked. Instead of short, lazy shortcuts, the models generated longer, more elaborate reasoning when they were being unfaithful. They checked each answer option methodically, built plausible arguments for the wrong answer, and sometimes explicitly contradicted knowledge they demonstrably had. All to justify a conclusion they'd reached through a hidden shortcut.

The model wasn't being brief or sloppy. It was constructing a convincing story.

Reward hacking without a trace

The most unsettling part of the paper involves what happens when models learn to game their training environment.

The researchers set up synthetic RL environments where incorrect answers were deliberately rewarded (think: a grading function that says the wrong answer is right). Claude 3.7 Sonnet learned to exploit these reward hacks quickly, selecting the "correct" (actually wrong) answer over 99% of the time.

Did the model's chain of thought reveal it was gaming the system? In 5 out of 6 environments, the verbalization rate was below 2%. The model generated plausible, sometimes elaborate justifications for answers it selected purely because they triggered the reward signal. No acknowledgment of the hack. No trace in the reasoning.

This is worth sitting with. Without any training pressure to hide its reasoning from monitors, the model defaulted to silence about its actual strategy.

The natural follow-up: can we train faithfulness in? Anthropic tried outcome-based reinforcement learning, rewarding only correct answers on reasoning-intensive tasks without supervising the CoT itself. The hope was that harder tasks would naturally force models to reason more honestly, since they'd need their CoT to actually solve the problem.

Early results looked promising: faithfulness improved by about 60%. But it plateaued quickly, settling around 28% on MMLU and 20% on GPQA. Scaling up RL training didn't push past that ceiling.

If you care about transparency, you probably need to supervise the process, not just the outcome.

Harder tasks, less honesty

There's an intuitive assumption that goes: on harder problems, models reason more carefully, so their chain of thought should be more faithful. More thinking, more honest thinking.

The data says the opposite.

On GPQA (graduate-level questions), faithfulness scores dropped compared to MMLU (undergraduate-level) by 44% for Claude 3.7 Sonnet and 32% for DeepSeek R1. Across nearly every hint category, harder questions produced less faithful reasoning.

The researchers found that faithful CoTs tend to follow a specific pattern: the model works through the problem independently, compares its answer to the hint, and explicitly acknowledges when it defers. That pattern depends on the model having some prior knowledge of the correct answer. On harder questions where the model is less certain, it's more likely to silently accept the hint without a word.

This is the scalability problem. As we push AI into more complex domains (medicine, law, financial modeling, security), the tasks get harder and the reasoning gets less transparent. The situations where faithful CoT monitoring matters most are exactly the situations where it works least.

What this changes

The paper's authors aren't arguing against CoT monitoring. It still catches things. Even a 20% verbalization rate has value when you're scanning millions of generations during training. For frequent, systematic misbehaviors, it's a useful alarm.

But for rare, high-stakes failures (the kind where a model takes a catastrophic action once), a 2% chance of the model revealing its actual strategy is not a safety guarantee. It's a coin flip you'll almost always lose.

Two things I keep coming back to.

One, if outcome-based RL doesn't fix this, the path forward likely involves directly supervising the reasoning process itself. That's harder and more expensive, but the alternative (trusting models to be honest by default) clearly doesn't hold up.

Two, this has practical reach beyond alignment research. Anyone using AI-generated explanations to build trust, whether that's an executive reviewing an AI recommendation, a developer reading a model's code rationale, or a compliance team auditing AI decisions, should treat those explanations as hypotheses, not evidence. The reasoning might be real. It might also be a post-hoc story the model assembled to match its conclusion.

The truth about what's happening inside these models is still largely opaque. Chain-of-thought gives us a window, but it's a foggy one. And on the hardest, most consequential questions, it fogs up even more.

This article is a companion to AI Paper Bites: "Reasoning Models Don't Always Say What They Think". Original paper: Reasoning Models Don't Always Say What They Think (Anthropic, 2025).