“Let’s think step by step” is the most repeated line in prompt engineering. It earned that status honestly. On the models of 2022 and 2023 it produced large, measurable gains on reasoning tasks, and it cost nothing to add.
On the models most people are using now, adding it to every prompt is mostly a way to pay for tokens you do not need.
What chain-of-thought was for
Early models had a specific failure. Ask for the answer to a multi-step problem and they would produce the shape of an answer immediately, skipping the work. Because each token is generated from the ones before it, arriving at the final number without generating the intermediate steps meant there was nothing for the answer to be derived from.
Telling the model to work through it forced those intermediate tokens into existence. The arithmetic now had somewhere to happen. That is the entire mechanism, and it explains both why it worked so well and why its usefulness was always going to be temporary: it was a workaround for models that did not reason unless told to.
What changed
Current frontier models are trained to do that work internally. Give one a multi-step problem and it generates reasoning whether or not you asked, either visibly or in a separate reasoning pass you are billed for but do not see.
Which means the instruction now lands in one of three ways:
- Redundant. The model was going to reason anyway. You added tokens and changed nothing.
- Duplicative. On a reasoning model, you get internal reasoning and a written-out version of roughly the same thing. You pay for both.
- Harmful. On a simple task, you have turned a three-token answer into four paragraphs of justification for a conclusion that was never in doubt.
The evidence
Two findings are worth knowing before you decide.
Wharton’s prompting research in 2025 tested the standard step-by-step instruction across current models and found the benefit had largely evaporated on models that already reason by default. On some tasks the accuracy difference was within noise. On others the instruction produced longer answers with no accuracy gain at all.
The second finding is less comfortable. Explicit chain-of-thought has been associated with higher rates of invented detail on complex tasks, with reported increases of around 12% in some evaluations. The mechanism is straightforward once you see it: each reasoning step is generated the same way as any other text. A wrong intermediate step is not flagged as wrong. It is written down, treated as established, and everything after it inherits the error, dressed in the authority of a numbered list.
A confident wrong answer with five steps of visible support is harder to catch than a confident wrong answer on its own. The reasoning does not make it more correct. It makes it more persuasive.
When explicit reasoning still wins
This is not an argument for deleting it everywhere. There are four situations where writing the reasoning out is the point.
| Situation | Why it still earns its tokens |
|---|---|
| Audit trail | Someone must be able to see why a decision was made. A loan assessment, a moderation call, a medical triage note. The reasoning is a deliverable, not a technique. |
| Controlling the path | You do not just want a correct answer, you want it reached a particular way. “Check eligibility first, then amount, then date.” Naming the steps enforces the order. |
| Debugging | A prompt is failing and you cannot tell where. Turn reasoning on temporarily to see where it goes wrong, then turn it off. |
| Weaker or smaller models | Cheap models routed for cost reasons often do not reason by default. The original mechanism still applies to them. |
Notice that three of the four are about visibility rather than accuracy. That is the honest case for chain-of-thought in 2026: it is an inspection tool.
When it is pure token burn
| Simple task | Complex task | |
|---|---|---|
| Reasoning-native model | Remove it. Pure waste. | Remove it unless you need the trail. The model reasons anyway. |
| Classic / small model | Remove it. Nothing to reason about. | Keep it. This is the case it was designed for. |
Three of the four cells say remove. That ratio is roughly why the instruction has quietly become a default cost on a large number of production prompts.
The cost is not trivial at volume. A classification prompt that returns one word costs a handful of output tokens. The same prompt with step-by-step reasoning returns 150 to 400. Output tokens are the expensive kind. Run that a hundred thousand times a month and the reasoning nobody reads is a line item, which is the subject of measuring a prompt properly before it ships.
What to do instead
The instinct behind adding step-by-step is usually right: the answers are sloppy and you want more rigour. The fix is to constrain the output rather than instruct the thinking.
// Old habit
"Let's think step by step. First consider the eligibility
criteria, then the amount, then explain your reasoning
before giving a final decision."
// Better: specify what must be in the answer
Return exactly:
decision: approve | reject | escalate
reason: one sentence, max 20 words
evidence: exact quote from the application
rule: the rule ID you appliedThe second version gets you more, not less. You have an auditable decision in four fields instead of a paragraph you have to read. The evidence field forces the model to point at something real rather than assert. The rule field makes the decision checkable against your policy. And it costs a fraction of the tokens.
The general principle: ask for artefacts, not for effort. “Think carefully” is unenforceable. “Quote the clause you applied” either happens or it does not.
A worked comparison
A moderation task: decide whether a product review breaks the guidelines. Ten thousand reviews a day. Here is the same job written both ways.
// Version A - the habit
Read the review below. Let's think step by step.
First consider whether it contains profanity, then whether
it contains personal information, then whether it is spam.
Explain your reasoning for each, then give a final verdict.
REVIEW: {{review}}// Version B - artefacts instead of effort
Check the REVIEW against three rules.
Return exactly:
profanity: yes | no
personal_info: yes | no
spam: yes | no
verdict: allow | flag | remove
trigger: exact phrase from REVIEW, or NONE
REVIEW: """{{review}}"""| Version A | Version B | |
|---|---|---|
| Typical output length | 200–350 tokens | ~25 tokens |
| Machine-readable? | No. Verdict buried in prose. | Yes. Five fields, fixed values. |
| Auditable? | In theory. Nobody reads 10,000 paragraphs. | Yes. The trigger field points at the exact phrase. |
| Failure mode | Reasons its way to a wrong verdict, convincingly. | Returns a value outside the allowed set, which you can catch. |
| Reviewable by a human | Read the paragraph, form a judgement. | Glance at the trigger phrase. Two seconds. |
Version B is roughly a tenth of the output cost and more accountable, not less. The trigger field is the substitute for reasoning: it forces the model to point at something in the text rather than assert a conclusion. If the verdict is remove and the trigger is NONE, you have a contradiction you can detect automatically. No amount of reasoning prose gives you that.
The reasoning-model trap
One more trap specific to models that reason internally. If you also ask for visible step-by-step reasoning, you can end up paying twice for the same work: once for the internal reasoning pass, once for the written narrative that restates it. The two are generated separately, and on some tasks they do not even agree with each other, which is its own kind of unsettling.
If you are on a reasoning model and you need visibility, look first at whether the platform exposes a reasoning summary you are already being billed for. Reading what you have paid for beats generating a second copy of it.
Rewriting an old prompt
- Find every step-by-step instruction in your prompt library. Grep for “step by step”, “think through”, “reason about”, “before answering”, “explain your thinking”.
- Ask what you do with the reasoning. If nobody reads it and nothing is logged from it, it is decoration.
- Replace it with fields. Whatever you were hoping the reasoning would give you, name it as a required output field instead.
- Run your test set both ways. Same inputs, both versions, and compare the fields that matter. Do not judge by reading two examples.
- Keep it where it earned its place. Audit trails and ordering constraints stay. Everything else goes.
Step four is where people get surprised. Removing reasoning sometimes improves accuracy, because it removes the chance of a wrong intermediate step anchoring the final answer.
Frequently asked questions
Is chain-of-thought dead?
No. It has moved from a general-purpose accuracy technique to a targeted one. It still earns its place where you need an auditable trail, where the order of reasoning matters, while debugging, and on smaller models that do not reason by default. Applying it to everything is the part that has aged badly.
Does “think step by step” still help on current frontier models?
Usually not enough to justify the tokens. Models trained to reason internally do so whether or not you ask. Recent testing found the improvement on such models is often within noise, while answer length and cost go up reliably.
Why would chain-of-thought increase invented detail?
Because reasoning steps are generated the same way as the answer, with no verification. A wrong step is written down, treated as settled, and everything after it builds on it. More generated text means more surface for an error to appear on, and a numbered list makes the result look better supported than it is.
How do I keep an audit trail without paying for full reasoning?
Ask for evidence rather than narrative. An exact quote, a rule identifier, a confidence label. These are short, checkable, and far more useful in a log than a paragraph of prose explaining what the model felt.
What should I replace my old chain-of-thought prompts with?
A constrained output schema. Name the fields the decision needs, cap the length of any free-text field, and require at least one field to be quoted directly from the input. You get rigour that can be tested rather than rigour you are asked to trust.
The shift that makes it click
Chain-of-thought was never about making the model think. It was about making the thinking happen somewhere it could be used. Newer models already do that, so the useful question is no longer whether to ask for reasoning but whether you need to see it.
If the answer is no, spend those tokens on a tighter specification instead. And if what is really bothering you is not the reasoning but the answers changing between runs, that is a different problem with a different fix, covered in why AI gives you a different answer every time.
The AI Prompt Engineering book covers how these models read instructions and why techniques age, and the seven lesson mini-course walks through rebuilding a prompt around output fields one exercise at a time.