Ask a model for a citation and it will give you one. Author, title, year, page number. Confident, formatted, plausible. And sometimes entirely invented.
The usual advice is to tell the model to be accurate. That advice does not work, and it is worth being precise about why.
Why “be accurate” does nothing
A phrase like be accurate or do not make things up asks the model to apply a standard it has no way of checking. The model is not consulting a store of facts and reporting what it finds. It is producing the next token that fits the pattern of the text so far. A fabricated citation fits that pattern beautifully, because real citations look exactly like that.
Instructions of this kind assume the model knows which of its outputs are true and is choosing not to tell you. It does not have that information. Adding the phrase changes the tone of the answer and leaves the failure rate roughly where it was.
Worse, one common fix makes things worse. Asking the model to reason step by step through a complex task has been associated with a higher rate of invented detail, not lower, because each additional reasoning step is another opportunity to generate something plausible and unchecked.
What does work is removing the need to guess.
The four places invention comes from
Lumping every wrong answer under one word hides the fact that they have different causes and different fixes. In practice they fall into four groups.
| Type | What happens | What fixes it |
|---|---|---|
| Knowledge gap | The fact was never in training, or was too rare to stick. The model produces something shaped like the answer. | Retrieval. Put the fact in the prompt. |
| Over-extension | The source says something narrow; the answer states something broader. “Sales rose in Q3” becomes “sales are growing”. | Ban inference explicitly. Require a quote per claim. |
| Conflation | Two similar entities get merged. Two people with the same surname, two product versions, two years of figures. | Trim the context so only one of them is present. |
| Format pressure | You asked for five bullet points and the source supports three. The model writes five. | Make the count a maximum, not a target. Allow short answers. |
That last one is the most self-inflicted and the easiest to miss. If your template says “list five benefits” and the document contains three, you have instructed the model to invent two. Write “list up to five” and the pressure disappears.
The four levers that actually work
| Lever | What it fixes | What it does not fix |
|---|---|---|
| Retrieve | Missing knowledge. The answer is in a document, so the model reads instead of recalling. | The model can still misread or over-extend what it was given. |
| Abstain | The gap between “I don’t know” and a confident guess. Gives the model a legal exit. | Facts that are present but wrong in the source. |
| Verify | Claims that survived the first pass but do not hold up against the source. | Systematic errors the checker shares with the writer. |
| Constrain | Free-form prose where invention hides. Narrow output leaves less room. | Wrong values inside a correctly shaped output. |
None of these is a prompt phrase you sprinkle on top. Each one changes the structure of the request.
Grounding: hand it the source, close the door
Grounding means the model answers from text you supply rather than from what it absorbed during training. The important half is the part people leave out: explicitly closing the door on outside knowledge.
A grounded prompt that only says “here is a document, answer questions about it” still allows the model to blend the document with whatever it already believes. You have to forbid that.
Answer ONLY from the SOURCE below.
Do not use outside knowledge. Do not infer beyond what is written.
Every claim in your answer must be traceable to a specific line.
If the answer is not in the SOURCE, reply exactly:
NOT_IN_SOURCE
SOURCE:
"""
{{document}}
"""
QUESTION: {{question}}Three things are doing the work here. The word ONLY. The explicit ban on inference. And a named, exact string for the failure case, so a refusal is machine-detectable rather than a paragraph of apology you have to parse.
The abstain clause
Most models will guess before they will admit ignorance, because guessing produces the kind of text that looks like a good answer. You change that by making the refusal cheap and specific.
Compare the two versions:
| Weak | Strong |
|---|---|
| “If you are unsure, say so.” | “If the SOURCE does not state it, output exactly: NOT_IN_SOURCE” |
| Unsure is undefined. The model decides what counts. | The condition is objective: is it in the text or not. |
| Output is prose you have to interpret. | Output is a token you can test for. |
Once the refusal is a fixed string, you can count it. A sudden rise in NOT_IN_SOURCE means your retrieval is failing. A rate of zero across a hundred varied questions usually means the model has quietly stopped abstaining, which is worth investigating before you trust the rest.
The verification pass
For anything that matters, run a second call whose only job is to check the first. Not “is this good” but a mechanical comparison.
You are checking an ANSWER against a SOURCE.
For each factual claim in the ANSWER, output one row:
claim | SUPPORTED | exact quote from SOURCE
claim | UNSUPPORTED | (no quote available)
Do not correct anything. Do not add commentary.
SOURCE:
"""{{document}}"""
ANSWER:
"""{{answer}}"""Requiring an exact quote is the whole trick. A model asked “is this supported?” will say yes. A model asked to produce the specific line that supports it either finds one or has to write UNSUPPORTED. If any row comes back unsupported, the answer does not ship.
Less context beats more context
A large context window invites you to paste everything and let the model sort it out. This costs you twice.
- Relevant detail gets diluted. The passage that answers the question competes with forty pages that do not.
- Contradictions creep in. Feed two document versions and the model may merge them into a third that matches neither.
- You pay for it. Every irrelevant page is billed on every single call.
Retrieve the two or three passages that bear on the question and send those. A short, clean source is easier to ground against and easier to verify afterwards.
A worked example
A support team wants the model to answer refund questions from their policy page. The first version reads like this:
You are a helpful support agent for Acme.
Be accurate and never make things up.
Policy: {{policy_text}}
Customer: {{question}}It works on the obvious questions and fails on the interesting ones. Asked “can I get a refund after 45 days if the item is faulty?” when the policy only covers 30 days and says nothing about faults, it produces a confident, sympathetic, invented exception. The customer is now holding a promise the company never made.
The second version changes the structure rather than the tone:
Answer ONLY from the POLICY below. Do not infer.
Quote the clause you are relying on.
If the POLICY does not cover the question, output exactly:
NOT_COVERED
Format:
answer:
clause:
POLICY:
"""{{policy_text}}"""
QUESTION: {{question}} Same model, same policy. The 45-day question now returns NOT_COVERED, which routes to a human. The routine questions return an answer with the clause attached, so the agent can see at a glance whether it is right. Nothing about the model changed. The room to guess was removed.
Notice also that two sentences max is doing quiet work. Length is where invention hides; a model given a paragraph to fill will fill it.
The pre-publish checklist
| # | Check | Fail looks like |
|---|---|---|
| 1 | Is the source in the prompt, not assumed from training? | The prompt asks about a document it never includes. |
| 2 | Is outside knowledge explicitly forbidden? | No sentence containing ONLY or its equivalent. |
| 3 | Is there a named abstain token? | “Say you don’t know” with no fixed string. |
| 4 | Is the output format constrained? | Free prose where a field list would do. |
| 5 | Are numbers and names required to be quoted, not paraphrased? | Figures that appear nowhere in the source. |
| 6 | Is there a verification pass for high-stakes output? | One call, shipped straight to a user. |
| 7 | Have you run the same input five times? | Tested once, declared working. |
| 8 | Do you have a case where the answer genuinely is not in the source? | Every test question is answerable, so abstain is never exercised. |
| 9 | Is the context trimmed to what the question needs? | Whole manual pasted for a one-line lookup. |
| 10 | Does a failure alert someone? | UNSUPPORTED logged and never read. |
Frequently asked questions
Why doesn’t telling the model to “be accurate” work?
Because accuracy is not a setting the model can switch on. It generates the most plausible continuation of your text, and invented detail is plausible by construction. The instruction asks it to filter by a criterion it cannot evaluate, so the phrasing changes and the error rate largely does not.
Does retrieval eliminate hallucinations completely?
No. It removes the category caused by missing knowledge, which is the largest one. It leaves misreading, over-extension beyond what the passage supports, and errors that were already in your source. Retrieval plus an abstain clause plus verification covers considerably more than retrieval alone.
Does chain-of-thought reduce or increase invented detail?
It can increase it on complex tasks. Each reasoning step is generated the same way as the answer, so a wrong intermediate step gets treated as established and everything after it inherits the error. Reason step by step when you need an auditable path, not as a general accuracy measure.
How do I make the model say “I don’t know”?
Give it an exact string to output and an objective condition for using it. “If the SOURCE does not state it, output NOT_IN_SOURCE” works. “Say if you are unsure” does not, because unsure is left to the model to define and it will define it generously.
Does a bigger context window mean fewer errors?
Not by itself, and often the reverse. More text means more competing material, more chance of contradiction, and a higher bill on every call. Precision in what you retrieve matters more than capacity.
Where this leaves you
Stop treating invented detail as a flaw in the model to be scolded out of it, and start treating it as a gap in the specification. The model guessed because you left room for a guess. Close the room.
Once the structure is right, measure it. Testing a prompt before it reaches real users covers how to build a fixed set of cases so a change that helps one input is not quietly breaking four others. And if your answers move between runs without any invention involved, that is a different problem entirely, covered in why AI gives you a different answer every time.
The AI Prompt Engineering Checklist puts the ten checks above into a fillable page you can run before anything ships, and all six pieces come together in the Complete Bundle.