It is late, your script has just crashed, and the reason is almost funny. You asked the model for a list of categories. It gave you a list of categories, wrapped in a friendly sentence, inside a code fence, with a closing remark underneath. Your parser wanted JSON. It got prose with JSON somewhere in the middle. Getting reliable structured output from AI is less about asking nicely and more about removing the room to improvise.
Nothing about the answer was wrong. It simply was not machine readable, and machine readable is the whole point when software is on the receiving end.
This guide covers the whole chain: why the wrapper appears, how to specify a schema the model cannot misread, when to use provider-level JSON mode instead of instructions, how to keep reasoning out of the payload, and what your code should do on the run that still comes back broken.
Why asking for structured output from AI is not enough
Write return the result as JSON and you will usually get JSON. Usually is the problem. One run in twenty arrives with an explanation in front of it, or a trailing note, or single quotes instead of double. Your code does not care that it was close.
The model is not ignoring you. It has been trained heavily on helpful conversational replies, and helpfulness includes framing an answer. Left to its defaults, it will wrap.
It is worth naming the specific ways it breaks, because each one has a different fix:
| What comes back | Why | Fix |
|---|---|---|
Here is the JSON you asked for: then the object | Conversational framing | Anchor the first character |
| Object inside triple backticks | Trained on markdown code blocks | Forbid code fences explicitly |
| Trailing note after the closing brace | Helpful closing habit | Say output ends at the brace |
Single quotes, or Python True | Confusion with Python literals | Say strict JSON, double quotes |
Trailing comma before } | Common in training data | Validate and retry |
| Renamed or reordered fields | Format was described, not shown | Paste the literal skeleton |
Number returned as "42" | Type never stated | Declare the type per field |
| Cut off mid-object | Hit the token ceiling | Raise max tokens, shrink schema |
Give the exact shape, not a description of it
The single highest leverage change is to stop describing the format and start showing it. Paste the literal skeleton you want back, with the field names and types spelled out.
A description leaves the model to invent field names, ordering and nesting. A skeleton removes those decisions entirely. It is no longer authoring a structure, it is filling one in.
WEAK
Return the analysis as JSON with the sentiment, the main
topics and a confidence score.STRONG
Return strict JSON matching exactly this schema. Double
quotes only. No trailing commas. No additional keys.
{
"sentiment": "positive" | "neutral" | "negative",
"topics": [string], // 1 to 5 items, lowercase
"confidence": number, // 0.0 to 1.0, two decimals
"status": "ok" | "cannot_process"
}
If the input cannot be analysed, set status to
"cannot_process", topics to [] and confidence to 0.0.Be explicit about the boring parts too. Which fields are required. What goes in a field when the answer is unknown. Whether a list can be empty. Every unstated case is a case the model will improvise, and improvisation is exactly what you are trying to remove.
Close the door on the wrapper
Three instructions do most of the work:
- Output the object and nothing else. No preamble, no explanation, no closing line.
- Do not wrap the output in a code fence. Say it directly or you will keep stripping backticks in code.
- Begin the response with the opening brace. Anchoring the very first character is surprisingly effective.
That last one works because of how generation runs. Once the first token is a brace, every plausible continuation is inside a JSON object. The wrapper never gets a chance to start. Some APIs let you go further and prefill the assistant turn with { yourself, which makes the anchor absolute rather than requested.
Use JSON mode when your provider offers it
If your provider offers structured output, JSON mode or tool calling, use it. Enforcing the grammar at the decoding level is stronger than any instruction you can write, because invalid output becomes impossible rather than unlikely.
| Approach | Guarantees | Use when |
|---|---|---|
| Instruction in the prompt | None. High compliance, not certainty. | Prototyping, or no other option |
| JSON mode | Valid JSON syntax, any shape | You control the schema in code anyway |
| Strict schema / structured output | Valid JSON matching your schema | Production, always prefer this |
| Tool / function calling | Typed arguments matching the signature | The output triggers an action |
One caution: schema enforcement guarantees the shape, never the content. A strictly valid object can still contain a fabricated order number. Structure and truth are separate problems, and only the first one is solved at the decoding layer.
Choose the right format for the job
- JSON for anything a program will consume. Universal, compact, easy to validate.
- XML style tags when the payload is long prose. Tags survive messy content better than braces do, and models are very comfortable closing a tag. Long text inside a JSON string means escaping every quote and newline, which is exactly where malformed output comes from.
- Plain delimited lines for simple flat lists. Do not reach for JSON when a newline would do.
A useful hybrid for reports: tags on the outside, JSON on the inside. The prose sits in <summary> tags where quotes cannot break anything, and the structured fields sit in a small JSON block next to it.
Broken JSON is mistake number four
Seven AI prompt mistakes that cost hours, with the fix for each: an 11-page reference plus a 12-page fillable PDF. Output format is one of them, and it is rarely the one people expect.
See the sevenInstant download · unlimited downloads · no expiry · one payment, no subscriptionKeep the thinking out of the payload
Reasoning often improves answer quality, and it also pollutes structured output. Ask a model to think step by step and return only JSON and you have given it two contradictory orders. It will pick one, and you will not get to choose which.
The fix is to give the reasoning its own place: a dedicated field inside the object that your code reads and discards, separate from the field your application actually uses.
{
"reasoning": "Two order numbers present; the second is
quoted from an earlier email, so the first
is the active one.",
"order_id": "A-4471",
"status": "ok"
}Put the reasoning field first. Generation is sequential, so a field written before the answer can genuinely inform it. A reasoning field placed after the answer is a justification written to fit a decision already made, which is worth much less.
Validate, then decide what happens on failure
Assume malformed output will happen eventually and plan the branch now. Four layers, cheapest first:
1 PARSE try to decode. Fail closed, never trust the string.
2 SHAPE required keys present? types correct?
3 VALUES enum members valid? numbers in range? length capped?
4 TRUTH does each extracted value appear in the source text?Layers one to three are pure code and cost nothing. Layer four is the one people skip, and it is the only one that catches fabrication. If the model returns an order number, check that string actually occurs in the input. If it does not, the field is invented, no matter how well formed the object is.
On failure, retry once with a short corrective instruction that includes the parser error. Do not retry in a loop; if two attempts fail, the prompt is the problem and another call will not fix it. Log the raw text every time it fails, because that log is where you learn what your prompt is missing.
Resist the temptation to write a clever repair function that hunts for the first brace and the last brace. It works for months and then silently extracts the wrong object out of a response that contained two. Fix the prompt instead.
The habit worth building
Before a prompt goes anywhere near production, run it against five deliberately awkward inputs: a normal one, an empty one, a very long one, one full of quotes and special characters, and one written to confuse it. Run each three times. If all fifteen responses parse, you have something you can build on. The full loop is in how to test an AI prompt before it reaches real users.
The quotes case deserves special attention here. Text containing " and newlines is the most common single cause of broken JSON from a model, and it is the case least likely to appear in your own hand-written test input.
Two follow-ups if the output is still not clean. Examples are usually the fastest lever here, and how many examples a prompt actually needs covers how many and how to choose them. Where those examples and the schema itself should live is covered in system prompt vs user prompt.
Frequently asked questions
Why does the model add a code fence even when I tell it not to?
Because markdown fences surround almost every JSON example in its training data, so a fence is the statistically expected continuation. Instructions push against that; anchoring the first character to {, or prefilling the assistant turn with it, removes the opportunity altogether.
Does temperature 0 guarantee valid JSON?
No. It reduces variation, which reduces the failure rate, but the model is still free to choose a wrapper if a wrapper is the most likely continuation. Low temperature is a helpful setting, not a guarantee. Schema-level enforcement is the guarantee.
Should I use JSON or XML tags for output?
JSON when a program consumes fixed fields. Tags when the payload is long prose, because every quote and line break inside a JSON string is a chance to produce something unparseable. Many production prompts use both, tags outside and a small JSON block inside.
How do I stop it inventing values for missing fields?
State the null behaviour per field in the schema, then verify in code. Writing if no order number is present, order_id is null removes the ambiguity; checking that the returned string actually appears in the source text catches it when the model ignores you anyway.
Is retrying on a parse error good enough?
Once, with the parser error included in the retry, is reasonable. A retry loop is not. Repeated failure means the schema is underspecified or the response is being truncated by your token limit, and both are prompt problems that another call will not solve.
The short version
Show the skeleton, do not describe it. Declare types and null behaviour. Forbid the fence and anchor the first character. Use schema enforcement when the provider offers it. Put reasoning in its own field, first. Parse, check shape, check values, check the values were really in the input. Retry once, then fix the prompt.
The Build Fail-Proof AI Prompts guide covers schema enforcement and silent reasoning in full, and the production readiness checklist turns the testing habit into sixteen concrete checks.
All six pieces — book, guide, checklist, listicle, mini-course and audio series — come together in the AI Prompt Engineering Complete Bundle.
About the author. Written by Said Sihame, founder of DigiBog. I make the planners, workbooks and checklists sold on this site, and these articles come out of the same work. Everything here is what I use and test myself — take what fits and leave the rest.