Most prompts are tested the same way: you try it once, the answer looks good, you ship it. That is not testing. That is a demo, run by the one person on earth who knows exactly what the prompt expects.
Real users do not behave like that. They paste half a document. They leave the field blank. They type in a language you did not plan for. The gap between a prompt that works for you and one that works in production is mostly made of cases you never tried.
Build a small set of hostile inputs
You do not need a large test suite. Five well chosen inputs catch most failures, and you can assemble them in ten minutes.
- Standard. A clean, typical case. This is the one you already tried.
- Empty. Nothing at all, or whitespace. Does it fail loudly, or invent an answer from nothing?
- Noise. Far too long, badly formatted, full of quotes and line breaks and special characters.
- Adversarial. Input containing an instruction, to see whether your containment holds.
- Nonsense. Off topic or gibberish. A good prompt says it cannot help. A weak one produces confident nonsense.
Keep these five in a spreadsheet next to the prompt. Every time you edit the prompt, run all five again. That is the entire discipline.
Check the shape before you read the words
When software consumes the output, structure fails before content does, and structure is far cheaper to check.
- Did it return valid, parseable output?
- Are all required fields present?
- Are the types right, a number where you expect a number?
- Is there anything outside the payload, a greeting, a code fence, a closing note?
If the parser throws, the prompt failed. It does not matter how good the prose inside was.
Define what good looks like before you judge it
The most common testing mistake is evaluating by feel. You read the answer, it seems fine, you move on. Feel does not survive contact with a hundred users.
Write down what a correct answer must contain before you run anything. Two or three concrete criteria are enough: it must cite only information present in the input, it must stay under a set length, it must never guess a value it was not given. Now you are checking against a standard instead of a mood.
Run it more than once
These models sample with some randomness, so one good run proves very little. Run each of your five inputs three times. If the answers vary in ways that matter, the prompt is underspecified and the variation is telling you exactly where.
Version your prompts
Prompts drift. Someone adds a line to fix one complaint and quietly breaks two behaviours nobody was watching. Treat a prompt like code, because it is code.
- Keep old versions rather than editing in place.
- Note what changed and why, in one line.
- Re-run the five cases after every edit. This is your regression test.
Watch it after it ships
Testing does not end at launch. Log the inputs that produced failures and read them weekly. Real users will find cases you would never have imagined, and those logs are the highest quality test cases you will ever get, because they actually happened.
The short version
Five inputs. Three runs each. Written criteria. Check structure first. Version everything. Read your failure logs.
None of it is complicated, and it is the difference between a prompt that works and one you can depend on.
The production readiness checklist turns this into sixteen checks you can tick off before shipping, and the seven mistakes workbook covers the errors that show up most often in real prompts.