A prompt in a product and a prompt in a chat are different objects

Prompt engineering for developers · Lesson 1 / 22

The same text, different requirements

In a chat you write a request, read the answer with your own eyes and, if you do not like it, rewrite the request. The feedback loop takes seconds, you are the judge, and the volume is a single input. In a product the same text becomes the body of a function that is called thousands of times a day on inputs you have never seen, and the result is consumed by a parser rather than a person.

The whole course grows out of that difference. A prompt is code without types and without tests: no compiler will tell you that you made a typo, and no linter will catch a contradiction between two paragraphs of the instruction. The only way to control it is a set of inputs with expected results that runs on every change.

What changes when you move into a product

Input diversity. Your ten test examples sit in the middle of the distribution. Real traffic brings empty strings, text in three languages inside one field, emoji instead of a description, a forty-page document instead of a paragraph.
The consumer of the answer is a parser. A person forgives a cheerful "Sure! Here is the result:" in front of the JSON. Code either crashes on it or, worse, silently writes garbage into the database.
Failures are invisible to the author. In a chat you see every bad answer. In a product you only see the ones a user complained about, which is roughly one in fifty.
Changes accumulate. Six months later the prompt holds twenty rules added by different people for different incidents, and half of them conflict with each other.
Cost and latency become a budget. One extra paragraph of context is money on every call and milliseconds in the user's response.

Why a prompt worked on ten examples and broke on a hundred

You picked the ten examples yourself, so they resemble each other and the picture you held in your head while writing the instruction. The hundred come from the product and contain the tail of the distribution. A failure rate simply cannot be measured on a sample of ten: one failure is ten percent, zero failures means "under twenty percent at any reasonable confidence". To tell a prompt with a two percent error rate apart from one with eight percent, you need hundreds of runs.

Minimum contract for a production prompt
1. What comes in   - field types and bounds, what to do with empty
2. What goes out   - schema, enumeration of allowed values
3. What on refusal - which answer means "could not do it"
4. What is banned  - what must never appear in the output
5. How it is checked - a set of inputs with expected results

Insight. A prompt without item 3 is a prompt that will invent an answer on a confusing input instead of honestly returning a refusal. Models cannot stay silent by default: if you have not defined the shape of a refusal, the model fills the gap with plausible text.

Common mistake. Judging a prompt by a couple of runs in a sandbox. The model is non-deterministic: the same input yields different answers, and one lucky run proves nothing. Measure the failure rate on a fixed set instead of collecting impressions.

Pro tip. Start a file of problematic inputs on the first day of development. Every incident becomes a new line. In a month you will have a regression set that you would never have assembled on purpose.

Cheat sheet

A prompt in a product is a component with a contract, not text for a human.
Checking ten convenient examples does not measure the failure rate.
Five contract items: input, output, refusal, bans, check set.
Non-determinism means you can only judge by a sample.

1. Why does a prompt that worked perfectly on ten examples break on real traffic?

2. What does the claim 'a prompt is code without types and without tests' mean?

3. What happens if the prompt does not define the shape of a refusal?

🔒 Answer the question correctly to move on to the next lesson.