Messages, roles and the system instruction

Working with LLM APIs: advanced · Lesson 2 / 21

There is no conversation, there is an array

The model does not store your conversation. On every call it receives the full list of messages again and remembers nothing from the previous request. The feeling of memory is produced by your code, which stitches the history together and sends all of it. One practical consequence follows: dialogue length converts directly into money and latency, and it grows quadratically if you send the whole history on every step.

Roles and what they mean

system — behavior rules for the whole call. It usually comes first and carries extra weight when it conflicts with the rest.
user — whatever came from a human. This is always untrusted data.
assistant — past model answers, including its requests to call tools.
tool — results your code returned for a model request. Each such result must reference the call identifier.

messages:
  system    rules, format, boundaries
  user      first user request
  assistant tool call request (call id)
  tool      call result (same id)
  assistant final text
  user      next request

What goes into the system instruction

The system message sets the role, the answer format, the boundaries and the refusal rules. It does not change between calls, which makes it the first candidate for prompt caching of the prefix — if the provider supports it and the system part is long and constant, you pay for it at a reduced rate. Changing user data must not go there: it breaks the cache and mixes trusted content with untrusted content.

Separating roles is not cosmetic, it is your first line of defense. The model is trained to treat system rules as higher priority. If you glue your instruction and the user text into one string under the user role, you give that advantage away for free: the line "ignore previous instructions" inside a customer review gets exactly the same standing as your rules.

How to truncate history

Window of recent messages. Cheap and predictable, but it loses facts from the start of the conversation.
Compression into a summary. Old messages are replaced by a short state text. It costs one extra call but keeps length constant.
Structured state. Store parsed fields — the order, the step, the confirmed parameters — instead of dialogue text, and build the prompt from them.

The third option is almost always better when the task is not free-form chat. It gives constant context length, reproducibility, and the ability to check state with ordinary tests.

Message order matters too

The model treats the beginning and the end of the array differently. Instructions placed at the very end sit closer to the moment of generation and are followed more willingly than the same words at the start of a long context. A practical trick: keep general behavior rules in the system message, and repeat the format requirement for this specific answer as the last line of the user message. That duplication costs a dozen tokens and noticeably lowers the share of wrong-format answers on long prompts.

Insight. The system message costs money on every call. A 900-token instruction at a million calls a month is a separate budget line that nobody notices, because it is spread thin.

Common mistake. Putting variable data into the system instruction — user name, date, cart contents. That resets the prefix cache and makes every call more expensive for no benefit.

Pro tip. Keep the system instruction in a version-controlled file, not in a string literal in the middle of your code. Then a change in product behavior shows up in the change history and can be rolled back.

Cheat sheet

Conversation memory is created by your code; the model stores nothing.
Roles separate trusted from untrusted — do not glue them together.
The system part is constant: it is a candidate for prefix caching.
For scripted tasks store state as a structure, not as conversation text.

1. Why does a long conversation get expensive faster than it seems?

2. Where should user data not be placed?

3. What makes structured state better than history text?

Task — checked by AI

Take one real model call from your code or project. Write out its messages array in full, marking the role of each message and the token count of each. Separately note which parts change from call to call and which stay constant. Then estimate what this array becomes after ten conversation turns, and describe which truncation strategy you would choose and why exactly that one.

← Back

🔒 Answer the question correctly to move on to the next lesson.