How a request to the model is built

How a request to the model is built

From a code perspective an LLM API is an ordinary HTTP endpoint: you send JSON, you get JSON back. No magic. But this request has quirks that determine both answer quality and your bill at the end of the month.

The minimal request

POST https://api.openai.com/v1/chat/completions
Authorization: Bearer $OPENAI_API_KEY
Content-Type: application/json

{
  "model": "gpt-4o-mini",
  "messages": [
    { "role": "system", "content": "You are a terse assistant." },
    { "role": "user", "content": "Explain HTTP 429 in one sentence." }
  ],
  "temperature": 0.2
}

Three roles and why they exist

• system — rules for the whole conversation: tone, format, constraints. The model treats it as a priority instruction.
• user — the user's request.
• assistant — the model's previous replies. You send them back so the model "remembers" the conversation.

The point that trips up newcomers: the API is stateless. Every request is independent. Conversation "memory" is simply the messages array you resend in full each time. That's also why long conversations get expensive.

The response shape

{
  "id": "chatcmpl-...",
  "choices": [{
    "message": { "role": "assistant", "content": "429 means rate limited." },
    "finish_reason": "stop"
  }],
  "usage": { "prompt_tokens": 28, "completion_tokens": 9, "total_tokens": 37 }
}

Two fields to read always, not just while debugging:

• finish_reason — stop (fine), length (hit max_tokens — the answer is truncated!), tool_calls (the model wants a tool), content_filter.
• usage — actual token spend. Log it from day one or the bill will be a surprise.

Parameters that actually matter

• temperature (0–2). 0–0.3 for extraction, classification, code. 0.7–1.0 for prose and ideas. Keep it low in production logic: reproducibility beats "creativity".
• max_tokens — the cap on the answer. Always set it: it protects you from runaway generation and runaway cost.
• top_p — an alternative to temperature. Change one, not both.
• seed (where supported) — improves reproducibility without guaranteeing it.

Cheat sheet

• Stateless API: context = the whole messages array.
• Roles: system (rules), user, assistant (history).
• Always read finish_reason and usage.
• Low temperature for logic; always set max_tokens.