Calling a model is a network call to someone else's service

Working with LLM APIs: advanced · Lesson 1 / 21

The model sits behind a wire

From your code the model call looks like a single library line. In reality it is an HTTP request into someone else's data center: a TLS handshake, a load balancer, a queue on the provider side, a quota counter tied to your key, and a real chance the connection drops halfway through the response. Everything you already know about external dependencies — timeouts, retries, degradation, caching — carries over unchanged.

Two things are different, and both are unpleasant. The call lasts seconds or tens of seconds instead of milliseconds, so the usual three-second client timeout is useless. And the call costs money, so a failed retry burns budget, not just time.

What a request is made of

POST /v1/chat/completions
Authorization: Bearer [[key]]
Content-Type: application/json

{
  "model": "model-name",
  "messages": [list of messages with roles],
  "temperature": 0.2,
  "max_tokens": 512,
  "stream": false
}

The request body is data, not a command. The model does not execute your JSON. It receives it as context and continues the text. Everything else follows from that: there is no guaranteed match between request and response, only a probabilistic one that you raise with parameters and schemas.

What a response is made of

{
  "id": "call identifier",
  "model": "version that actually served the call",
  "choices": [
    { "message": {"role": "assistant", "content": "text"},
      "finish_reason": "stop" }
  ],
  "usage": {"prompt_tokens": 812, "completion_tokens": 143, "total_tokens": 955}
}

The usage field is the only trustworthy source of money data: count spend from it, not from your own estimates of text length. The model field shows which version actually ran, and it disagrees with the name you requested more often than people assume — aliases such as latest move to a new version without asking you.

finish_reason matters more than the content

stop — the model finished on its own. The answer can be treated as complete.
length — you hit max_tokens. The text breaks mid-word and the JSON is invalid. This is not an HTTP error; the status is 200.
tool_calls — the model is not answering with text, it is asking you to call a function. Your content is empty in this case.
content_filter — filtering fired. There is no answer, and retrying the same request will not help.

Code that reads choices[0].message.content without looking at finish_reason will one day store truncated text in your database and never notice.

What to measure from day one

A model call has three quantities you must capture right away, because later there is nowhere to get them from: full latency from send to last byte, time to first token, and token spend in each direction. The first number sets your timeouts, the second sets how fast the product feels, the third sets the bill. All three depend heavily on context length, so averaging them across every call is meaningless: break them down by request type.

Insight. Status 200 does not mean the answer is usable. The answer is usable when finish_reason equals stop and the content passes your validation — those are two separate conditions, and you must check both.

Common mistake. Measuring spend by string length in characters. The same text in Cyrillic and in Latin script yields different token counts, and the system part of the prompt is not in your string at all.

Pro tip. Log the response id and the model field next to every result. When a quality complaint arrives a month later, that is the only way to tell which version produced it.

Cheat sheet

A model call is an unstable network dependency, not a function call.
Take spend from usage, not from string length.
Always check finish_reason: length and content_filter arrive with status 200.
The actual model version lives in the response, not in your config.

1. What does finish_reason with the value length mean?

2. Where should token spend data come from?

3. Why is it worth logging the model field from the response?

🔒 Answer the question correctly to move on to the next lesson.