Long context, fine-tuning, plain search: when they win

Four ways to give a model your knowledge, and three of them sometimes win

Retrieval-augmented generation is neither the only option nor always the best one. The engineering decision follows from the size of the collection, how often it changes, how strong the verifiability requirement is and what budget you accept per request. It is worth reviewing the alternatives honestly, because building a pipeline you do not need costs more than not building one you do.

Everything into the context

If the whole collection fits into the model window with room to spare, you do not need retrieval: put the collection in the prompt. This wins on accuracy, because no chunk can be lost when nothing is discarded. The approach has two limits. The first is money and latency: a context of a hundred thousand tokens is paid for on every request, and time to first token grows. Prefix caching removes part of that cost but not all of it. The second is the quality of attention: over a long context the model is worse at locating a single fact, especially in the middle. A practical guideline: up to a few tens of thousands of tokens of stable text, long context is usually simpler and more accurate than a pipeline; above that it starts losing on cost before it loses on quality.

Fine-tuning

Fine-tuning is good at changing behaviour: tone, answer format, adherence to domain terminology, the habit of filling a particular structure. It works badly as a way to memorise facts. After fine-tuning the model still cannot tell a recalled fact from an invented one, still cannot cite a document, and any change to a fact demands a new training cycle. A sensible combination is common: fine-tuning owns the form, retrieval owns the content.

Plain search without generation

If the user wants a document rather than an answer, generation only gets in the way: it adds latency, cost and the risk of distorting the wording. A lawyer looking for a clause of a contract wants to see the clause. For navigational queries a result list with snippets beats a paraphrase.

A structured query against a database

Questions about how many, over what period and in what status are not text search at all. The answer lives in a table, and the correct architecture is to generate a query against the database instead of looking for similar paragraphs. Vector retrieval gives systematically wrong answers on aggregate questions, because the total is not written down in any document.

collection small and stable        --> long context
a document is needed, not an answer --> plain search
tone and format are needed         --> fine-tuning
numbers and aggregates are needed  --> query against a database
a lot of changing text,
a citation is required             --> retrieval-augmented generation

Insight. The choice is more often driven by the verifiability requirement than by data volume. As soon as an answer must be backed by a link to a specific document, fine-tuning drops out regardless of how large the collection is.

Common mistake. Building a retrieval pipeline over a collection of two hundred pages. It fits into the context whole, and you have added chunking, an index, selection and four new sources of error in exchange for savings that do not exist at that size.

Pro tip. Before designing anything, sort a hundred real user questions into types: factual, navigational, aggregate, procedural. The share of aggregate questions often turns out large enough that text retrieval alone cannot close the task, and it is much better to learn this before release.

Cheat sheet

A small stable collection is simpler and more accurate in long context.
Fine-tuning sets the form of the answer, not the facts.
A navigational query is closed by a document list with no generation.
Questions about quantities go to a database query, not to vector retrieval.

1. Why is fine-tuning a poor fit for factual answers grounded in documents?

2. Which type of question is worst served by retrieving similar chunks?

3. What most often makes long context uneconomical before it becomes inaccurate?

Task — checked by AI

Take a real task of your own and justify your choice of architecture with numbers. Estimate the size of the collection in tokens, how often it changes, whether a citation to the source is required and what budget per answer you are willing to pay. Then sort 30 real user questions into four types: factual, navigational, aggregate, procedural, and give the shares. Finish by writing which approach you choose for each type and which of them should not be closed with text retrieval at all.

← Back

🔒 Answer the question correctly to move on to the next lesson.