Skilvy

Three ways to give an LLM your knowledge

There are several approaches to make an LLM "know" your data: put it all in the context (prompt), fine-tune the model, or use RAG. Each has its place. Understanding when to use what is an important engineering decision. Let us cover the differences.

RAG fits best when data is extensive, changing and source citations are needed.

Three approaches

All in context: insert the needed data directly into the prompt. Simple, but the data must be small (context limit), and costly to repeat a large context each time.
Fine-tuning: fine-tune the model on your data. Good for style and behavior, but costly, hard to update, and the model still tends to invent facts.
RAG: find the relevant in your data and feed it into the context dynamically. Good for working with extensive, changing data on facts.

When to choose RAG, and when fine-tuning or just context?

Guides for the choice: 1) Just insert into context — when the data is SMALL and fits in the prompt (a few paragraphs, a short document). No RAG complexity needed. But if the data is large or the context repeats on each request — costly and you hit the limit. 2) RAG — when there is a LOT of data (does not fit in context), it CHANGES (updated a document — RAG uses the new right away, no retraining), you need FACTUALLY ACCURATE answers over your data, and/or you need SOURCE CITATIONS (RAG knows where it took the answer). This is the most common choice for "answers over a knowledge base / documents". 3) Fine-tuning — when you need to change the STYLE, FORMAT, BEHAVIOR of the model (speak in a certain tone, follow a complex format, a specific task) rather than add factual knowledge. Important: fine-tuning is poor for "remember facts" — the model can still invent, and updating is costly. For facts over your data RAG is almost always better. A common beginner mistake: thinking "answers over our documents" needs fine-tuning. Usually no — you need RAG: it is cheaper, easily updated (just change the data), gives source citations and invents less. Practical rule: little data to context; a lot of changing data with facts to RAG; change style/behavior to fine-tuning. Often combined: RAG for knowledge + possibly fine-tuning for style. For most "AI over our data" tasks the answer is RAG.

A common misconception: "must fine-tune"

A common beginner mistake is thinking that "answers over our documents" needs fine-tuning the model. Usually no. Fine-tuning is good for style and behavior but poor for remembering facts (the model still invents, and updating is costly). For factual answers over your data RAG is almost always better: cheaper, easily updated (just change the data), gives source citations, hallucinates less. Do not rush to fine-tune where RAG is needed.

RAGs strengths

RAG is especially good when: there is a lot of data (does not fit in context), it changes (updated — RAG uses the new at once), accurate factual answers are needed, source citations are needed (important for trust — you can show where the answer came from). That is why RAG is the standard solution for knowledge-base assistants, document support, internal-data search. Later in the course — how to build it.

Rule: little data to context; a lot of changing data with facts and citations needed to RAG; change style/behavior to fine-tuning. For "answers over our data" you usually need RAG, not fine-tuning.

🧠 What is usually needed for "answers over our documents"?

← Back Next →