From words to numbers: why vectors are needed

Vector databases and embeddings · Lesson 1 / 20

Word overlap is a poor substitute for meaning overlap

Classic search is built on an inverted index. A document is split into tokens, every token keeps a list of documents, and relevance is scored with a formula like BM25: the rarer a word is across the collection and the more often it occurs in a document, the larger its contribution. This works as long as the query and the document are written with the same words. The query 'how do I send an item back' and the article 'processing a product return' share no meaningful token. The document will not land in tenth place — it will not appear in the results at all, because it is physically not there.

Where lexical search breaks down

Synonymy. 'Car', 'automobile', 'vehicle' are three tokens and one object. Synonym dictionaries patch this, but they are maintained by hand and do not carry over between domains.
Paraphrase. The user asks with verbs ('the email never arrives'), the document is titled with nouns ('notification delivery'). The overlap is zero.
Morphology and word forms. Stemming fixes endings, but it does not fix 'the payment did not go through' against 'the charge was declined'.
Multi-word concepts. 'Refusal to process a payment' is four tokens; each one is noisy on its own, together they mean one thing.

The replacement idea: turn the object into numbers

An embedding is a function that maps an object — a chunk of text, an image, a product, a user — into a fixed-length vector: a list of 256, 384, 768 or 1536 floating point numbers. The model is trained so that objects treated as similar in its training data receive close vectors. After that, the question 'are these two texts similar' turns into the question 'what is the value of one number computed over two arrays' — an operation the processor runs in nanoseconds and can execute in batches of a million.

query   ──► embedder ──► [0.021, -0.113, ... , 0.008]   768 numbers
                                  |
                                  v  compared with document vectors
doc A   ──► embedder ──► [ ... ]      similarity 0.81
doc B   ──► embedder ──► [ ... ]      similarity 0.34

What this changes in search architecture

A lexical index answers the question 'which documents contain these tokens'. A vector index answers a fundamentally different one: 'which documents are closest to this point'. The second question always returns an answer. For a nonsense query, for a query outside the topic of the collection, for an empty string — you get a top-10 with some similarity scores attached. Lexical search knows how to stay silent, vector search does not. An empty result set is a useful signal; ten irrelevant documents scored 0.62 are not a signal, and you will have to cut them off with a threshold that you set, not the model.

Insight. An embedding encodes not meaning in general, but the notion of similarity the model formed on its training data. If your task treats objects as similar that the model pushed into different corners of the space, no amount of index tuning will repair that.

Common mistake. Assuming that vector search replaces lexical search. Exact identifiers — part numbers, error codes, version numbers, surnames — are blurred by a vector model: SKU-4471 and SKU-4472 are almost the same thing to it, while for the warehouse they are different objects.

Pro tip. Before you build an index, collect 30 real queries and mark by hand the document that must be found for each one. This list will be useful in every lesson that follows and will repay the hour it costs ten times over.

Cheat sheet

Lexical search looks for tokens, vector search looks for the nearest points; these are different questions to ask of your data.
An embedding is a fixed-length vector, usually 256 to 1536 numbers.
Vector search always returns a result, even when the correct answer is 'there is nothing'.
A labelled query set is prepared before the index, not after the first complaint.

1. Why does the query 'how do I send an item back' fail to find the article 'processing a product return' in a lexical index?

2. How does vector search structurally differ from lexical search in its behaviour on a bad query?

3. What does the claim that an embedding does not encode meaning in general actually mean?

🔒 Answer the question correctly to move on to the next lesson.