A map of text tasks and choosing the right framing

Framing decides more than model choice

A request to "run an analysis of our reviews" is not a task. You cannot assemble a training set from it, you cannot write annotation guidelines from it, and you cannot compute whether anything got better or worse. The first thing an engineer does is translate the request into one of a handful of standard forms. Each form has its own answer format, its own metrics, and its own cost of error.

Five forms that cover almost everything

• Classification. Input is a span of text, output is a label from a closed list: request topic, sentiment, spam, language, priority. One label or several, but the list is known in advance.
• Extraction. Input is a document, output is a set of spans with their types and positions: an amount, a date, a contract number, an organisation name. The answer is anchored to a place in the text.
• Search and similarity. Input is a query, output is a ranked list of documents. The course on vectors and semantic search covers this in depth; here it is enough to remember that the metric is rank-based, not per-item.
• Transformation. Input is text, output is different text: translation, shortening, rewriting in another register. A single gold answer does not exist.
• Question answering. Input is a question plus a source, output is an answer together with a pointer to where it came from. The key difference from transformation is that the answer must be supported by the source.

One business need, several framings

"Understand what customers are unhappy about" unfolds into at least three different systems. Classification over a list of twelve complaint reasons gives you numbers for a dashboard, but it requires that the list already exists and is complete. Extracting the mentioned objects and the claims made against them gives you detail, but collecting statistics over free-form wording is much harder. Clustering without a list finds themes nobody thought of, but its output cannot be compared across quarters. There is no correct answer here. There is a choice, and it is made before data collection, not after it.

customer request         framing                     output            metric
"sort the mail"       -> classification, 8 classes   label             recall on rare
"pull the details"    -> extraction, 6 fields        fields + spans    precision per field
"find similar ones"   -> ranking                     list of 10        hit in top-5
"give me a summary"   -> transformation              text              human rating

Testing the framing before you start

A workable framing passes three tests. First: you can take one real example and write down the correct answer in full, inventing nothing along the way. Second: two people who read your instructions will produce the same answer on that example. Third: there is a number that will be computed on a held-out set, and you already know which value counts as good enough. If even one of these tests fails, you are not yet at the stage of choosing a model. Go back and rewrite the framing until all three pass, because everything downstream inherits the ambiguity you leave here.

Cheat sheet

• A text task reduces to classification, extraction, search, transformation, or answering with a source.
• One need unfolds into several framings with different costs and different metrics.
• A framing works if the answer can be written by hand, reproduced by a second person, and measured by a number.
• Twenty annotated examples come before the choice of tooling.