Preprocessing: what to clean and what you must not break

Natural language processing (NLP) · Lesson 2 / 24

Every deleted character is a hypothesis

The standard text cleaning script looks the same in a hundred repositories: lowercase, drop punctuation, drop digits, drop stop words, collapse whitespace. Every line of that script is a claim that whatever it removes carries no information for your task. Half of those claims are usually false, and that gets discovered in production.

What almost always gets cleaned

Structural markup. HTML tags, leftover markdown, email signatures, quoted conversation history. In a support ticket, three quarters of the characters can be a quote of the previous message, and the model learns from that instead of from the new question.
Invisible characters. Non-breaking space, soft hyphen, zero-width space, several kinds of quotes and dashes. They turn one word into two different tokens and break exact matching.
Duplicates. The same text landing in the sample forty times skews both training and evaluation. A duplicate that leaks across the train and test split is especially dangerous.
Encoding garbage. Mojibake, double-encoded text, random runs of question marks. That is a sign of a broken source, not of text.

What you must not break

Case carries information. A message typed in all caps carries sentiment, and after lowercasing the abbreviation US is indistinguishable from the pronoun us, while IT collapses into it. Punctuation carries sentence boundaries, and exclamation marks and ellipses carry emotion; for sentiment analysis, removing them costs several points of quality. Digits carry the substance of financial and medical text: strip them and you have equated a nine thousand invoice with a nine million one. Stop words contain negation: delete not and without and you have turned does not work into works. Emoji in reviews and chat are a strong sentiment signal, not noise.

operation             safe for                 dangerous for
lowercasing           topics, search           entities, abbreviations, sentiment
dropping punctuation  bag of words             sentence splitting, emotion
dropping digits       topics                   finance, medicine, logistics
dropping stop words   keyword search           negation, intent

The ordering rule

Cleaning runs before tokenization and must be identical at training time and at inference time. A mismatch here gives you the nastiest class of bugs: metrics on the test set look excellent, production is worse, and the difference is invisible to the eye. Here is how to check it. Take a hundred random documents, run them through the pipeline, and read them before and after in two columns. Half an hour of that reading usually turns up three errors that would otherwise have lived for months.

Insight. A modern general-purpose model digests dirty text better than a classical pipeline does: it needs neither lowercasing nor stop word removal. Aggressive cleaning in front of such a model hurts more often than it helps.

Common mistake. Copying a cleaning function out of somebody else's project together with its stop word list. Standard English lists contain not, no, never and without, and negation leaves the corpus along with them, taking half the meaning of every complaint with it.

Pro tip. Store the raw text alongside the cleaned text, not instead of it. When it turns out a month later that cleaning ate something you needed, redoing it costs one line instead of a fresh data collection round.

Cheat sheet

Every cleaning operation is a testable hypothesis that what it removes is uninformative.
Quoted email history, invisible characters and duplicates get removed almost always.
Case, negation, digits and emoji are removed only with proof that they are not needed.
The cleaning pipeline is identical in training and in production, otherwise the metrics lie.

1. What is the danger of removing a standard stop word list from a corpus of complaints?

2. Why must the cleaning pipeline match between training and inference?

3. How should you generally treat aggressive cleaning in front of a general-purpose model?

Task — checked by AI

Run a hundred random documents from your corpus through your current cleaning step and lay them out in two columns: before and after. Find and describe at least five cases where cleaning removed something meaningful, and at least three cases where garbage survived. For every operation in the pipeline write one line: what hypothesis stands behind it, and whether that hypothesis holds on your corpus. State which operations you removed or added as a result.

← Back

🔒 Answer the question correctly to move on to the next lesson.