SSkilvy

Different questions to an image — different tasks

CV is not one task but a spectrum. It is important to understand the difference between the main types, because the tool choice and complexity depend on it. They all answer different questions to an image, with growing precision and complexity. Let us cover the main ones.

Classification: what it isDetection: what and whereSegmentation: which pixels CV tasks Growing precision
Classification answers "what", detection "what and where", segmentation "which exact pixels".

Three main types

  • Classification — "what is it?": one label for the whole image. "The photo is a cat". The simplest task.
  • Detection — "what and where?": boxes around objects with labels. "Here are three cars and two people, each in its box". Harder.
  • Segmentation — "which exact pixels?": a precise mask of the objects shape, not a box. "Here is the exact object contour pixel by pixel". Even more precise.
Explain the difference between classification, detection and segmentation with an example.
Let us cover three tasks on one example — a city-street photo: 1) CLASSIFICATION — "what is this image?": The model gives ONE label for the whole photo. E.g. "a city street" or, if the task is the main object, "a car". One question — one answer for the whole image. — The simplest. — Does not say WHERE the objects are and how many — only the general category. — Application: sorting photos by category, "defect/normal", scene-type detection, face mood. 2) OBJECT DETECTION — "what and where?": The model finds EACH object, draws a box (bounding box) around it and puts a label. On the street photo: a "car" box around each car, a "person" box around each pedestrian, a "sign" box around a sign. — Harder than classification: not one label but MANY objects, each with its position. — Gives the what, where and how many. — Application: counting objects (how many cars), video surveillance, car perception, retail shelf analytics. 3) SEGMENTATION — "which exact pixels belong to the object?": Even more precise than detection. Instead of a rectangular box — a PRECISE MASK: which specific pixels are the car, which are the road, which are the sky. Outlines the exact contour/shape, not a rough rectangle. — The most precise and complex of the three. — Needed when the exact shape/boundary matters: medical scans (the exact tumor contour), extracting an object for photo editing, precise road separation for cars. — Kinds: semantic (all "car" pixels marked as a car) and per-object (distinguishes individual cars). The key distinction in one phrase: — Classification: WHAT is in the picture (one label). — Detection: WHAT and WHERE (boxes around objects). — Segmentation: WHICH EXACT PIXELS (a shape mask). Growing precision = growing complexity = usually more data and computation needed. The practical meaning of the choice: take the task for the REAL need, no more complex than necessary. — Need to just categorize a photo? Classification — simpler, cheaper, less data. — Need to find and count objects, know their location? Detection. — Need the exact object shape/boundary? Segmentation — but it is more complex and demanding. A common mistake is taking a more complex task than needed (segmentation where classification would suffice): more data, computation, complexity without benefit. Determine which QUESTION you ask the image, and choose the minimally sufficient task type. There are other tasks too (OCR — text, face recognition, pose estimation, video tracking), but these three are the foundation of understanding CV. Understanding the difference, you correctly formulate the task and choose the tool.

Growing precision = growing complexity

Note the pattern: from classification to segmentation both the answer precision (label to boxes to pixel mask) and the complexity (usually more data and computation) grow. This matters for the choice: a more precise task is more demanding. Maximum precision is not always needed — often a simpler task suffices, which is cheaper and needs less data.

Take the minimally sufficient task

A practical choice principle: determine which question you ask the image, and take the minimally sufficient task type. Need to just categorize a photo — classification (simpler, cheaper). Need to find and count objects — detection. Need the exact shape/boundary — segmentation (but harder). A common mistake is taking a task more complex than needed (segmentation where classification would suffice): more data, computation, complexity without benefit. There are other tasks too (OCR, faces, pose, tracking), but these three are the foundation of understanding CV.

Rule: classification — "what" (a label), detection — "what and where" (boxes), segmentation — "which pixels" (a mask). Growing precision = growing complexity. Take the minimally sufficient task for your question.

🧠 How does object detection differ from classification?