presentsUNAIVERSE
The UN AI timeline

    Learn AI /Computer Vision and Language Technology /Image segmentation

    Image segmentation

    Also called: semantic segmentation.

    Detail

    In five seconds

    What it is

    Image segmentation divides an image into pixel-level regions; semantic segmentation labels them as roads, buildings or water.

    In plain words

    It colours in an image so each tiny part is tagged with what it shows.

    An everyday example

    A planning team is preparing a large paper map of a city before a journey planning exercise. Instead of putting one big box around the centre, they colour every tiny block by what it is: road, park, river, housing or industrial area. After that, they can trace exactly where a walking route crosses roads, enters green space or reaches the riverbank. The map becomes useful not just for naming what exists, but for showing precisely where each type of place begins and ends.

    Where the story stops working

    A city map is already a simplified drawing with clear categories chosen by people. Real photographs are messier: boundaries can be fuzzy, lighting changes, objects overlap, and the system does not start from a clean map. Also, segmentation labels pixels in an image plane, not real-world ownership, land rights or legal land use.

    Remember thisImage segmentation assigns a class to each pixel, producing a map of regions rather than one label or a set of rough boxes. The boundaries remain model predictions and require validation.

    This term is contested

    The broad idea is settled, but people sometimes use "image segmentation" loosely to cover several subtypes, including semantic, instance and panoptic segmentation, so the exact scope depends on context.

    Understand it

    Understand it

    Why it matters

    For a non-specialist, the value is not the label itself but the kind of decision it enables. Pixel-level outlines can support area estimates, route planning, change monitoring, crop mapping, flood extent review, medical image analysis and quality control. Compared with a simple box, segmentation can tell you how much of an image is actually relevant and where boundaries fall. That can improve downstream analysis, but only if the classes fit the real task, the training data reflect the real setting, and humans check whether the masks are reliable enough for the stakes involved.

    From photo to pixel labels

    After viewing this diagram, a reader should be able to explain how a system turns one image into a labelled pixel map and why this is more precise than drawing a box.

    1. 01

      Input image

      Start with one photo, such as a street scene containing road, pavement, cars, people and buildings.

      Show a rectangular photo thumbnail on the left with simple illustrated objects. Add a caption: Original image.

    2. 02

      Pixel grid

      The image is made of many pixels. The task is to assign one class to each pixel.

      Show the same rectangle with a visible grid overlay. Highlight a few individual squares and label them pixel.

    3. 03

      Prediction

      A trained model estimates which class each pixel most likely belongs to by using local detail and wider image context.

      Place a central processing box labelled trained segmentation model. Use arrows from the pixel grid into the box and from the box to the next panel.

    4. 04

      Label map

      The output is a mask: a second image where colours represent classes such as road, person, building and sky.

      Show a flat coloured version of the image on the right, with a small legend mapping colours to class names.

    5. 05

      Useful result

      Because the labels follow boundaries closely, the result can support measurement, routing, counting areas or later analysis.

      Show the original image with a semi-transparent coloured overlay and a callout pointing to an exact road boundary, contrasting it with a faint dashed bounding box as a side note.

    A left-to-right flow with four panels: original street image, image split into many pixels, model output as a coloured mask, and final overlay showing exact labelled regions. Read from input to output to see how pixel-level labels create usable boundaries.

    Hovering or tapping a class in the legend highlights only that class in the mask and overlay; hovering the dashed box note shows the text: boxes are rough location, masks are exact area.

    Describe this without the diagram

    The diagram explains image segmentation as a five-step flow. First, there is an original street image. Second, the same image is shown as a grid to stress that it is made of pixels. Third, a trained segmentation model receives that grid. Fourth, the output is a label map, also called a mask, where each pixel has a class shown by colour, such as road, person, building or sky. Fifth, the mask is overlaid on the original image to show precise boundaries. A side note contrasts this with a dashed bounding box, which only gives a rough location. The main point is that segmentation labels every pixel, not just the whole image or a box around an object.

    With reduced motion: Show the same five panels at once in a static left-to-right row with arrows between them and a fixed legend. Keep the dashed bounding box comparison visible without animation.

    Image segmentation is a computer vision task that divides an image into regions at pixel level. Some forms produce a labelled map. In semantic segmentation, each pixel is assigned to a class such as road, sky, person or tree.

    The main idea is finer spatial detail. Image classification gives one or more labels for the whole image. Object detection finds chosen object instances and marks where they appear, often with boxes. Segmentation instead marks pixel-level regions. Semantic segmentation labels regions by class, while instance segmentation can separate individual objects.

    Many learned segmentation models are trained on images paired with human-made masks. A mask is a second image where colours or numbers show the reference region or class for each pixel. During training, the model adjusts its parameters to reduce the difference between its predicted and reference masks.

    For a new image, such a model uses local patterns such as edges, textures and shapes, along with wider image context. The result might be a coloured overlay, a black-and-white mask for one region, or a data file recording region or class labels.

    This term is often confused with object detection. Detection marks where chosen object instances appear. Segmentation marks their pixel-level extent or divides the image into regions. It also differs from image captioning, which produces text rather than a pixel-level map.

    How “Marking a city map by land use” maps onto the real thing

    In the storyIn the AI system
    colouring each tiny block on the mapassigning a class label to each pixel
    different land-use colours such as road, park and riversegmentation classes such as road, vegetation and water
    seeing exactly where a route crosses into another areafinding precise boundaries between regions in an image
    not relying on one big box around the city centregoing beyond object detection with bounding boxes

    A worked example

    For the situation report, the statistician reviews a georeferenced image tile containing 1,000 by 1,000 pixels and needs an estimate of the share classified as road surface.

    Starting from: One image tile, with each pixel to be labelled road, building, vegetation, water or other.

    1. The model processes the image and predicts one class for each of its 1,000,000 pixels.
    2. The predicted mask contains 420,000 road pixels, 310,000 building pixels, 240,000 vegetation pixels, no water pixels and 30,000 pixels labelled other.
    3. The statistician overlays the mask on the source image and checks boundaries, shadows and parked vehicles, which the model may have confused with road surface.
    4. The road-pixel count is divided by the total pixel count. If the georeferenced pixels represent equal ground areas, the pixel share also estimates the road share of the mapped tile.

    Result: The predicted road share is 420,000 ÷ 1,000,000 × 100 = 42 per cent. This is an estimate derived from the segmentation mask, not proof that every boundary or pixel label is correct.

    In a United Nations workplace

    A statistician in a regional bureau is finalising the surface-water section of a situation report before the regional bulletin closes. Using an approved imagery tool, the statistician segments a satellite image into water, roads, buildings, vegetation and other ground, then sends the resulting area estimates and map to the reporting officer for clearance.

    A pixel-level mask can turn a large image into comparable area estimates and show where roads or built-up areas overlap with detected water. It helps focus manual review when the reporting window is short, but it does not establish conditions on the ground by itself.

    Before you try thisDo not upload security-sensitive imagery, precise protected locations or restricted geospatial data to a tool that has not been approved for them. Cloud, shadow, seasonal water and image resolution can all distort the mask, so a qualified analyst must validate it against the source image and other available information before it informs reporting or operational decisions.

    Where you may hear it

    • A regional bureau review of satellite-derived indicators
    • A vendor demonstration for an approved imagery platform
    • A procurement unit meeting on geospatial analysis requirements
    • An inter-agency working group reviewing a common map
    • A data unit quality review before a monitoring report is cleared

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • It simply finds objects in pictures.

      Not exactly. Object detection usually returns boxes around objects or regions. Segmentation assigns a class to every pixel, so it can outline exact shapes and background regions such as sky, road or water, not only countable objects.

    • Its output is automatically accurate because the boundaries look precise.

      A crisp-looking mask can still be wrong. Shadows, reflections, smoke, low resolution and unusual viewpoints can mislead the model. Precision of appearance is not the same as correctness, so evaluation and human review still matter.

    • It always separates each individual object.

      Semantic segmentation often groups all pixels of the same class together, so two people standing side by side may both be labelled person without being separated into person A and person B. Separating each instance is a related but different task.

    Quick check

    Which output best matches image segmentation?

    Go deeper

    Go deeper

    The simple version

    It colours in an image so each tiny part is tagged with what it shows.

    The technical version

    Image segmentation is a dense prediction task in computer vision. Given an image, the model outputs a label for each pixel rather than one label for the whole image. In semantic segmentation, pixels are assigned to classes such as road, water or building, and all objects of the same class may share the same label map. Training usually uses annotated masks and an optimisation process that reduces the difference between predicted and reference masks. Architectures commonly combine local feature extraction with broader context so the model can place boundaries while still using scene-level cues. Outputs may be represented as per-pixel class probabilities, a discrete mask, or an overlay for inspection. This differs from object detection, which predicts object locations, often with bounding boxes, and from image classification, which predicts only image-level categories.

    Reviewed external resources

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    Image segmentation assigns a class to each pixel, producing a map of regions rather than one label or a set of rough boxes. The boundaries remain model predictions and require validation.