Semantic vs Instance vs Panoptic Segmentation: The Difference

Semantic segmentation gives every pixel of an image a class: this pixel is building, that one is road, the next is tree. Instance segmentation goes one step further and separates the objects of a class from each other: this pixel is building number 12, that one is building number 13.

The difference in one line

Semantic answers "how much". Instance answers "how many, and which one". Panoptic, the third term, answers both on the same map.

The same picture, labelled two different ways. Semantic segmentation paints every building the same colour. Instance segmentation gives each one its own outline and its own number.

Classification, detection, segmentation

Segmentation is the most detailed of three steps. Image classification says what an image contains, with no location. Object detection says where each object is, with a rectangle. Segmentation says which pixels belong to what, per IBM.

A box is a poor footprint: it holds background pixels and cannot follow a roof's shape, as Roboflow notes. That is why mapping work moves from detection to segmentation.

What each one gives you on a map

A semantic model gives you surfaces to measure, and an instance model gives you objects to count.

Semantic is a land cover map with sharp edges: a building mask, a road mask, a vegetation mask. Right for area statistics, impervious surface, canopy cover, change between two dates. Wrong for anything that needs a count.

Instance is an inventory. Every roof, tree crown or pool arrives as its own polygon with its own row in the attribute table. You can count, measure each one, join a height to each one. The cost is that the model has to decide where one object ends, which is hard on a terraced row, and the failures look like it: two houses merged, one crown split in three.

Instance models also cost more to build. Per viso.ai, they need better and more training data, more complex code and more compute than semantic ones.

Where the words come from. Both are computer vision terms. In 2014, Long, Shelhamer and Darrell showed that a network trained "end-to-end, pixels-to-pixels" could do semantic segmentation. In 2017, Mask R-CNN added to an object detector "a branch for predicting an object mask", which is instance segmentation. In 2023, Segment Anything was trained on "over 1 billion masks on 11M licensed and privacy respecting images", and that is the family the QGIS instance tools now run.

Panoptic segmentation, the third term

Panoptic segmentation labels every pixel with a class, like semantic, and also gives each countable object its own ID, like instance. Kirillov and colleagues named the task in 2018 and defined it as one that "unifies the typically distinct tasks of semantic segmentation (assign a class label to each pixel) and instance segmentation (detect and segment each object instance)".

The paper splits classes into two kinds. Things are countable: buildings, cars, trees. Stuff is amorphous: grass, water, road surface. A panoptic map counts the things and measures the stuff, in one layer with no gaps and no overlaps. The same paper proposed a single score for both, panoptic quality (PQ), "for all classes (stuff and things)".

Where GIS uses it. Farmland is the clearest case. PASTIS, a French benchmark, holds 124,422 agricultural parcels across 2,433 Sentinel-2 time series at 10 m a pixel, covering about 4,000 km² of France. Every pixel carries both a crop type and a parcel index. A semantic model on it says "wheat here". A panoptic one says "wheat, parcel 57", so you can count fields and measure each one. The paper behind it calls pixel-precise panoptic segmentation of parcels a task with "major economic and environmental implications".

In practice, a GIS team rarely needs a panoptic model. Run an instance tool for the things, a semantic one for the stuff, and stack the two layers in QGIS. That gives the same map in two steps.

What a semantic mask counts: three French tiles, measured

Polygonise a semantic building mask and you count blocks, not buildings: on 1 km² of central Lens, IGN's AI land cover map, made from 20 cm aerial photos, gives 269 building blobs where the national building register has 3,006 buildings. We measured this on 23 September 2026 on three 1 km² tiles, each checked against IGN's BD TOPO buildings.

A finer mask does not fix the count. The mask finds 96 to 97% of the buildings on each tile, but a terrace or an apartment block shares its walls, so no background pixel ever separates two houses, at 20 cm or at 2 cm. In Lens and Paris a building found by the mask sits, at the median, in a blob of about 30.

Liévin shows where a semantic mask does count. Detached houses come out as one blob each, and 344 of the 926 buildings found there have a blob to themselves. The semi-detached pairs and the rows of the old mining estate come out as one blob each, which is why the median building sits in a blob of two.

For area, the semantic mask is close. The AI mask comes out 7 to 17% above the sum of footprints, which is expected: it sees roofs, which overhang the walls BD TOPO traces, and small annexes. The 10 m "built-up" class is 1.8 to 3.8 times the footprint area, because a 10 m pixel holding a roof, a yard and a pavement is labelled built-up. It measures urban fabric, not buildings.

Trees merge too

On the Paris tile, 1,296 of the 1,583 trees the city inventories sit in the AI mask's tree canopy (within 2 m of the trunk), and 89% of them share a canopy blob with another inventoried tree. The planted promenade along boulevard Richard-Lenoir is a single blob holding 254 inventoried trees. Counting crowns from a canopy mask works for isolated trees and fails along a planted street.

How we measured this

Tiles are 1 km squares in Lambert 93 centred on Lens (50.432 N, 2.832 E), Cité Pasteur in Liévin (50.433 N, 2.762 E) and boulevard Richard-Lenoir (48.860 N, 2.370 E). A blob is a group of touching mask pixels of 20 m² or more, which is what QGIS Polygonize makes with its default settings, and a building counts as found when half its footprint is under the mask. CoSIA is published only as a styled map, so we read it at about 0.4 m and mapped each colour back to its class. The few edge pixels lost that way split blobs, so the shares above are, if anything, low. The photos behind CoSIA predate the register we compared them with. BD TOPO sometimes draws a semi-detached pair as one building. The Paris inventory leaves out private trees. Three tiles are three places, not a national figure.

How each one is scored

Each type has its own accuracy score, so numbers from two papers rarely compare.

• Semantic: mean IoU. IoU measures the overlap between a predicted shape and the true one, per Ultralytics. Averaged over classes it gives mean IoU, or mean IU. The FCN paper reported 62.2% mean IU on PASCAL VOC 2012. Encord lists PSPNet at 85.4% mIoU on the same benchmark.
• Instance: average precision (AP). AP is the area under the precision-recall curve. mAP averages it over classes, and mAP50-95 averages it over IoU thresholds from 0.50 to 0.95 (Ultralytics).
• Panoptic: panoptic quality (PQ). One score for stuff and things, as defined above.

A good mIoU does not mean you can count. In the Lens tile above, the mask finds 96 to 97% of the buildings and still gives 269 blobs for 3,006 buildings. For an inventory, test your count against a register, not only the benchmark score.

From mask to polygons in QGIS

A semantic mask needs three or four processing steps to become usable polygons, and an instance tool hands you the polygons directly. Here is each path with the QGIS tools that do it.

From a semantic mask

1. Remove speckle. Run Raster > Analysis > Sieve. It "removes raster polygons smaller than a provided threshold size (in pixels)" and fills them with the value of the largest neighbour, per the QGIS manual. The default threshold is 10 pixels. At 20 cm, 10 pixels is only 0.4 m², so raise it to match the smallest object you care about.
2. Polygonize. Run Raster > Conversion > Polygonize (Raster to Vector). It "creates vector polygons for all connected regions of pixels in the raster sharing a common pixel value", with the value in a field named DN by default, per the manual. Leave "Use 8-connectedness" off. With it on, two pixels touching only at a corner join, so two buildings meeting at one corner become one polygon.
3. Keep the class. Background pixels form polygons too. Select "DN" = 1 (or your class value) and save the selection as a new layer.
4. Decide what one feature means. For a class surface, run Vector > Geoprocessing Tools > Dissolve with no field: you get one multipart feature and one area figure. For blobs, tick "Keep disjoint features separate" (QGIS 3.32 or later), or run Multipart to singleparts from the Processing Toolbox, and each blob becomes its own row.
5. Clean the edges. Pixel edges come out as staircases. The raster to vector guide covers simplify, smooth and an area filter.

What you have after step 4 is one polygon per blob, not per object. The Lens table above is what that difference looks like: 269 blobs for 3,006 buildings.

This is also how large footprint datasets are made. Microsoft's global building footprints are built in two stages: "Semantic Segmentation", recognizing building pixels with deep neural networks, then "Polygonization", converting those pixels into polygons. A pixel counts as building when its probability is above 0.5. The polygon step is where the separation into buildings happens, and it is extra work a semantic model leaves to you.

From an instance tool

1. Run the tool on a zone. Each object arrives as its own polygon, one row in the attribute table.
2. Check for multiparts. Some tools return an object in several pieces. Multipart to singleparts splits them, then check the ones that were split before you count.
3. Measure each one. Add a field with $area in the field calculator, and filter out slivers below the smallest real object.
4. Join what you know. Each polygon can take a height, an address or a register ID with a spatial join, which a class surface cannot.

With AI Segmentation steps 1 to 3 collapse into one: type "building" or "swimming pool", pick a zone, and each object found comes back as its own polygon in a new layer. With AI Agent, the semantic path above runs from one sentence in a chat panel inside QGIS: you describe the layer you want, and it runs the same QGIS tools, Sieve, Polygonize and Dissolve, then checks the map before it answers.

Which one the QGIS tools produce

Most QGIS plugins built on land cover models are semantic, and the tools built on Segment Anything are instance.

• Semantic: Deepness and most of the models in its zoo, the Semi-Automatic Classification Plugin.
• Instance: the Segment Anything family, so Geo-SAM, samgeo and the SAM plugins for QGIS. One object per click.
• AI Segmentation, the plugin we make at TerraLab for QGIS, is an instance tool. You name the object, a building, a tree, a pool, or draw one example of it, and it returns one polygon per object found in the zone. The building footprint guide shows what that looks like on a dense block. Where the buildings are already mapped, the free footprint downloader gives you those instance polygons as GeoJSON without running anything.

Questions people ask

The short answers below cover what people search next to these two terms.

What is the difference between semantic, instance and panoptic segmentation?

Semantic segmentation gives each pixel a class. Instance segmentation separates each object of a class into its own mask. Panoptic segmentation does both at once: every pixel gets a class, and each countable object, such as a building or a car, also gets its own ID.

Which one should I use for a GIS project?

Use semantic when the answer is an area or a share: canopy cover, impervious surface, flooded land. Use instance when the answer is a count or a per-object attribute: buildings, trees, pools, solar panels. If you need both, run one of each and stack the layers.

Is object detection the same as instance segmentation?

No. Object detection returns a rectangle around each object. Instance segmentation returns its outline, pixel by pixel. For a map you almost always want the outline.

Is YOLO used for segmentation?

Yes. YOLO started as an object detector, and some versions add a segmentation head, so they return one mask per object, which is instance segmentation (viso.ai). Mask R-CNN, SOLO and YOLACT are other common instance models, and U-Net, DeepLab and PSPNet are common semantic ones (GeeksforGeeks). Pick by task and test on your own imagery, not by leaderboard.

Can I turn a semantic mask into instances?

Sometimes. Polygonise the mask, then split the polygons where they neck. It works on separated things like tree crowns in an orchard. It fails on a terrace, where there is no neck to cut at.

Does the resolution of the imagery change the answer?

Yes, but less than people expect. At 10 m a house is one or two pixels and there is nothing to separate, so satellite land cover is semantic by necessity. At 20 cm a house is thousands of pixels and instance segmentation becomes possible, yet a semantic mask at 20 cm still merges every house that shares a wall: in central Lens, 97.5% of the buildings it finds share a blob.

Why does my polygonized mask give one polygon for a whole block?

Because the houses in that block share walls, so no background pixel separates them in the mask. Polygonize joins every connected pixel of the same value into one polygon. Only an instance model, which draws each object on its own, separates them.

Is Segment Anything instance segmentation?

Yes, in its output: it returns one mask per object you point at, so two touching roofs come back as two masks. The original model gives no class label, though. It separates objects without naming them, and the QGIS tools built on it take the class from your click or your prompt.

Both semantic and instance segmentation sit inside the wider GeoAI category. What GeoAI means covers the term these two are examples of. Tree canopy mapping shows both sides on trees: canopy cover from a mask, or one polygon per crown. The QGIS AI hub collects what AI can and cannot do inside QGIS.