Segment Anything Model (SAM) Explained: Promptable Segmentation

The Segment Anything Model (SAM) is a computer vision foundation model developed by Meta AI that generates pixel-level segmentation masks from prompts, allowing users to select objects without training a separate model for every new task. This approach, known as promptable segmentation, makes object selection more flexible for image editing, annotation and computer vision workflows.

But how does SAM turn a simple click or bounding box into an accurate object mask? Understanding its architecture and prompts also helps explain how the model evolved into video segmentation and text-guided object tracking. This guide covers the original SAM, the SA-1B dataset, newer model versions, practical applications and a Python example.

What is the Segment Anything Model (SAM)?

The Segment Anything Model (SAM) is a promptable image segmentation model introduced by Meta in 2023. Given an image and a prompt indicating what to select, it predicts a mask identifying the relevant pixels. The original model was designed to generalise to unfamiliar images and objects without requiring task-specific retraining for every new category.

How is SAM Different From Traditional Image Segmentation?

Traditional segmentation models are often trained for specific classes or datasets. A model designed to segment road surfaces, for example, may not automatically handle unfamiliar objects.

SAM image segmentation offers a more flexible approach: users guide the model towards an object using visual prompts. It can segment many objects beyond a fixed set of training categories, although results depend on image quality, object boundaries and prompt clarity. Promptable segmentation does not guarantee perfect masks in every situation.

What does a Segmentation Mask Represent?

A segmentation mask identifies which pixels belong to a selected object. A bounding box only indicates the approximate region containing it.

For example, a box can locate a person in a photograph, while a mask can trace the person’s visible outline, separating them from the background.

Once the output is clear, the next step is understanding how users guide SAM towards the right object.

How does Promptable Segmentation Work in SAM?

SAM promptable segmentation uses visual guidance to determine which object or region to select. Users can provide points, boxes or an existing mask, then refine the result with additional prompts.

Point Prompts

SAM point prompts identify foreground or background regions. A foreground point tells the model what to include, while a background point indicates what to exclude.

For example, clicking on a dog and adding a background point on nearby grass can help SAM distinguish the animal from its surroundings.

Box Prompts

SAM box prompts use a rectangle around the target object. They are useful when an object is easy to locate but its exact outline is difficult to draw manually.

Mask Prompts and Iterative Refinement

SAM mask prompts provide an existing mask to guide further prediction in supported workflows. Additional points or other prompts can help correct missed areas or remove unwanted pixels.

The original SAM primarily uses visual prompts. Later versions, particularly SAM 3, extend the interaction model with text and visual exemplar prompts.

What is the Architecture of SAM?

The original segment anything model architecture contains three core components: an image encoder, a prompt encoder and a mask decoder. Together, they convert image information and user guidance into candidate segmentation masks.

Image Encoder

The SAM image encoder processes the input image into a visual embedding. Because image features can be reused, the model does not need to recompute the entire image representation each time a user tests another prompt.

Prompt Encoder

The SAM prompt encoder converts points, boxes or mask prompts into representations that guide segmentation towards the intended object.

Mask Decoder

The SAM mask decoder combines image and prompt information to generate candidate masks and associated quality scores. These candidates help the user or application select a suitable result.

The overall workflow is:

This is the original SAM architecture; later versions introduce additional components for tasks such as video tracking and text-guided segmentation.

What is the SA-1B Dataset?

The SA-1B dataset, short for Segment Anything 1 Billion, was released alongside the original SAM. According to Meta’s official dataset documentation, it contains 11 million images and 1.1 billion segmentation masks.

How was SA-1B Created?

Meta developed a model-assisted data engine in which human annotators used interactive SAM tools to create annotations. These annotations helped develop the model and dataset, while the released masks were generated automatically by SAM. A random sample was reviewed by human annotators.

Why does SA-1B Matter?

SA-1B provides a large and diverse collection of images and class-agnostic masks for research into general-purpose segmentation. Unlike conventional datasets, its masks are not organised around a fixed set of object-category labels.

The dataset is intended for permitted research purposes. Check Meta’s SA-1B data licence before considering any commercial use.

SAM vs SAM 2 vs SAM 3.1: What has Changed?

The SAM family has expanded from still-image segmentation to video understanding, text-guided identification and object tracking. As of October 2026, Meta’s official materials identify SAM 3.1 as the latest version.

Model	Main capability	Key distinction
SAM	Image segmentation	Point, box and mask prompts
SAM 2 / SAM 2.1	Image and video segmentation	Memory supports object consistency across frames
SAM 3	Image and video detection, segmentation and tracking	Adds text and visual exemplar prompts
SAM 3.1	More efficient multi-object video processing	Object multiplexing processes multiple tracked objects jointly

How does SAM 2 Handle Video?

SAM 2 extends promptable segmentation to video using a memory mechanism that helps maintain object identity across frames. Users can select an object in a frame, propagate the segmentation through a video and add prompts to correct predictions.

Its training resources include the Segment Anything Video (SA-V) dataset, which contains video annotations known as masklets—object masks tracked over time.

What is New in SAM 3 and SAM 3.1?

SAM 3 supports short text descriptions, visual exemplars and visual prompts to identify, segment and track matching objects in images and videos.

SAM 3.1 introduces object multiplexing, allowing up to 16 objects to be tracked in a single forward pass. Meta reports that this doubles processing throughput from 16 to 32 frames per second for videos with a medium number of objects on a single H100 GPU. These are reported results under specific conditions, not a guarantee of performance on every system.

The versions are not architecturally identical: SAM 2 introduces video memory, while SAM 3 and SAM 3.1 build on broader detection and tracking capabilities.

What are the Use Cases of SAM?

SAM can support workflows that require object masks, while later versions extend these capabilities to video and text-guided tasks.

Image Editing and Background Removal

SAM masks can isolate foreground objects, select image regions and support background removal. The resulting mask may still need refinement around hair, transparent objects or complex edges.

Dataset Annotation for Computer Vision

SAM image annotation can speed up the creation of segmentation labels for training datasets. Human review remains important when annotation quality affects model performance.

Video Editing and Object Tracking

SAM 2 and later versions can follow selected objects across video frames, supporting object-focused editing and analysis. SAM 3 and SAM 3.1 add capabilities for identifying and tracking objects described by text or visual examples.

Scientific and Industrial Imaging

Potential applications include robotics, inspection and scientific image analysis. However, specialist applications require domain-specific validation; SAM should not be assumed to provide reliable medical or industrial results without testing.

How can You Use SAM in Python?

The original SAM repository provides an inference API for generating masks from point and box prompts. The following SAM Python example uses the original implementation, not SAM 3.1.

Basic Python Inference Example

import numpy as np
import torch
from PIL import Image
from segment_anything import (
    sam_model_registry,
    SamPredictor,
)
# Download the official checkpoint separately.
checkpoint = "sam_vit_h_4b8939.pth"
model_type = "vit_h"
sam = sam_model_registry[model_type](
    checkpoint=checkpoint
)
device = "cuda" if torch.cuda.is_available() else "cpu"
sam.to(device=device)
predictor = SamPredictor(sam)
image = np.array(
    Image.open("image.jpg").convert("RGB")
)
predictor.set_image(image)
# Pixel coordinates: (x, y); 1 means foreground.
masks, scores, _ = predictor.predict(
    point_coords=np.array(),
    point_labels=np.array([1]),
    multimask_output=True,
)
best_mask = masks[np.argmax(scores)]

What should Developers Know Before Running It?

Install dependencies and download the appropriate checkpoint using the official Segment Anything GitHub repository. The point coordinates must match the supplied image, and the model may return multiple candidate masks. Inspect the scores and visual output before selecting a result.

For current SAM 3.1 setup and inference instructions, use the separate official SAM 3 repository; do not assume the original SAM API works unchanged.

Conclusion

The Segment Anything Model has made computer vision segmentation more flexible by allowing users to identify objects through prompts rather than relying solely on task-specific models. From the original SAM’s image segmentation capabilities to SAM 2’s video understanding and SAM 3.1’s more efficient multi-object tracking, the evolution of the SAM family demonstrates how promptable AI is expanding the possibilities of visual understanding. Understanding these models helps developers choose the right approach for their computer vision applications.

Ready to take your understanding of Generative AI beyond individual models and explore how these technologies work together in real-world applications? Explore the Generative AI programme by IIT Delhi, delivered on Varsity by InterviewBit to build a stronger foundation in modern AI systems, vision-language models and emerging Generative AI techniques.

Frequently Asked Questions

1. Does SAM understand the meaning of an object, or only its visual boundaries?

The original SAM primarily predicts visual boundaries based on prompts rather than interpreting an object’s semantic meaning. SAM 3 adds text-guided identification of concepts, although its capabilities have limits.

2. Can SAM segment several objects in the same image?

Yes. The original SAM includes an automatic mask generator that can propose masks for multiple objects and regions in an image.

3. Does SAM work on images captured in low light or with motion blur?

It may produce masks, but poor lighting, blur and unclear boundaries can reduce reliability. Results should be inspected for the intended application.

4. Can SAM produce instance masks when objects overlap?

It can generate candidate masks for individual objects, but overlapping objects and occlusion can make separation difficult. Additional prompts may improve the result.

5. Can SAM be fine-tuned for a specialised industry or dataset?

Adaptation may be possible using version-specific training workflows and labelled data. The appropriate method depends on the SAM version and the target task.

6. What hardware is needed to run SAM locally?

Requirements depend on the model variant, image size and workload. A compatible GPU is recommended for faster inference, but actual memory and speed requirements should be checked against the chosen checkpoint and setup.

7. Can SAM be used in a commercial product?

Do not assume all SAM-family models and datasets share the same terms. Review the specific model’s licence, dataset conditions and any applicable usage restrictions before commercial deployment.

8. How should segmentation masks be evaluated for accuracy?

Common measures include intersection over union (IoU), which compares predicted and reference masks, and boundary-focused metrics. Choose metrics that reflect the needs of the application.

9. Can SAM replace manual annotation completely?

Not reliably in every workflow. SAM can accelerate annotation, but difficult boundaries, missed objects and domain-specific requirements may still require human review.

10. How does SAM relate to open-vocabulary object detection?

Open-vocabulary detection identifies objects using flexible category descriptions rather than only fixed labels. SAM 3 combines concept detection with segmentation and tracking, producing masks as well as object localisation.

Resources

1. Meta AI: Segment Anything
2. Segment Anything 1 Billion (SA-1B) Dataset
3. Original Segment Anything Research Paper
4. Official Segment Anything GitHub Repository
5. SAM 2: Segment Anything in Images and Videos
6. SAM 2 Research Paper
7. Meta AI: Segment Anything Model 3
8. Official SAM 3 GitHub Repository
9. SAM 3 Research Paper