📝 Computer Vision (CV)

Description

< What is it? >

Definition: Computer Vision (CV) develops methods for extracting information from images and video. It connects numerical pixel values to descriptions of objects, locations, shapes, motion, and scenes.
Example: given a street photograph, a vision system might identify a car, draw a box around each pedestrian, or mark the pixels belonging to the road. These are different tasks with different output formats.
Scope: this page introduces visual representations, common tasks, and evaluation. The Computer Vision (CV) Fine-Tuning subpage contains the Stanford Cars dataset-preparation, model-adaptation, and training exercises.

Key points

< Images and video as numerical data >

Pixels and channels: a digital image is a grid of pixels. A grayscale image has one intensity channel; an RGB image has red, green, and blue channels. In an 8-bit RGB image, each channel ranges from 0 to 255: (255, 0, 0) represents red, while (0, 0, 0) represents black.
Image dimensions: an RGB image with height 480 and width 640 can be represented as an array of shape (480, 640, 3), or height–width–channels. Many PyTorch vision models receive batches shaped (B, 3, 480, 640), where B is the batch size. The required layout, resolution, and numerical scaling depend on the model.
Spatial structure: neighboring pixels and their arrangement carry information about edges, textures, and shapes. Two images can contain the same colors in different positions and depict different objects.
Video: a video is an ordered sequence of frames. Motion and temporal context help distinguish events that a single frame cannot resolve, such as a person sitting down versus standing up.

< Common computer-vision tasks >

Task and output: choose the task according to the information the application needs. A class label, bounding box, pixel mask, and embedding answer different questions.

Task	Question answered	Example input → output
Image classification	What category describes the image?	Car photograph → `sports car`
Object detection	Which objects are present, and where?	Street photograph → `car` and `person`, each with a bounding box and score
Semantic segmentation	Which class does each pixel belong to?	Street photograph → road, car, person, and background pixel labels
Instance segmentation	Which pixels belong to each individual object?	Two cars → a separate mask for each car
Keypoint detection	Where are important landmarks?	Person photograph → shoulder, elbow, and wrist coordinates
Optical Character Recognition (OCR)	What text appears in the image?	Receipt photograph → recognized text and, depending on the system, text locations
Depth estimation	How far away are surfaces?	Scene image → relative or metric depth map, depending on the model
Object tracking	Which object is the same one across frames?	Video → object locations with persistent identities
Image retrieval	Which stored images are visually or semantically similar?	Product photograph → ranked matching products

See the Hugging Face guides to object detection and image segmentation for box- and mask-based outputs.

Classification variants: single-label classification assigns one class from a defined set. Multi-label classification can assign several labels to one image, such as beach, sunset, and people.
Zero-shot image classification: a pretrained model can classify images without additional labeled training for the target task. For example, CLIP ranks candidate class descriptions against an image. Zero-shot prompting instead describes a prompt without worked demonstrations; the two concepts can overlap. See Zero-shot prompting vs. zero-shot image classification.

< From pixels to visual features >

Features: visual features represent useful patterns such as edges, textures, parts, or object-level appearance. Classical methods may explicitly calculate descriptors; neural networks learn representations from data. Google's Machine Learning Crash Course introduces how neural networks learn nonlinear relationships.
Convolutional Neural Networks (CNNs): convolutional filters process local neighborhoods with shared weights. Successive layers can combine local patterns into more complex representations. See Convolutional Neural Network (CNN).
Vision Transformers (ViTs): a common design divides an image into patches, embeds them, and uses attention to model relationships between patches. Positional information preserves their arrangement. See Vision Transformer (ViT).
Backbone and task head: a backbone extracts visual representations; a task head converts them into the required output, such as class scores or bounding boxes. A visual embedding can instead support retrieval by comparing compatible representations of query and stored images. Torchvision's model documentation illustrates several vision tasks and model families.

< A typical vision inference pipeline >

Processing stages: an application decodes the image, applies the model's required preprocessing, runs inference, and interprets the output.

Image or video frame
        ↓
Decode, resize, and normalize as required
        ↓
Vision model
        ↓
Class scores, boxes, masks, or embeddings
        ↓
Application result: label, annotation, measurement, or retrieval ranking

Coordinate handling: resizing or cropping changes image coordinates. Boxes and masks must be mapped back to the original image before displaying them. Some processors perform this conversion through a postprocessing function.
Output interpretation: a single-label classifier commonly produces one logit per class; softmax converts these into class probabilities. Selecting the largest probability gives the predicted class. A high score does not guarantee correctness or probability calibration.

< Evaluation and common challenges >

Match the metric to the task: classification can use accuracy, precision, recall, F1, and a confusion matrix. Detection commonly uses mean Average Precision (mAP) under specified box-overlap criteria. Segmentation often uses Intersection over Union (IoU), the area of overlap between predicted and reference masks divided by their union.
Variation in appearance: lighting, viewpoint, scale, blur, and partial occlusion can change an object's pixels substantially without changing its identity. Evaluation should represent the conditions in which the system will operate.
Generalization: models can rely on background cues instead of the intended object. For example, a car classifier may learn showroom backgrounds that correlate with car labels. Similar frames from one video or photographs of the same subject across data splits can also produce misleading evaluation results.
Training and inference: training updates parameters using a learning objective; inference applies the resulting model to new inputs. Fine-tuning is one way to adapt a pretrained model, covered separately in the CV fine-tuning walkthrough.

Comparison

< Image processing, computer vision, and multimodal understanding >

Relationship: these areas overlap, but emphasize different outputs. Image processing often prepares or transforms images; computer vision extracts structured information; multimodal systems connect visual information with another modality, such as language.

Area	Typical purpose	Example
Image processing	Transform or improve image data	Resize a photograph or reduce noise
Computer vision	Recognize, locate, or measure visual content	Detect cars and estimate their positions
Vision–language understanding	Relate visual content to language	Answer “Which car is closest to the entrance?” about a photograph

Examples

< One street scene, different outputs >

Sample input: a 640 × 480 street photograph contains two cars, one pedestrian, and a road.

Illustrative outputs: the following examples describe possible outputs; they are not predictions from an executed model. Bounding boxes use (x_min, y_min, x_max, y_max) pixel coordinates, with the origin at the upper-left corner.

Task	Illustrative result	Information retained
Scene classification	`street`	One category for the whole image
Object detection	`car: (40, 180, 220, 330)`; `car: (300, 160, 470, 290)`; `person: (500, 150, 550, 350)`	Separate object locations, without exact outlines
Semantic segmentation	A `480 × 640` label map with background `0`, road `1`, car `2`, and person `3`	Pixel-level classes; both cars share class ID `2`
Instance segmentation	Three object masks: car A, car B, and pedestrian	Pixel-level outlines and separate object identities

Selecting the output: use detection when object locations are sufficient. Use instance segmentation when individual outlines matter, such as estimating how much of each car is visible.

Diffusion Models explains image generation, ControlNet guidance, and masked inpainting.
Computer Vision (CV) Fine-Tuning adapts a pretrained classifier to Stanford Cars with connected code examples.
Convolutional Neural Network (CNN) explains convolutional image features.
Vision Transformer (ViT) processes images as sequences of patches.
Multimodal Models connects vision, language, and other modalities.
AI Pipelines includes an image background-removal example.
Confusion Matrix explains classification outcomes and related metrics.

Description​

< What is it? >​

Key points​

< Images and video as numerical data >​

< Common computer-vision tasks >​

< From pixels to visual features >​

< A typical vision inference pipeline >​

< Evaluation and common challenges >​

Comparison​

< Image processing, computer vision, and multimodal understanding >​

Examples​

< One street scene, different outputs >​

Related ideas​

Reference​