CNNs and Computer Vision

Why CNNs for Images?

A 224×224 RGB image has 224×224×3 = 150,528 values. An MLP applied to this flat vector has no awareness that neighboring pixels are related, and the parameter count explodes. CNNs exploit three properties of images:

1. Local connectivity: nearby pixels are more correlated than distant ones
2. Translation equivariance: a cat in the top-left is still a cat in the bottom-right
3. Compositionality: complex patterns are composed of simpler local patterns

These are implemented via shared convolutional filters and pooling.

The Convolution Operation

A 2D convolution slides a filter K (kernel) over the input image I:

(I ∗ K)[i,j] = Σₘ Σₙ I[i+m, j+n] · K[m,n]

For each spatial position, compute the dot product between the kernel and the corresponding patch of the input.

Key hyperparameters: - Kernel size: 3×3 (most common), 5×5, 1×1 (channel mixing) - Stride: step size of the sliding window (stride=2 halves spatial size) - Padding: 'same' keeps output size; 'valid' reduces size by (k−1)/2 - Number of filters (output channels): each filter learns to detect a different feature

Output size formula: H_out = ⌊(H_in + 2P − K) / S⌋ + 1

where P=padding, K=kernel size, S=stride.

Parameter count for one conv layer: K × K × C_in × C_out + C_out (biases)

This is independent of spatial size H × W — the key efficiency of shared weights.

import torch.nn as nn

# Conv2d(in_channels, out_channels, kernel_size, stride, padding)
conv = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1)
# Input: (batch, 3, 224, 224) → Output: (batch, 64, 224, 224)

# Params: 3*3*3*64 + 64 = 1,792 (vs 150528*64 = 9.6M for MLP)

Pooling Layers

Pooling reduces spatial dimensions and introduces limited translation invariance.

pool = nn.MaxPool2d(kernel_size=2, stride=2)  # halves H and W
gap  = nn.AdaptiveAvgPool2d((1, 1))           # (batch, C, H, W) → (batch, C, 1, 1)

Standard CNN Architecture Pattern

Input → [Conv → BN → ReLU → Pool] × N → Global Avg Pool → FC → Softmax

Each convolutional block: - Increases channel count (32 → 64 → 128 → 256) - Decreases spatial resolution (halved by stride or pooling)

class ConvBlock(nn.Module):
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.block = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 3, stride=stride, padding=1, bias=False),
            nn.BatchNorm2d(out_ch),
            nn.ReLU(inplace=True)
        )
    def forward(self, x): return self.block(x)

class SimpleCNN(nn.Module):
    def __init__(self, n_classes):
        super().__init__()
        self.features = nn.Sequential(
            ConvBlock(3, 32),    ConvBlock(32, 32, stride=2),   # /2
            ConvBlock(32, 64),   ConvBlock(64, 64, stride=2),   # /2
            ConvBlock(64, 128),  ConvBlock(128, 128, stride=2), # /2
        )
        self.classifier = nn.Sequential(
            nn.AdaptiveAvgPool2d(1), nn.Flatten(),
            nn.Linear(128, n_classes)
        )
    def forward(self, x): return self.classifier(self.features(x))

Landmark CNN Architectures

ResNet — Residual Learning

ResNet solves the "degradation problem" (deeper networks having higher error):

y = F(x, {Wᵢ}) + x

The residual block learns the residual function F(x) = target − input, which is easier to learn than the full transformation directly.

class ResBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn1   = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn2   = nn.BatchNorm2d(channels)
        self.relu  = nn.ReLU(inplace=True)

    def forward(self, x):
        residual = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return self.relu(out + residual)

EfficientNet — Compound Scaling

Instead of scaling only depth, width, or resolution independently, EfficientNet scales all three simultaneously with a compound coefficient φ:

depth: d = α^φ, width: w = β^φ, resolution: r = γ^φ subject to α·β²·γ² ≈ 2

This achieves better accuracy/efficiency tradeoff than prior architectures.

Vision Transformer (ViT)

Apply the Transformer architecture directly to image patches:

1. Split image into 16×16 patches (224×224 image → 196 patches)
2. Linearly embed each patch (patch embedding)
3. Add positional encoding + class token
4. Run through standard Transformer encoder
5. Use class token output for classification

ViT matches CNN accuracy with enough data (>14M images); struggles with less. Hybrid models (e.g., Swin Transformer) add local inductive biases.

Transfer Learning for Computer Vision

Pre-trained models on ImageNet extract powerful visual features. Fine-tune for your task:

import torchvision.models as models
import torch.nn as nn

# Load pre-trained ResNet-50
model = models.resnet50(weights="IMAGENET1K_V2")

# Freeze all layers except the classification head
for param in model.parameters():
    param.requires_grad = False

# Replace final FC layer for new task
num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, n_classes)  # only this will be trained

# Optionally unfreeze later layers (fine-tuning)
for param in model.layer4.parameters():
    param.requires_grad = True

optimizer = torch.optim.AdamW([
    {"params": model.layer4.parameters(), "lr": 1e-4},
    {"params": model.fc.parameters(), "lr": 1e-3}
])

When to freeze vs. fine-tune:

Data Augmentation for Vision

Augmentation is critical for visual tasks — models must learn appearance invariance:

from torchvision import transforms

# Standard augmentation for training
train_transforms = transforms.Compose([
    transforms.RandomResizedCrop(224, scale=(0.7, 1.0)),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(brightness=0.3, contrast=0.3, saturation=0.3),
    transforms.RandomGrayscale(p=0.1),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])  # ImageNet stats
])

# Validation: only resize and center crop
val_transforms = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])

Advanced augmentation: - CutMix: paste a patch from one image into another, mix labels - Mixup: blend two images linearly, blend labels - AugMix: chain multiple transformations - RandAugment: random selection from 14 augmentation types

Object Detection

Detect and localize multiple objects in an image (bounding boxes + class labels).

Metrics: mAP (mean Average Precision) at IoU thresholds (0.5, 0.5:0.95).

IoU (Intersection over Union): IoU = Area(A ∩ B) / Area(A ∪ B)

Image Segmentation

U-Net (encoder-decoder with skip connections) is the standard architecture for medical imaging segmentation:

Encoder: downsample + extract features
Bottleneck: lowest resolution
Decoder: upsample + concatenate skip connections from encoder
Output: pixel-wise class map

Common Computer Vision Tasks and Approaches