Linear Algebra for ML

Why Linear Algebra?

Nearly every ML operation is a matrix computation: a forward pass through a neural network is a sequence of matrix multiplications; PCA is an eigendecomposition; least-squares regression is a matrix equation. Understanding these building blocks lets you read papers, debug shapes, and reason about computation cost.

Scalars, Vectors, Matrices, Tensors

In numpy notation, a column vector is shape (n, 1) and a row vector is (1, n).

import numpy as np

a = 3.14                        # scalar
v = np.array([1, 2, 3])        # vector shape (3,)
A = np.array([[1,2],[3,4]])     # matrix shape (2,2)
T = np.zeros((32, 224, 224, 3))# tensor (batch of images)

Vector Operations

Dot Product

u · v = Σᵢ uᵢvᵢ = ‖u‖ ‖v‖ cos θ

Geometric meaning: projection of one vector onto another. If u · v = 0 the vectors are orthogonal.

u = np.array([1, 2, 3])
v = np.array([4, 5, 6])
dot = np.dot(u, v)   # 1*4 + 2*5 + 3*6 = 32

Norms (vector length)

• L1 norm: ‖v‖₁ = Σ|vᵢ| (Manhattan distance)
• L2 norm: ‖v‖₂ = √(Σvᵢ²) (Euclidean distance)
• L∞ norm: ‖v‖∞ = max|vᵢ|

L1 and L2 norms appear in regularization (Lasso and Ridge respectively).

l1 = np.linalg.norm(v, ord=1)
l2 = np.linalg.norm(v)        # default is L2

Matrix Operations

Matrix Multiplication

For A (m×k) and B (k×n), C = AB is (m×n):

Cᵢⱼ = Σₗ Aᵢₗ Bₗⱼ

Key rules: - Not commutative: AB ≠ BA in general - Associative: (AB)C = A(BC) - Transpose: (AB)ᵀ = Bᵀ Aᵀ

A = np.random.randn(3, 4)
B = np.random.randn(4, 5)
C = A @ B    # shape (3, 5)

Transpose

(Aᵀ)ᵢⱼ = Aⱼᵢ — flip rows and columns.

In ML, the normal equations for least-squares use Xᵀ.

Element-wise (Hadamard) Product

A ⊙ B: same shape, multiply corresponding elements. Used in LSTM gates, dropout masks.

D = A * B    # numpy broadcasting handles element-wise

Matrix Properties

Rank

The number of linearly independent rows (or equivalently columns). A full-rank (m×n) matrix has rank min(m, n). Rank deficiency causes singular systems with no unique solution.

Determinant

For square matrix A: det(A) is a scalar encoding "volume scaling factor" of the linear transformation. det(A) = 0 ↔ matrix is singular (non-invertible).

Inverse

A⁻¹ exists only if A is square and det(A) ≠ 0:

A A⁻¹ = I

In ML, avoid computing explicit inverses — use np.linalg.solve(A, b) instead of np.linalg.inv(A) @ b for numerical stability.

Trace

tr(A) = Σᵢ Aᵢᵢ — sum of diagonal elements. Appears in covariance matrix analysis and nuclear norm.

Eigenvalues and Eigenvectors

For square A, eigenvector v and eigenvalue λ satisfy:

A v = λ v

The eigenvector is a direction that the matrix only stretches (by λ), never rotates. A (n×n) matrix has up to n eigenvalue-eigenvector pairs.

Applications in ML: - PCA finds eigenvectors of the covariance matrix (principal components) - PageRank is the dominant eigenvector of the link graph - Spectral clustering uses eigenvectors of the graph Laplacian

A = np.array([[4, 2], [1, 3]])
eigenvalues, eigenvectors = np.linalg.eig(A)
# eigenvalues ≈ [5, 2]; eigenvectors are columns

Singular Value Decomposition (SVD)

Any (m×n) matrix A can be decomposed:

A = U Σ Vᵀ

• U: (m×m) orthogonal matrix — left singular vectors
• Σ: (m×n) diagonal matrix — singular values σ₁ ≥ σ₂ ≥ … ≥ 0
• V: (n×n) orthogonal matrix — right singular vectors

Truncated SVD (rank-k approximation): take only the top-k singular values. This is the theoretical basis for PCA, matrix factorization in recommendation systems, and latent semantic analysis.

U, s, Vt = np.linalg.svd(A, full_matrices=False)
# Reconstruct with top-k components
k = 2
A_approx = U[:, :k] @ np.diag(s[:k]) @ Vt[:k, :]

Linear Systems and Least Squares

A linear system Ax = b is: - Consistent (exact solution) if b lies in the column space of A - Overdetermined (more equations than unknowns, typical in ML) — solved approximately by least squares

Least-squares solution (normal equations):

x = (AᵀA)⁻¹ Aᵀ b*

This is the closed-form solution for linear regression where A = feature matrix X, b = targets y.

# Closed-form least squares
X = np.column_stack([np.ones(n), x_feature])  # add bias column
theta = np.linalg.lstsq(X, y, rcond=None)[0]  # numerically stable

Special Matrices

Broadcasting (numpy/pytorch)

Broadcasting lets you operate on arrays of different shapes without explicit replication:

# Add scalar to matrix
A = np.ones((3, 4))
A + 5               # adds 5 to every element

# Add row vector to matrix (broadcasts over rows)
bias = np.array([1, 2, 3, 4])   # shape (4,)
A + bias            # shape (3, 4) — adds bias to each row

# Neural net: add bias to batch
W = np.random.randn(512, 256)   # weight matrix
x = np.random.randn(32, 512)    # batch of 32 inputs
b = np.random.randn(256)        # bias vector
out = x @ W + b                 # shape (32, 256)

Rules: dimensions align from the right; a size-1 dimension is stretched to match the other.

Computational Complexity

This is why large-scale deep learning avoids explicit inverses and prefers iterative gradient-based methods.

Common ML Shapes to Remember