Skip to main content

Linear Algebra for Machine Learning

Introduction​

This linear algebra course is structured around Gilbert Strang's Introduction to Linear Algebra and designed for machine learning practitioners. The material is organized into three sections progressing from foundational theory through spectral analysis to applied methods.

The Three Sections​

Section 1: Core Linear System Theory​

Fundamental algebraic structures and solution spaces.

Topics:

  • Vector spaces, subspaces, basis, dimension, and rank
  • The Four Fundamental Subspaces
  • Matrix inverse and invertibility
  • Linear transformations (kernel, image, rank-nullity)
  • Matrix multiplication (four interpretations), LU decomposition, special matrices
  • Determinants: properties, geometric interpretation, characteristic polynomial

Section 2: Spectral Theory & Matrix Decompositions​

Eigenvalue analysis, positive definiteness, and the SVD.

Topics:

  • Eigenvalues, eigenvectors, and diagonalization
  • Spectral theorem for symmetric matrices
  • Positive definite matrices: tests, quadratic forms, Cholesky, Rayleigh quotient
  • Singular Value Decomposition: geometry, low-rank approximation, pseudoinverse

Section 3: Applied Linear Algebra​

Orthogonality, projections, least squares, optimization, and numerical computation.

Topics:

  • Orthogonal vectors and subspaces, orthogonality of the four subspaces
  • Projections onto lines and subspaces, projection matrices
  • Least squares, linear regression, QR decomposition, Gram-Schmidt
  • Gradient descent and condition number, conjugate gradient
  • SGD, momentum methods (Nesterov, Adam), convexity
  • Constrained optimization: Lagrange multipliers, KKT systems
  • Numerical stability, matrix norms, iterative solvers, sparse techniques
  • Randomized methods: randomized SVD, random projections (Johnson-Lindenstrauss)