Skip to main content

Optimization Methods - Deep Dive

Mathematical Foundations​

Rigorous treatment of optimization in linear algebra.

Optimization Theory​

Critical Points​

Theorem: If f is differentiable and x* is a local minimum, then ∇f(x*) = 0

Proof: (To be completed)

Second Derivative Test​

Theorem: For f(x) = ½xᵀAx - bᵀx:

  • If A is positive definite, x* = A⁻¹b is the global minimum
  • If A has negative eigenvalues, x* is a saddle point

Proof: (To be completed)

Constrained Optimization​

Lagrange Multipliers​

Theorem: If x* minimizes f(x) subject to g(x) = 0, then ∇f(x*) = λ∇g(x*) for some λ

Proof: (To be completed)

KKT Conditions​

(To be completed)

Finite Element Method​

Variational Formulation​

Theorem (Principle of Minimum Energy): (To be completed)

Weak Form​

(To be completed)

Convergence Theory​

(To be completed)

Connection to Machine Learning​

Least Squares as Optimization​

(To be completed)

Gradient Descent​

(To be completed)

Exercises​

(Advanced problems to be completed)