Section 3: Integral Calculus and Optimization
Overview
Focus on integration as it arises in probabilistic ML (expectations, marginalizations, normalizing constants) and on the calculus-based theory of optimization.
Topics Covered
Chapter 1: Integration for ML
- Computing expectations and marginalizations
- Normalizing constants and partition functions
- Monte Carlo integration
- Change of variables with the Jacobian determinant
Chapter 2: Calculus of Optimization
- First-order and second-order optimality conditions
- Convexity characterization via the Hessian
- Gradient descent convergence analysis
- Newton's method and Lagrange multipliers
Learning Objectives
- Compute expectations and marginalizations using integration
- Understand Monte Carlo methods as integral approximations
- Apply first- and second-order conditions to classify critical points
- Analyze gradient descent convergence for convex and strongly convex functions