Catalog

Matrix Calculus for Machine Learning and Beyond

Mathematics MIT CC BY-NC-SA 4.0 17 lectures

We all know that calculus courses such as {{% resource_link "54fb3d8f-10d8-493c-94d7-17d7859a8c87" "*18.01 Single Variable Calculus*" %}} and {{% resource_link "6fcb92d1-f2a8-467a-b462-14945ef4fca6" "*18.02 Multivariable Calculus*" %}} cover univariate and vector calculus, respectively. Modern applications such as machine learning and large-scale optimization require the next big step, "matrix calculus" and calculus on arbitrary vector spaces. This class covers a coherent approach to matrix calculus showing techniques that allow you to think of a matrix holistically (not just as an array of scalars), generalize and compute derivatives of important matrix factorizations and many other complicated-looking operations, and understand how differentiation formulas must be reimagined in large-scale computing.

Syllabus

  1. 1 Lecture 1 Part 1: Introduction and Motivation
  2. 2 Lecture 1 Part 2: Derivatives as Linear Operators
  3. 3 Lecture 2 Part 1: Derivatives in Higher Dimensions: Jacobians and Matrix Functions
  4. 4 Lecture 2 Part 2: Vectorization of Matrix Functions
  5. 5 Lecture 3 Part 1: Kronecker Products and Jacobians
  6. 6 Lecture 3 Part 2: Finite-Difference Approximations
  7. 7 Lecture 4 Part 1: Gradients and Inner Products in Other Vector Spaces
  8. 8 Lecture 4 Part 2: Nonlinear Root Finding, Optimization, and Adjoint Gradient Methods
  9. 9 Lecture 5 Part 1: Derivative of Matrix Determinant and Inverse
  10. 10 Lecture 5 Part 2: Forward Automatic Differentiation via Dual Numbers
  11. 11 Lecture 5 Part 3: Differentiation on Computational Graphs
  12. 12 Lecture 6 Part 1: Adjoint Differentiation of ODE Solutions
  13. 13 Lecture 6 Part 2: Calculus of Variations and Gradients of Functionals
  14. 14 Lecture 7 Part 1: Derivatives of Random Functions
  15. 15 Lecture 7 Part 2: Second Derivatives, Bilinear Forms, and Hessian Matrices
  16. 16 Lecture 8 Part 1: Derivatives of Eigenproblems
  17. 17 Lecture 8 Part 2: Automatic Differentiation on Computational Graphs

Course materials