Matrix Calculus for

Machine Learning &

Beyond

Alan Edelman, Steven G. Johnson - MIT

1.  Introduction and Motivation

2. Derivatives as Linear Operators

3. Derivatives in Higher Dimensions: Jacobians and Matrix Functions

4. Vectorisation of Matrix Functions

5. Kronecker Products and Jacobians

6. Finite-Difference Approximations

7. Gradients and Inner Products in other Vector Spaces

8. Nonlinear Root Finding, Optimisation, and Adjoint Gradient Methods

9. Derivative of Matrix Determinant and Inverse

10.  Forward Automatic Differentiation via Dual Numbers

11. Differentiation on Computational Graphs

12.  Adjoint Differentiation of ODE Solutions

13. Calculus of Variations and Gradients of Functionals

14. Derivatives of Random Functions

15. Second Derivatives, Bilinear Forms, and Hessian Matrices

16. Derivatives of Eigen-problems

17. Automatic Differentiation on Computational Graphs