Matrix and tensor factorization techniques for machine learning

- Fundamental notions of linear and multilinear algebra.

- Old and new ML methods leveraging matrix and tensor decomposition: PCA/CCA, collaborative filtering, spectral graph clustering, spectral methods for HMM, K-FAC, spectral normalization, tensor method of moments, NN/MRF compression, tensor regression/completion, etc.

- Open problems.

- Linear algebra fundamentals
- Vector spaces, linear maps, matrices, rank, change of basis, Hilbert spaces (orthogonality), etc.
- Matrix decomposition: eigenvalues/eigenvectors, diagonalizability (Jordan canonical form), Gram-Schmidt algorithm, QR decomposition, singular value decomposition.
- Linear algebra and matrix factorization techniques for ML
- Linear algebra is everywhere: PCA, CCA, LLE, ...
- Spectral graph clustering
- Spectral learning of HMMs, PCFGs and related models
- Collaborative filtering, matrix completion and low rank matrix recovery
- Multilinear algebra and tensor factorization techniques for ML:
**Tensors are the new matrices!**(quoting David Steurer) - Presentation of several tensor decomposition formats: CanDecomp/ParaFac (CP), Tucker, Tensor Train (TT), etc.
- Tensor networks: a unifying tool
- Some tensor decomposition algorithms: alternating least squares, SVD based algorithms (HOSVD, TT-SVD), DMRG like algorithms, etc.
- Tensor method of moments: consistent estimators from observable tensors
- Algorithms for CP decomposition of symmetric tensors: Jenrich's algorithm, tensor power method, sum of squares hierarchy...
- Multilinear extensions: tensor completion, tensor recovery, tensor regression, ...
- Compressing ML models (e.g. NN, MRF) with tensor factorization

- The matrix cookbook: a very useful collection of matrix identities
- Essence of linear algebra video series by Grant Sanderson (3blue1brown): a must watch, both as a refresher and to gain some geometric intuition on linear algebra.
- Gilbert Strang's Linear Algebra course from MIT opencourseware.
- Linear Algebra done Right (abridged)
- Matrix Analysis by Horn and Johnson: goes in considerably more depth than what we will need in the course but a standard reference to know about.
- Some lecure notes I used for the preparation of the Linear Algebra refresher lectures:
- Lecture notes on linear algebra by Prof. David Lerner from University of Kansas
- ENGG 5781 Matrix Analysis and Computations course by Prof. Wing-Kin (Ken) Ma at the Chines University of Hong-Kong
- ECE 712 Course Notes by Prof. James Reilly at McMaster University.
- Numerical Methods for Solving Large Scale Eigenvalue Problems course by Prof. Peter Arbenz at ETH Zürich.

- Two surveys:
- (Kolda and Bader, 2009): very good introductory survey
- (Grasedyck, Kressner and Tobler, 2013): contains a long list of references and pointers