IFT 6760A - Winter 2022 [fr]
Matrix and tensor factorization techniques for machine learning
- The first class will be on Tuesday January 11th at 12:30
- The page of last 2020's edition of the class (with lecture notes and bibliography) is available here for references.
The goal of this course is to present an overview of linear and multilinear algebra techniques for designing/analyzing ML algorithms and models, and to engage students with new research in the area.
- Fundamental notions of linear and multilinear algebra.
- Old and new ML methods leveraging matrix and tensor decomposition: PCA/CCA,
collaborative filtering, spectral graph clustering, spectral methods for HMM, K-FAC,
tensor method of moments, NN/MRF compression, tensor regression/completion, etc.
- Open problems.
Tuesdays 12:30-2:30PM and Thursdays 11:30-1:30PM
First day of class: Tuesday January 11th
Where: Online until Jan. 17th (then TBD, either online or at Mila)
Instructor: Guillaume Rabusseau
Office hours: Tuesdays after class
Class project (30%) + Assignments (30%) + Quizzes (10%) + Paper presentation (30%)
Tentative list of topics (to be updated as we go along)
- Linear algebra fundamentals
- Vector spaces, linear maps, matrices, rank, change of basis, Hilbert spaces (orthogonality), etc.
- Matrix decomposition: eigenvalues/eigenvectors, diagonalizability (Jordan canonical form), Gram-Schmidt algorithm, QR decomposition, singular value decomposition.
- Linear algebra and matrix factorization techniques for ML
- Linear algebra is everywhere: PCA, CCA, LLE, ...
- Spectral graph clustering
- Spectral learning of HMMs, PCFGs and related models
- Collaborative filtering, matrix completion and low rank matrix recovery
- Multilinear algebra and tensor factorization techniques for ML: Tensors are the new matrices!
- Presentation of several tensor decomposition formats: CanDecomp/ParaFac (CP), Tucker, Tensor Train (TT), etc.
- Tensor networks: a unifying tool
- Some tensor decomposition algorithms: alternating least squares, SVD based algorithms (HOSVD, TT-SVD), DMRG like algorithms, etc.
- Tensor method of moments: consistent estimators from observable tensors
- Algorithms for CP decomposition of symmetric tensors: Jenrich's algorithm, tensor power method, sum of squares hierarchy...
- Multilinear extensions: tensor completion, tensor recovery, tensor regression, ...
- Compressing ML models (e.g. NN, MRF) with tensor factorization