January 7
Linear algebra refresher.
- Vector space, subspaces, span, linear independence, basis, orthogonality, etc.
- The four fundamental spaces of a matrix, matrix rank, rank nullity theorem
Reading suggestions:
Linear algebra refresher (cont'd).
- Orthogonality, projections
- Singular value decomposition
Reading suggestions:
Linear algebra refresher (cont'd).
- QR decomposition
- Basics about matrix inverse and determinant
- Moore-Penrose pseudo-inverse
- Eigenvalues / eigenvectors definition
Reading suggestions:
Linear algebra refresher (cont'd).
- Diagonalizability, spectral theorem, Schur decomposition, Jordan canonical form
- Proof of the SVD
Reading suggestions:
Linear algebra refresher (cont'd).
- Matrix norms
- Low rank approximation (Eckart-Young-Mirsky theorem)
- Variational characterization of eigenvalues (Rayleigh quotient), Min-max theorem
Reading suggestions:
Dimensionality reduction
- Principal component analysis (PCA)
- Canonical correlation analysis (CCA)
- Deep CCA
Reading suggestions:
Markov chains, eigenvalues and Page Rank
- Markov chains and connections to spectral theory
- Existence and uniqueness of a stationnary distribution
- Convergence of Markov chains
- Page rank algorithm
Reading suggestions (material I used to prepare the lecture):
- Lecture notes from the Computer Engineering course at ETH Zurich.
- Lecture notes of the Combinatorial Models and Stochastic Algorithms course from Pekka Orponen: part 1 part 2
Spectral learning of HMMs and related models
Reading suggestions:
February 6th
Spectral learning of HMMs and related models (cont'd)
Introduction to tensor decomposition
- Basic tensor operations, Kronecker product, ...
- Notions of tensor rank
- CP and Tucker decomposition
- Jennrich's algorithm and Alternating Least Squares (ALS) for CP and Tucker decomposition
Reading suggestions:
Tensor networks and SVD based tensor decomposition algorithms
- Introduction to tensor networks
- Tensor Train (TT), Tensor Ring (TR), Hierarchical Tucker (HT) decomposition
- SVD based decompostion algorithms for Tucker and TT
Reading suggestions:
February 18th
Tensor networks and SVD based tensor decomposition algorithms (cont'd)
Tensor Train decomposition:
- Tensor Train (TT) decomposition
- Parametrization of ML models with the TT format
- DMRG-like algorithms for optimization in TT format
Reading suggestions:
More on the tensor train decomposition and tensor networks
- SVD-based and DMRG-like algorithms for optimization in TT format
- Beyond TT: PEPS, tensor ring, hierarchical Tucker...
Reading suggestions:
Learning latent variable models via tensor decomposition
- Single topic model and mixture of Gaussians
- Method of moments and tensor decomposition
- Power method for matrix and tensor decomposition
Reading suggestions:
March 10th
- Theoretical Analysis of the Expressive Power of Deep Nets using Tensor Decomposition.
Reading suggestions:
March 12th
March 24th
March 26th
March 31st
Measure Theory (in the smallest nutshell you can find):
- Lebesgue measure, null sets and continuous distribution
- Non-zero polynomial functions are non-zero almost everywhere [Caron and Traynor, 2005]
- A random matrix has full rank with probability one
April 7th
Correction of Homework 1
- Gradients of Tensor Networks and Generalized Tensor Decomposition
Reading suggestions:
- Unsupervised Learning with Matrix Product States
Reading suggestions:
Resources
Linear Algebra
Tensor Decomposition
Class Bibliography
PCA / CCA / ICA
Clustering
Spectral Learning
Tensor Decomposition
Tensor Networks
- Supervised Learning with Quantum-Inspired Tensor Networks (Stoudenmire and Schwab, 2016)
- Exponential Machines (Novikov et al., 2016)
- Learning Relevant Features of Data with Multi-scale Tensor Networks (Stoudenmire, 2017)
- Modeling Sequences with Quantum States: A Look Under the Hood (Bradley et al., 2019)
- A Multi-Scale Tensor Network Architecture for Classification and Regression (Reyes and Stoudenmire, 2020)
- Unsupervised Generative Modeling Using Matrix Product States (Han et al., 2017)
- Supervised learning with generalized tensor networks (Glasser et al., 2018)
- Spectral Methods from Tensor Networks (Moitra and Wein, 2019)
- Tensor Ring Decomposition (Zhao et al., 2016)
- Expressive power of tensor-network factorizations for probabilistic modeling, with applications from hidden Markov models to quantum machine learning (Glassner et al. 2019)
Model Compression
Latent Variable Models
Neural Network Analysis with Tensors
Tensor Completion
Various Applications of Tensor Methods
Misc.
- Neural Word Embedding as Implicit Matrix Factorization (Levi and Goldberg, 2014)
- Understanding Composition of Word Embeddings via Tensor Decomposition (Frandsen and Ge, 2019)
- Incremental Truncated LSTD (Gehring et al., 2016)
- Boosted Sparse and Low-Rank Tensor Regression (He et. al, 2018)
- Learning from Multiway Data: Simple and Efficient Tensor Regression (Yu and Liu, 2016)
- Multi-Dimensional Causal Discovery (Schaechtle et al., 2013)
- Learning to Reason with Third-Order Tensor Product (Schlag and Schmidhuber, 2018)
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature (Martens and Grosse,2015)
- Multilinear PageRank (Gleich et al., 2015)
- Multilinear Dynamical Systems for Tensor Time Series
- Implicit Regularization in Deep Matrix Factorization (Arora et al., 2019)