1. CIS 5590: Advanced Topics in Large-Scale Machine Learning
Dictionary Learning & Matrix Factorization
Instructor: Kai Zhang
Temple University, Spring 2018

2. Outlines Sparsity Dictionary Learning
Compressive sensing, sparse coding, LASSO
Dictionary Learning
K-means
K-SVD algorithm
Non-negative Matrix Factorization

3. Sparsity
Many zeros in a vector or matrix/tensor

4. Sparsity
Acoustics

5. Compressive Sensing
Definition: sense the data in a compressed form i.e., at a lower sampling rate.
Assumption: If a signal has a sparse representation in a certain domain (or under a known transform, represented in a certain basis), then CS allows sensing the signal with potentially large reduction in the sampling rate for accurate reconstruction

6. Compressive Sensing Conditions for recovery
The sampling matrix should be different from the transform
Measurement matrix and sparse representation should be different

7. Sparse Coding
Image Representation

8. Sparse Coding Biological motivation: V1 visual cortex
Example on MNIST handwritten digits
An image of size 28x28 pixels can be represented using a small combination of codes from a basis set.

9. Sparse Coding
Topic Modelling

10. Sparse coding
Sparse linear model

11. Sparse Learning (LASSO)

12. L1-induced Sparsity

13. Sparse Regression
Sparse linear models

14. Sparse Regression

15. Small Sample Problem

16. Sparse Regression
Simultaneous feature and model selection

17. Solving LASSO How to solve LASSO Quadratic programing
𝛼= 𝛼 + - 𝛼 − , 𝛼 + ≥0, 𝛼 − ≥0, 𝛼 1 = 𝛼 + + 𝛼 −
Sub-gradient method (or coordinate descent)

18. Solving LASSO How to Solve LASSO
Soft thresholding (coordinate descent)
Matching pursuit, Basis pursuit,…
SLEP Package
ftp://ftp.math.ucla.edu/pub/camreport/cam09-17.pdf
Starting from 𝑥 (0)

19. Variants of LASSO Elastic Net Adaptive LASSO Group LASSO formulation
Select more features, stabilized regularization path, grouping effect
Adaptive LASSO
Formulation
Improved convergence property
Group LASSO
Group structured sparsity ( 𝐼 𝑔 specifies group)
Useful for multitask learning

20. Mixed norm regularization