1. Matrix Extensions to Sparse Recovery Yi Ma 1,2 Allen Yang 3 John Wright 1 CVPR Tutorial, June 20, 2009 1 Microsoft Research Asia 3 University of California Berkeley 2 University of Illinois at Urbana-Champaign TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA A A A A AA A

2. FINAL TOPIC – Generalizations: sparsity to degeneracy The tools and phenomena underlying sparse recovery generalize very nicely to low-rank matrix recovery ???

3. FINAL TOPIC – Generalizations: sparsity to degeneracy The tools and phenomena underlying sparse recovery generalize very nicely to low-rank matrix recovery Matrix completion: Given an incomplete subset of the entries of a low-rank matrix, fill in the missing values. Robust PCA: Given a low-rank matrix which has been grossly corrupted, recover the original matrix.

4. ??? Face images Degeneracy: illumination models Errors: occlusion, corruption Relevancy data Degeneracy: user preferences co-predict Errors: Missing rankings, manipulation Video Degeneracy: temporal, dynamic structures Errors: anomalous events, mismatches… Examples of degenerate data: THIS TALK – From sparse recovery to low-rank recovery

5. KEY ANALOGY – Connections between rank and sparsity Sparse recoveryRank minimization Unknown Vector x Matrix A Observations y = Ax y = L[A] (linear map) Combinatorial objective Convex relaxation Algorithmic tools Linear programming Semidefinite programming

6. KEY ANALOGY – Connections between rank and sparsity This talk: exploiting this connection for matrix completion and RPCA Sparse recoveryRank minimization Unknown Vector x Matrix A Observations y = Ax y = L[A] (linear map) Combinatorial objective Convex relaxation Algorithmic tools Linear programming Semidefinite programming

7. CLASSICAL PCA – Fitting degenerate data If degenerate observations are stacked as columns of a matrix then

8. CLASSICAL PCA – Fitting degenerate data If degenerate observations are stacked as columns of a matrix then Principal Component Analysis via singular value decomposition: Stable, efficient computation Optimal estimate of under iid Gaussian noise Fundamental statistical tool, huge impact in vision, search, bioinformatics

9. CLASSICAL PCA – Fitting degenerate data If degenerate observations are stacked as columns of a matrix then But… PCA breaks down under even a single corrupted observation. Principal Component Analysis via singular value decomposition: Stable, efficient computation Optimal estimate of under iid Gaussian noise Fundamental statistical tool, huge impact in vision, search, bioinformatics

10. ROBUST PCA – Problem formulation … … D - observation A – low-rankE – sparse error … Properties of the errors: Each multivariate data sample (column) may be corrupted in some entries Corruption can be arbitrarily large in magnitude (not Gaussian!) Problem: Given recover. Low-rank structureSparse errors

11. ROBUST PCA – Problem formulation Problem: Given recover. Low-rank structureSparse errors Numerous heuristic methods in the literature: Random sampling [Fischler and Bolles ‘81] Multivariate trimming [Gnanadesikan and Kettering ‘72] Alternating minimization [Ke and Kanade ‘03] Influence functions [de la Torre and Black ‘03] No polynomial-time algorithm with strong performance guarantees! … … D - observation A – low-rankE – sparse error …

12. ROBUST PCA – Semidefinite programming formulation Seek the lowest-rank that agrees with the data up to some sparse error:

13. ROBUST PCA – Semidefinite programming formulation Seek the lowest-rank that agrees with the data up to some sparse error: Not directly tractable, relax:

14. ROBUST PCA – Semidefinite programming formulation Seek the lowest-rank that agrees with the data up to some sparse error: Not directly tractable, relax: Semidefinite program, solvable in polynomial time Convex envelope over

15. MATRIX COMPLETION – Motivation for the nuclear norm Related problem: we observe only a small known subset of entries of a rank- matrix. Can we exactly recover ?

16. MATRIX COMPLETION – Motivation for the nuclear norm Related problem: recover a rank matrix from a known subset of entries Convex optimization heuristic [Candes and Recht] : Spectral trimming also succeeds with for For incoherent, exact recovery with [Keshavan, Montanari and Oh] [Candes and Tao]

17. ROBUST PCA – Exact recovery? CONJECTURE :If with sufficiently low-rank and exactly recovers. Sparsity of error sufficiently sparse, then solving Empirical evidence: probability of correct recovery vs rank and sparsity Perfect recovery Rank

18. Decompose as or ? ROBUST PCA – Which matrices and which errors? Fundamental ambiguity – very sparse matrices are also low-rank: rank-1 rank-0 0-sparse1-sparse Obviously we can only hope to uniquely recover that are incoherent with the standard basis. Can we recover almost all low-rank matrices from almost all sparse errors?

19. ROBUST PCA – Which matrices and which errors? Random orthogonal model (of rank r) [Candes & Recht ‘08] : independent samples from invariant measure on Steifel manifold of orthobases of rank r. arbitrary.

20. ROBUST PCA – Which matrices and which errors? Random orthogonal model (of rank r) [Candes & Recht ‘08] : independent samples from invariant measure on Steifel manifold of orthobases of rank r. arbitrary. Bernoulli error signs-and-support (with parameter ): Magnitude of is arbitrary.

21. MAIN RESULT – Exact Solution of Robust PCA “Convex optimization recovers almost any matrix of rank from errors affecting of the observations!”

22. BONUS RESULT – Matrix completion in proportional growth “Convex optimization exactly recovers matrices of rank, even with entries missing!”

23. MATRIX COMPLETION – Contrast with literature [Candes and Tao 2009]: Correct completion whp for Does not apply to the large-rank case This work: Correct completion whp for even with Proof exploits rich regularity and independence in random orthogonal model. Caveats: - [C-T ‘09] tighter for small r. - [C-T ‘09] generalizes better to other matrix ensembles.

24. MAIN RESULT – Exact Solution of Robust PCA “Convex optimization recovers almost any matrix of rank from errors affecting of the observations!”

25. ROBUST PCA – Solving the convex program Semidefinite program in millions of unknowns. Scalable solution: apply a first-order method with convergence to Sequence of quadratic approximations [Nesterov, Beck & Teboulle] : Solved via soft thresholding (E), and singular value thresholding (A).

26. ROBUST PCA – Solving the convex program Iteration complexity for suboptimal solution. Dramatic practical gains from continuation

27. SIMULATION – Recovery in various growth scenarios Correct recovery with and fixed, increasing. Empirically, almost constant number of iterations: Provably robust PCA at only a constant factor more computation than conventional PCA.

28. SIMULATION – Phase Transition in Rank and Sparsity Fraction of successes with, varying (10 trials) Fraction of successes with, varying (65 trials) [0,.5] x [0,.5][0,1] x [0,1] [0,.4] x [0,.4]

29. EXAMPLE – Background modeling from video Video Low-rank appx.Sparse error Static camera surveillance video 200 frames, 72 x 88 pixels, Significant foreground motion

30. EXAMPLE – Background modeling from video Video Low-rank appx.Sparse error Static camera surveillance video 550 frames, 64 x 80 pixels, significant illumination variation Background variation Anomalous activity

31. EXAMPLE – Faces under varying illumination … … RPCA 29 images of one person under varying lighting:

32. EXAMPLE – Faces under varying illumination … … RPCA 29 images of one person under varying lighting: Self- shadowing Specularity

33. EXAMPLE – Face tracking and alignment Initial alignment, inappropriate for recognition:

34. EXAMPLE – Face tracking and alignment

43. Final result: per-pixel alignment

44. EXAMPLE – Face tracking and alignment Final result: per-pixel alignment

45. SIMULATION – Phase Transition in Rank and Sparsity Fraction of successes with, varying (10 trials) Fraction of successes with, varying (65 trials) [0,.5] x [0,.5][0,1] x [0,1] [0,.4] x [0,.4]

46. CONJECTURES – Phase Transition in Rank and Sparsity 1 1 0 0 Hypothesized breakdown behavior as m  ∞

47. CONJECTURES – Phase Transition in Rank and Sparsity 1 1 0 0 What we know so far: This work Classical PCA

48. CONJECTURES – Phase Transition in Rank and Sparsity 1 1 0 0 CONJECTURE I: convex programming succeeds in proportional growth

49. CONJECTURES – Phase Transition in Rank and Sparsity 1 1 0 0 CONJECTURE II: for small ranks, any fraction of errors can eventually be corrected. Similar to Dense Error Correction via L1 Minimization, Wright and Ma ‘08

50. CONJECTURES – Phase Transition in Rank and Sparsity 1 1 0 0 CONJECTURE III: for any rank fraction,, there exists a nonzero fraction of errors that can eventually be corrected with high probability.

51. CONJECTURES – Phase Transition in Rank and Sparsity 1 1 0 0 CONJECTURE IV: there is an asymptotically sharp phase transition between correct recovery with overwhelming probability, and failure with overwhelming probability.

52. CONJECTURES – Connections to Matrix Completion Our results also suggest the possibility of a proportional growth phase transition for matrix completion. 1 1 0 0 How do the two breakdown points compare? How much is gained by knowing the location of the corruption? Robust PCA Matrix Completion Similar to Recht, Xu and Hassibi ‘08 Matrix CompletionRobust PCA

53. FUTURE WORK – Stronger results on RPCA? RPCA with noise and errors: Tradeoff between estimation error and robustness to corruption? Deterministic conditions on the matrix Simultaneous error correction and matrix completion: bounded noise (e.g., Gaussian) Conjecture: stable recovery with we observe

54. Faster algorithms: Smarter continuation strategies Parallel implementations, GPU, multi-machine Further applications: Computer vision: photometric stereo, tracking, video repair Relevancy data: search, ranking and collaborative filtering Bioinformatics System Identification FUTURE WORK – Algorithms and Applications

55. Reference: Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Matrices by Convex Optimization submitted to the Journal of the ACM Collaborators: Prof. Yi Ma (UIUC, MSRA) Dr. Zhouchen Lin (MSRA) Dr. Shankar Rao (UIUC) Arvind Ganesh (UIUC) Yigang Peng (MSRA) Funding: Microsoft Research Fellowship (sponsored by Live Labs) Grants NSF CRS-EHS-0509151, NSF CCF-TF-0514955, ONR YIP N00014-04-1-0633, NSF IIS 07-03756 REFERENCES + ACKNOWLEDGEMENT

56. Questions, please? THANK YOU! John Wright Robust PCA: Exact Recovery of Corrupted Low-Rank Matrices