1. USPACOR: Universal Sparsity-Controlling Outlier Rejection
G. B. Giannakis, G. Mateos, S. Farahmand, V. Kekatos, and H. Zhu
ECE Department, University of Minnesota
Acknowledgments: NSF grants no. CCF , EECS ,
May 24, 2011

2. Robust learning
Motivation: (statistical) learning from high-dimensional data
DNA microarray
Traffic surveillance
Preference modeling
Outliers: data not adhering to postulated models
Resilience key to: model selection, prediction, classification, tracking,…
Our goal: `universally’ robustify learning algorithms
Major innovative claim: sparsity control robustness control 2 2

3. Robustifying linear regression
Least-trimmed squares regression [Rousseeuw’87]
(LTS)
is the -th order statistic among
residuals discarded
Q: How should we go about minimizing nonconvex (LTS)?
A: Try all subsets of size , solve, and pick the best
Simple but intractable beyond small problems
Near-optimal solvers [Rousseeuw’06]; RANSAC [Fischler-Bolles’81] 3 3

4. Modeling outliers Outlier variables s.t. Remarks
otherwise
Nominal data obey ; outliers something else
-contamination [Fuchs’99], Bayesian framework [Jin-Rao’10]
Remarks
Both and are unknown
If outliers sporadic, then vector is sparse!
Natural (but intractable) nonconvex estimator 4 4

5. LTS as sparse regression
Lagrangian form
(P0)
Tuning parameter controls sparsity in number of outliers
Proposition 1: If solves (P0) with chosen s.t ,
then in (LTS).
The result
Formally justifies the regression model and its estimator (P0)
Ties sparse regression with robust estimation 5 5

6. Just relax! (P0) is NP-hard relax ; e.g., [Tropp’06]
(P1) convex, and thus efficiently solved
Role of sparsity-controlling is central
Q: Does (P1) yield robust estimates ?
A: Yap! Huber estimator is a special case
where 6 6

7. Minimizers of (P1) are fully determined by
Lassoing outliers
Suffices to solve Lasso [Tibshirani’94]
Proposition 2:
as and
Minimizers of (P1) are fully determined by
Enables effective data-driven methods to select
Lasso solvers return entire robustification path (RP)
Cross-validation (CV) fails with multiple outliers [Hampel’86] 7 7

8. Robustification paths
Coeffs.
Lasso path of solutions is piecewise linear
LARS returns whole RP [Efron’03]
Same cost of a single LS fit ( )
Lasso is simple in the scalar case
Coordinate descent is fast! [Friedman ‘07]
Exploits warm starts, sparsity
Other solvers: SpaRSA [Wright et al’09], SPAMS [Mairal et al’10]
Leverage these solvers consider a grid
values of with 8 8

9. Selecting Relies on RP and knowledge on the data model
Number of outliers known: from RP, obtain range of s.t Discard outliers (known), and use CV to determine
Variance of the nominal noise known: from RP, for each on the grid, find the sample variance
The best is s.t.
Variance of the nominal noise unknown: replace above with a robust estimate , e.g., median absolute deviation (MAD) 9 9

10. USPACOR vs. RANSAC , i.i.d. Nominal: , , i.i.d. Outliers: i.i.d. for10 10

11. Beyond linear regression
Nonparametric (kernel) regression
Doubly-robust Kalman smoother [Farahmand et al’10]
Fixed-lag KS
Fixed-lag DRKS
State:
Measurement:
Loss functions: quadratic, , Huber, -insensitive
General criteria
Regularization for and : ridge, (group)-Lasso, adaptive Lasso,… 11 11

12. Unsupervised learning
Sparsity control for robust PCA [Mateos-Giannakis’10]
Low-rank factor analysis model:
Original
Robust PCA
`Outliers’
Outlier-aware robust clustering [Forero et al’11]
Generative model for K-means:
Data
Clustering result 12 12

13. OUTLIER-RESILIENT ESTIMATION
Concluding remarks
Universal sparsity-controlling framework for robust learning
Tuning along RP controls:
OUTLIER-RESILIENT ESTIMATION
CONVEX OPTIMIZATION
LASSO
Degree of sparsity in model residuals
Number of outliers rejected
Universality
Information used for selecting
Nominal data model
Criterion adopted to fit the chosen model
More on USPACOR:
Now! - `Robust nonparametric regression by controlling sparsity’
Friday - `Outlier-aware robust clustering’ 13 13