1. Noisy Sparse Subspace Clustering with dimension reduction
Yining Wang, Yu-Xiang Wang, Aarti Singh
Machine Learning Department
Carnegie mellon university

2. Subspace Clustering

3. Subspace Clustering Applications
Motion Trajectories tracking1
1 (Elhamifar and Vidal, 2013), (Tomasi and Kanade, 1992)

4. Subspace Clustering Applications
Face Clustering1
Network hop counts, movie ratings, social graphs, …
1 (Elhamifar and Vidal, 2013), (Basri and Jacobs, 2003)

5. Sparse Subspace Clustering
(Elhamifar and Vidal, 2013), (Wang and Xu, 2013).
Data: 𝑋= 𝑥 1 , 𝑥 2 ⋯, 𝑥 𝑁 ⊆ 𝑅 𝑑
Key idea: similarity graph based on l1 self-regression
𝑥 1
No False Connections
𝑥 2
𝑥 3
𝑥 𝑁
𝑥 1
𝑥 2
𝑥 3
𝑥 𝑁

6. Sparse Subspace Clustering
(Elhamifar and Vidal, 2013), (Wang and Xu, 2013).
Data: 𝑋= 𝑥 1 , 𝑥 2 ,⋯, 𝑥 𝑁 ⊆ 𝑅 𝑑
Key idea: similarity graph based on l1 self-regression
𝑐 𝑖 = argmin 𝑐 𝑖 𝑐 𝑖 1
s.t. 𝑥 𝑖 = 𝑗≠𝑖 𝑐 𝑖𝑗 𝑥 𝑗
Noiseless data
𝑐 𝑖 = argmin 𝑐 𝑖 𝑥 𝑖 − 𝑗≠𝑖 𝑐 𝑖𝑗 𝑥 𝑗 +𝜆 𝑐 𝑖 1
Noisy data

7. SSC with dimension reduction
Real-world data are usually high-dimensional
Hopkins-155: 𝑑=112~240
Extended Yale Face-B: 𝑑≥1000
Computational concerns
Data availability: values of some features might be missing
Privacy concerns: releasing the raw data might cause privacy breaches.

8. SSC with dimension reduction
Dimensionality reduction:
𝑋 =Ψ𝑋, Ψ∈ 𝑅 𝑝×𝑑 , 𝑝≪𝑑
How small can p be?
A trivial result: 𝑝=Ω(𝐿𝑟) is OK.
L: the number of subspaces (clusters)
r: the intrinsic dimension of each subspace
Can we do better?

9. Main Result Pr ∀𝒙∈𝑺, Ψ𝑥 2 2 ∈ 1±𝜖 𝑥 2 2 ≥1−𝛿
𝑝=Ω 𝐿𝑟 ⟹𝑝=Ω(𝑟 log 𝑁 ),if Ψ is a subspace embedding
Random Gaussian projection
Fast Johnson-Lindenstrauss Transform (FJLT)
Uniform row subsampling under incoherence conditions
Sketching ……
Lasso SSC should be used even if data are noiseless.

10. Proof sketch
Review of deterministic success conditions for SSC (Soltanolkotabi and Candes, 12)(Elhamifar and Vidal, 13)
Subspace incoherence
Inradius
Analyze perturbation under dimension reduction
Main results for noiseless and noisy cases.

11. Review of SSC success condition
Subspace incoherence
Characterizing inter-subspace separation
𝜇 ℓ ≔ max 𝑥∈𝑋\ X (ℓ) max 𝑖 normalize 𝑣 𝑥 𝑖 ℓ , 𝑥
where 𝑣(𝑥) solves
max 𝑣 𝑣, 𝑥 − 𝜆 2 𝑣 2 2
𝑠.𝑡 𝑋 𝑇 𝑣 ∞ ≤1
Lasso SSC formulation
Dual problem of Lasso SSC

12. Review of SSC success condition
Inradius
Characterzing inner-subspace data point distribution
Large inradius 𝜌
Small inradius 𝜌

13. Review of SSC success condition
(Soltanolkotabi & Candes, 2012)
Noiseless SSC succeeds (similarity graph has no false connection) if
𝝁<𝝆
With dimensionality reduction:
𝜇→ 𝜇 , 𝜌→ 𝜌
Bound 𝝁− 𝝁 , 𝝆− 𝝆

14. Perturbation of subspace incoherence 𝜇
𝜈= argmax 𝜈: 𝑋 𝑇 𝜈 ∞ ≤1 𝜈,𝑥 − 𝜆 2 𝜈 2 2
𝜈 = argmax 𝜈: 𝑋 𝑇 𝜈 ∞ ≤1 𝜈, 𝑥 − 𝜆 2 𝜈 2 2
We know that 𝜈,𝑥 ≈ 𝜈, 𝑥 …
So 𝜈≈ 𝜈 because of strong convexity

15. Perturbation of inradius 𝜌
Main idea: linear operator transforms a ball to an ellipsoid

16. Main result
SSC with dimensionality reduction succeeds (similarity graph has no false connection) if
𝝁+ 𝟑𝟐 𝝐/𝝀 +𝟑𝝐 <𝝆
Noisy case: (𝜂 is the adversarial noise level)
𝜇 𝜂 2 𝜌 ℓ + 8(𝜖+3𝜂) 𝜆 +3𝜖<𝜌
Takeaways: the geometric gap Δ=𝜌−𝜇 is a resource that can be traded-off for data dimension reduction
Regularization parameter of Lasso
Error of approximate isometry
Lasso SSC required even for noiseless problem
O(1) if 𝑝=Ω(𝑟 log 𝑁 )

17. Simulation results (Hopkins 155)

18. Conclusion
SSC provably succeeds with dimensionality reduction
Dimension after reduction 𝑝 can be as small as Ω(𝑟 log 𝑁 )
Lasso SSC is required for provable results.
Questions?

19. References
M. Soltanolkotabi and E. Candes. A Geometric Analysis of Subspace Clustering with Outliers. Annals of Statistics, 2012.
E. Elhamifar and R. Vidal. Sparse Subspace Clustering: Algorithm, Theory and Applications. IEEE TPAMI, 2013
C. Tomasi and T. Kanade. Shape and Motion from Image Streams under Orthography. IJCV, 1992.
R. Basri and D. Jacobs. Lambertian Reflection and Linear Subspaces. IEEE TPAMI, 2003.
Y.-X., Wang and H., Xu. Noisy Sparse Subspace Clustering. ICML, 2013.