1. Spectral Matting Anat Levin 1,2 Alex Rav-Acha 1 Dani Lischinski 1 1 School of CS&Eng The Hebrew University 2 CSAIL MIT

2. Hard segmentation and matting Hard segmentation compositing matte compositing Source image

3. Previous approaches to segmentation and matting InputHard outputMatte output

4. Previous approaches to segmentation and matting Unsupervised InputHard outputMatte output Spectral segmentation: Spectral segmentation: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Zelnik and Perona 05 Tolliver and Miller 06

5. Previous approaches to segmentation and matting Unsupervised Supervised InputHard outputMatte output July and Boykov01 Rother et al 04 Li et al 04

6. Previous approaches to segmentation and matting Unsupervised Supervised InputHard outputMatte output Trimap interface Trimap interface: Bayesian Matting (Chuang et al 01) Poisson Matting (Sun et al 04) Random Walk (Grady et al 05) Scribbles interface: Scribbles interface: Wang&Cohen 05 Levin et al 06 Easy matting (Guan et al 06)

7. Previous approaches to segmentation and matting Unsupervised Supervised InputHard outputMatte output ?

8. Unsupervised matting Automatically computed matting components Automatically computed hard segments (Yu and Shi 03) Input

9. Using components =++ Building foreground object by simple components addition

10. Generalized compositing equation 2 layers compositing = xx +

11. Generalized compositing equation 2 layers compositing = xx + K layers compositing = xx + + xx + Matting components

12. Generalized compositing equation “Sparse” layers- 0/1 for most image pixels Matting components: K layers compositing = xx + + xx +

13. Goals: Automatically extract matting components from an image Derive analogy between hard spectral segmentation and matting, and use similar tools. Use matting components to automate matte extraction process and suggest new modes of user interaction

14. Spectral segmentation: Analyzing smallest eigenvectors of a graph Laplacian L Spectral segmentation E.g.: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Maila and shi 01 Zelnik and Perona 05 Tolliver and Miller 06

15. Spectral segmentation Fully separated classes: class indicator vectors belong to Laplacian nullspace General case: class indicators approximated as linear combinations of smallest eigenvectors Null Binary indicating vectors Laplacian matrix

16. Spectral segmentation Fully separated classes: class indicator vectors belong to Laplacian nullspace General case: class indicators approximated as linear combinations of smallest eigenvectors Smallest eigenvectors- class indicators only up to linear transformation Zero eigenvectors Binary indicating vectors Laplacian matrix Smallest eigenvectors Linear transformation

17. The matting Laplacian (Levin, Lischinski and Weiss CVPR06) = Result: F,B can be eliminated from the matting cost Local correlation between matte and image:

18. semidefinite sparse matrix local function of the image: The matting Laplacian (Levin, Lischinski and Weiss CVPR06)

19. The matting Laplacian and user constrains Levin et al CVPR06- Input: Image+ user scribbles

20. The matting Laplacian and user constrains Levin et al CVPR06- Input: Image+ user scribbles Our goal: Matting components from matting Laplacian- without user input Build on hard spectral segmentation ideas

21. Matting components and the matting Laplacian Claim: For an image consisting of “well separated” layers, the matting components belong to the matting Laplacian nullspace In the general case, matting components are reasonably approximated as linear combinations of smallest eigenvectors Null Matting Laplacian Matting components

22. From eigenvectors to matting components linear transformation

23. Hard segmentation- matting analogy Smallest eigenvectorsLinear transformation Traditional LaplacianMatting LaplacianBinary class indicatorsContinuous matting components

24. From eigenvectors to matting components Smallest eigenvectorsProjection into eigs space K-means 1) Initialization: projection of hard segments 2) Non linear optimization for sparse components

25. Components (our approach) Levin et al cvpr06 Wang&Cohen 05 Random Walk Poisson Components with the scribble interface

26. Components (our approach) Levin et al cvpr06 Wang&Cohen 05 Random Walk Poisson Components with the scribble interface

27. Direct component picking interface =++ Building foreground object by simple components addition

28. InputGround truth matte70 eigs approximation Limitations Need to set number of components: Too few - may not contain desired matte Too many - complicates computation and user interaction Cluttered images require a large number of components

29. Conclusions Derived analogy between hard spectral segmentation to image matting Automatically extract matting components from eigenvectors Automate matte extraction process and suggest new modes of user interaction Ground truth data and code available online: vision.huji.ac.il/SpectralMatting =++