Spectral Matting Anat Levin 1,2 Alex Rav-Acha 1 Dani Lischinski 1 1 School of CS&Eng The Hebrew University 2 CSAIL MIT

Hard segmentation and matting Hard segmentation compositing matte compositing Source image

Previous approaches to segmentation and matting InputHard outputMatte output

Previous approaches to segmentation and matting Unsupervised InputHard outputMatte output Spectral segmentation: Spectral segmentation: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Zelnik and Perona 05 Tolliver and Miller 06

Previous approaches to segmentation and matting Unsupervised Supervised InputHard outputMatte output July and Boykov01 Rother et al 04 Li et al 04

Previous approaches to segmentation and matting Unsupervised Supervised InputHard outputMatte output Trimap interface Trimap interface: Bayesian Matting (Chuang et al 01) Poisson Matting (Sun et al 04) Random Walk (Grady et al 05) Scribbles interface: Scribbles interface: Wang&Cohen 05 Levin et al 06 Easy matting (Guan et al 06)

Previous approaches to segmentation and matting Unsupervised Supervised InputHard outputMatte output ?

Unsupervised matting Automatically computed matting components Automatically computed hard segments (Yu and Shi 03) Input

Using components =++ Building foreground object by simple components addition

Generalized compositing equation 2 layers compositing = xx +

Generalized compositing equation 2 layers compositing = xx + K layers compositing = xx + + xx + Matting components

Generalized compositing equation “Sparse” layers- 0/1 for most image pixels Matting components: K layers compositing = xx + + xx +

Goals: Automatically extract matting components from an image Derive analogy between hard spectral segmentation and matting, and use similar tools. Use matting components to automate matte extraction process and suggest new modes of user interaction

Spectral segmentation: Analyzing smallest eigenvectors of a graph Laplacian L Spectral segmentation E.g.: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Maila and shi 01 Zelnik and Perona 05 Tolliver and Miller 06

Spectral segmentation Fully separated classes: class indicator vectors belong to Laplacian nullspace General case: class indicators approximated as linear combinations of smallest eigenvectors Null Binary indicating vectors Laplacian matrix

Spectral segmentation Fully separated classes: class indicator vectors belong to Laplacian nullspace General case: class indicators approximated as linear combinations of smallest eigenvectors Smallest eigenvectors- class indicators only up to linear transformation Zero eigenvectors Binary indicating vectors Laplacian matrix Smallest eigenvectors Linear transformation

The matting Laplacian (Levin, Lischinski and Weiss CVPR06) = Result: F,B can be eliminated from the matting cost Local correlation between matte and image:

semidefinite sparse matrix local function of the image: The matting Laplacian (Levin, Lischinski and Weiss CVPR06)

The matting Laplacian and user constrains Levin et al CVPR06- Input: Image+ user scribbles

The matting Laplacian and user constrains Levin et al CVPR06- Input: Image+ user scribbles Our goal: Matting components from matting Laplacian- without user input Build on hard spectral segmentation ideas

Matting components and the matting Laplacian Claim: For an image consisting of “well separated” layers, the matting components belong to the matting Laplacian nullspace In the general case, matting components are reasonably approximated as linear combinations of smallest eigenvectors Null Matting Laplacian Matting components

From eigenvectors to matting components linear transformation

Hard segmentation- matting analogy Smallest eigenvectorsLinear transformation Traditional LaplacianMatting LaplacianBinary class indicatorsContinuous matting components

From eigenvectors to matting components Smallest eigenvectorsProjection into eigs space K-means 1) Initialization: projection of hard segments 2) Non linear optimization for sparse components

Components (our approach) Levin et al cvpr06 Wang&Cohen 05 Random Walk Poisson Components with the scribble interface

Components (our approach) Levin et al cvpr06 Wang&Cohen 05 Random Walk Poisson Components with the scribble interface

Direct component picking interface =++ Building foreground object by simple components addition

InputGround truth matte70 eigs approximation Limitations Need to set number of components: Too few - may not contain desired matte Too many - complicates computation and user interaction Cluttered images require a large number of components

Conclusions Derived analogy between hard spectral segmentation to image matting Automatically extract matting components from eigenvectors Automate matte extraction process and suggest new modes of user interaction Ground truth data and code available online: vision.huji.ac.il/SpectralMatting =++