1. Image Matting and Its Applications Chen-Yu Tseng Advisor: Sheng-Jyh Wang 2012-10-29

2. Image Matting A process to extract foreground objects from an image, along with an alpha matte (the opacity of the foreground color) Input ImageAlpha MatteExtracted Foreground

3. Two Approaches of Image Matting Supervised Matting With User’s Guidance Unsupervised Matting Without User’s Guidance Input ImageUser’s Guidance e.g. Trimap: White  Foreground Black  Background Unknown  Gray

4. Two Schemes of Supervised Matting Propagation-based Scheme Infer Alpha Matte with Propagation through a Graphical Model A Global-based Approach Sampling-based Scheme Infer Alpha Matte with Some Color Samples A Local-based Approach Foreground Pixel Background Pixel Unknown Pixel Foreground Color Set Background Color Set Unknown Pixel

5. Propagation-based scheme - Matting Laplacian Approach A Graphical Model with Connectivity between Pixels The Connectivity Is Learned from the Image Structure Capability for Dealing with Both Supervised Matting (Inference Problem) Unsupervised Matting (Decomposition Problem) Foreground Pixel Background Pixel Unknown Pixel

6. Reference of Matting Laplacian Approach First proposed by Levin et al. for supervised matting (closed-form matting) A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008. Extended to unsupervised matting (spectral matting) A. Levin, A. Rav-Acha, D. Lischinski. “Spectral Matting,” IEEE T. PAMI, vol. 30, no. 10, pp. 1699-1712, Oct. 2008. Extended to learning-based matting Y. Zheng and C. Kambhamettu. “Learning based digital matting,” In ICCV, pages 889–896, 2009. Extended to multi-layer matting D. Singaraju, R. Vidal. “Estimation of Alpha Mattes for Multiple Image Layers,” IEEE T. PAMI, vol. 33, no. 7, pp. 1295-1309, July 2011.

7. Matting Laplacian Input Image Estimating Pair-wise Affinity Graphical Model Node: Image Pixels Edge: Affinity Supervised Matting Background Foreground Matting Laplacian Matrix: Recording the Connectivity between Pair of Pixels

8. Introduction of Graph Laplacian 2 2 3 3 1 1 4 4 5 5 A Graph with Five Vertexes 01100 10100 11000 00001 00010 1 2 3 4 5 12345 Vertex Index

9. Introduction of Graph Laplacian 2 3 1 4 5 A Graph with Five Vertexes 01100 10100 11000 00001 00010 1 2 3 4 5 12345 Vertex Index 2 00 2 00 200 0001 000 1 20000 02000 00200 00010 00001

10. Cutting Cost Function with Graph Laplacian Cost Function for Cutting Criterion Low-cost Assignment High-cost Assignment 2 2 3 3 1 1 4 4 5 5 2 2 3 3 1 1 4 4 5 5

11. Construction of Matting Laplacian Color-model-based Approach (Original) Estimating Affinity Based on Relative Color Distance Learning-based Approach (Extended) Learning Affinity Based on Image Structure

12. Construction of Matting Laplacian Color-model-based Approach Color Distribution Input Image A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008. g r b

13. Construction of Matting Laplacian Learning-based Approach Learning Affinity among Local Pixels Linear Alpha-color Model for Single Pixel: Extending to a Local Patch q Assuming all Pixels Sharing the Same Linear Coefficient

14. Construction of Matting Laplacian Learning-based Approach Derived Linear Coefficient Rewritten Linear Model

15. Construction of Matting Laplacian Local Cost Function Input Image Patch q Local Linear Model

16. Construction of Matting Laplacian Local  Global Input Image Patch q

17. Supervised Matting (Closed-form Matting) Foreground Pixel Background Pixel Unknown Pixel Input Image ForegroundBackgroundUnknown 10- 110 Cost Function for Supervised Matting Affinity Cost Data Cost Optimal Solution

18. Experimental Results Input ImageAlpha MatteSynthesized Result

19. Unsupervised Matting (Spectral Matting) Solving Alpha Matte without User’s Guidance Procedures Decomposing Image into Several Matting Components Combining Matting Components into Alpha Matte

20. Spectral Clustering 1.L is symmetric and positive semi-definite. 2.The smallest eigenvalue of L is 0, the corresponding eigenvector is the constant one vector 1. 3.L has n non-negative, real-valued eigenvalues 0= λ 1 ≦ λ 2 ≦... ≦ λ n. 2 2 3 3 1 1 4 4 5 5 A Graph Example 2 00 2 00 200 0001 000 1 1 2 3 4 5 12345 0.047 0.577 0 0

21. Spectral Clustering & Matting Components 2 0000 2 0000 20000 0001100 000 00 0000011 00000 1 1 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 1 Zero-Eigenvectors Binary Indicating Vectors Linear Transformation

22. Overview of Spectral Matting Input Image Smallest Eigenvectors Matting Components K-means Clustering & Linear Transformation Matting Laplacian

23. Spectral Clustering & K-means Input Image s-smallest Eigenvectors … Pixel i s-dimensional Space K-means Clustering

24. Generating Matting Components Smallest Eigenvectors Projection into Eigen Space K-means ………

25. Reconstructing Alpha Matte from Matting Components =++ Input Image Matting Components Selected Matting ComponentsAlpha Matte

26. Reconstructing Alpha Matte by Grouping Matting Components Matting cost function Alpha Matte Generation Evaluating All Grouping Hypothesis to Derive the Optimal Alpha Matte

27. Results by Levin et al.

28. Summary Constructing Matting Laplacian Solving Supervised Matting Problem Solving Unsupervised Matting Problem

29. Proposed Approaches Efficient Cell-based Framework for Reducing Computations Multi-scale Analysis Extended Applications (Depth Image Reconstruction) Input Image Reconstructed Depth Depth Reconstruction from Single Image Depth Reconstruction in Shape From Focus (SFF) Input ImageReconstructed Depth

30. Cell-based Framework Image Pixel-wise Data Distribution Cell-wise Data Distribution Conventional Matting Laplacian Cell-based Matting Laplacian Pixel-wise Affinity Cell-wise Affinity

31. Multi-scale Affinity Learning Image & Computation Patterns Pixel-based Approach Cell-based Approach

32. Multi-scale Affinity Learning … Finest Level Coarsest Level … Cell-based Graph

33. Results of Reconstructed Alpha Matte 1 st Rank 2 nd Rank (a) Grouping Results by Levin et al. (b) Grouping Results by Levin et al. with Coarse-to-fine Scheme. (c) Ours Input

34. Results (a) Input images(b) Levin’s result(c) Our result

35. Proposed Approaches Efficient Cell-based Framework for Reducing Computations Multi-scale Analysis Extended Applications (Depth Image Reconstruction) Input Image Reconstructed Depth Depth Reconstruction from Single Image Depth Reconstruction in Shape From Focus (SFF) Input ImageReconstructed Depth

36. Depth Reconstruction in Shape From Focus (SFF) Optical Direction Multi-focus Image Sequence Optical Direction Focus Value W1W1 W2W2 W2W2 W1W1

37. Low-SNR Problem Spatially Varying Precision Low-texture  Low-SNR Leading Noisy Result Input Image Observation High- precision Low- precision

38. Proposed Maximum-a-posteriori Estimation Multi-focus Image Sequence Learning-based Graph Local Learning Inference Reconstructed Depth

39. Proposed Maximum-a-posteriori Estimation PosteriorLikelihoodPrior Local Observation with Spatial-varying Precision Learned from Image

40. Likelihood Model Input Observation Precision Result High- precision Low- precision PosteriorLikelihoodPrior Local Observation with Spatial-varying Precision

41. Prior Model PosteriorLikelihoodPrior Learning from Input Image Learning-based Graph Local Learning Multi-focus Image Sequence

42. Maximum-a-posteriori Estimation for Depth Reconstruction Input ImageObservationReconstructed Depth

43. Results of Shape from Focus Input Image M. Mahmood, 2012T. Aydin, 2008Ours S. Nayar, 1994

44. Conclusions Construction of Matting Laplacian Conventional Approach Multi-scale Cell-based Approach Supervised Matting Spectral Matting Depth Reconstruction