SOFT SCISSORS: AN INTERACTIVE TOOL FOR REALTIME HIGH QUALITY MATTING International Conference on Computer Graphics and Interactive Techniques ACM SIGGRAPH.

Slides:



Advertisements
Similar presentations
New Segmentation Technique
Advertisements

A Graph based Geometric Approach to Contour Extraction from Noisy Binary Images Amal Dev Parakkat, Jiju Peethambaran, Philumon Joseph and Ramanathan Muthuganapathy.
Interactively Co-segmentating Topically Related Images with Intelligent Scribble Guidance Dhruv Batra, Carnegie Mellon University Adarsh Kowdle, Cornell.
Cutting Images: Graphs and Boundary Finding Computational Photography Derek Hoiem, University of Illinois 09/15/11 “The Double Secret”, Magritte.
Graph cut Chien-chi Chen.
Presenter : Kuang-Jui Hsu Date : 2011/5/12(Tues.).
Spectral Matting Anat Levin 1,2 Alex Rav-Acha 1 Dani Lischinski 1 1 School of CS&Eng The Hebrew University 2 CSAIL MIT.
I Images as graphs Fully-connected graph – node for every pixel – link between every pair of pixels, p,q – similarity w ij for each link j w ij c Source:
1School of CS&Eng The Hebrew University
Image Matting with the Matting Laplacian
電腦視覺 Computer and Robot Vision I Chapter2: Binary Machine Vision: Thresholding and Segmentation Instructor: Shih-Shinh Huang 1.
Kapitel 13 “Interactive Segmentation” – p. 1 Interactive Segmentation  Live-wire approach  Random walker segmentation TexPoint fonts used in EMF. Read.
GrabCut Interactive Image (and Stereo) Segmentation Carsten Rother Vladimir Kolmogorov Andrew Blake Antonio Criminisi Geoffrey Cross [based on Siggraph.
GrabCut Interactive Foreground Extraction using Iterated Graph Cuts Carsten Rother Vladimir Kolmogorov Andrew Blake Microsoft Research Cambridge-UK.
A Gimp Plugin that uses “GrabCut” to perform image segmentation
Graph-Based Image Segmentation
Jue Wang Michael F. Cohen IEEE CVPR Outline 1. Introduction 2. Failure Modes For Previous Approaches 3. Robust Matting 3.1 Optimized Color Sampling.
Interactive Image Segmentation using Graph Cuts Mayuresh Kulkarni and Fred Nicolls Digital Image Processing Group University of Cape Town PRASA 2009.
1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.
Grayscale Image Matting And Colorization Tongbo Chen, Yan Wang, Volker Schillings, Christoph Meinel FB IV-Informatik, University of Trier, Trier 54296,
Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.
1 s-t Graph Cuts for Binary Energy Minimization  Now that we have an energy function, the big question is how do we minimize it? n Exhaustive search is.
Graph-based image segmentation Václav Hlaváč Czech Technical University in Prague Faculty of Electrical Engineering Department of Cybernetics Prague, Czech.
Image Matting and Its Applications Chen-Yu Tseng Advisor: Sheng-Jyh Wang
GrabCut Interactive Image (and Stereo) Segmentation Joon Jae Lee Keimyung University Welcome. I will present Grabcut – an Interactive tool for foreground.
Lecture 6 Image Segmentation
Learning to Detect A Salient Object Reporter: 鄭綱 (3/2)
CS448f: Image Processing For Photography and Vision Graph Cuts.
Boundary matting for view synthesis Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Computer Vision and Image Understanding 103 (2006) 22–32.
CS 376b Introduction to Computer Vision 04 / 08 / 2008 Instructor: Michael Eckmann.
1 Learning to Detect Objects in Images via a Sparse, Part-Based Representation S. Agarwal, A. Awan and D. Roth IEEE Transactions on Pattern Analysis and.
Natural and Seamless Image Composition Wenxian Yang, Jianmin Zheng, Jianfei Cai, Senior Member, IEEE, Susanto Rahardja, Senior Member, IEEE, and Chang.
A Closed Form Solution to Natural Image Matting
Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006.
Abstract Extracting a matte by previous approaches require the input image to be pre-segmented into three regions (trimap). This pre-segmentation based.
An Iterative Optimization Approach for Unified Image Segmentation and Matting Hello everyone, my name is Jue Wang, I’m glad to be here to present our paper.
Image Segmentation Today’s Readings Forsyth & Ponce, Chapter 14
Graph-based Segmentation
Image Segmentation Rob Atlas Nick Bridle Evan Radkoff.
MRFs and Segmentation with Graph Cuts Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 03/31/15.
Mean-shift and its application for object tracking
Graph Cut & Energy Minimization
Graph-based Segmentation. Main Ideas Convert image into a graph Vertices for the pixels Vertices for the pixels Edges between the pixels Edges between.
MRFs and Segmentation with Graph Cuts Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 02/24/10.
Object Stereo- Joint Stereo Matching and Object Segmentation Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on Michael Bleyer Vienna.
Intelligent Scissors for Image Composition Anthony Dotterer 01/17/2006.
Feature-Based Stereo Matching Using Graph Cuts Gorkem Saygili, Laurens van der Maaten, Emile A. Hendriks ASCI Conference 2011.
Non-Photorealistic Rendering and Content- Based Image Retrieval Yuan-Hao Lai Pacific Graphics (2003)
Gaussian Mixture Models and Expectation-Maximization Algorithm.
Interactive Image Cutout- Lazy Snapping
Journal of Visual Communication and Image Representation
October 1, 2013Computer Vision Lecture 9: From Edges to Contours 1 Canny Edge Detector However, usually there will still be noise in the array E[i, j],
CS654: Digital Image Analysis Lecture 28: Advanced topics in Image Segmentation Image courtesy: IEEE, IJCV.
Computational Photography lecture 8 – segmentation CS (future 572) Spring 2016 Prof. Alex Berg (Credits to many other folks on individual slides)
Improving Image Matting using Comprehensive Sampling Sets CVPR2013 Oral.
Implementing the By: Matthew Marsh Supervisors: Prof Shaun Bangay Mrs Adele Lobb segmentation technique as a plugin for the GIMP.
Cutting Images: Graphs and Boundary Finding Computational Photography Derek Hoiem, University of Illinois 09/20/12 “The Double Secret”, Magritte.
MRFs and Segmentation with Graph Cuts Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 03/27/12.
Graphcut Textures:Image and Video Synthesis Using Graph Cuts
Cutting Images: Graphs and Boundary Finding
Digital Visual Effects, Spring 2009 Yung-Yu Chuang 2009/5/21
GrabCut Interactive Foreground Extraction using Iterated Graph Cuts Carsten Rother Vladimir Kolmogorov Andrew Blake Microsoft Research Cambridge-UK.
Dingding Liu* Yingen Xiong† Linda Shapiro* Kari Pulli†
Iterative Optimization
PRAKASH CHOCKALINGAM, NALIN PRADEEP, AND STAN BIRCHFIELD
Lecture 31: Graph-Based Image Segmentation
“grabcut”- Interactive Foreground Extraction using Iterated Graph Cuts
Announcements Project 1 is out today help session at the end of class.
“Traditional” image segmentation
Presentation transcript:

SOFT SCISSORS: AN INTERACTIVE TOOL FOR REALTIME HIGH QUALITY MATTING International Conference on Computer Graphics and Interactive Techniques ACM SIGGRAPH 2007 JUE WANG, MANEESH ARGAWALA, MICHAEL F. COHEN

Outline  Introduction  Related Work  Algorithms  Results and Evaluation  Conclusion

Introduction  C p = α p F p + (1 − α p )B p  C → input image  F → foreground image  B → background image  α p → the foreground opacity of the pixel

Introduction  Previous matting algorithm  Require the user to roughly segment the image into a trimap.  Problem? trimap If the initial results are not satisfactory, the user must the refine the trimap and run the algorithm again until the process converges. Very Inefficient

Introduction – The Aim  To Provide a tool that can generate high quality mattes in realtime. The user roughly specifies the foreground boundary using an intelligent paint stroke. The system automatically updates the matte and foreground colors according to the newly-add information along the stroke to instantly reveal a local region of the final composite.

Related Work  GRADY, L Random walks for image segmentation. IEEE Trans. Pattern Analysis and Machine Intelligence.  WANG, J., AND COHEN, M. F Optimized color sampling for robust matting. In Proceedings of IEEE CVPR.

The Soft Scissors Algorithms - Overview  Flowchart:  Assume that the scissor stroke implicitly defines a trimap. (left : background, right : foreground, middle : unknown)

The Soft Scissors Algorithms - Overview  The boundary conditions and the width of the scissor stroke can be set manually by the user or dynamically adjusted by the system base on an analysis of the image statistics. M t affects the alpha values of surrounding pixels The matting region Ω t

The Soft Scissors Algorithms - Overview  1. Estimate the alpha values for pixels in Ω t using a robust matte solver.  2. The foreground colors of pixels in Ω t are updated by a foreground color solver that uses the newly computed alpha values.

Solving for the Matte  Central component → robust matting algorithm ★ Solving for α in the region as a soft graph- labeling problem.  For a pair of foreground and background colors and, the estimated alpha value is

Solving for the Matte  We have to compute nonparametric models of the foreground and background color distributions. Virtual node representing pure foreground Virtual node representing pure background Represent unknown pixels

Solving for the Matte  A data weight is defined between each pixel and a virtual node to enforce the data constraint. 1. γ is a free parameter. 2. δ is boolean function returning 0 or α and f are estimated alpha and confidence values. 1. γ is a free parameter. 2. δ is boolean function returning 0 or α and f are estimated alpha and confidence values.

Solving for the Matte  Confidence value for a sample pair as  is a distance ratio. , are two weights.

Solving for the Matte  An edge weight is defined between two neighboring pixels to enforce the neighborhood constraint.  The edge weights constrain nearby pixels to have similar alpha values. 3×3 Original algorithm : 3×3 windows Here is a different!! In this paper, each pixel is connected to its 25 spatial neighbors in this formulation. 3×3 Original algorithm : 3×3 windows Here is a different!! In this paper, each pixel is connected to its 25 spatial neighbors in this formulation.

Solving for the Matte – Random Walk  Once the graph is constructed, we solve the graph- labeling problem as a Random Walk, which minimizes the total graph energy over real values.  How Random Walk solver work?  1. The Random Walk solver determines the alpha values by placing a random walker at pixel i that can walk to any neighboring node j with probability.

Solving for the Matte – Random Walk  2. The walker then moves from j to another neighbor k in the same manner.  3. Do this process iterates until the walker reaches one of the boundary nodes.  Alpha value of pixel i : the probability that the walker ends up at the foreground virtual node.

Solving for the Foreground Colors  In addition to computing alpha values we also estimate the true foreground color F for each pixel in the unknown region.  This allows the foreground to be composed onto a new background without bringing the colors of the old background into the new composite.  In the matte estimation step, we just select a few foreground samples for each pixel, these samples are chosen individually without enforcing smoothness constraints.

Solving for the Foreground Colors  After the matte estimation step, the composite may contain visual artifacts. Left: Initial estimates of foreground colors after the matte estimation step. Right: Final foreground colors after optimization.

Solving for the Foreground Colors  To achieve higher quality composites: ★ Refine the estimated foreground colors by solving a second graph-labeling problem using Random Walk. Define a color edge weight between two neighbors as Unknown pixels : Those pixels in Ω t whose alpha values are strictly between 0 and 1 Each unknown pixel is connected to its 4 spatial neighbors.

Solving for the Foreground Colors  The edge weight encodes explicit smoothness prior on F, which are stronger in the presence of matte edges. For foreground pixels, we use their true colors as boundary conditions. For background pixels, we use their initially estimated foreground colors in the matte estimation step as boundary conditions.

Solving the Update Region  This system only update the alpha and foreground colors for small portion of the image on each iteration.  Given a new input region we compute the set of pixels that might be affected by the new information as the update region Ω t.  To determine the update region Ω t → solve a graph-labeling problem.

Solving the Update Region  Newly marked pixels are treated as boundary pixels. (assigned label of 1)  The label represents the impact of the new input region on the pixel.  All other pixels that were marked in previous iterations are treated as unknown pixels.

Solving the Update Region  We solve the problem using the way similar to the alpha estimation.  Each pixel is assigned a label measuring how much impact it receives from the new input region.  A smoother local image region will result in a large Ω t, since the weight between neighboring pixels are high, and vice versa.

Choosing the scissor brush width  Wider scissors are appropriate for object edges that are very fuzzy.  Narrower scissors are better for sharper edges as they provide tighter bounds on the solution and greatly improve computation efficiency.

Choosing the scissor brush width  At each time t,  1. Compute the current scissor path direction  2. Crete a side “look-ahead” region by extending the current path of the scissor along that direction. Look-ahead region

Choosing the scissor brush width  Then,  1. Treat all pixels in this region as unknowns and include them in Ω t for alpha estimation.  2. To estimate the matte profile we sample a group of pixels sparsely distributed along lines perpendicular to the current scissor path direction.

Choosing the scissor brush width  For each sampled point on a line compute a weight as α p is the estimated alpha value of the point  The center of the 1D distribution along the line is computed as x p is the distance form a sampling point to the extended scissor path  The width of the alpha profile:

Determining the boundary conditions  Initially, assume that the user orients the scissor brush strokes.  This system dynamically adjust the scissor boundary conditions by building color models of the foreground and background colors.  Once the user has created a short stroke and marked enough foreground/background pixels under the initial assumptions → Gaussian Mixture Models (GMM) Background region Foreground region

Determining the boundary conditions  Use GMM models to solve the problem.  How?  1. Sample a group of pixels along the newly added left edge.  2. Compute a foreground and background probability and by fitting samples with foreground and background GMMs, and normalize them so they sum to 1.

Determining the boundary conditions  3. If, set the left edge to be foreground. If, set the left edge to be background. is a difference threshold  If the difference between the two probabilities is smaller than → keep the current boundary condition unchanged.  Re-compute the GMMs using the new samples after a set number of new foreground and background samples are marked by the user.

Automatic vs. Manual Parameter Selection  If, we discard the automatically determined boundary conditions.  At any time the user can disable either automatic algorithm to regain full control over the brush parameters.

Results and Evaluation

 Higher quality results  Using Bayesian matting. (a). Original image. (b). Ground- truth matte. (c). Target image on which we apply matting algorithms. (a). Original image. (b). Ground- truth matte. (c). Target image on which we apply matting algorithms.

Results and Evaluation  Compare with other mattes:

Results and Evaluation  Two visual examples:  Soft Scissors extracts mattes with the highest quality. (even better than Robust Matting)

Results and Evaluation  Work well with more solid foreground objects.  Compare with binary cutout tools.

Preliminary User Study These results clearly suggest that soft scissors provides an effective interface for foreground matting.

Conclusion  The first realtime tool for generating high quality mattes and composites as the user roughly paints along the foreground boundary.  The scissor brush width and boundary conditions adjust automatically.

References  CHUANG, Y.-Y., CURLESS, B., SALESIN, D. H., AND SZELISKI, R A bayesian approach to digital matting. In Proceedings of IEEE CVPR, 264–271.  GRADY, L Random walks for image segmentation. IEEE Trans. Pattern Analysis and Machine Intelligence.  LEVIN, A., LISCHINSKI, D., AND WEISS, Y A closed form solution to natural image matting. In Proceedings of IEEE CVPR.  LI, Y., SUN, J., TANG, C.-K., AND SHUM, H.-Y Lazy snapping. In Proceedings of ACM SIGGRAPH, 303–308.  LISCHINSKI, D., FARBMAN, Z., UYTTENDAELE, M., AND SZELISKI, R Interactive local adjustment of tonal values. In Proceedings of ACM SIGGRAPH.  MORTENSEN, E., AND BARRETT, W Intelligent scissors for image composition. In Proceedings of ACM SIGGRAPH.  ROTHER, C., KOLMOGOROV, V., AND BLAKE, A Grabcut - interactive foreground extraction using iterated graph cut. In Proceedings of ACM SIGGRAPH, 309–314.  SUN, J., JIA, J., TANG, C.-K., AND SHUM, H.-Y Poisson matting. In Proceedings of ACM SIGGRAPH, 315–321.  WANG, J., AND COHEN, M An iterative optimization approach for unified image segmentation and matting. In Proceedings of ICCV 2005, 936–943.  WANG, J., AND COHEN, M. F Optimized color sampling for robust matting. In Proceedings of IEEE CVPR.