Computational Photography, Fall 2013 Matting http://vimeo.com/35769675 Computational Photography, Fall 2013 Connelly Barnes Slides from Alexei Efros, James Hays
How does Superman fly? Super-human powers? OR Image Blending Image Matting? Matting vs Blending
Physics of Alpha Matting Semi-transparent objects Or, why segmentation isn’t good enough. Pixels too large
Alpha channel Add one more channel: Image(R,G,B,alpha) Sprite! Encodes transparency (or pixel coverage): Alpha = 1: opaque object (complete coverage) Alpha = 0: transparent object (no coverage) 0<Alpha<1: semi-transparent (partial coverage) Example: alpha = 0.7 Partial coverage or semi-transparency
Alpha Blending Icomp = aaIa + (1-aa)abIb acomp = aa + (1-aa)ab So far we assumed that one image (background) is opaque. If blending semi-transparent sprites (the “A over B” operation): Icomp = aaIa + (1-aa)abIb acomp = aa + (1-aa)ab Note: sometimes alpha is premultiplied: im(aR,aG,aB,a): Icomp = Ia + (1-aa)Ib (same for alpha!)
Matting an Image Problem Definition: Hard problem The separation of an image C into A foreground object image Co, a background image Cb, and an alpha matte a Co and a can then be used to composite the foreground object into a different image Hard problem Even if alpha is binary, this is hard to do automatically (background subtraction problem) For movies/TV, manual segmentation of each frame is difficult Need to make a simplifying assumption…
Average/Median Image What can we do with this?
Background Subtraction - =
Crowd Synthesis (with Pooja Nath) Do background subtraction in each frame Find and record “blobs” For synthesis, randomly sample the blobs, taking care not to overlap them
Background Subtraction A largely unsolved problem… One video frame Estimated background Difference Image Thresholded Foreground on blue
Blue Screen
Blue Screen matting Most common form of matting in TV studios & movies Petros Vlahos invented blue screen matting in the 50s. His Ultimatte® is still the most popular equipment. He won an Oscar for lifetime achievement. A form of background subtraction: Need a known background Compute alpha as SSD(C,Cb) > threshold Or use Vlahos’ formula: a = 1-p1(B-p2G) Hope that foreground object doesn’t look like background no blue ties! Why blue? Why uniform?
The Ultimatte p1 and p2
Blue screen for superman?
Semi-transparent mattes What we really want is to obtain a true alpha matte, which involves semi-transparency Alpha between 0 and 1
Matting Problem: Mathematical Definition
Why is general matting hard?
Solution #1: No Blue!
Solution #2: Gray or Flesh
Triangulation Matting (Smith & Blinn) How many equations? How many unknowns? Does the background need to be constant color?
The Algorithm
Triangulation Matting Examples
More Examples
More examples
Intelligent Scissors [Mortensen, SIGGRAPH 1995]
Intelligent Scissors
Intelligent Scissors Initial Cost Matrix
Intelligent Scissors Distance from Seed Point (Dynamic Programming)
Trimap [Juan and Keriven 2005]
Poisson Matting [Sun et al, SIGGRAPH 2004]
Poisson Matting Matting Equation (not alpha-premultiplied)
Poisson Matting Differentiate. Now what simplifying assumptions can we make?
Poisson Matting Approximation. Solve in least square sense
Poisson Matting Least squares (variational) problem. Now differentiate again.
Poisson Matting Same equation as Project 2 – Poisson Image Editing
Poisson Matting Iteratively solve for (1) a, (2) F - B Poisson Matting Paper
Poisson Matting
Problems with Matting Solution: Modify the Matting Equation Images do not look realistic Lack of Refracted Light Lack of Reflected Light Solution: Modify the Matting Equation
Environment Matting and Compositing slides by Jay Hetler Douglas E. Zongker ~ Dawn M. Werner ~ Brian Curless ~ David H. Salsin SIGGRAPH 99
Environment Matting Equation C = F + (1- a)B + F C ~ Color F ~ Foreground color B ~ Background color a ~ Amount of light that passes through the foreground F ~ Contribution of light from Environment that travels through the object Alpha holds values from zero to one…
Explanation of F R – reflectance image T – Texture image
Series of structured backgrounds Environment Mattes
Performance Calibration Matting: 10-20 minutes extraction time for each texture map (Pentium II 400Mhz) Compositing: 4-40 frames per second Real-Time? 3-6 hours
How much better is Environment Matting? Alpha Matte Environment Matte Photograph
How much better is Environment Matting? Alpha Matte Environment Matte Photograph
Shree K. Nayar Gurunandan G. Krishnan Columbia University Fast Separation of Direct and Global Images Using High Frequency Illumination Shree K. Nayar Gurunandan G. Krishnan Columbia University Michael D. Grossberg City College of New York Ramesh Raskar MERL SIGGRAPH Conference Boston, July 2006 Support: ONR, NSF, MERL
Direct and Global Illumination participating medium D : Volumetric translucent surface E E : Diffusion C C : Subsurface A A : Direct surface source B B : Interrelection P camera
Direct and Global Components: Interreflections surface source j BRDF and geometry i camera direct global radiance
High Frequency Illumination Pattern surface source i camera fraction of activated source elements +
- High Frequency Illumination Pattern + i surface source camera fraction of activated source elements
Separation from Two Images direct global
Other Global Effects: Subsurface Scattering translucent surface source i j camera
Other Global Effects: Volumetric Scattering participating medium surface source i j camera
Diffuse Interreflections Specular Volumetric Scattering Subsurface Diffusion
Scene
Scene Direct Global
More Real World Examples:
Eggs: Diffuse Interreflections Direct Global
Wooden Blocks: Specular Interreflections Direct Global
Kitchen Sink: Volumetric Scattering Chandrasekar 50, Ishimaru 78 Direct Global
Peppers: Subsurface Scattering Direct Global
Hand Direct Global Skin: Hanrahan and Krueger 93, Uchida 96, Haro 01, Jensen et al. 01, Cula and Dana 02, Igarashi et al. 05, Weyrich et al. 05 Direct Global
Face: Without and With Makeup Without Makeup Global Direct With Makeup
Blonde Hair Direct Global Hair Scattering: Stamm et al. 77, Bustard and Smith 91, Lu et al. 00 Marschner et al. 03 Direct Global
www.cs.columbia.edu/CAVE