Natural Video Matting with Depth Jonathan Finger Oliver Wang University of California, Santa Cruz {jfinger,

Motivation Given a video, replace the background with something different  Isolate the find foreground in each frame Image courtesy of Yung-Yu Chuang, Brian Curless, David Salesin, Richard Szeliski

Our Method Use a depth camera to automate foreground extraction Use Bayesian matting Improve the matting algorithm to get more realistic video

The Matting Problem Separation of a foreground image from a background image Image obtained from Corel Knockout's tutorial.

The Easy Direction Background (known) Foreground (known) Composite (unknown) 2 knowns, 1 unknown

The Hard Direction Background (unknown) Foreground (unknown) Composite (known) 1 known, 2 unknowns

The Matting Problem Actually there is another unknown  Represents areas that are a combination of foreground and background 0 1 transparent opaque ::

The Matting Problem =1=1  =0.5 =0=0 Foreground

The Matting Problem How do we isolate the foreground?  Use an alpha mask Alpha Mask  An image who's color represents foreground and background

The Matting Problem original alpha mask

The Masking Problem Basic pipeline Original composite Alpha mask Isolated foreground New background New composite

The Masking Problem But, how do you get an alpha mask?

Previous Work Blue Screen Matting Petro Vlahos (1964) Hollywood Special Effects pioneer Can isolate the foreground if the background is a constant color

Previous Work Background is known so it is easy to make a mask Image courtesy of A. Smith and J. Blinn

The Matting Problem How can this be done with an unknown background?  Use a general matting algorithm input: original composite + trimap output: alpha mask

Trimaps A three color image (usually drawn by hand)  Black = 100% background  White = 100% foreground  Gray = unknown

Trimaps The matting algorithm fills in the gray area with estimated alpha values

Natrual Matting Algorithms The matting equation For each 2D location in the image, there is a given composite pixel C We are to find F, B, and  at each pixel where C =  F + (1 -  )B

Natural Matting Original compositeTrimap Foreground estimation Background estimationAlpha mask

Natural Matting Algorithms alpha maskbackground removedclose up Knockout Ruzon and Tomasi Bayesian Image courtesy of Yung-Yu Chuang, Brian Curless, David Salesin1, Richard Szeliski

Problem with Natural Matting These all require a manual trimap Our goal is to do this with video  We do not want to make trimaps by hand

Previous Work Defocus Video Matting (McGuire) Two cameras  one focused on the background  one focused on the foreground

Previous Work A trimap can be generated from the defocused foreground However, apertures have to be very specific and can be thrown off by lighting Also requires texture in the scene Image courtesy M. McGuire, W. Matusik, H. Pfister, J. Hughes, and F. Durand.

Previous Work Bayesian Matting Using Learned Image Priors (Apostoloff, Fitzgibbon) Sequences of frames can be compared in order to find movement Image courtesy N. Apostoloff and A. Fitzgibbon

Previous Work assumptions  foreground is moving  nothing else is moving Image courtesy N. Apostoloff and A. Fitzgibbon

Previous Work The Z-Cam is able to separate a video scene into depth plains, but does not calculate alpha values.

Our Contribution Automatically generated trimaps  Does not depend on lighting, texture or movement Improved Bayesian Matting using depth information Hella trimaps

Overview Low res depth Original composite High res depthTrimap Alpha mask SupersampleBayesian matting

Our Method Canesta depth camera  Uses infrared lasers to detect distances from the camera

Our Method Optical imageDepth image Canesta takes 64x64 resolution image  Optical images are 640x640 or more

Trimap Overview To get a trimap 1) Upsample depth image to resolution of optical image 2) Threshold to separate into two colors 3) Erode/dilate to create a gray border around the foreground

Upsampling Use Qing's supersampled depth method  Use edge cues from high resolution color image  Can increase the depth resolution to up to 100X

Thresholding Assumption  Foreground is in front of background  Threshold on a distance plane Done once for entire animation

Erode/Dilate Grow unknown area around edges

Improved Bayesian Matting Bayesian matting is ill defined when the foreground and background are similar colors Original image Alpha mask

Improved Bayesian Matting Use depth information in Bayesian Matting optimization step Original image Bayesian matting Depth map

Improved Bayesian Matting

Bayesian MattingImproved Bayesian Method

Results video

Conclusion Video matting can be done without the user having to manually tweak any individual frames We were able to improve Bayesian Matting using depth information