1. Natural Video Matting with Depth Jonathan Finger Oliver Wang University of California, Santa Cruz {jfinger, owang}@soe.ucsc.edu

2. Motivation Given a video, replace the background with something different  Isolate the find foreground in each frame Image courtesy of Yung-Yu Chuang, Brian Curless, David Salesin, Richard Szeliski

3. Our Method Use a depth camera to automate foreground extraction Use Bayesian matting Improve the matting algorithm to get more realistic video

4. The Matting Problem Separation of a foreground image from a background image Image obtained from Corel Knockout's tutorial.

5. The Easy Direction Background (known) Foreground (known) Composite (unknown) 2 knowns, 1 unknown

6. The Hard Direction Background (unknown) Foreground (unknown) Composite (known) 1 known, 2 unknowns

7. The Matting Problem Actually there is another unknown  Represents areas that are a combination of foreground and background 0 1 transparent opaque ::

8. The Matting Problem =1=1  =0.5 =0=0 Foreground

9. The Matting Problem How do we isolate the foreground?  Use an alpha mask Alpha Mask  An image who's color represents foreground and background

10. The Matting Problem original alpha mask

11. The Masking Problem Basic pipeline Original composite Alpha mask Isolated foreground New background New composite

12. The Masking Problem But, how do you get an alpha mask?

13. Previous Work Blue Screen Matting Petro Vlahos (1964) Hollywood Special Effects pioneer Can isolate the foreground if the background is a constant color

14. Previous Work Background is known so it is easy to make a mask Image courtesy of A. Smith and J. Blinn

15. The Matting Problem How can this be done with an unknown background?  Use a general matting algorithm input: original composite + trimap output: alpha mask

16. Trimaps A three color image (usually drawn by hand)  Black = 100% background  White = 100% foreground  Gray = unknown

17. Trimaps The matting algorithm fills in the gray area with estimated alpha values

18. Natrual Matting Algorithms The matting equation For each 2D location in the image, there is a given composite pixel C We are to find F, B, and  at each pixel where C =  F + (1 -  )B

19. Natural Matting Original compositeTrimap Foreground estimation Background estimationAlpha mask

20. Natural Matting Algorithms alpha maskbackground removedclose up Knockout Ruzon and Tomasi Bayesian Image courtesy of Yung-Yu Chuang, Brian Curless, David Salesin1, Richard Szeliski

21. Problem with Natural Matting These all require a manual trimap Our goal is to do this with video  We do not want to make trimaps by hand

22. Previous Work Defocus Video Matting (McGuire) Two cameras  one focused on the background  one focused on the foreground

23. Previous Work A trimap can be generated from the defocused foreground However, apertures have to be very specific and can be thrown off by lighting Also requires texture in the scene Image courtesy M. McGuire, W. Matusik, H. Pfister, J. Hughes, and F. Durand.

24. Previous Work Bayesian Matting Using Learned Image Priors (Apostoloff, Fitzgibbon) Sequences of frames can be compared in order to find movement Image courtesy N. Apostoloff and A. Fitzgibbon

25. Previous Work assumptions  foreground is moving  nothing else is moving Image courtesy N. Apostoloff and A. Fitzgibbon

26. Previous Work The Z-Cam is able to separate a video scene into depth plains, but does not calculate alpha values.

27. Our Contribution Automatically generated trimaps  Does not depend on lighting, texture or movement Improved Bayesian Matting using depth information Hella trimaps

28. Overview Low res depth Original composite High res depthTrimap Alpha mask SupersampleBayesian matting

29. Our Method Canesta depth camera  Uses infrared lasers to detect distances from the camera

30. Our Method Optical imageDepth image Canesta takes 64x64 resolution image  Optical images are 640x640 or more

31. Trimap Overview To get a trimap 1) Upsample depth image to resolution of optical image 2) Threshold to separate into two colors 3) Erode/dilate to create a gray border around the foreground

32. Upsampling Use Qing's supersampled depth method  Use edge cues from high resolution color image  Can increase the depth resolution to up to 100X

33. Thresholding Assumption  Foreground is in front of background  Threshold on a distance plane Done once for entire animation

34. Erode/Dilate Grow unknown area around edges

35. Improved Bayesian Matting Bayesian matting is ill defined when the foreground and background are similar colors Original image Alpha mask

36. Improved Bayesian Matting Use depth information in Bayesian Matting optimization step Original image Bayesian matting Depth map

37. Improved Bayesian Matting

38. Bayesian MattingImproved Bayesian Method

39. Results video

40. Conclusion Video matting can be done without the user having to manually tweak any individual frames We were able to improve Bayesian Matting using depth information