Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San Diego

Related Work Studio Matting – Vlahos 71, Wallace 82, Smith and Blinn 96, Ben-Ezra 00, Debevec et. al 02, McGuire et al. 06 User Assisted Natural Matting – Ruzon and Tomasi 00, Chuang et al. 01, Chuang et al. 02, Sun et al. 04, Li et al. 04, Rother et al 04, Li et al. 05, Wang et al. 05, Wang and Cohen 05, Levin et al. 06 Automated Natural Matting – Wexler et al. 02, Zitnick et al. 04, McGuire et al. 05, Kolmogrov et al. 05 – Our Method

Contributions Extending the matting equation to use the variance of pixel measurements Using a camera array for alpha matting An automatic method for computing alpha mattes that is linear in the number of pixels A near real-time system for natural video matting – 2 to 5 FPS

Algorithm Overview Foreground Object Background Camera Array  

Using a Camera Array

Foreground and Background Parallax

Aligning to a Foreground Plane Foreground Plane     I0I0 InIn I0I0 InIn IiIi IiIi

Aligned Images

Variance across cameras Taking the Variance Background Foreground Transparent (Mixture) Transparent I0I0 InIn IiIi

Key Idea alpha is a function of: variance across cameras images

Traditional Alpha Matting [Smith and Blinn 96] I =  F + (1 -  )B I = observed image  = transparency F = foreground B = background

Alpha Matting with Multiple Images I =  F + (1 -  )B I, F, and B are continuous random variables  is a scalar

I =  F + (1 -  ) B Solving for alpha  2 var( F ) + (1 -  ) 2 var( B ) - var( I ) = 0 var( I ) = var[  F + (1 -  ) B ]

Solving for alpha and  F  2 var( F ) + (1 -  ) 2 var( B ) - var( I ) = 0 a  2 + b  + c = 0 a = var( F ) + var( B ) b = -2var( B ) c = var( B ) - var( I )

Triangulation Matting: A Special Case of our Method I1I1 I2I2 [Smith and Blinn 96]

 F is a function of: [mean(I),mean(B)] Variance and Mean Statistics We observe I F and B can’t be directly measured  is a function of: [var(I),var(F),var(B)] ? ? ?

Foreground = Low var(I) Background = High var(I) Threshold var(I) to get Trimap

Propagating Variance and Mean For each pixel in the unknown (gray) region: – var(F) = var(I nf ) where I nf is the nearest pixel in the foreground – var(B) = var(I nb ), mean(B) = mean(I nb ) where I nb is the nearest pixel in the background

Algorithm Steps 1. Calibrate camera array and film scene 2. Find the foreground depth 3. Compute var(I) and mean(I) 4. Threshold var(I) to get trimap 5. Estimate var(F), var(B), mean(B) 6. Compute  and  F

System and Implementation Details 8 640x480 (Bayer pattern) Basler Video Cameras 1 Desktop PC (P4 3.0 Ghz) Calibration: – Color w/ Macbeth chart – Bundle Adjustment Average Running Time: 2 to 5 FPS at 320 x 240

Our methodUsing a known background

Our method Using a known background

Limitations Self-occlusion: alpha is not view independent Known Background Our Method Need background variation, i.e., var(B) should be large X

Conclusions Standard matting equation extended to work with variance and mean statistics Running time is linearly proportional to the number of pixels Automatic, near real-time alpha matting using a camera array

Acknowledgments SIGGRAPH Reviewers Joe Marks, John Barnwell, Frédo Durand, Matthias Zwicker, Bennett Wilburn, Dan Morris, and Merrie Morris Our willing and patient “Actors”: Lavanya Sharan, Chien-i Tu, Bernd Bickel, Yuanzhen Li MERL (for my internship) NSF IGERT Grant (for additional funding)