Che-An Wu Background substitution

Background Substitution AlphaMa p Trimap Depth Map Extract the foreground object and put into another background Objective

Environment Platform Laptop vs Kinect in Java with library OpenNI and Processing Assumption fixed camera position the interest parts in the video are human human in the scene are closer than background

Depth color picture By the method SimpleOpenNI.depthImage() Black : pixels without depth measurement Depth information from Kinect Extract the foreground object

Trimap User-generated mask map made up of three zones Black : Background White : Foreground Gray : Unknown Generate a trimap Extract the foreground object

Generate a trimap Extract the foreground object

Generate a trimap Extract the foreground object

Develop algorithm to determine Unknown area Extract the foreground object

Cross Bilaterial Filter[1] Extract the foreground object

Alpha matte – Result (right) The unknown area color value is been determined by substituting the alpha value estimated from the algorithm into the compositing equation Cross Bilaterial Filter[1] Extract the foreground object

Che-An Wu

Known Alpha value of unknown area New background image Foreground value from original scene Determine the color of new image Putting into another background

background substitution Alpha map Determine the color of new image Putting into another background

Results Putting into another background Real-time parameters setting Depth threshold = 2m Neighborhood window size = 13*13; Spatial Filter Gaussian sigma = 13*13*2% Range Filter Gaussian sigma 30 Improvement Adjusting the neighborhood window definition => or blue area is the pixels in the neighborhood window The result is almost the same but we can adjust the parameter of window size twice to be 19*19

Problem (Ongoing) Putting into another background The FOV of the rgb camera and depth camera (combine to camera in order to generate depth image) is slightly different It still need to calibrate both camera in order to have the good result of background substitution. The result C=aF+(1-a)B the alpha location cannot match the foreground location.

References [1] Crabb, Ryan, et al. "Real-time foreground segmentation via range and color imaging.“ Computer Vision and Pattern Recognition Workshops, CVPRW'08. IEEE Computer Society Conference on. IEEE, 2008.

Che-An Wu