Background Removal of Multiview Images by Learning Shape Priors Yu-Pao Tsai, Cheng-Hung Ko, Yi-Ping Hung, and Zen-Chung Shih

ABSTRACT Image-based rendering has been successfully used to display 3-D objects for many applications. A well- known example is the object movie, which is an image- based 3-D object composed of a collection of 2-D images taken from many different viewpoints of a 3-D object.

INTRODUCTION

MVI ( Multiview Image) Ο ( omicron ) Θ ( thet ) : pan angle Φ ( phi ) : tilt angle MVI has three basic characteristics: 1) When an equi-tilt set of the MVI is captured, a large proportion of the background scene is static. 2) Only one interesting object is presented in every image of the MVI. 3) The foreground and background color distributions are distinct in most cases.

Proposed Flowchart The proposed approach aims to let every single image segmentation,rather than only those in neighboring viewing directions To take the shape prior into account, the user is required to selectd a subset of acceptably segmented images. The 3-D shape is then generated from these selected images.

AUTOMATIC INITIAL SEGMENTATION : Graph Cut Image Segmentation Graph cut image segmentation requires the user to interactively mark some pixels as being inside the foreground objects, and others as a part of the background scene. All the other pixels are considered to be unknown, and then they can be classified into the foreground or background by Markov random field (MRF) optimization.

AUTOMATIC INITIAL SEGMENTATION : Trimap Labeling Trimap consisting of labels drawn from Trimap labeling method = β-labeling and ξ-labeling β : based on the color difference. zero-mean normalized cross correlation (ZNCC) ξ : based on the foreground model. ξ : xi. β: beta.μ:mu. 1.If the color of a pixel varies, the pixel should be the background and labeled 2.mathematical morphology is applied to filter out the remained noises such that only one region exists 3.Each pixel whose color differs widely from the background model can be labeled. Where is a strict threshold to ensure that only the pixels that differ widely from the background model are labeled ξ The β and μ regions are collected and clustered by using K-means. Let denote the mean color of the th cluster for image Each pixel p with the label μ in the image is examined and labeled ξ

AUTOMATIC INITIAL SEGMENTATION : Trimap Labeling β Labeling ξ Labeling Background: black Foreground : white Unknow : gray Mathematical morphology in (a) MVI

SEGMENTATION WITH SHAPE PRIORS : Volumetric Graph Cuts In (6), the first integral tends toward a photo-consistent surface, while the second, called the ballooning term, prefers a fatter reconstructed model. For each voxel, let be the photo-consistency score of, where a lower value represents a better photo-consistency. Let be the volume between and the base surface. The true surface is determined by finding the global minimum of the energy function among all candidate surfaces.

Discrete Medial Axis Constraint: Energy Function Analysis: Let be a neighborhood system defined for, which containing the set of all pairs of neighboring voxels. Let be a family of random variables defined on the set, in which each variable takes a label from D(p) is the penalty according to how well the voxel fits into the given label, while B(p,q) indicates whether the surface is likely to pass through the edge between p and q. B(p,q) can maintain the smoothness prior

Discrete Medial Axis Constraint: Imposing the DMA Constraint The medial axis is represented by a set of discrete voxels interior to the 3-D object, called discrete medial axis (DMA). To compute the DMA of the base surface, which is assumed to be an adequate approximation of the DMA of the true surface. let be the set of voxels in the DMA. Let dp be the minimum distance from the voxel to its nearest voxel in (12) guarantees that the voxels in are always labeled as being inside the surface.

Discrete Medial Axis Constraint: Segmentation Refinement C1.C2.C4 built hull. C3.C5 projection

EXPERIMENTS: Initial Segmentation Results Error MVI Trimap Labeling (β Labeling + ξ Labeling) Error

EXPERIMENTS: Learning Shape Prior Without DMA and ballooning increased a.Visual hull b.DMA of ( a ) c.MVI d.Our method a.Visual hull base surface b. DMA of ( a ) c.d. reconstructed model

EXPERIMENTS: Rectification of Segmentation Errors Projection of the reconstructed

EXPERIMENTS: Rectification of Segmentation Errors Projection of the reconstructed Trimap Labeling = β Labeling +ξ Labeling Automatic initial segmentation MVI Shape priors

EXPERIMENTS: Rectification of Segmentation Errors Shape prior1 use 10 images Shape prior2 use 20 images

Thank you for your listening ! listening !