Multi Scale CRF Based RGB-D Image Segmentation Using Inter Frames Potentials Taha Hamedani Robot Perception Lab Ferdowsi University of Mashhad The 2 nd RSI International Conference on Robotics and Mechatronics (ICROM 2014) Oct 2014

Outline  Image segmentation Definition  Kinect Range Data and its Noises  Conditional Random Field (CRF)  Region Extraction based on Geometrical Features  Energy Function  Experimental Result  Future Work 2

Image Segmentation  Partitioning the image in to the similar and disjoint regions  These regions have the similar features such as RGB, orientation, texture 3

Microsoft Kinect Sensor  Estimate Depth data by structure light method  Project pseudo Random Pattern IR  Estimation based on comparison with position of received IR ray 4

2.5 Dimension Data and Noises 5

Conditional Random Field (CRF) According to Hammersley-Clifford Theorem 6

Multi scale CRF  Build a Pyramid of variant resolutions of image  Compute unary and pairwise potentials for each level of pyramid based on coarsest level of pyramid 7

Energy minimization  Solve energy minimization problem as a top down strategy  Projection  relaxation 8

Inter level Cliques  Consider inter level cliques beside singleton and doubleton cliques in each level (Kato) 9

Region Extraction  Extract the main region of the scene based on geometrical features  RGB edges (canny edge detector)  Depth edges (cosine of angle between normals of two neighbor pixels)  Sum two edges  Morphological opening in order to construct regions of the scene 10

Region Extraction  A simple scene with three side wall 11

Region Extraction  More complex scene ( NYU V2) 12

Inter Frame Cliques  Consider inter frame cliques between current pixel and previous frame labeling 13

Energy Function  Define our energy function based on these new cliques and regions 14

Experimental Result  Data set  New York v2 (NYUV2)  Release in 2012  RGB-D images from Kinect  464 images of 26 different indoor scenes  Annotated for 1000 classes 15

Experimental Result 16

Experimental Result  Hausdorff Distance 17

Future Work  Using previous frame data as a more effective manner  Information updating after each minimization iteration such as Normal vector correction 18

Question ? Thanks for your attention ? 19