Fast Census Transform-based Stereo Algorithm using SSE2 Young Ki Baik* Kyoung Mu Lee Computer Vision Lab. School of Electrical Engineering and Computer Science Seoul National University

Contents Stereo Vision Census Transform Stereo Vision Fast approaches Experimental result Conclusion and Future work These are contents what I’ll present you. At first, I’ll introduce stereo vision and census transform stereo vision. After that, I’ll show you fast approaches for stereo vision and experimental result. Finally, I’ll finish my presentation with conclusion and future work.

Introduction What is the stereo vision? The stereo vision is the method to extract 3D information using image from different view points. Topographical survey Obstacle detection Object tracking Face recognition What is the stereo vision? Stereo vision is the method to extract 3D information using image from different view points. And It can be applied to Topographical survey, obstacle detection, object tracking, face recognition and so on.

Introduction Trade off of algorithms Algorithm for accurate results Complex computation and iteration Slow processing time Unable to realize real-time system Algorithm for fast processing time Simple computation and no iteration Fast processing time Unable to realize accurate system Stereo vision has the trade off. For accurate results, algorithm has complex computation and iteration, slow processing time. For these reason, algorithms are unable to realize real-time system. For fast processing time, algorithm has simple computation and no iteration, fast processing time. But it is unable to realize accurate system. Our approach, we will focus on fast processing algorithm for stereo vision.

Introduction Fast stereo vision algorithm Window size invariant method Box filtering method “Box-filtering techniques”, M.J.McDonnell (CGIP-81’) “Real time correlation-based stereo : algorithm, implementations and applications”, Olivier Faugeras , Zhengyou Zhang , … (Tech.Rep.RR-2013, INRIA,1993) Disparity range invariant method Rectangular subregioning method “Rectangular Subregioning and 3-D Maximum-Surface Techniques for Fast Stereo Matching”, Changming Sun (CVPR-2001) Parallel processing technique For the fast stereo vision algorithm, these 3 items can be considered. At first, window size invariant method or box filtering method was introduced from these references. Also, disparity range invariant method was introduced by Changming sun. Finally, parallel processing technique is used for fast stereo vision.

Introduction Problem Census transform Real images from grabbers can not assure of brightness consistency in corresponding region. Intensity correlation method is not proper for real images. Census transform Census transform has been evaluated as the method robust to radiometric distortion. J. Banks and P. Corke, "Quantitative evaluation of matching methods and validity measures for stereo vision," Int. J. Robotics Research, vol. 20, pp. 512-532, July 2001. Heiko Hirschmller, "Improvements in Real-Time Correlation Based Stereo Vision", Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision, pp. 141-148, Kauai, Hawaii, December 2001. In common stereo vision algorithm, it does not consider following problem. The problem is that Real images from grabbers can not assure of brightness consistency in corresponding region. Therefore Intensity correlation method is not proper for real images. Generally, LOG filtering method are used for solving these problem. But, census transform has been evaluated as the method robust to radiometric distortion from these papers. So, we use census transform method in order to implement more accurate and fast stereo vision algorithm.

Census Transform Stereo Vision Census transform converts relative intensity difference to 0 or 1 and deforms 1 dimensional vector as much as window size of census transform. 210 159 99 86 39 198 170 32 67 47 45 78 102 98 30 40 33 115 109 31 26 130 121 1 X Census transform window (CTW) I’ll talk about census transform. Census transform converts relative intensity difference to 0 to 1 and deforms 1 dimensional vector as much as window size of census transform. And vector will be multiple proportion of 8, therefore we can use parallel processing effectively.

Census Transform Stereo Vision Result of census transform Census transform makes data of (image size * vector size). (Square size of CTW)-1 Height Width Height And, census transform makes data of image size by vector size. With this result… Width

Census Transform Stereo Vision Sum of Hamming distance The Hamming distance of two transformed vectors with correlation windows is used to find corresponding region. Sum of Hamming distance Disparity range The Hamming distance of two transformed vectors with correlation windows is used to find corresponding region. And we can obtain 3d information from this 3d space. This is census transform based stereo vision. Right census transformed vector Left census transformed vector 3D disparity space

Census Transform Stereo Vision Complexity of algorithm Census transform-based stereo vision (CTSV) has high complexity. N : Searching window size D : Disparity range C : Census transform window size Method Complexity Conventional stereo vision ND Census transform CND But census transform-based stereo vision has high complexity. As you can see in this table, complexity of census transform-based stereo vision is CND where N is searching window size. D is disparity range C is Census transform window size In order to overcome this slow processing time, we try to apply fast approaches to CTSV.

Fast Census Transform Stereo Vision Hamming distance Parallel processing - SSE2 8bit look-up table Parallel processing - SSE2 Correlation fast census transform-based stereo vision consists of these 3 parts. We use parallel processing for census transform, 8-bit look-up table or parallel processing for computing Hamming distance, Moving window technique and parallel processing for correlation. Moving window technique Parallel processing - SSE2

Fast Census Transform Stereo Vision SSE2 (Streaming SIMD Extension 2) 128-bit SIMD packed integer & floating point arithmetic operation Cache and memory management operation Continuous memory structure is required No advantages in separate data SSE2 offers 128-bit SIMD packed integer and floating point arithmetic operation and It provides cache and memory management operation. In order to use SSE2, continuous memory structure is required and there are no advantages in separate data.

Fast Census Transform Stereo Vision Fast approaches (census transform) Usage of parallel processing (SSE2) 16 pixels are loaded to XMM(SSE2 memory) and computed at once. For the census transform, we use SSE2. In this process, 16 pixels are loaded to XMM and computed at once.

Fast Census Transform Stereo Vision Fast approaches (sum of Hamming distance) Usage of 8bit look-up table (LUT) Parallel processing : SSE2 Parallel processing is faster than 8 bit LUT For computing sum of Hamming distance, we use 8 bit look up table or parallel processing. And, we can verify that parallel processing is faster than 8 bit look up table.

Fast Census Transform Stereo Vision Fast approaches (correlation) Moving window technique For correlation, we use moving window technique which considers near memory access with box filtering.

Fast Census Transform Stereo Vision Fast approaches (correlation) Combination of moving window and SIMD Moving window technique for x, y-axis SSE2 for d-axis In order to make faster approach for correlation, We combine moving window for x, y-axis and SSE2 for d-axis.

Experimental result Environment Condition System : Pentium-IV 2.4GHz Cache memory : 512Kbyte Camera : Stereo Mega-D (Videre design) Condition Image size : 320 x 240 gray stereo images Census transform window size :5x5, 7x7, 9x9 Disparity searching range : 32 Correlation window size : 11x11 I’ll show you experimental result. We use pentinu-4 2.4GHz and 512Kbyte cache memory, and stereo mega-d camera from videre design. We use 320x240 gray stereo images. And we test alternative census transform window size. Disparity searching range is 32. And correlation window size is 11x11.

Experimental result Detail processing time 32 disparity searching range Method Census Transform Window Time (msec) Frame Rate CT Cal HD Corr Total with SSE2 5 5.9 17.5 5.8 29.2 34.2 7 11.6 29.6 5.6 46.8 21.4 9 35.4 52.6 4.3 92.3 10.8 without 34.3 24.1 14.2 72.5 13.8 66.2 35.1 14.1 115.4 8.7 208.3 66.5 289.0 3.5 For the census transform-based stereo vision with SSE2, we can get 34.2 frame rate.

Experimental result Performance of census transform stereo vision And This is the graph of performance for census transform stereo vision

Conclusion and Future work Moving window technique reduces processing time to constant except in transforming stage SSE2 instructions reduces running time by 2.5 to 3 times Possibility for faster result Specialized Instruction 16 bit look-up table Fixed window size of census transform Future work Applying real-time approach to another stereo algorithm Combine stereo system to other applications We can verify that moving window technique reduces processing them to constant except in transforming stage And sse2 instructions reduces running time by 2.5 to 3 times. We can expect faster results with specialized instruction in sse2, 16 bit look-up table and fixed census transform window size. For future works, we will apply real-time approaches to another stereo algorithm and combine stereo system to other applications. This is end of my talk. Thank you for your attention. Is there any question?