Improved Census Transforms for Resource-Optimized Stereo Vision Wade S. Fife, Member, IEEE, James K. Archibald, Senior Member, IEEE IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 23, NO. 1, JANUARY 2013

Outline Introduction Related Work Proposed Algorithm Sparse Census Transform Generalized Census Transform Hardware Implementation Experimental Results Conclusion

Introduction

Introduction The challenges: It is critical to… The enormous amount of computation required to identify the corresponding points in the images. It is critical to… maximize the accuracy and throughput of the stereo system while minimizing the resource requirements

Objective Propose the sparse census transforms : Reduce the resource requirements of census-based systems Maintain correlation accuracy Propose the generalized census transforms : A new class of census-like transforms Increase the robustness and flexibility

Related Work

Related Work Census Transform : Color Gradient

Related Work After aggregation step: Census on colors Census on gradients

Related Work Sparse census[6] : Half of the bits The computation costs for the hamming distances are quite large. X [6] C. Zinner, M. Humenberger, K. Ambrosch, and W. Kubinger, “An optimized software-based implementation of a census-based stereo matching algorithm,” in Proc. 4th ISVC, 2008, pp. 216–227.

Related Work Mini-census[8] : X [8] N.-C. Chang, T.-H. Tsai, B.-H. Hsu, Y.-C. Chen, and T.-S. Chang,“Algorithm and architecture of disparity estimation with mini-census adaptive support weight,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 6, pp. 792–805, Jun. 2010.

Related Work Mini-census[8] : Mini-census adaptive support weight [8] N.-C. Chang, T.-H. Tsai, B.-H. Hsu, Y.-C. Chen, and T.-S. Chang,“Algorithm and architecture of disparity estimation with mini-census adaptive support weight,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 6, pp. 792–805, Jun. 2010.

Related Work Mini-census[8] :

Proposed Algorithm

Sparse Census Transform Definition : N: the set of points within a T × T window around p 𝑁 : a new set of N P’ P

Transform Point Selection Goal : minimize the size of the census transform vector Challenge: Must quantify how much each point in the transform window contributes to overall correlation accuracy Test correlation accuracy: Define a sparse census transform consisting of a single point (| 𝑵 | = 1) Determine how consistently this point leads to correct correlation 13 × 13 correlation window (aggregation)

Transform Point Selection Go Tsukuba Venus Average Bright: Higher correlation accuracy 25 × 25 neighborhood Teddy Cones

Transform Point Selection Further from the center : value decreasing Very near the center : less effective It is best to choose points that are neither too far from nor too close to the center pixel. Optimal distance : 2 pixels If the image is noisy should be slightly further from the center

Transform Point Selection With Gaussian noise ( 𝝈 = 5.12) Tsukuba Tsukuba Venus Venus Average Bright: Higher correlation accuracy 37 × 37 neighborhood Teddy Teddy Cones Cones

Proposed Sparse Census Transform Very good correlation accuracy can be achieved using very sparse transforms. 16-point 12-point 8-point 4-point 2-point 1-point

Experimental Results

Generalized Census Transform Goal : greater freedom in choosing the census transform design Definition : redrawing the transform as a graph 3 × 3 correlation (aggregation) 3 × 3 census

Generalized Census Transform As.. (1)transform neighborhoods become more and more sparse (2)fewer pixels are used in the correlation process selection of points to include in the transform becomes more critical Horizontal + Vertical 2-edge 2-point

Generalized Census Transform symmetric

Proposed Generalized Census Transform Benefits : Often require a smaller census transform window (memory) Increased robustness under varying conditions (noise) 16-edge 12-edge 8-edge 4-edge 2-edge 1-edge

Experimental Results

Experimental Results

Hardware Implementation Pipelining : to increase throughput in an FPGA implementation (Field Programmable Gate Array) 3 2 1 0 3 2 1 0 3 2 1 0 3 2 1 0 3 2 1 0 One input pixel per clock cycle & Output one disparity result per clock cycle Range : 0~3

Hardware Implementation Correlation window sum (Aggregation) :

Experimental Results

Left Image Ground Truth Full 7x7 census 12-edge 4-edge

Left Image Ground Truth Full 7x7 census 12-edge 4-edge

Left Image Full 7x7 census 12-edge 4-edge

Experimental Results 𝟖𝟖%↓ 𝟔𝟏%↓ LUTs (look-up tables) : the amount of logic required to implement the method FFs : the number of 1-bit registers (the amount of pipelining used) RAMs : the number of 18-kbit block memories Freq. : the maximum operating frequency reported by synthesis

Conclusion

Conclusion Proposed and analyzed in this paper: A range of sparse census transforms reduce hardware resource requirements attempting to maximize correlation accuracy. often better than or nearly as good as the full census Generalized census transforms increased robustness in the presence of image noise