1. 1 Automobile Lane Detection System- on-Chip Integrated with Mixed Signal Mode CMOS Image Sensor IEEE 9th International Symposium on Consumer Electronics (ISCE 2005) Pei-Yung Hsiao¹, Hsien-Chein Cheng¹, Chun-Wei Yeh¹, Shih-Shinh Huang², and Li-Chen Fu² ¹ Department of Electronic Engineering Chang Gung University ² Department of Electrical Engineering National Taiwan University

2. Chang Gung University & National Taiwan University 2 OUTLINE The Problem & Motive Brief Introduction to Algorithms Architecture and Circuits description Simulation and Results Conclusion

3. Chang Gung University & National Taiwan University 3 Original Idea for the Problem Capture Image Land map We are interested in developing a System- on-Chip, Soc, which can capture image as well as produce vehicle lane map at the same time.

4. Chang Gung University & National Taiwan University 4 Motive The areas of Intelligent Transportation System, ITS, include lane detection, obstacle recognition, vehicle detection, car following, etc. Our goal in this investigation is to develop a CMOS imager to achieve real-time image capture and lane detection, simultaneously, for intelligent automotive driver awareness/assistance system.

5. Chang Gung University & National Taiwan University 5 Automotive IC Design Vehicle Detection & Tracking Driver Assistance System Lane Departure Prevention Automobile Lane Detection SoC Built in The target chip can be defined as a component device for intelligent vehicles.

6. Chang Gung University & National Taiwan University 6 Widespread Applications The proposed imager without demanding extra ADC circuits for signal transformation is a single low-cost and compact chip for used in the thousands of consumer electronics not limited to ITS.

7. Chang Gung University & National Taiwan University 7 Introduction(1/4) From the referenced literatures, there are a lot of vision-based lane detection algorithms proposed in recent 10 years [1-6] (1995-2004). In 1995, Kluge and Lakshmanan [3] proposed the LOIS (Likelihood of Image Shape) lane detection, which is able to detect lanes even in situations with shadows or broken lanes by using a stochastic optimization procedure.

8. Chang Gung University & National Taiwan University 8 Introduction (2/4) In 1995, Broggi [5] proposed an edge-based road detection algorithm, while it is effective only for the well-painted road. In 1999, Takahashi, etc. [4] divided the parameter space of the lane model to generate the lane marking patterns and then applied the voting scheme to find the lane boundary.

9. Chang Gung University & National Taiwan University 9 Introduction (3/4) In 2004, Huang, etc [1] proposed an on-board vision system for lane recognition and front- vehicle detection to enhance driver's awareness. Regarding to high recognition rate and hard- wired regularity, we adopted Peak-Finding based lane detection algorithm from Huang, etc [1], which has high recognition rate about 96%.

10. Chang Gung University & National Taiwan University 10 Introduction (4/4) According to Huang’s algorithm, we also developed an auto-regulated threshold circuit to automatically adjust the threshold for the lane detector to adapted to different weather conditions.

11. Chang Gung University & National Taiwan University 11 Architecture and Circuit description Our chip can be divided into three parts, such as analogue capturing and processing circuits, digital processing circuits and digital control unit. The analogous circuits include 2-D pixel cell array, CDS module, 1-D Gaussian filter and Peak-Finding module. The digital processing circuits are composed of Line Point Allocation module, column selector and row selector.

12. Chang Gung University & National Taiwan University 12 SOC for real-time image capture and lane detection

13. Chang Gung University & National Taiwan University 13 Pixel Cell & Sensor Array The developed sensor array consists of two types of pixel cells. Our sensor array prototype is made up of 64*64 effective pixels. The upper region containing 16*64 pixels is ignored in back-end processing to promote the computing efficiency. The other regions are horizontally partitioned into three sub-regions. Each sub-region consists of 16 rows. The 12th row in the sub-region or in the upper region is defined as a 1-D sample array. Consequently, we have four 1-D sample arrays in our sensor array.

14. Chang Gung University & National Taiwan University 14 Normal cell Sampling cell

15. Chang Gung University & National Taiwan University 15 Dual 1-D Gaussian Filters Each 1-D Gaussian filter includes 64 Gaussian mask cells and a current divider. Each Gaussian mask cell consists of 3 current mirrors in 7 transistors and two OR gates. The Gaussian filter module is used for smoothing the selected pixel by referring to a couple of right and left neighbors to eliminate noisy points in the original image.

16. Chang Gung University & National Taiwan University 16 Dual 1-D Gaussian Filters

17. Chang Gung University & National Taiwan University 17 Peak-Finding Module (1/3) The 1st part of the Peak-Finding Module can accumulate and average current, Iavg, from the aforementioned sample arrays, Ips. The averaged current from sample array, Iavg, was generated according to the following equation.

18. Chang Gung University & National Taiwan University 18 Lane-Point Output Peak-Point Output Peak-Finding Module Line-Point Allocation

19. Chang Gung University & National Taiwan University 19 Peak-Finding Module (2/3) The 2nd part of PFM is called as auto- regulated threshold circuit. It compares average current, Iavg, with four preconfigured currents, Irefn, and then produces the threshold current, Isth. The total threshold current, I TH can be noted by the following equation., where

20. Chang Gung University & National Taiwan University 20 Peak-Finding Module (3/3) Inside the auto-regulated threshold circuit, each threshold mapping circuit control a threshold current. It can be noted by the following equation: According to the 3rd part of the PFM, If the current pixel is a peak point, the output value will be 1, otherwise, it should be 0., where

21. Chang Gung University & National Taiwan University 21 Line-Point Allocation The Lane-Point Allocation Module expressed as the following equation is composed of two digital functions, such as the line segment filter and the lane point selector. Lane points, Lb(i,j), are obtained, and represented by only one pixel width in each row.

22. Chang Gung University & National Taiwan University 22 The clock frequency of the Row Selector (clk R) is 64 times of the Column Selector (clk C). In this case, the frequency of the Column Selector is 25MHz and the frequency of the Row Selector is 0.78MHz. Timing Diagram

23. Chang Gung University & National Taiwan University 23 HSPICE Simulation (1/2) (a) (b) (c) Peak-Point (a) is the output current of the current Gaussian Filter. (b) is the output current of the previous Gaussian Filter. (c) is the results of the Peak-Finding Module. 1.5us

24. Chang Gung University & National Taiwan University 24 HSPICE Simulation (2/2) (a) (b) (c) (d) (a) is the simulation results of the current Gaussian Filter. (b) is the simulation results of the previous Gaussian Filter. (c) is the simulation result of the Peak-Finding Module. (d) is the simulation results of the Lane-Point Allocation Module. 5us25us

25. Chang Gung University & National Taiwan University 25 Software Simulation in C (a) Original image (512x512 > 320x240[1]) (b) Lane map points generated by Peak-Finding algorithm. (c) Original image (64x64) (d) Lane map points generated by Peak-Finding algorithm. From [1], noises are to be removed by the followed post processing.

26. Chang Gung University & National Taiwan University 26 Experimental Results (a) Original image in 32 * 32 (c) Result generated by our chip (b) Result generated by software

27. Chang Gung University & National Taiwan University 27 Chip Layout in 64 * 64

28. Chang Gung University & National Taiwan University 28 Specification of The Proposed CMOS Imager ItemValues Pixel Count64(H) X 64(V) Pixel Size18.45 um(H) X 21.8 um (V) Aperture Size12.45 um(H) X 9.6 um (V) Fill Factor29.7 % Image Size1217.7 um (H) X 1455.05 um (V) Chip Size2191.4 um (H) X 2389.8 um (V) Operation Clock25MHz Operation Voltage3.3 V Power consumption159.4mW

29. Chang Gung University & National Taiwan University 29 Conclusion Our investigation includes a 2-D image sensor array embedded with four modularized circuits: ----- four 1-D sample array by different pixel cell design for accumulating the sampled currents; ----- dual 1-D Gaussian filers coupling as an analogue image smoothing module; ----- an analogue design for Peak-Finding function associating with a novel auto-regulated threshold operation; ----- a sophisticated digital implementation for Lane-Point allocation.

30. Chang Gung University & National Taiwan University 30 Conclusion, cont. A new current-mode mixed signal design of CMOS image sensor integrated with Peak-Finding based lane detection algorithm is developed. The proposed low-cost and one compact chip solution can grab the road images from the real world and successfully detect the lane markers simultaneously, in real time.

31. Chang Gung University & National Taiwan University 31 Reference (1/4) [1] Shih-Shinh Huang, Chung-Jen Chen, Pei-Yung Hsiao, and Li- Chen Fu, “On-Board Vision System for Lane Recognition and Front-Vehicle Detection to Enhance Driver's Awareness”, IEEE International Conference on Robotics and Automation, vol. 3, 26 April – 1 May, 2004, pp.2456–2461. [2] Yue Wang, Eam Khwang Teoh and Dinggang She, “ Lane detection using B-snake”,, International Conference on Information Intelligence and Systems, 31 Oct. - 3 Nov., 1999, pp.438 – 443. [3] Kluge, K., and S. Lakshmanan,, “A Deformable-Template Approach to Lane Detection”, in I. Masaky, editor, Proceedings IEEE Intelligent Vehicle’95, Detroit, 25-26 Sept., 1995, pp.54-59. [4] Takahashi, A., Ninomiya, Y., Ohta, M., and Tange, K., “A Robust Lane Detection Using Real-Time Voting Processor”, IEEE/IEEJ/JSAI International Conference on Intelligent Transportation Systems, 5-8 Oct., 1999, pp.577–580.

32. Chang Gung University & National Taiwan University 32 Reference (2/4) [5] A. Broggi, “Robust Real-Time Lane and Road Detection in Critical Shadow Conditions,” Computer Vision, 1995, Proceedings., International Symposium on, 21-23 Nov. 1995 pp.353-358 [6] Li, Q., Zheng, N., and Cheng, H.,“ Springrobot: A Prototype Autonomous Vehicle and Its Algorithms for Lane Detection”, IEEE Transactions on Intelligent Transportation System, vol. 5, no. 4, Dec., 2004, pp.300-308. [7] Coulombe, J., Sawan, M. and Wang, C., “Variable Resolution CMOS Current Mode Active Sensor”, IEEE International Symposium on Circuits and Systems, vol. 2, 28-31 May, 2000, pp.293 – 296. [8] Tabet, M., Hornsey, R.,“ CMOS Image Sensor Camera with Focal Plane Edge Detection”, Canadian Conference on Electrical and Computer Engineering, vol. 2, 13-16 May, 2001, pp.1129 – 1133.

33. Chang Gung University & National Taiwan University 33 Reference (3/4) [9] Pei-Yung Hsiao, Yu-Chun Hsu, Wen-Ta Lee, Chia-Chun Tsai, and Chia-Hao Lee, ”An Embedded Analog Spatial Filter Design of The Current-Mode CMOS Image Sensor”, IEEE Transactions on Consumer Electronics, vol. 50, no. 3, Aug., 2004, pp.945– 951. [10] N. Yang and G. Jianhong, “ A 256x256 Pixel Smart CMOS Image Sensor for Line Based Stereo Vision Applications”, IEEE Journal of solid state circuits, vol. 35, no. 7, July, 2000, pp.1055-1061 [11] Yuan, J. and Svensson, C., “High-speed CMOS circuit technique,” IEEE J. Solid-state Circuits, vol. 24, no. 2, 1989, pp. 62-70. [12] Byungsoo Chang, Joonbae Park and Wonchan Kim,“A 1.2 GHz CMOS dual-modulus prescaler using new dynamic D-type flip- flops”, IEEE Journal of Solid-State Circuits, vol. 31, no. 5, May, 1996. pp.749-752

34. Chang Gung University & National Taiwan University 34 Reference (4/4) [13] Fish, A and Yadid-Pecht, o.,”CMOS current/voltage mode winner-take-all circuit with spatial filtering”, The IEEE International Symposium on Circuits and Systems, vol. 3, 6-9 May, 2001, pp.636-639. [14] Nakamura, J., Pain, B., Nomoto, T., Nakamura, T. and Fossum, E.R., “On-focal-plane signal processing for current- mode active pixel sensors”, IEEE Transactions on Electron Devices, vol. 44, no. 10, Oct., 1997, pp.1747 – 1758