1. f Networked Embedded Applications and Technologies Lab Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN Implementation of In-Vehicle Multi-sensor Information Fusion Gateway for Cooperative Driving Authors: Ting-Ying Wei, Zhi-Liang Qiu, Chung-Ping Young Presenter: Zhi-Liang Qiu

2. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 2 NEAT Outline  Introduction  Multi-Sensor Information Fusion  Hardware architecture  Vehicle Gateway  Electronic Control Units (ECUs)  Communication  Sensor  Software  Conclusion

3. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 3 NEAT Introduction  The traffic crashes caused by the human factor in US are about 80%-90%  The safety concern is the most important issue for vehicle operation  Cooperative driving  Enhance the driving information  Increase the response time

4. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 4 NEAT Introduction  Precaution warning or compensatory control before the pre-crash stage of vehicle safely  Heterogeneous multi-core processor:  One ARM core for I/O control and system management  Two DSP cores for intensive computation of information fusion  Signal processing  Location mapping  Trajectory prediction  Risk assessment

5. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 5 NEAT Introduction  Two types of tasks:  Real-time  Non-real-time  The ARM processor:  General Purpose Processor (GPP) for the non-real-time tasks  PAC DSP:  Special Purpose Processor(SPP) for the real-time tasks  Electronic Control Unit(ECU):  Gathering the surrounding information

6. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 6 NEAT Multi-Sensor Information Fusion  Vehicle gateway implements the information fusion  In-vehicle sensor data  Wireless communication packets  GPP will inform the data to Information fusion engine in SPP whenever the data is received  SPP generates several safety indexes and produce warning messages

7. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 7 NEAT Multi-Sensor Information Fusion  Procedure:  Retrieving information  Integrating the information in a time sequential linked list  Cross-evaluating the information  Mapping each car in the tree structure onto a highway map  Predicting the upcoming path  Generating warning messages  Broadcasting to surrounding vehicles

8. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 8 NEAT Multi-Sensor Information Fusion  Procedure:

9. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 9 NEAT Multi-Sensor Information Fusion  Time synchronization base on the GPS signal from each car  Image processing DSPs detect the lane departure and blind spot  Camera frame rate: 30fps  Sensor sampling rate: 200Hz

10. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 10 NEAT Multi-Sensor Information Fusion  Location mapping:  GPS data is only be used for global positioning route planning or track recording  Merging the signals from radar, accelerometer, wheel speed sensor, and yaw sensor generates the relative distance, position and velocity

11. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 11 NEAT Multi-Sensor Information Fusion  Trajectory prediction:  Using the current and past vehicle information as well as the circumstance event to estimate the next few seconds path  If there are any specific events, like emergency brake, we will receive the warning from the event car

12. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 12 NEAT Multi-Sensor Information Fusion  Risk assessment:

13. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 13 NEAT Vehicle Gateway  The vehicle gateway is a core of this system, it includes many different interfaces of communication as shown below

14. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 14 NEAT Vehicle Gateway  The realization of vehicle gateway

15. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 15 NEAT Vehicle Gateway  PAC Duo DSP development platform The PAC Duo platform is developed by Industry Technology Research Institute (ITRI) in Taiwan Each DSP core communicates with ARM core through AXI-AHB bus

16. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 16 NEAT Electronic Control Units  Microchip PIC18Fxx8 microcontroller  Renesas R32C/118 The microcontrollers are the middle layers between PAC Duo and the surrounding sensors

17. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 17 NEAT Communication  There are several wired or wireless of communication interfaces are required  CAN serial bus protocol  OBDII  IEEE 802.15.4  DSRC  3G

18. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 18 NEAT Sensors  Different types of sensors are used for acquiring vehicle diagnosis or driving status  Infrared night vision camera  The GPS receiver

19. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 19 NEAT Software  The software is separated into several parts  Non-real-time OS, Meego  Open source Linux-based mobile OS  Supported by Intel and Nokia  Provide the GUI  Real-time on PAC DSP  Non-OS on each microcontroller board

20. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 20 NEAT Conclusion  A heterogeneous multi-core platform is establishing.  The multi-sensor information fusion is implemented for computing the relative position, velocity and acceleration to the host vehicle.  Assess the potential of collision risk and provide the cooperative driving feature to enhance the active safety

21. Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN 21 NEAT Thank you for attention!