1. NTU CSIE 2003 1 研 究 生：方瓊瑤 A Vision-Based Driver Assistance System Based on Dynamic Visual Model 指導教授：陳世旺博士 傅楸善博士

2. NTU CSIE 20032 Outline Introduction Dynamic visual model (DVM) Neural modules Road sign recognition system System to detect changes in driving environments System to detect motion of nearby moving vehicles Conclusions

3. NTU CSIE 20033 Introduction (1) -- ITS Intelligent transportation system (ITS) Advanced traffic management systems (ATMS) Advanced traveler information systems (ATIS) Commercial vehicle operations (CVO) Advanced public transportation systems (APTS) Advanced rural transportation systems (ARTS) Advanced vehicle control and safety systems (AVCSS) Driver assistance systems (DAS)

4. NTU CSIE 20034 Introduction (2) -- DAS Driver assistance systems (DAS) Safety Passive Active Driving is a sophisticated process The technology of vehicle The temperament of the driver

5. NTU CSIE 20035 Introduction (3) -- VDAS Vision-based driver assistance systems (VDAS) Difficulties of VDAS Weather and illumination Daytime and nighttime Vehicle motion and camera vibration

6. NTU CSIE 20036 Introduction (4) Subsystems of VDAS Road sign recognition system System to detect changes in driving environments System to detect motion of nearby moving vehicles

7. NTU CSIE 20037 Introduction (5) -- DVM DVM: dynamic visual model A computational model for visual analysis using video sequence as input data Two ways to develop a visual model Biological principles Engineering principles Artificial neural networks

8. NTU CSIE 20038 Dynamic Visual Model Conceptual component Perceptual component Sensory component Information acquisition CART neural module STA neural module Yes No Video images Focuses of attention Spatialtemporal information Categorical features Category Feature detection Pattern extraction CHAM neural module Patterns Data transduction Action Episodic Memory

9. NTU CSIE 20039 Human Visual Process Transducer Sensory analyzer Class of input stimuli Perceptual analyzer Conceptual analyzer Physical stimuli Data compression Low-level feature extraction High-level feature extraction Classification and recognition

10. NTU CSIE 200310 Neural Modules Spatial-temporal attention (STA) neural module Configurable adaptive resonance theory (CART) neural module Configurable heteroassociative memory (CHAM) neural module

11. NTU CSIE 200311 STA Neural Network (1) akak Output layer (Attention layer) njnj Inhibitory connection Excitatory connection Input layer w ij aiai xjxj nknk nini

12. NTU CSIE 200312 STA Neural Network (2) The input to attention neuron n i due to input stimuli x : The linking strengths between the input and the attention layers corresponding neurons w kj nini njnj nknk Input neuron Attention layer rkrk Gaussian function G

13. NTU CSIE 200313 STA Neural Network (3) The input to attention neuron n i due to lateral interaction: Lateral distance “Mexican-hat” function of lateral interaction Interaction +

14. NTU CSIE 200314 STA Neural Network (4) The net input to attention neuron n i : : a threshold to limit the effects of noise where 1< d <0

15. NTU CSIE 200315 STA Neural Network (5) t p 1 pd 1 The activation of an attention neuron in response to a stimulus. stimulus activation

16. NTU CSIE 200316 ART2 Neural Network (1) CART r p u w v x q y Input vector i Input representation field F 1 Attentional subsystem Orienting subsystem G G G G G Category representation field F 2 Reset signal ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ － － － － － Signal generator S

17. NTU CSIE 200317 ART2 Neural Network (2) The activities on each of the six sublayers on F 1 : where I is an input pattern where where the J th node on F 2 is the winner

18. NTU CSIE 200318 ART2 Neural Network (3) Initial weights: Top-down weights: Bottom-up weights: Parameters:

19. NTU CSIE 200319 HAM Neural Network (1) CHAM j Output layer (Competitive layer) Excitatory connection Input layer w ij xjxj i vivi v1v1 v2v2 vnvn

20. NTU CSIE 200320 HAM Neural Network (2) The input to neuron n i due to input stimuli x : n c : the winner after the competition

21. NTU CSIE 2003 21 Road Sign Recognition System

22. NTU CSIE 200322 Objective Get information about road Warn drivers Enhance traffic safety Support other subsystems

23. NTU CSIE 200323 Problems

24. NTU CSIE 200324 Perceptual Component

25. NTU CSIE 200325 Conceptual Component— C lassification results of CART Training Set Test Set

26. NTU CSIE 200326 Conceptual Component — Training Patterns for CHAM

27. NTU CSIE 200327 Experimental Results

28. NTU CSIE 200328

29. NTU CSIE 200329

30. NTU CSIE 200330 Other Examples

31. NTU CSIE 200331 Discussion Vehicle and camcorder vibration Incorrect recognitions Input patterns Recognition results Correct patterns

32. NTU CSIE 2003 32 System to Detect Changes in Driving Environments

33. NTU CSIE 200333 Definition The environmental changes in expressways: Left-lane-change Right-lane-change Tunnel-entry Tunnel-exit Expressway-entry Expressway-exit Overpass-ahead

34. NTU CSIE 200334 Objective Coordinate DAS subsystems Update parameters Detect unexpected changes Detect rapid changes

35. NTU CSIE 200335 Results of the Sensory Component

36. NTU CSIE 200336 Results of the Perceptual Component

37. NTU CSIE 200337 The Prototypical Attention Patterns

38. NTU CSIE 200338 Experimental Results

39. NTU CSIE 200339 Experimental Results

40. NTU CSIE 200340 Experimental Results

41. NTU CSIE 200341 Experimental Results

42. NTU CSIE 200342 Experimental Results

43. NTU CSIE 200343 Discussion Curved roads Shadows Multiple environmental changes

44. NTU CSIE 2003 44 System to Detect Motion of Nearby Moving Vehicles

45. NTU CSIE 200345 Introduction Motions of the Vehicles Lane change Speed change Objective Simple motion detection Complex motion detection

46. NTU CSIE 200346 Simple Motion Patterns

47. NTU CSIE 200347 Improved DVM Two problems: The motions of vehicles may occur anywhere on the road. Training a CART neural network to recognize various complex motions is quite difficult. Solutions: Feature extraction Attention map partition Collection of classification results Temporal integral process

48. NTU CSIE 200348 No Yes Attention maps Windowing Feature extraction CART 1 CART 2 CART n-1 CART n Decision making Confirm? Output bnbn b n-1 b2b2 b1b1 st1st1 st2st2 s t n-1 stnstn Feature extraction Flowchart for Conceptual Component

49. NTU CSIE 200349 Attention Map Partition b4b4 b5b5 b1b1 b2b2 b3b3

50. NTU CSIE 200350 Feature Extraction (1) --- Skewness features gi1gi1 i gi1gi1 i 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10

51. NTU CSIE 200351 Feature Extraction (2) The horizontal skewness features: g i1 : the skewness of intensity value m i2, m i3 : the normalized second and third moments, respectively :the column means of intensity values, : the mean horizontal position of the intensity means

52. NTU CSIE 200352 Classification Results CART 1 CART 2 CART n CART i D t

53. NTU CSIE 200353 Temporal Fuzzy Integral (1) Let n be the number of CART neural networks : the output strings of labels of CART i from time t-r i +1 to t, k = 1, 2, …, r i : the set of all labels, where l 0 is null label p j : the stored pattern corresponding to label l j P : the set containing all stored patterns r i : time period

54. NTU CSIE 200354 Temporal Fuzzy Integral (2) Fuzzy measure function where #p : the number of non-zero pixels of one stored pattern : the number of such pixels falling in the union of windows i or j.

55. NTU CSIE 200355 Some Values of Fuzzy Measure Function

56. NTU CSIE 200356 Temporal Fuzzy Integral (3) Confidence function where j, k = 1, 2, …, r i, : a distance between p j and p k, : weight functions, : positive parameters

57. NTU CSIE 200357 Temporal Fuzzy Integral (4) Fuzzy integral : the integral value for : the fuzzy intersection characterized by a t-norm

58. NTU CSIE 200358 Intermediate Decision of Individual CART i where : a distance threshold

59. NTU CSIE 200359 Collection of Classification Results The final classification set where, : the corresponding integral value of : a threshold

60. NTU CSIE 200360 Experimental Results b4b4 b5b5 b1b1 b2b2 b3b3

61. NTU CSIE 200361 Experimental Results b4b4 b5b5 b1b1 b2b2 b3b3

62. NTU CSIE 200362 Complex Motion Sequence A B C

63. NTU CSIE 200363 Experimental Results Simple motion sequences 12 sequences accuracy rate: 97.9% Complex motion sequences 18 sequences accuracy rate: 93.3%

64. NTU CSIE 200364 Experimental Results

65. NTU CSIE 200365 Discussion Improve attention map partition Detect other dynamic obstacles

66. NTU CSIE 200366 Conclusions A neural-based dynamic visual model Three major components: Sensory component Perceptual component Conceptual component Three DAS subsystems: Road sign recognition system Driving environmental change detection system Nearby vehicle motion detection system

67. NTU CSIE 200367 Future Research Potential applications Lane marks detection Obstacle recognition Drowsy driver detection Coordinating system Improvement of the DVM structure Acoustic and tactile processes Reasoning: Intellectual and associative thinking DVM implementation Genetic algorithms Fuzzy theoretic methods