1. National Taiwan Universityviolet@ice.ntnu.edu.tw1 A Road Sign Recognition System Based on a Dynamic Visual Model C. Y. Fang Department of Information and Computer Education National Taiwan Normal University, Taipei, Taiwan, R. O. C. C. S. Fuh Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R. O. C. S. W. Chen Department of Computer Science and Information Engineering National Taiwan Normal University, Taipei, Taiwan, R. O. C. P. S. Yen Department of Information and Computer Education National Taiwan Normal University, Taipei, Taiwan, R. O. C.

2. National Taiwan Universityviolet@ice.ntnu.edu.tw2 Outline Introduction Dynamic visual model (DVM) Neural modules Road sign recognition system Experimental Results Conclusions

3. National Taiwan Universityviolet@ice.ntnu.edu.tw3 Introduction -- DAS Driver assistance systems (DAS) The method to improve driving safety Passive methods: seat-belts, airbags, anti-lock braking systems, and so on. Active methods: DAS Driving is a sophisticated process The better the environmental information a driver receives, the more appropriate his/her expectations will be.

4. National Taiwan Universityviolet@ice.ntnu.edu.tw4 Introduction -- VDAS Vision-based driver assistance systems (VDAS) Advantages: High resolution Rich information Road border detection or lane marking detection Road sign recognition Difficulties of VDAS Weather and illumination Daytime and nighttime Vehicle motion and camera vibration

5. National Taiwan Universityviolet@ice.ntnu.edu.tw5 Subsystems of VDAS Road sign recognition system System to detect changes in driving environments System to detect motion of nearby vehicles Lane marking detection Obstacle recognition Drowsy driver detection ……

6. National Taiwan Universityviolet@ice.ntnu.edu.tw6 Introduction -- DVM DVM: dynamic visual model A computational model for visual analysis using video sequence as input data Two ways to develop a visual model Biological principles Engineering principles Artificial neural networks

7. Dynamic Visual Model Conceptual component Perceptual component Sensory component Information acquisition CART neural module STA neural module Yes No Video images Focuses of attention Spatialtemporal information Categorical features Category Feature detection Pattern extraction CHAM neural module Patterns Data transduction Action Episodic Memory

8. National Taiwan Universityviolet@ice.ntnu.edu.tw8 Human Visual Process Transducer Sensory analyzer Class of input stimuli Perceptual analyzer Conceptual analyzer Physical stimuli Data compression Low-level feature extraction High-level feature extraction Classification and recognition

9. National Taiwan Universityviolet@ice.ntnu.edu.tw9 Neural Modules Spatial-temporal attention (STA) neural module Configurable adaptive resonance theory (CART) neural module Configurable heteroassociative memory (CHAM) neural module

10. National Taiwan Universityviolet@ice.ntnu.edu.tw10 STA Neural Network (1) akak Output layer (Attention layer) njnj Inhibitory connection Excitatory connection Input layer w ij aiai xjxj nknk nini

11. National Taiwan Universityviolet@ice.ntnu.edu.tw11 STA Neural Network (2) The input to attention neuron n i due to input stimuli x : The linking strengths between the input and the attention layers corresponding neurons w kj nini njnj nknk Input neuron Attention layer rkrk Gaussian function G

12. National Taiwan Universityviolet@ice.ntnu.edu.tw12 STA Neural Network (3) The input to attention neuron n i due to lateral interaction: Lateral distance “Mexican-hat” function of lateral interaction Interaction +

13. National Taiwan Universityviolet@ice.ntnu.edu.tw13 STA Neural Network (4) The net input to attention neuron n i : : a threshold to limit the effects of noise where 1< d <0

14. National Taiwan Universityviolet@ice.ntnu.edu.tw14 STA Neural Network (5) t p 1 pd 1 The activation of an attention neuron in response to a stimulus. stimulus activation

15. National Taiwan Universityviolet@ice.ntnu.edu.tw15 ART2 Neural Network (1) CART r p u w v x q y Input vector i Input representation field F 1 Attentional subsystem Orienting subsystem G G G G G Category representation field F 2 Reset signal ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ － － － － － Signal generator S

16. National Taiwan Universityviolet@ice.ntnu.edu.tw16 ART2 Neural Network (2) The activities on each of the six sublayers on F 1 : where I is an input pattern where where the J th node on F 2 is the winner

17. National Taiwan Universityviolet@ice.ntnu.edu.tw17 ART2 Neural Network (3) Initial weights: Top-down weights: Bottom-up weights: Parameters:

18. National Taiwan Universityviolet@ice.ntnu.edu.tw18 HAM Neural Network (1) CHAM j Output layer (Competitive layer) Excitatory connection Input layer w ij xjxj i vivi v1v1 v2v2 vnvn

19. National Taiwan Universityviolet@ice.ntnu.edu.tw19 HAM Neural Network (2) The input to neuron n i due to input stimuli x : n c : the winner after the competition

20. National Taiwan Universityviolet@ice.ntnu.edu.tw20 Road Sign Recognition System Objective Get information about road Warn drivers Enhance traffic safety Support other subsystems

21. National Taiwan Universityviolet@ice.ntnu.edu.tw21 Problems contrary light side by side shaking occlusion

22. National Taiwan Universityviolet@ice.ntnu.edu.tw22 Information Acquisition Color information Example: Red color Shape information Example: Red color edge

23. National Taiwan Universityviolet@ice.ntnu.edu.tw23 Results of STA Neural Module — Adding Pre-attention

24. National Taiwan Universityviolet@ice.ntnu.edu.tw24 Locate Road Signs — Connected Component

25. National Taiwan Universityviolet@ice.ntnu.edu.tw25 Categorical Feature Extraction Normalization: 50X50 pixels Remove the background pixels Features: Red color horizontal projection: 50 elements Green color horizontal projection: 50 elements Blue color horizontal projection: 50 elements Orange color horizontal projection: 50 elements White and black color horizontal projection: 50 elements Total: 250 elements in a feature vector

26. National Taiwan Universityviolet@ice.ntnu.edu.tw26 Conceptual Component — Classification results of the CART Training Set Test Set

27. National Taiwan Universityviolet@ice.ntnu.edu.tw27 Conceptual Component — Training and Test Patterns for the CHAM

28. National Taiwan Universityviolet@ice.ntnu.edu.tw28 Conceptual Component — Training and Test Patterns for the CHAM

29. National Taiwan Universityviolet@ice.ntnu.edu.tw29 Conceptual Component — Another Training Patterns for the CHAM

30. National Taiwan Universityviolet@ice.ntnu.edu.tw30 Experimental Results of the CHAM

31. National Taiwan Universityviolet@ice.ntnu.edu.tw31 Experimental Results

33. National Taiwan Universityviolet@ice.ntnu.edu.tw33 Other Examples

34. National Taiwan Universityviolet@ice.ntnu.edu.tw34 Discussion Vehicle and camcorder vibration Incorrect recognitions Input patterns Recognition results Correct patterns

35. National Taiwan Universityviolet@ice.ntnu.edu.tw35 Conclusions (1) Test data: 21 sequences Detection rate (CART): 99% Misdetection: 1% (11 frames) Recognition rate (CHAM): 85% of detected road signs Since our system only outputs a result for each input sequence, this ratio is enough for our system to recognize road signs correctly.

36. National Taiwan Universityviolet@ice.ntnu.edu.tw36 Conclusions (2) A neural-based dynamic visual model Three major components: sensory, perceptual and conceptual component Future Researches Potential applications Improvement of the DVM structure DVM implementation