National Taiwan Normal A System to Detect Complex Motion of Nearby Vehicles on Freeways C. Y. Fang Department of Information and Computer Education National Taiwan Normal University, Taipei, Taiwan, R. O. C. C. P. Chen Department of Computer Science and Information Engineering National Taiwan Normal University, Taipei, Taiwan, R. O. C. C. S. Fuh Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R. O. C. S. W. Chen Department of Computer Science and Information Engineering National Taiwan Normal University, Taipei, Taiwan, R. O. C.

National Taiwan Normal Outline Introduction Dynamic visual model (DVM) Detect complex motion of nearby vehicles Feature extraction Attention map partition Collection of classification results Temporal integral process Experimental Results Conclusions

National Taiwan Normal Introduction -- VDAS System to detect motion of nearby vehicles: a Vision- based driver assistance system (VDAS) Advantages: High resolution Rich information Difficulties of VDAS Weather and illumination Daytime and nighttime Vehicle motion and camera vibration

National Taiwan Normal Introduction -- DVM DVM: dynamic visual model A computational model for visual analysis using video sequence as input data Two ways to develop a visual model Biological principles Engineering principles Artificial neural networks

National Taiwan Normal Improved DVM Two problems: The motions of vehicles may occur anywhere on the road. Training a CART neural network to recognize various complex motions is quite difficult. Solutions: Feature extraction Attention map partition Collection of classification results Temporal integral process

The Improved Dynamic Visual Model Conceptual component Perceptual component Category Sensory component Information acquisition CART STA neural module Yes No Video images Focuses of attention Spatialtemporal information Category Data transduction Action Episodic Memory Windowing Feature extraction Features CART Decision making Confirm? No Yes

National Taiwan Normal Introduction Motions of the Vehicles Lane change Speed change Objective Simple motion detection Complex motion detection

National Taiwan Normal Simple Motion Patterns

National Taiwan Normal Attention Map Partition b4b4 b5b5 b1b1 b2b2 b3b3

National Taiwan Normal Feature Extraction (1) --- Skewness features gi1gi1 i gi1gi1 i

National Taiwan Normal Feature Extraction (2) The horizontal skewness features: g i1 : the skewness of intensity value m i2, m i3 : the normalized second and third moments, respectively :the column means of intensity values, : the mean horizontal position of the intensity means

National Taiwan Normal Classification Results CART 1 CART 2 CART n CART i D t

National Taiwan Normal Temporal Fuzzy Integral (1) Let n be the number of CART neural networks : the output strings of labels of CART i from time t-r i +1 to t, k = 1, 2, …, r i : the set of all labels, where l 0 is null label p j : the stored pattern corresponding to label l j P : the set containing all stored patterns r i : time period

National Taiwan Normal Temporal Fuzzy Integral (2) Fuzzy measure function where #p : the number of non-zero pixels of one stored pattern : the number of such pixels falling in the union of windows i or j.

National Taiwan Normal Some Values of Fuzzy Measure Function CARTs Motion cases CART 1 CART 2 CART 3 CART 4 CART 5 (a) Vehicle ahead slows down (b) Right front vehicle changes lane to the left (c) Left front vehicle changes lane to the right (d) Vehicle ahead changes lane to the right (e) Vehicle ahead changes lane to the left

National Taiwan Normal Temporal Fuzzy Integral (3) Confidence function where j, k = 1, 2, …, r i, : a distance between p j and p k, : weight functions, : positive parameters

National Taiwan Normal Temporal Fuzzy Integral (4) Fuzzy integral : the integral value for : the fuzzy intersection characterized by a t-norm

National Taiwan Normal Intermediate Decision of Individual CART i where : a distance threshold

National Taiwan Normal Collection of Classification Results The final classification set where, : the corresponding integral value of : a threshold

National Taiwan Normal Experimental Results b4b4 b5b5 b1b1 b2b2 b3b3

National Taiwan Normal Experimental Results b4b4 b5b5 b1b1 b2b2 b3b3

National Taiwan Normal Experimental Results (1) b4b4 b5b5 b1b1 b2b2 b3b3

National Taiwan Normal b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (1)

National Taiwan Normal b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (2)

National Taiwan Normal b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (3)

National Taiwan Normal b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (4)

National Taiwan Normal b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (5)

National Taiwan Normal b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (6)

National Taiwan Normal Complex Motion Sequence A B C

National Taiwan Normal Experimental Results Simple motion sequences 12 sequences accuracy rate: 97.9% Complex motion sequences 18 sequences accuracy rate: 93.3% Since our system only outputs a result for each input sequence, this ratio is enough for our system to recognize road signs correctly.

National Taiwan Normal Experimental Results

National Taiwan Normal Discussion Improve attention map partition Detect other dynamic obstacles

National Taiwan Normal Conclusions A neural-based dynamic visual model Three major components: sensory, perceptual and conceptual component Future Researches Potential applications Improvement of the DVM structure DVM implementation

National Taiwan Normal Human Visual Process Transducer Sensory analyzer Class of input stimuli Perceptual analyzer Conceptual analyzer Physical stimuli Data compression Low-level feature extraction High-level feature extraction Classification and recognition

National Taiwan Normal Neural Modules Spatial-temporal attention (STA) neural module Configurable adaptive resonance theory (CART) neural module

National Taiwan Normal STA Neural Network (1) akak Output layer (Attention layer) njnj Inhibitory connection Excitatory connection Input layer w ij aiai xjxj nknk nini

National Taiwan Normal STA Neural Network (2) The input to attention neuron n i due to input stimuli x : The linking strengths between the input and the attention layers corresponding neurons w kj nini njnj nknk Input neuron Attention layer rkrk Gaussian function G

National Taiwan Normal STA Neural Network (3) The input to attention neuron n i due to lateral interaction: Lateral distance “Mexican-hat” function of lateral interaction Interaction +

National Taiwan Normal STA Neural Network (4) The net input to attention neuron n i : : a threshold to limit the effects of noise where 1< d <0

National Taiwan Normal STA Neural Network (5) t p 1 pd 1 The activation of an attention neuron in response to a stimulus. stimulus activation

National Taiwan Normal ART2 Neural Network (1) CART r p u w v x q y Input vector i Input representation field F 1 Attentional subsystem Orienting subsystem G G G G G Category representation field F 2 Reset signal ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ － － － － － Signal generator S

National Taiwan Normal ART2 Neural Network (2) The activities on each of the six sublayers on F 1 : where I is an input pattern where where the J th node on F 2 is the winner

National Taiwan Normal ART2 Neural Network (3) Initial weights: Top-down weights: Bottom-up weights: Parameters: