Multimodal Information Analysis for Emotion Recognition (Tele Health Care Application) Malika Meghjani Gregory Dudek and Frank P. Ferrie Good Morning, My name is Malika. I am from the department of Electrical and Computer Engineering at McGill University. And I would be presenting my research entitled “Multimodal Information Analysis for Emotion Recognition”. This work is done under the supervision of Prof. Gregory Dudek from school of Computer Science and Prof. Frank Ferrie from ECSE.

Content Our Goal Motivation Proposed Approach Results Conclusion In my presentation, I will talk about…..

Our Goal Automatic emotion recognition using audio-visual information analysis. Create video summaries by automatically labeling the emotions in a video sequence.

NURSING INTERVENTIONS Motivation Map Emotional States of the Patient to Nursing Interventions. Evaluate the role of Nursing Interventions for improvement in patient’s health. intervention - care provided to improve a situation (especially medical procedures or applications that are intended to relieve illness or injury) We have, recorded video sequences of a patient interacting with a nurse on a remote end in a video conference call. During the conversation, the patient shares various issues she has relating her health and the life in general. Based on the concerns and the emotional state of the patient, the nurse provides the required support for the patient to better manage her health. Later these recorded sequences are watched manually and an inference is drawn between the emotional state of the patient and the nurse’s intervention to evaluate how the intervention helped the nurse in the improvement of the patient’s health. This was a very tedious task which we aim at automating. We want a computer to look at the video sequences and classify it in one of the emotional classes. NURSING INTERVENTIONS

Proposed Approach Visual Feature Extraction Visual based Emotion Classification Recognized Emotional State Data Fusion Decision Level Fusion Audio Feature Extraction Audio based Emotion Classification Emotions are mainly expressed through facial expressions, gestures and/or in the voice. Here we use two sources for emotion recognition: vocal information and facial expression. First we perform emotion recognition using one of the individual modalities and then we combine the two modalities to compare how well the system performs. There are basically two ways of combining the information. We can either take decision of the two modalities and see how confident each one is and then combine based on the confidence measure. The second method combines the two sources before any decision is made and give entire set of information to the recognizer and the emotion classification is performed. Feature Level Fusion

Visual Analysis Face Detection (Voila Jones Face Detector using Haar Wavelets and Boosting Algorithm) Feature Extraction (Gabor Filter: 5 Spatial Frequencies and 4 orientations, 20 filter responses for each frame) Feature Selection (Select the most discriminative features in all the emotional classes) SVM Classification (Classification with probability estimates)

Visual Feature Extraction = X ( 5 Frequencies X 4 Orientation ) Frequency Domain Filters Feature Selection Automatic Emotion Classification

Audio Analysis Audio Pre-Processing (Remove leading and trailing edges) Feature Extraction (Statistics of Pitch and Intensity contours and Mel Frequency Cepstral Coefficients) Feature Normalization (Remove inter speaker variability) SVM Classification (Classification with probability estimates)

Audio Feature Extraction Automatic Emotion Classification Speech Rate Pitch Intensity Spectrum Analysis Mel Frequency Cepstral Coefficients (Short-term Power Spectrum of Sound) Audio Signal Fourier Transform of audio signal. Map the powers of the spectrum to Mel Frequency. Take log of the powers. Perform discrete cosine transform. MFCC are amplitudes of the spectrum.

SVM Classification Maximize the margin Have a penalty term for misclassified points. Map the data to higher dimensional plane where it is easier to classify with linear decision surface.

Feature Selection Feature selection method is similar to the SVM classification. (Wrapper Method) Generates a separating plane by minimizing the weighted sum of distances of misclassified data points to two parallel planes. Suppress as many components of the normal to the separating plane which provide consistent results for classification. Average Count of Error Features Distance Selected A and B are two point sets. ‘w’  Normal to the separating plane. e  Vector of ones. Bounding Planes

Data Fusion Decision Level: Obtaining probability estimate for each emotional class using SVM margins. The probability estimates from two modalities are multiplied and re-normalized to the give final estimation of decision level emotional classification. 2. Feature Level: Concatenate the Audio and Visual feature and repeat feature selection and SVM classification process.

Database and Training Database: Visual only posed database Audio Visual posed database Training: Audio segmentation based on minimum window required for feature extraction. Corresponding visual key frame extraction in the segmented window. Image based training and audio segment based training.

Experimental Results Statistics Database No. of Training Examples No. of Subjects No. of Emotional State % Recognition Rate Validation Method Posed Visual Data Only (CKDB) 120 20 5+Nuetral 75% Leave one subject out cross validation Posed Audio Visual Data (EDB) 270 9 6 82% 76% Decision Level Feature Level 14

Time Series Plot Surprise Sad Angry Disgust Happy 75% Leave One Subject Out Cross Validation Results ( * Cohen Kanade Database, Posed Visual only Database) 15

(*eNTERFACE 2005 , Posed Audio Visual Database) Feature Level Fusion (*eNTERFACE 2005 , Posed Audio Visual Database)

(*eNTERFACE 2005 , Posed Audio Visual Database) Decision Level Fusion (*eNTERFACE 2005 , Posed Audio Visual Database)

Confusion Matrix

Demo

Conclusion Combining two modalities (Audio and Visual) improves overall recognition rates by 11% with Decision Level Fusion and by 6% with Feature Level Fusion Emotions where vision wins: Disgust, Happy and Surprise. Emotions where audio wins: Anger and Sadness Fear was equally well recognized by the two modalities. Automated multimodal emotion recognition is clearly effective. 20

Things to do… Inference based on temporal relation between instantaneous classifications. Tests on natural audio-visual database (on-going).