Audio-based Emotion Recognition for Advanced Information Retrieval in Judicial Domain ICT4JUSTICE 2008 – Thessaloniki,October 24 G. Arosio, E. Fersini, E. Messina, F. Archetti Dipartimento di Informatica, Sistemistica e Comunicazione Università degli Studi di Milano-Bicocca

 Affective Computing  Learning the emotional state of a human being  Learning from:  Vocal signals  Facial expressions  Biometric signals  Multimodal sources  Applications  Games (personal robots)  Call centers  Automotive  JUMAS: Emotion Recognition in Judicial Domain for Semantic Retrieval Emotion Recognition

JUMAS Project Audio&Video Document Current Scenario Manual Transcription Manual Retrieval Manual Information Extraction Automatic Recording Manual Retrieval Manual Information Extraction Automatic Recording Audio Stream Analogical / Digital Acquisition Video Stream Future Scenario Audio&Video Document Digital Acquisition Automatic Audio Transcription Automatic Audio&Video Annotation Automatic Information Extraction Automatic Semantic Retrieval Audio&Video Stream Emotion Annotation for

Neutral Fear Emotion Recognition  Output: XML Searchable Tags Neutral

 Challenges:  What features are able to describe and discriminate different emotional states?  Which kind of environment influences emotional state recognition?  Which kind of learning models produces the optimal performance? Emotion Recognition

 Italian DB: 391 samples  Sentences from movies  5 emotional states:  Anger  Happiness  Sadness  Neutral  Fear Step 1 – Vocal Signature Acquisition Emotion Recognition from vocal signatures  German DB: 531 samples  Acted sentences: emotion on request  7 emotional states  Anger  Fear  Happiness  Sadness  Neutral  Disgust  Boredom

Preliminary Experimental Results Flat Models  Learning Models are biased by:  Language  Gender  Neutral emotional state

Multi-Layer Support Vector Machines  Hierarchical Classification:  Multi-Layer Support Vector Machines

Experimental Results  Consclusion  Multi-layer SVMs outperform traditional learning techniques  Fututre Work  Dynamic Techniques  Integration with Semantic Information Retrieval System  Cooperation with Deception Recognition Flat vs Multi-Layer Multi-Layer Flat German DB Italian DB