1. ICT Networking Session « SEMANTIC ADAPTATION IN AFFECTIVE INTERACTION » STEFANOS KOLLIAS National Technical University of Athens Computer Science Division School of Electrical and Computer Engineering Lyon, France, November 25, 2008

2. Introduction Affective computing is of major importance in Human Machine Interaction; it involves perception, interpretation, cognition, expression. Affective computing is of major importance in Human Machine Interaction; it involves perception, interpretation, cognition, expression. Related International conferences include ACII2005, ACII2007, LREC2007, LREC2008, ACII2009. Related International conferences include ACII2005, ACII2007, LREC2007, LREC2008, ACII2009. EU IST projects and networks (Ermis, Interface, Safira, Humaine, Callas, Semaine, Feelix-Growing, Metabo) investigate(d) different issues of affective computing: theories and models of emotional processes, computational modelling, emotional databases, input signal analysis, emotion recognition, generation of embodied conversational agents. EU IST projects and networks (Ermis, Interface, Safira, Humaine, Callas, Semaine, Feelix-Growing, Metabo) investigate(d) different issues of affective computing: theories and models of emotional processes, computational modelling, emotional databases, input signal analysis, emotion recognition, generation of embodied conversational agents.

3. State-of-the-art Affective computing systems: model and analyse single or multi-modal affective cues; model and analyse single or multi-modal affective cues; extract and use statistical information and rules; extract and use statistical information and rules; create data sets and environments to perform user state detection; create data sets and environments to perform user state detection; include Embodied Conversational Agent (ECA) synthesis and interaction. include Embodied Conversational Agent (ECA) synthesis and interaction.

4. Requirements and achievements Common frameworks: Common frameworks: i) discrete, dimensional, component, appraisal emotional & affective models, i) discrete, dimensional, component, appraisal emotional & affective models, ii) cognitive & goal-based interaction computational models ii) cognitive & goal-based interaction computational models iii) Facial Action Coding System (FACS) and MPEG-4 models for facial, viseme and body motion analysis and synthesis, iii) Facial Action Coding System (FACS) and MPEG-4 models for facial, viseme and body motion analysis and synthesis, iv) features extracted from paralinguistic speech analysis, iv) features extracted from paralinguistic speech analysis, v) affect indicating words from linguistic speech recognition, v) affect indicating words from linguistic speech recognition, vi) physiological characteristics and cues, vi) physiological characteristics and cues, vii) multi-modal integration through feature or decision fusion. vii) multi-modal integration through feature or decision fusion. Need to address emerging requirements: humanlike interactions, less constrained environments, adaptive artificial systems Need to address emerging requirements: humanlike interactions, less constrained environments, adaptive artificial systems

5. Problems encountered Synchronisation, contradiction, co-existence lack, evolution. Synchronisation, contradiction, co-existence lack, evolution. Fusion of different signals and cues requires inferring on contradictory or ambiguous emotional cues Fusion of different signals and cues requires inferring on contradictory or ambiguous emotional cues Assuring that interaction is acceptable and appropriate in terms of user experiences. Assuring that interaction is acceptable and appropriate in terms of user experiences. Lack of common knowledge framework consistently representing taxonomies, rules and correlations in affective computing.  Recognition performance and fusion of multi-modal inputs is not satisfactory across varying and different environments,  Lack of interaction cycles where, in real time, the analysis loop feeds the synthetic one with info which is then reused further through user’s affective feedback,  Including the cognitive component and the context of interaction in the loop makes the situation much harder and systems almost impossible to benchmark.

6. Problems encountered (2) Test databases utilize sets of well-defined multi-modal (aural, visual, biosignal) data, captured in controlled or partly uncontrolled environments, depicting humans engaged in predetermined tasks. Test databases utilize sets of well-defined multi-modal (aural, visual, biosignal) data, captured in controlled or partly uncontrolled environments, depicting humans engaged in predetermined tasks. Moreover, knowledge obtained from one specific dataset may only be efficiently reused in the same environment. This refers to: Moreover, knowledge obtained from one specific dataset may only be efficiently reused in the same environment. This refers to:  low-level requirements, e.g., similar lighting conditions or absence of occlusion in visual signal analysis,  user-oriented restrictions, e.g., same subjects with same physical characteristics and similar expressivities,  similar cognitive or interaction tasks, e.g. reading or speaking to an artificial listener, classifying hand gestures in predefined categories.

7. SAFE Concept Generate a common adaptable semantic representation framework, i.e., a consistent environment to serve as knowledge substrate for affective interaction in real, uncertain, environments. Generate a common adaptable semantic representation framework, i.e., a consistent environment to serve as knowledge substrate for affective interaction in real, uncertain, environments. Basis: Basis:  state-of-the-art imperfect (uncertain, incomplete, vague, inconsistent) knowledge representation formalisms,  novel alignment, mapping and reasoning tools,  semantic adaptation and learning.

8. SAFE Concept (2) Generate a learning, evolving and adapting cognitive model Generate a learning, evolving and adapting cognitive model Start with basic knowledge about the nature of possible interactions, users and the environment, Start with basic knowledge about the nature of possible interactions, users and the environment, Include powerful sensing and reasoning mechanisms, along with the ability to infer from expert knowledge reinforced by accumulated experiences, Include powerful sensing and reasoning mechanisms, along with the ability to infer from expert knowledge reinforced by accumulated experiences, Result in a system gradually evolving its knowledge to incorporate its observations along with its own or the user’s evaluation. Result in a system gradually evolving its knowledge to incorporate its observations along with its own or the user’s evaluation.

9. SAFE Model

10. SAFE Technologies Formal Knowledge Formal Knowledge It stores the terminology, axioms, assertions, constrains that describe affective interaction. It stores the terminology, axioms, assertions, constrains that describe affective interaction. - HCI Ontologies module (formal ontological description representing the concepts and relationships of the field, providing formal definitions and axioms that hold in every HCI environment. - HCI Ontologies module (formal ontological description representing the concepts and relationships of the field, providing formal definitions and axioms that hold in every HCI environment. * abstract descriptions of users’ affective states * abstract descriptions of users’ affective states * low level descriptions of multimodal users’ info * low level descriptions of multimodal users’ info * context description (user expressivity, goals of * context description (user expressivity, goals of interaction, environmental characteristics). interaction, environmental characteristics).

11. SAFE Technologies (2) Real environments cause inconsistencies in the Formal Knowledge Real environments cause inconsistencies in the Formal Knowledge For example, the personality and expressivity of the specific user make some of the axioms and constraints of the HCI Ontology non-applicable or even wrong, according to logical entailments or user feedback. For example, the personality and expressivity of the specific user make some of the axioms and constraints of the HCI Ontology non-applicable or even wrong, according to logical entailments or user feedback. These inconsistencies make the formal use of knowledge that the SAFE Reasoner provides rather problematic. These inconsistencies make the formal use of knowledge that the SAFE Reasoner provides rather problematic. How to solve: How to solve:

12. SAFE Technologies (3) The reasoner detects the inconsistency The reasoner detects the inconsistency It follows a paraconsistent reasoning approach taking into account that the interaction with the user is assumed to be continuous and the user feedback will provide the system with additional important information needed in order to resolve the inconsistency. It follows a paraconsistent reasoning approach taking into account that the interaction with the user is assumed to be continuous and the user feedback will provide the system with additional important information needed in order to resolve the inconsistency. The Knowledge Adaptation component of SAFE resolves the inconsistency through a recursive learning process. The Knowledge Adaptation component of SAFE resolves the inconsistency through a recursive learning process.

13. SAFE Technologies (4) Knowledge adaptation Knowledge adaptation It determines the minimal set(s) of axioms that cause the inconsistency, It determines the minimal set(s) of axioms that cause the inconsistency, - Inconsistency Handling module - Inconsistency Handling module The minimal sets get represented in connectionist models and, with the aid of learning algorithms, are adapted and then re-inserted in the knowledge base. - The minimal sets get represented in connectionist models and, with the aid of learning algorithms, are adapted and then re-inserted in the knowledge base. - Some parts of the knowledge represent properties of objects that are rigid, while others are highly dynamic. Some parts of the knowledge represent properties of objects that are rigid, while others are highly dynamic. For the latter, the adaptation will be performed more drastically, with high learning rates, while for the former a more careful strategy will be followed. For the latter, the adaptation will be performed more drastically, with high learning rates, while for the former a more careful strategy will be followed.

14. SAFE Target Semantic adaptation in affective interaction. Semantic adaptation in affective interaction. Key words are learning and knowledge. Key words are learning and knowledge. Successfully combine these main components of cognition and machine intelligence. Successfully combine these main components of cognition and machine intelligence. Adaptation and evolution of ontological knowledge to effectively handle the context of interactions, i.e., specific user characteristics, goals & behaviours, or environmental changes. Adaptation and evolution of ontological knowledge to effectively handle the context of interactions, i.e., specific user characteristics, goals & behaviours, or environmental changes.

15. SAFE Target (2) The SAFE accomplishments will follow and closely relate to current state-of-the-art developments: The SAFE accomplishments will follow and closely relate to current state-of-the-art developments:  in W3C (RIF, OWL Development groups, EMOXG, MMI, Emotion Incubator)  in the Humaine Association (http://emotion- research.net/association), so that the SAFE system and technologies can be easily distributed and shared by the R&D community for affective interaction.

16. Advancing the State-of-the-Art Computational models of Affect Computational models of Affect  Taxonomies of affective user states, related to computational models of affect and emotions (Scherer’s proposal for distinguishing classes of affective states:  Taxonomies of affective user states, related to computational models of affect and emotions (Scherer’s proposal for distinguishing classes of affective states: * Emotions (e.g., angry, sad, joyful, fearful, ashamed, proud, elated, desperate) * Emotions (e.g., angry, sad, joyful, fearful, ashamed, proud, elated, desperate) * Moods (e.g., cheerful, gloomy, irritable, listless, depressed, buoyant) * Moods (e.g., cheerful, gloomy, irritable, listless, depressed, buoyant) * Interpersonal stances (e.g., distant, cold, warm, supportive, contemptuous) * Interpersonal stances (e.g., distant, cold, warm, supportive, contemptuous) * Preferences/ Attitudes (e.g., liking, loving, hating, valuing, desiring) * Preferences/ Attitudes (e.g., liking, loving, hating, valuing, desiring) * Affect dispositions (e.g., nervous, anxious, reckless, morose, hostile) * Affect dispositions (e.g., nervous, anxious, reckless, morose, hostile)

17. Advancing the State-of-the-Art (2) Interpretation of Affective User Behaviours Interpretation of Affective User Behaviours Context Modelling Context Modelling Imperfect Knowledge Representation and Reasoning Imperfect Knowledge Representation and Reasoning Connectionist model for ontology adaptation Connectionist model for ontology adaptation

18. Applications Robotics (Emotion aware, Knowledge Aggregation, Context Analysis): FEELIX- GROWING Robotics (Emotion aware, Knowledge Aggregation, Context Analysis): FEELIX- GROWING Human Computer Interaction in Emotional Environments (Knowledge and Emotion) : CALLAS Human Computer Interaction in Emotional Environments (Knowledge and Emotion) : CALLAS Analysis of status of children/students in e-learning Environments (Behaviour Analysis, Knowledge of User States): AGENT-DYSL Analysis of status of children/students in e-learning Environments (Behaviour Analysis, Knowledge of User States): AGENT-DYSL Analysis of status of car driver (monitoring safety): METABO Analysis of status of car driver (monitoring safety): METABO

19. Prospects Requirement for Novel Cognitive Systems interweaving Knowledge & Learning/Adaptation Technologies Requirement for Novel Cognitive Systems interweaving Knowledge & Learning/Adaptation Technologies Theoretical and Technological Advancing of the State-of-the-art Theoretical and Technological Advancing of the State-of-the-art Novel Applications: Emotion, Context and Knowledge Aware Robots, Tutors, Systems,… Novel Applications: Emotion, Context and Knowledge Aware Robots, Tutors, Systems,… New FP7 ICT Call for funding. New FP7 ICT Call for funding.

20. Thank you for your attention. Thank you for your attention. contact details: stefanos@cs.ntua.gr contact details: stefanos@cs.ntua.gr