Comparative Approaches to Emotion-Oriented Architectures (WP 7: Emotion in Cognition and Action) Last Plenary :-/ :-) Lola Cañamero (UH) Plenary 3, 4-6 June 2007, Paris, France

WP7: The area Scope: investigating computational models of emotional influences in cognition and action Enhance behavior & interactions of emotion-oriented systems Feedback to emotion theorists (synthetic approach, operationalize) Exemplar: Comparative approaches to emotion-oriented architectures: assumptions, integration challenges, and guidelines for future research => Output: edited collection Divided in 4 elements: Emotion in embodied cognition and action Emotion in reflective cognition and action Emotions in bridging the gap between embodied and reflective C&A Emotions in social cognition and interaction Groups: UH, OFAI, Bari, Paris8, DIST, GERG, HW, EMPL38, CNR, USC, ICCS, KCL, UM, INESC-IST, EFPL, Miralab, FT-RD, UOXF, USFD humaine WP7: Emotion in Cognition and Action

WP7 Exemplar: the four elements 1 2 3 4 4 humaine WP7: Emotion in Cognition and Action

Where are we? exemplar timeline Stage 1: Critical analysis of state of the art and needs Months 1 - 18: iterations to define problems and exemplar; uncover assumptions and needs Stage 2: Integration challenges and key development goals Month 19: workshop Months 19-39: theoretical and practical work on integration challenges Stage 3: Conclusions and guidelines for future research Month 40: start developing “guidelines” for future research; chapter proposals and abstracts => moved to earlier date (D7e, Month 35) Month 42 (JUNE 15!!!!!): drafts of chapters due Months 43-48: chapters reviewed and revised Month 48: book to publisher humaine WP7: Emotion in Cognition and Action

Months 1-5 (D7b): state of the art, analysis (1) Review of key achievements: Emotion-based architectures (action selection, learning, memory) Appraisal and cognitive systems User modeling ECAs and virtual environments Key conceptual problems Mechanisms underlying the involvement of emotions in cognition and action Emotion elicitors (which factors activate those mechanisms?) Emotions as cognitive modes Relations among emotion, value systems, motivation and action humaine WP7: Emotion in Cognition and Action

Months 1-5 (D7b): state of the art, analysis (2) Key integration challenges Problems arising from theories and models (diversity, poor understanding) Diversity of computational frameworks & modeling approaches Embodied AI, dynamical systems Symbolic AI Hybrid systems Social simulation Key development goals “Grounding problem” of artificial emotions Dissolving the “mind-body” problem Untangling the “knot of cognition”: links emotion – intelligence Measuring progress & the contributions of emotions to our systems humaine WP7: Emotion in Cognition and Action

Months 6-11 (D7c): approach to exemplar Best approach to fulfill horizontal goals in our area: Comparative approaches to emotion-oriented architectures: assumptions, integration challenges, and guidelines for future research Key ideas: “Comparative approaches” welcome the diversity of conceptual and computational models and frameworks de-emphasize idea of a “unified” model for an emotion-based architecture (misleading goal at this point) -> complements “blueprint” “Assumptions, integration challenges and guidelines for future research” stresses the nature of our principled integration effort in setting sound grounds humaine WP7: Emotion in Cognition and Action

Months 6-11 (D7c): elements of exemplar Focused working groups, integration at various levels: WG1: Emotion in “lower-lever” cognition and action UH, GERG, CNRS EPML 38, KCL WG2: Emotion in “higher-level” cognition and action Uni. Bari, France Telecom RD, GERG, CNR-ISTC, QUB, UA WG3: Bridging gap between “lower-” and “higher-level” C & A OFAI, HW, INESC-ID, IST, EPFL, USC WG4: Emotion in Social Cognition and Interaction OFAI, UH, Paris8, MIRALab, DIST, DFKI Output: Edited book Reflection based on “proof-of-concept” designs and implementations humaine WP7: Emotion in Cognition and Action

Support to Network activities Presentations (posters) at Plenary 1 Presentations at WP3 workshop: “Contributions from robotic models of emotions” & several “hands-on demonstrations” Presentations at WP4 workshop: links wp7-wp4, Markov-based analysis Presentations & posters at WP6 workshop (ToM, affect-based imitation) Cross-WPs links (meetings in Saarbruecken, Geneva, Santorini, Paris): WP3, blueprint; conceptual clarification WP4, constraints from cognition-action to signals/signs processing WP6, integration internal models-expressive behaviors WP8, mental states underlying external manifestations of persuasion Co-organization (with WP3) of symposium on architectures of computational models at Plenary 2, May 2005 Working visits (UH & OFAI to GERG & MIRALab, etc) Support actions (sessions) to WPs 3, 6, 8 planed at workshop (July 2005) humaine WP7: Emotion in Cognition and Action

Dissemination & “external” activities Symposium “Architectures for Modeling Emotion”, AAAI Spring Symposium, Stanford, March 2004 Symposium “Dimensions of Sociality”, Vienna, Nov. 2004 Symposium “Motivational and Emotional Roots of Cognition and Action”, AISB’05 @ UH, April 2005 Symposium “Mind-Minding Agents”, AISB’05 Co-organization (with WP3) symposium “Architecture of Computational Models” at ISRE in Bari, July 2005 Co-edition (with WP6) special issue Humanoid Robots Various press reports, numerous scientific articles published or submitted (e.g. contribution to special issue NNets) humaine WP7: Emotion in Cognition and Action

Input from March 2005 project review No “recommendations” for WP7 but some “comments”: C1: Establish more clearly synergies with other WPs WP3: interplay theories / implementations meetings WP6 (+WP4): complementarity towards human-like capabilities (special issue); emotion-attention interplay for social interaction WP8: cognitive emotion models for dialog, communication, persuasion WP10: working towards standards (joint handbook chapter); ethics C2: Distinction between “lower-level” and “higher-level” misleading Elements renamed to make focus more precise and avoid confusion C3: Provide more details on plans to “bridge the gap” Element 3 re-structured and made more concrete C4: Robotic implementations shouldn’t be toy demonstrations of problems Closer integration with emotion theory and formal analysis More prominent use of ECAs humaine WP7: Emotion in Cognition and Action

Core achievements in 2005 Definition of exemplar (D7d) WP 7 workshop, London, July 2005 (D7a) Other workshops to develop/support WP7 exemplar but not funded by HUMAINE New partner (CNRS-EPML38, development), involvement of other new partners (USC, CNR) Contribution to 2005 “high quality” dissemination deliverable and co-edition of 2006 one Joint conceptual, design, implementation work Publications Other dissemination activities and esteem factors humaine WP7: Emotion in Cognition and Action

Feedback from March 2006 review Develop potential synergies with other projects IST’06 Networking Session Further improve links with the other workpackages and provide clear assessment means of this progress Co-edition of special journal issue with WP6 Cross-currents symposium ACE’06 symposium Involvement of WP4 in follow-up proposal humaine WP7: Emotion in Cognition and Action

Core achievements in 2006 Good progress in the 4 elements of the exemplar Increased links with other WPs & projects Sessions: Cross-currents, Summer School, IST’06, ACE’06 Co-edition of 2006 “high quality” dissemination deliverable with WP6 (IJHR special issue) Publications 9 joint (4 journal, 5 conf / wksp), 21 single institution (8 journal, 13 conf / w) Outline book submissions Other dissemination activities and esteem factors Edition (4), conf. organization (ACII, ACE, Ro-Man, EpiRob), inv. talks (10) Follow-up project merging E1 + E4 (+ WP4-WP6): FEELIX GROWING humaine WP7: Emotion in Cognition and Action

E1: Emotion in embodied cognition and action Interactions between emotion & cognition-action as occurring through the “body” UH, EPML38, Paris8, KCL , GERG Subtasks: E1.1 – Emotional modulation of perception-action in embodied agents: proximal causes, development, evolution E1.2 – Analysis of embodied emotion-oriented architectures and behavior of robots: ethological + mathematical E1.3 – Novelty detection and emotion-attention interactions (ECAs) humaine WP7: Emotion in Cognition and Action

Modulation of Per-Ac loops Motivations Mfatigue Mcold Bfeed Bwarmup Behaviors Physiology sheat sfood dtemp denergy External Stimulus Bumper Bavoid Bsearch Actuator Sensors E D Fixed Nodes Depot Node Internal Sensors Possible Initial Network E D Evolution Proximal causation Development humaine WP7: Emotion in Cognition and Action

E2: Emotion in reflective cognition and action Influence of emotions in cognition-action from the perspective of subjective perception and reasoning (introspection, linguistic accounts) Bari, CNR, FT-RD, UM, USC Subtasks: Role of BDI&E models and relation to rationality and psych. theory: Emotional conflict, cognitive dissonance Emotion and anticipation Validation of cognitive models of emotion activation by means of ‘sensitivity analysis’ & their extension to the ‘interpretation’ of emotional expressions displayed by the user Application & comparison of models for emotion activation and recognition to dialogs humaine WP7: Emotion in Cognition and Action

Emotional Mind humaine WP7: Emotion in Cognition and Action Dynamic Emotion table Selecting a context and a personality Selecting an event Firing the event Simulation history Mind at T0 Setting a personality set and a context set Emotional Mind humaine WP7: Emotion in Cognition and Action

Emotional Mind in action humaine WP7: Emotion in Cognition and Action

E3: Emotions in bridging gap Role of emotions in relating behavioral meaning and symbolic representations OFAI, HW, USC, INESC, IST; GERG, KCL, UOXF, UM Subtasks: E3.1 – A scenario-based survey of bridging functions of emotions E3.2 – Improving upon symbolic models of reflective cognition & action E3.3 – Improving upon embodied models of cognition & action E3.4 – Bridging the gap between micro- (individual-based) and macro- (social) views on social functions of emotion humaine WP7: Emotion in Cognition and Action

Scenario-based evaluation and design Goal: Understand state of modeling across different disciplines Challenge: Substantial differences in concrete scenarios addressed (over 12). Different affect-related phenomena modeled at different granularities in settings of different complexity Approach: to compare systems, the functional role of “emotion” (use of term) must be explicated Focus on architectural building blocks: Data structures, processes, interactions between processes Fixed vs. dynamic/implicit paths of communication Explicit differentiation of contexts of information processing (“modules”, “levels”, “stages”,…) Bridging between such contexts Derive best practices for the development of computational models of emotion humaine WP7: Emotion in Cognition and Action

Improving upon symbolic reflective models Suitable building blocks to model emotional processes? Question foundations of symbolic architectures: symbolic “shortcuts” need to be motivated explicitly Parallel embodied processes as basic behavioral components Challenge: realize reflective and symbolic processes “on top” Concurrent processes and resource management Resources Processes Meta-Processes humaine WP7: Emotion in Cognition and Action

Improving upon embodied models Further extension of previous work on emergent affective and personality model that integrates perception, motivation, action selection, planning and memory Autobiographical memory Group level dynamics Autobiographic Memory Model (Wan Ching Ho) Emotional Parameters Synthetic Group Dynamics humaine WP7: Emotion in Cognition and Action

Bridging micro-macro gap From inwards-oriented appraisal towards social-communicative behavior Social emotions as result of supra-individual process of co-regulated reactions Extension of appraisal theory model Assessment of sequential evaluation check model Improving sensing and rapidly reacting to human emotional signals Integration of socially situated theory emphasizing centrality of social goals and contingent behavior Planned workshop (October 2007, USC/ISI) humaine WP7: Emotion in Cognition and Action

E4: emotions in social cognition and interaction Roles of emotions in social cognition and interaction; emotions, cognition and action not modeled from the perspective of the individual but of the interaction itself. OFAI, MIRALab, EPFL, ICCS-NTUA, DIST, UH, EPML-38, UBari, CNR, U. Sheffield Subtasks: E4.1 – Towards socially meaningful emotional agents: Closing the emotion recognition-generation-expression loop E4.2 – Socially situated nature of emotions: Socially situated affective dialogue humaine WP7: Emotion in Cognition and Action

Closing the emotion rec-gen-exp loop Emotion recognition from full body motion ECA copying observed expressive gestures humaine WP7: Emotion in Cognition and Action

The Premio Paganini experiment Measure of motoric activation: 4 special videocameras (50fps) Audio: recording of both direct violin and in ambience Physiological data (BioMuse): ECG (Electrocardiogram) EMG (Electromyogram) Multimodal integration: Data Synchronization and Analysis (EyesWeb XMI) Description of the set up Stimulus material : Canon from the Musical Offering (J.S Bach) humaine WP7: Emotion in Cognition and Action

Cross-currents symposium, June 2006 Dynamical systems as a framework to bridge gaps in emotion research? L. Cañamero (UH, coord), R. te Boekhorst (UH), A. Flykt (Mid Sweden U), P. Gaussier (EPML38), N. Korsten (KCL) Closing the emotion recognition-generation-expression loop J. Gratch (USC, coord), A. Blanchard (UH), G. Castellano (DIST), A. Egges (Miralab), K. Karpouizis (ICCS), C. Peters (Paris8) Beyond the blackbox vs process models alternative: reflective emotion models in comparison, with their mental ingredients, grain size, application perspectives and limits F. de Rosis (Bari, coord), Peter Goldie (UM), Stacy Marsella (USC), Sabine Payr (OFAI), Isabella Poggi (CNR) Avenues to bridge gaps between "embodied" and "reflective" systems P. Petta (OFAI, coord), Nienke Korsten (KCL), Robert Marsh (UH), Sandy Louchart (HW), Fiorella de Rosis (Bari) humaine WP7: Emotion in Cognition and Action

Networking Session @ IST06 “Embodied Emotion, Cognition and Action for Autonomous and Interactive Artifacts” Aims: provide framework to explore opportunities for interaction among projects with a common interest in embodied emotion and cognition draft a longer-term “research agenda” for this area Presentations FP6 projects: HUMAINE, euCognition, ICEA, ENACTIVE, MindRACES, S2S2, TAI-CHI, CALLAS Challenges, needs and other projects identified Over 100 participants Follow-up session @ Plenary07 humaine WP7: Emotion in Cognition and Action

Book: proposed submissions 4 chapters from E1: (1) modulation Per-Ac loops; (2) neuromodulation; (3) dynamical systems analysis; (4) novelty and attention 3 chapters E2: (5) emotional conflict; (6) empathic dialogue agent; (7) emotion and anticipation 4 chapters E3: (8) AS architecture for virtual humans; (9) hybrid affective mind; (10) improving upon symbolic models; (11) improving upon embodied models 5 chapters E4: (12) emotion sharing & understanding; (13) PerAc models of imitation; (14) analysis of movement dynamics for emotion recognition; (15) full-body motion and gesture analysis for recognition; (16) socially situated affective dialogue External input to each section humaine WP7: Emotion in Cognition and Action

Socio-emotional development ICCS FEEL, Interact, eXpress: a Global appRoach to develOpment With INterdisciplinary Grounding FP6-IST-045169, December 2006 – May 2010 humaine WP7: Emotion in Cognition and Action

Objectives Identification of key evaluation scenarios (types of problems) in global socially situated development of autonomous agents identify cross-disciplinary benchmarks (scenarios and methods) for a comparative evaluation Investigation of the roles of emotion, interaction, expression and their interplays in bootstrapping & driving socially situated development implementation and testing of robotic systems that improve existing work Integration of: (a) the above “capabilities” in at least 2 different robotic prototypes, and (b) feedback across the disciplines involved platform for grounded long-term multidisciplinary research (roadmap) Identification of needs towards achieving standards in: (a) design of scenarios and problem typologies, (b) evaluation metrics, (c) design of everyday robotic platforms and related technology. humaine WP7: Emotion in Cognition and Action

WP4: “Feel” and development Tasks Cross-disciplin. training & critical analysis models of emotion in development Sample key work in psychology to inspire / support robotic studies Emotion elicitation in spontaneous vs induced imitation Roles of + & - emotion in attachment and emotion regulation Implementing & testing in robots selected key aspects: Hedonic processes and their roles in motivation and emotion regulation in social interaction Selected mechanisms for the detection / recognition of emotions in social interactions (modal & amodal) Attachment processes & their roles in exploration, learning and adaptation to social environment humaine WP7: Emotion in Cognition and Action

“Interact” and development Tasks Cross-disciplin. critical analysis models of interaction in development Sample key work in psychology to inspire / support robotic studies Emotions in social referencing Emotion in joint attention (chimps w differential rearing conditions) Implementing & testing in robots selected key aspects: Joint attention, particularly the role of gaze direction Task learning by observation/imitation; effect of (emotional) user feedback Interaction and “emotional resonance” humaine WP7: Emotion in Cognition and Action

“Express” and development Tasks Cross-disciplinary critical analysis models of expression in development Sample key work in psychology to inspire / support robotic studies Normal & impaired development of emotional resonance & recognition Perception of emotion in human vs robot Use of FACS for robots Implementing & testing in robots selected key aspects: Development of emotional expression related to social interaction Use of expression as signalling for communication (no link to “internal” emotional state) Use of expression as manifestation of an “internal” emotional state humaine WP7: Emotion in Cognition and Action