1. Comparative Approaches to Emotion-Oriented Architectures
(WP 7: Emotion in Cognition and Action)
Last Plenary
:-/ :-)
Lola Cañamero (UH)
Plenary 3, 4-6 June 2007, Paris, France

2. WP7: The area
Scope: investigating computational models of emotional influences in cognition and action
Enhance behavior & interactions of emotion-oriented systems
Feedback to emotion theorists (synthetic approach, operationalize)
Exemplar:
Comparative approaches to emotion-oriented architectures: assumptions, integration challenges, and guidelines for future research => Output: edited collection
Divided in 4 elements:
Emotion in embodied cognition and action
Emotion in reflective cognition and action
Emotions in bridging the gap between embodied and reflective C&A
Emotions in social cognition and interaction
Groups: UH, OFAI, Bari, Paris8, DIST, GERG, HW, EMPL38, CNR, USC, ICCS, KCL, UM, INESC-IST, EFPL, Miralab, FT-RD, UOXF, USFD
humaine WP7: Emotion in Cognition and Action

3. WP7 Exemplar: the four elements1 2 3 4 4
humaine WP7: Emotion in Cognition and Action

4. Where are we? exemplar timeline
Stage 1: Critical analysis of state of the art and needs
Months : iterations to define problems and exemplar; uncover assumptions and needs
Stage 2: Integration challenges and key development goals
Month 19: workshop
Months 19-39: theoretical and practical work on integration challenges
Stage 3: Conclusions and guidelines for future research
Month 40: start developing “guidelines” for future research; chapter proposals and abstracts => moved to earlier date (D7e, Month 35)
Month 42 (JUNE 15!!!!!): drafts of chapters due
Months 43-48: chapters reviewed and revised
Month 48: book to publisher
humaine WP7: Emotion in Cognition and Action

5. Months 1-5 (D7b): state of the art, analysis (1)
Review of key achievements:
Emotion-based architectures (action selection, learning, memory)
Appraisal and cognitive systems
User modeling
ECAs and virtual environments
Key conceptual problems
Mechanisms underlying the involvement of emotions in cognition and action
Emotion elicitors (which factors activate those mechanisms?)
Emotions as cognitive modes
Relations among emotion, value systems, motivation and action
humaine WP7: Emotion in Cognition and Action

6. Months 1-5 (D7b): state of the art, analysis (2)
Key integration challenges
Problems arising from theories and models (diversity, poor understanding)
Diversity of computational frameworks & modeling approaches
Embodied AI, dynamical systems
Symbolic AI
Hybrid systems
Social simulation
Key development goals
“Grounding problem” of artificial emotions
Dissolving the “mind-body” problem
Untangling the “knot of cognition”: links emotion – intelligence
Measuring progress & the contributions of emotions to our systems
humaine WP7: Emotion in Cognition and Action

7. Months 6-11 (D7c): approach to exemplar
Best approach to fulfill horizontal goals in our area:
Comparative approaches to emotion-oriented architectures: assumptions, integration challenges, and guidelines for future research
Key ideas:
“Comparative approaches”
welcome the diversity of conceptual and computational models and frameworks
de-emphasize idea of a “unified” model for an emotion-based architecture (misleading goal at this point) -> complements “blueprint”
“Assumptions, integration challenges and guidelines for future research” stresses the nature of our principled integration effort in setting sound grounds
humaine WP7: Emotion in Cognition and Action

8. Months 6-11 (D7c): elements of exemplar
Focused working groups, integration at various levels:
WG1: Emotion in “lower-lever” cognition and action
UH, GERG, CNRS EPML 38, KCL
WG2: Emotion in “higher-level” cognition and action
Uni. Bari, France Telecom RD, GERG, CNR-ISTC, QUB, UA
WG3: Bridging gap between “lower-” and “higher-level” C & A
OFAI, HW, INESC-ID, IST, EPFL, USC
WG4: Emotion in Social Cognition and Interaction
OFAI, UH, Paris8, MIRALab, DIST, DFKI
Output:
Edited book
Reflection based on “proof-of-concept” designs and implementations
humaine WP7: Emotion in Cognition and Action

9. Support to Network activities
Presentations (posters) at Plenary 1
Presentations at WP3 workshop: “Contributions from robotic models of emotions” & several “hands-on demonstrations”
Presentations at WP4 workshop: links wp7-wp4, Markov-based analysis
Presentations & posters at WP6 workshop (ToM, affect-based imitation)
Cross-WPs links (meetings in Saarbruecken, Geneva, Santorini, Paris):
WP3, blueprint; conceptual clarification
WP4, constraints from cognition-action to signals/signs processing
WP6, integration internal models-expressive behaviors
WP8, mental states underlying external manifestations of persuasion
Co-organization (with WP3) of symposium on architectures of computational models at Plenary 2, May 2005
Working visits (UH & OFAI to GERG & MIRALab, etc)
Support actions (sessions) to WPs 3, 6, 8 planed at workshop (July 2005)
humaine WP7: Emotion in Cognition and Action

10. Dissemination & “external” activities
Symposium “Architectures for Modeling Emotion”, AAAI Spring Symposium, Stanford, March 2004
Symposium “Dimensions of Sociality”, Vienna, Nov. 2004
Symposium “Motivational and Emotional Roots of Cognition and Action”, UH, April 2005
Symposium “Mind-Minding Agents”, AISB’05
Co-organization (with WP3) symposium “Architecture of Computational Models” at ISRE in Bari, July 2005
Co-edition (with WP6) special issue Humanoid Robots
Various press reports, numerous scientific articles published or submitted (e.g. contribution to special issue NNets)
humaine WP7: Emotion in Cognition and Action

11. Input from March 2005 project review
No “recommendations” for WP7 but some “comments”:
C1: Establish more clearly synergies with other WPs
WP3: interplay theories / implementations meetings
WP6 (+WP4): complementarity towards human-like capabilities (special issue); emotion-attention interplay for social interaction
WP8: cognitive emotion models for dialog, communication, persuasion
WP10: working towards standards (joint handbook chapter); ethics
C2: Distinction between “lower-level” and “higher-level” misleading
Elements renamed to make focus more precise and avoid confusion
C3: Provide more details on plans to “bridge the gap”
Element 3 re-structured and made more concrete
C4: Robotic implementations shouldn’t be toy demonstrations of problems
Closer integration with emotion theory and formal analysis
More prominent use of ECAs
humaine WP7: Emotion in Cognition and Action

12. Core achievements in 2005 Definition of exemplar (D7d)
WP 7 workshop, London, July 2005 (D7a)
Other workshops to develop/support WP7 exemplar but not funded by HUMAINE
New partner (CNRS-EPML38, development), involvement of other new partners (USC, CNR)
Contribution to 2005 “high quality” dissemination deliverable and co-edition of 2006 one
Joint conceptual, design, implementation work
Publications
Other dissemination activities and esteem factors
humaine WP7: Emotion in Cognition and Action

13. Feedback from March 2006 review
Develop potential synergies with other projects
IST’06 Networking Session
Further improve links with the other workpackages and provide clear assessment means of this progress
Co-edition of special journal issue with WP6
Cross-currents symposium
ACE’06 symposium
Involvement of WP4 in follow-up proposal
humaine WP7: Emotion in Cognition and Action

14. Core achievements in 2006
Good progress in the 4 elements of the exemplar
Increased links with other WPs & projects
Sessions: Cross-currents, Summer School, IST’06, ACE’06
Co-edition of 2006 “high quality” dissemination deliverable with WP6 (IJHR special issue)
Publications
9 joint (4 journal, 5 conf / wksp), 21 single institution (8 journal, 13 conf / w)
Outline book submissions
Other dissemination activities and esteem factors
Edition (4), conf. organization (ACII, ACE, Ro-Man, EpiRob), inv. talks (10)
Follow-up project merging E1 + E4 (+ WP4-WP6): FEELIX GROWING
humaine WP7: Emotion in Cognition and Action

15. E1: Emotion in embodied cognition and action
Interactions between emotion & cognition-action as occurring through the “body”
UH, EPML38, Paris8, KCL , GERG
Subtasks:
E1.1 – Emotional modulation of perception-action in embodied agents: proximal causes, development, evolution
E1.2 – Analysis of embodied emotion-oriented architectures and behavior of robots: ethological + mathematical
E1.3 – Novelty detection and emotion-attention interactions (ECAs)
humaine WP7: Emotion in Cognition and Action

16. Modulation of Per-Ac loops
Motivations
Mfatigue
Mcold
Bfeed
Bwarmup
Behaviors
Physiology
sheat
sfood
dtemp
denergy
External Stimulus
Bumper
Bavoid
Bsearch
Actuator
Sensors E D
Fixed Nodes
Depot Node
Internal
Sensors
Possible Initial Network E D
Evolution
Proximal causation
Development
humaine WP7: Emotion in Cognition and Action

17. E2: Emotion in reflective cognition and action
Influence of emotions in cognition-action from the perspective of subjective perception and reasoning (introspection, linguistic accounts)
Bari, CNR, FT-RD, UM, USC
Subtasks:
Role of BDI&E models and relation to rationality and psych. theory:
Emotional conflict, cognitive dissonance
Emotion and anticipation
Validation of cognitive models of emotion activation by means of ‘sensitivity analysis’ & their extension to the ‘interpretation’ of emotional expressions displayed by the user
Application & comparison of models for emotion activation and recognition to dialogs
humaine WP7: Emotion in Cognition and Action

18. Emotional Mind humaine WP7: Emotion in Cognition and Action Dynamic
Emotion table
Selecting a context
and a personality
Selecting an event
Firing the event
Simulation
history
Mind at T0
Setting
a personality set
and a context set
Emotional Mind
humaine WP7: Emotion in Cognition and Action

19. Emotional Mind in action
humaine WP7: Emotion in Cognition and Action

20. E3: Emotions in bridging gap
Role of emotions in relating behavioral meaning and symbolic representations
OFAI, HW, USC, INESC, IST; GERG, KCL, UOXF, UM
Subtasks:
E3.1 – A scenario-based survey of bridging functions of emotions
E3.2 – Improving upon symbolic models of reflective cognition & action
E3.3 – Improving upon embodied models of cognition & action
E3.4 – Bridging the gap between micro- (individual-based) and macro- (social) views on social functions of emotion
humaine WP7: Emotion in Cognition and Action

21. Scenario-based evaluation and design
Goal: Understand state of modeling across different disciplines
Challenge: Substantial differences in concrete scenarios addressed (over 12). Different affect-related phenomena modeled at different granularities in settings of different complexity
Approach: to compare systems, the functional role of “emotion” (use of term) must be explicated
Focus on architectural building blocks:
Data structures, processes, interactions between processes
Fixed vs. dynamic/implicit paths of communication
Explicit differentiation of contexts of information processing (“modules”, “levels”, “stages”,…)
Bridging between such contexts
Derive best practices for the development of computational models of emotion
humaine WP7: Emotion in Cognition and Action

22. Improving upon symbolic reflective models
Suitable building blocks to model emotional processes?
Question foundations of symbolic architectures: symbolic “shortcuts” need to be motivated explicitly
Parallel embodied processes as basic behavioral components
Challenge: realize reflective and symbolic processes “on top”
Concurrent processes and resource management
Resources
Processes
Meta-Processes
humaine WP7: Emotion in Cognition and Action

23. Improving upon embodied models
Further extension of previous work on emergent affective and personality model that integrates perception, motivation, action selection, planning and memory
Autobiographical memory
Group level dynamics
Autobiographic Memory Model (Wan Ching Ho)
Emotional Parameters
Synthetic Group Dynamics
humaine WP7: Emotion in Cognition and Action

24. Bridging micro-macro gap
From inwards-oriented appraisal towards social-communicative behavior
Social emotions as result of supra-individual process of co-regulated reactions
Extension of appraisal theory model
Assessment of sequential evaluation check model
Improving sensing and rapidly reacting to human emotional signals
Integration of socially situated theory emphasizing centrality of social goals and contingent behavior
Planned workshop (October 2007, USC/ISI)
humaine WP7: Emotion in Cognition and Action

25. E4: emotions in social cognition and interaction
Roles of emotions in social cognition and interaction; emotions, cognition and action not modeled from the perspective of the individual but of the interaction itself.
OFAI, MIRALab, EPFL, ICCS-NTUA, DIST, UH, EPML-38, UBari, CNR, U. Sheffield
Subtasks:
E4.1 – Towards socially meaningful emotional agents: Closing the emotion recognition-generation-expression loop
E4.2 – Socially situated nature of emotions: Socially situated affective dialogue
humaine WP7: Emotion in Cognition and Action

26. Closing the emotion rec-gen-exp loop
Emotion recognition from
full body motion
ECA copying observed
expressive gestures
humaine WP7: Emotion in Cognition and Action

27. The Premio Paganini experiment
Measure of motoric activation: 4 special videocameras (50fps)
Audio: recording of both direct violin and in ambience
Physiological data (BioMuse):
ECG (Electrocardiogram)
EMG (Electromyogram)
Multimodal integration: Data Synchronization and Analysis (EyesWeb XMI)
Description of the set up
Stimulus material : Canon from the Musical Offering (J.S Bach)
humaine WP7: Emotion in Cognition and Action

28. Cross-currents symposium, June 2006
Dynamical systems as a framework to bridge gaps in emotion research?
L. Cañamero (UH, coord), R. te Boekhorst (UH), A. Flykt (Mid Sweden U), P. Gaussier (EPML38), N. Korsten (KCL)
Closing the emotion recognition-generation-expression loop
J. Gratch (USC, coord), A. Blanchard (UH), G. Castellano (DIST), A. Egges (Miralab), K. Karpouizis (ICCS), C. Peters (Paris8)
Beyond the blackbox vs process models alternative: reflective emotion models in comparison, with their mental ingredients, grain size, application perspectives and limits
F. de Rosis (Bari, coord), Peter Goldie (UM), Stacy Marsella (USC), Sabine Payr (OFAI), Isabella Poggi (CNR)
Avenues to bridge gaps between "embodied" and "reflective" systems
P. Petta (OFAI, coord), Nienke Korsten (KCL), Robert Marsh (UH), Sandy Louchart (HW), Fiorella de Rosis (Bari)
humaine WP7: Emotion in Cognition and Action

29. Networking Session @ IST06
“Embodied Emotion, Cognition and Action for Autonomous and Interactive Artifacts”
Aims:
provide framework to explore opportunities for interaction among projects with a common interest in embodied emotion and cognition
draft a longer-term “research agenda” for this area
Presentations FP6 projects: HUMAINE, euCognition, ICEA, ENACTIVE, MindRACES, S2S2, TAI-CHI, CALLAS
Challenges, needs and other projects identified
Over 100 participants
Follow-up Plenary07
humaine WP7: Emotion in Cognition and Action

30. Book: proposed submissions
4 chapters from E1:
(1) modulation Per-Ac loops; (2) neuromodulation; (3) dynamical systems analysis; (4) novelty and attention
3 chapters E2:
(5) emotional conflict; (6) empathic dialogue agent; (7) emotion and anticipation
4 chapters E3:
(8) AS architecture for virtual humans; (9) hybrid affective mind; (10) improving upon symbolic models; (11) improving upon embodied models
5 chapters E4:
(12) emotion sharing & understanding; (13) PerAc models of imitation; (14) analysis of movement dynamics for emotion recognition; (15) full-body motion and gesture analysis for recognition; (16) socially situated affective dialogue
External input to each section
humaine WP7: Emotion in Cognition and Action

31. Socio-emotional development
ICCS
FEEL, Interact, eXpress: a Global appRoach to develOpment With INterdisciplinary Grounding
FP6-IST , December 2006 – May 2010
humaine WP7: Emotion in Cognition and Action

32. Objectives
Identification of key evaluation scenarios (types of problems) in global socially situated development of autonomous agents
identify cross-disciplinary benchmarks (scenarios and methods) for a comparative evaluation
Investigation of the roles of emotion, interaction, expression and their interplays in bootstrapping & driving socially situated development
implementation and testing of robotic systems that improve existing work
Integration of: (a) the above “capabilities” in at least 2 different robotic prototypes, and (b) feedback across the disciplines involved
platform for grounded long-term multidisciplinary research (roadmap)
Identification of needs towards achieving standards in:
(a) design of scenarios and problem typologies,
(b) evaluation metrics,
(c) design of everyday robotic platforms and related technology.
humaine WP7: Emotion in Cognition and Action

33. WP4: “Feel” and development
Tasks
Cross-disciplin. training & critical analysis models of emotion in development
Sample key work in psychology to inspire / support robotic studies
Emotion elicitation in spontaneous vs induced imitation
Roles of + & - emotion in attachment and emotion regulation
Implementing & testing in robots selected key aspects:
Hedonic processes and their roles in motivation and emotion regulation in social interaction
Selected mechanisms for the detection / recognition of emotions in social interactions (modal & amodal)
Attachment processes & their roles in exploration, learning and adaptation to social environment
humaine WP7: Emotion in Cognition and Action

34. “Interact” and development
Tasks
Cross-disciplin. critical analysis models of interaction in development
Sample key work in psychology to inspire / support robotic studies
Emotions in social referencing
Emotion in joint attention (chimps w differential rearing conditions)
Implementing & testing in robots selected key aspects:
Joint attention, particularly the role of gaze direction
Task learning by observation/imitation; effect of (emotional) user feedback
Interaction and “emotional resonance”
humaine WP7: Emotion in Cognition and Action

35. “Express” and development
Tasks
Cross-disciplinary critical analysis models of expression in development
Sample key work in psychology to inspire / support robotic studies
Normal & impaired development of emotional resonance & recognition
Perception of emotion in human vs robot
Use of FACS for robots
Implementing & testing in robots selected key aspects:
Development of emotional expression related to social interaction
Use of expression as signalling for communication (no link to “internal” emotional state)
Use of expression as manifestation of an “internal” emotional state
humaine WP7: Emotion in Cognition and Action