1. Neural Approach for Personalized Emotional Model in Human-Robot Interaction Mária Virčíková, Martin Pala, Peter Smolar, Peter Sincak Technical University of Kosice, Slovakia IEEE WCCI 2012 Brisbane, Australia

2. Slovakia (5million inhabitants) 2/33

3. 3 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI Structure of the Talk A. Motivation & System Overview B. What is unique about our approach ? C. Technology - research D. Results & Future 3/33

4. How can we make the human - machine (humanoid) interaction with easier and more enjoyable for people ? Center for Intel. Technology, Technical University of Kosice, Slovakia, EU 4 Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI A. Motivation & System Overview 4/33

5. Visual events Audible events Touch events … events External events Battery state Motors, processors state Internal events Events Memory Events Control Reasoning Control Cognition Emotions Output robot behavior A. Motivation & System Overview Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 5/33

6. 6 A. Motivation & System Overview Robotic ExpressionsEmotion recognition Machine User Part of Human-Robot Communication Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 6/33

7. 7 A. Motivation & System Overview Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI Can this Robot Evoke my Emotions ???? Influence my decision? Can it enhance Human- Robot or Robot-Human communication ? 7/33

8. “ The question is NOT whether intelligent machines CAN HAVE emotions, but whether machines CAN BE intelligent without any emotions.” M. Minsky “ The question is NOT whether intelligent machines CAN HAVE emotions, but whether machines CAN BE intelligent without any emotions.” M. Minsky 8 “The question is NOT whether intelligent machines CAN EVOKE HUMAN EMOTIONS, but whether machines CAN BE intelligent without this ability.” M. Virčíkova at al. Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI A. Motivation & System Overview 8/33

9. 9 B. What is UNIQUE about our approach? Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 9/33

10. 10 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI User 1 SYSTEM User 3 User 2 B. What is unique about our approach? different people express same feelings in different ways 10/33

11. 11 User 1 SYSTEM User 2 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI B. What is unique about our approach? 8 basic emotions whole emotional spectrum Evolving Robots behaviors during interaction with users Easily preprogrammed for any humanoid-type robot 11/33

12. 12 4. Emotional Technology: Experiment User 1 SYSTEM User 3 User 2 “Is the personalized form of the emotion E better understandable than the pre-programmed form of the emotion E?” Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 12/33

13. 13 C.Technology - Research Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 13/33

14. Center for Intel. Technology, Technical University of Kosice, Slovakia, EU 14 Emotion model by R. Plutchik Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI Despite different forms of expression there are prototype patterns, that can be identified. There is a small number of basic emotions. All other emotions are mixed or derivative states. All emotions vary in their degree of similarity to one another. Each emotion can exist in varying degrees of intensity or levels of arousal. C. Technology - Research Plutchik’s psychoevolutionary model 14/33

15. 15 Emotion model by R. Plutchik Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI Despite different forms of expression there are prototype patterns, that can be identified. There is a small number of basic emotions. All other emotions are mixed or derivative states. All emotions vary in their degree of similarity to one another. Each emotion can exist in varying degrees of intensity or levels of arousal. C. Technology - Research Plutchik’s psychoevolutionary model: computational implementation NN recognition Fuzzy Logic Overlapped clusters create many types of mixed emotions from basic ones Center for Intel. Technology, Technical University of Kosice, Slovakia, EU 15/33

16. Center for Intel. Technology, Technical University of Kosice, Slovakia, EU 16 Emotion model by R. Plutchik Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI C. Technology - Research 16/33

17. 17 C. Technology - Research User 1 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI membership-function ARTMAP NN o clustering technique with supervision o incremental learning o calculates the membership function (MF) of the sample from the feature space to the fuzzy class 17/33

18. 18 C. Technology - Research User 1 SYSTEM User 3 User 2 Input pattern = M-dimensional vector (I 1,,,, I M ) I i = position of one point of the body in time Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI : Inputs motion capture system 18/33

19. 19 C. Technology - Research: Learning Process Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI When receives a feature vector, deduces the best-matching category by evaluating a distance measure against all memory category nodes Each new pattern is compared with all of the nodes (categories) 1. Memory empty - no categories recorded 2. 1st pattern to arrive - the 1st category created - saved in the memory 3. 2nd pattern reaches the network, a comparison between the arrived pattern and the saved category(s) is established 4. NN compares the current input with a trained class representation (emotion) using training set 19/33

20. 20 C. Technology - Research User 3 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI If new user starts to interact with the system, his/her emotions are added. The network is not learning from beginning, only this specific cluster is added to the original topology 20/33

21. 21 C. Technology - Research Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI not a crisp decision on whether the input sample is or not a member of a particular class a vector of values that describes the degree of membership of this sample to all of the classes (emotions) MF ARTMAP Result of classification Fuzzy Plutchik’s model of emotional responses 21/33

22. 22 C. Technology - Research User 1 SYSTEM User 3 User 2 1.input layer - 3D coordinates of body postures in time 2.comparison layer - partial membership functions 3.recognition layer - clusters - emotional expression of each of the user. Adaptation to the concrete user 4.mapfield layer - classes from previous clusters – from different users 5.cognitive layer - similarities between classes Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 22/33

23. 23 C. Technology - Research User 1 User 3 User 2 the degree of membership of the emotion of the new user to the existing class is 1 only 1 cluster for both of the users for Joy the degree of membership of his/her expression to the existing expression of Joy is 0 2 clusters for expressing Joy will exist Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 23/33

24. 24 C. Technology - Research User 1 SYSTEM User 3 User 2 following statement can be deducted: “expression of Anger of User A is similar to the expression of Surprise of User B with the value of the coefficient of similarity 0.8.” coefficient of similarity is fuzzified: “ class A is very similar/similar/not similar to the class B.” Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 24/33

25. 25 User 1 SYSTEM X1X1 x2x2 X n-1 XnXn Joy Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI X1X1 x2x2 X n-1 XnXn Sadness The importance of personalization – possible feedback for psychology n- dimensional space, n= 48 x time expressions in n-dimensional space 25/33

26. 26 D. Results & Future Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 26/33

27. 27 D. Results: Basic emotions User 1 SYSTEM User 3 User 2 Personalized better Equal to understand Preprogrammed better Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 27/33

28. 28 D. Results: Mixed emotions SYSTEM User 3 User 2 Personalized better Equal to understand Preprogrammed better Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 28/33

29. 29 SYSTEM User 3 User 2 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI Children’s Dpt. of Oncology, Hospital of Kosice Several Schools in Kosice (age 6-12) Center for People with Autism Geriatrics (elder people) D. Results 29/33

30. 30 SYSTEM User 3 User 2 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 30/33

31. 31 Future possible consideration...... SYSTEM User 3 User 2 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI expand our database of emotional expressions … CLOUD ROBOTICS! 31/33

32. 32 Summary (the last slide … ) SYSTEM User 3 User 2 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI P ersonalization of the robot to different users R eplicable model from human body to humanoid body O verlapped clusters create many types of mixed emotions from 8 basic emotions. F or new user – NN not learning from beginning, only this specific cluster is added to the topology. I nformation about the data structure. The cognitive layer - similarity of clusters (how different people express their emotions.) Our modified MF ARTMAP neural network enables: 32/33

33. 33 Video System in action Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI 33/33

34. 34 Fuzzy relation for 2D input space User 1 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI axes x and y - dimensions of the input space axe z - degree of membership of the samples to the fuzzy set (fuzzy clusters )

35. 35 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI Fuzzy cluster Fuzzy cluster is considered as a fuzzy set in multidimensional feature space. j=1, 2, …, m - index of the dimension of input space U = X1 x X 2 x...x X - input space n - number of samples in the training set i = 1, 2,..., n - index of the current sample p - number of previously formed fuzzy clusters k = 1,2,…,p - index of the current fuzzy cluster A(xi) - value of membership of sample xi to k Ek and Fk - parameters of the fuzzy cluster k Xsk - center of the cluster k

36. 36 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI more complex shapes of real clusters... the fuzzy relation may: 1. not sufficiently cover the samples which belong to the fuzzy class 2. cover even samples which do not belong to the fuzzy cluster. … problem … reason fuzzy relation cannot adapt sufficiently to the training samples

37. 37 Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI Modified fuzzy relation multidimensional Gaussian probab. distribution based on the covar. matrix able to adapt to the shape of the cluster made up from the input patterns Σ covariance matrix of input samples in the dimensional feature space X vector of input sample values µ vector of mean values

38. 38 User 1 SYSTEM User 3 User 2 Overlapping of clusters Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI

39. 39

40. 40 similarity between classes Jeffries-Matusita(JM) distance (if JM = 0... Identical clusters) Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI Cognitive layer i, j - class identifiers of compared classes α - Bhattacharyya distance for multidimensional Gaussian distributions

41. 41 coefficient of similarity of each of the classes 0 - different classes 1 – identical classes Center for Intel. Technology, Technical University of Kosice, Slovakia, EU Maria Vircikova: Neural Approach for Personalized Emotional Model in HRI Ci,j number of clusters of the class i similar to our class j Ci, Cj number of clusters of class i and j