Presentation is loading. Please wait.

Presentation is loading. Please wait.

Cues to Emotion: Face, Body, Brain Bob Coyne (cs6998) 1.

Similar presentations


Presentation on theme: "Cues to Emotion: Face, Body, Brain Bob Coyne (cs6998) 1."— Presentation transcript:

1 Cues to Emotion: Face, Body, Brain Bob Coyne (cs6998) 1

2 Outline Emotion in voice and brain – The voice of emotion: an FMRI study of neural responses to angry and happy vocal expressions – (2006) Tom Johnstone et al Facial Expressions and emotion – – Facial Action Coding System (FACS) – Ekman & Friesen – Microexpressions - Ekman – Poker tells: The Body Language of Poker (1984) Mike Caro – 3D graphics facial expressions – FaceGen software 2

3 The voice of emotion: an FMRI study of neural responses to angry and happy vocal expressions – (2006) Tom Johnstone et al Range of issues: – Emotions in general vs specific emotions – General regions of brain (left/right) vs localized – Perception vs production – Prosody vs semantic content in speech – Facial expressions vs speech – Attention vs inattention – Type of experiment, controls, and measurement (FRMI, MEG, PET, EEG, other) – What function does vocal affect have and how to test 3

4 Importance of response to vocal affect From the earliest stages of development, infants respond to affect-laden vocal expressions from their mothers (Fernald, 1989; Fernald and Morikawa, 1993) – Why might this be? – Relevance to the 4 approaches (darwinian, jamesian…)? Still important/relevant after? – When? – Emoticons? Emotion in voice recognized as well as faces across cultures (Scherer and Wallbott, 1994) – True? 4

5 Early Prior work Production/comprehension experiments – Production vs perception. Makes a difference? – Production in Right hemisphere (Hughlings-Jackson ) How did they know? – Neurological evidence for comprehension (Tucker et al., 1977) – Reduced affective speech comprehension in right-hemisphere- damaged listeners (bowers 1987, Heilman et al., 1984; Peper and Irle, 1997; Ross, 1981) What’s missing? – Where processed? – Controlled experiments? – How else to test/identify region? 5

6 More recent Prior work Specific regions for facial emotion processing (e.g. amygdala for fear, insula for disgust) Little prior work on localized regions for emotional speech for specific emotions. Propositional content vs emotional prosody – Right prefrontal activation (PET - George et al. 1996) – EEG scalp electroencephalography Pihan et al Accented syllable duration and fundamental frequency Are those correlated with emotional speech? Magnetoencephalography (MEG) (Imaizumi et al. 1998) – Also found some left-hemisphere processing of propositional and prosodic content 6

7 Recent Prior work non-differentiated per emotion Selective activation Congruence effect with facial and vocal Experiments – Mitchell et al. (2003) found areas of posterior middle temporal gyrus (MTG) and superior temporal sulcus (STS) that activated more when attending to affective prosody as compared with semantic content of spoken words – Grandjean et al. (2005) and Sander et al. (2005) have reported FMRI data that revealed a region in STS that showed greater activation in response to angry speech as compared with neutral speech – Wildgruber et al. (2005) identified a right hemisphere network consisting of posterior STS, and dorsolateral and orbitobasal prefrontal cortex that showed selective activation during an emotion recognition task. Differential activations for the five emotions were not observed. – Ethofer et al. (2006) identified regions in the right posterior MTG and STS and bilateral inferior/middle frontal gyrus that activated more when individuals identified affective prosody than when identifying the emotional content of the spoken words. No distinction was made between responses to the different expressed emotions studied – Pourtois et al. (2005) demonstrated an area in left MTG that showed heightened activation when congruent vocal expressions and facial expressions of happiness or fear were simultaneously presented, as compared with when only one expressive modality was presented 7

8 The Brain Whole Brain Atlas Top 100 Brain Structures: 8

9 Loci of Sulci and Gyri 1.Central sulcus 2.Precentral gyrus 3.Postcentral gyrus 4.Precentral sulcus 5.Postcentral sulcus 6.Middle frontal gyrus 7.Superior frontal gyrus 8.Superior parietal lobule 9.Occipital gyri 10.Longitudinal fissure 9

10 Highly specialized brain function? 10

11 Hypotheses The question remains, therefore, of whether specific neural regions are more engaged in the processing of some emotions than others. Based on the hypothesis that affiliative social vocal signals are prevalent throughout our lives and serve a fundamental purpose in social binding we hypothesize that emotional expressions of happiness would preferentially engage parts of the temporal cortex and inferior frontal regions previously shown to be involved in the processing of affective prosody. A further question concerning the perception of vocal expressions of emotion is how directed attention towards or away from the expressed emotion affects the associated neural response. – How to test this? Activation for face versus voice – most studies on face What types of emotional speech detected? All? 11

12 Experimental approach FMRI to detect localized regions for processing happiness vs anger in speech Why those two emotions? Reasons – Control for similar acoustic signal. – Common/basic. Easier to characterize. – social utility hypothesis regarding happiness Which theory of emotion (jamesian etc) What was experimental setup? Some factors – 40 subjects – Facial stimuli consisted of 16 grayscale images of posed expressions of anger and 16 expressions of happiness, half of them females. – Vocal stimuli consisted of short phrases (dates and numbers) lasting on average 1 s, spoken with either angry or happy prosody. – Half the participants make their decision on the basis of the facial expression, while the other half on the basis of the vocal expression – FMRI scan during happy/angry speech With corresponding or opposite facial expression 12

13 Experiment Issues/problems? – Brain regions necessarily responding to emotion? – Quality of audio? Affects results? – FMRI measures blood flow. Relevant to neural? – Right handed subjects. Relevant? – Faces not correlated to the speaker…. – Why normalized audio amplitude? – Why no semantic emotional content? – Why short phrases? 13

14 FRMI Functional MRI or functional Magnetic Resonance Imaging (fMRI) is a type of specialized MRI scan. It measures the haemodynamic response related to neural activity in the brain or spinal cord of humans or other animals. It is one of the most recently developed forms of neuroimaging. Since the early 1990s, fMRI has come to dominate the brain mapping field due to its low invasiveness, lack of radiation exposure, and relatively wide availability. 14

15 Experiment - results Happy voices elicited significantly more activation than angry voices in right anterior and posterior middle temporal gyrus (MTG), left posterior MTG and right inferior frontal gyrus. Attention-dependent results: In contrast with brain regions showing a main effect of vocal emotion regardless of attentional focus, a network of brain regions including the left insula, left amygdala and hippocampus, and rostral ACC responded more to happy voices than to angry voices when attending to the voice, but showed either no difference or greater activation to angry voices than to happy voices when attending to the face. Do these results support the social binding hypothesis? 15

16 Results - RT and accuracy 16 Why longer RT for “voice” case? Why measure RT and accuracy at all?

17 Results How do these graphs support the stated results? 17

18 Facial Expressions and Emotion - Paul Ekman Found facial expressions of emotion are not culturally determined, but universal across human cultures and thus biological in origin. – How did he determine this? – Fully true? Examples? – How does this square with the four approaches: Darwinian? Jamesian? Social constructivist? Appraisal? Expressions he found to be universal included those indicating anger, disgust, fear, joy, sadness, and surprise. Findings on contempt are less clear, though there is at least some preliminary evidence that this emotion and its expression are universally recognized. – examples of non-universal emotions? of culture-dependent expression of same emotion? Of individual-dependent expression of same emotion? Facial Action Coding System (FACS) taxonomize all human facial expression. (Ekman and Friesen) – 48 action descriptors, which are a contraction or relaxation of one or more muscles. – – Microexpressions – detect lying – How does this phenomenon square with the four approaches? 18

19 FACS – action descriptors 1 Inner Brow Raiser -- Frontalis (pars medialis) 2 Outer Brow Raiser -- Frontalis (pars lateralis) 4 Brow Lowerer -- Depressor glabellae, Depressor supercilii, Corrugator supercilii 5 Upper Lid Raiser -- Levator palpebrae superioris 6 Cheek Raiser -- Orbicularis oculi (pars orbitalis) 7 Lid Tightener -- Orbicularis oculi (pars palpebralis) 9 Nose Wrinkler -- Levator labii superioris alaeque nasi 10 Upper Lip Raiser -- Levator labii superioris, caput infraorbitalis 11 Nasolabial Deepener -- Zygomaticus minor 12 Lip Corner Puller -- Zygomaticus major 13 Cheek Puffer -- Levator anguli oris (also known as Caninus) 14 Dimpler -- Buccinator 15 Lip Corner Depressor -- Depressor anguli oris (also known as Triangularis) 16 Lower Lip Depressor -- Depressor labii inferioris 17 Chin Raiser -- Mentalis 18 Lip Puckerer -- Incisivii labii superioris and Incisivii labii inferioris 19 Tongue Out 20 Lip stretcher -- Risorius w/ platysma 21 Neck Tightener -- Platysma 22 Lip Funneler -- Orbicularis oris 23 Lip Tightener -- Orbicularis oris 24 Lip Pressor -- Orbicularis oris 25 Lips part -- Depressor labii inferioris or relaxation of Mentalis, or Orbicularis oris 26 Jaw Drop -- Masseter, relaxed Temporalis and internal pterygoid 27 Mouth Stretch -- Pterygoids, Digastric 28 Lip Suck -- Orbicularis oris 29 Jaw Thrust 30 Jaw Sideways 31 Jaw Clencher -- Masseter 32 Lip Bite 33 Cheek Blow 34 Cheek Puff 35 Cheek Suck 36 Tongue Bulge 37 Lip Wipe 38 Nostril Dilator 39 Nostril Compressor 41 Lid Droop 42 Slit 43 Eyes Closed -- Relaxation of Levator palpebrae superioris; Orbicularis oculi (pars palpebralis) 44 Squint 45 Blink -- Relaxation of Levator palpebrae superioris; Orbicularis oculi (pars palpebralis) 46 Wink -- Relaxation of Levator palpebrae superioris; Orbicularis oculi (pars palpebralis) 19

20 Facial Action Coding System - FACS FACS can be used to distinguish two types of smiles: insincere and voluntary Pan American smile: contraction of zygomatic major alone sincere and involuntary Duchenne smile: contraction of zygomatic major and inferior part of orbicularis oculi. 20

21 Pictures of Facial Affect (resource) – Ekman and Friesen 110 pictures of posed facial emotions – 10 second display to subjects (male and female 50% mix) – 70% accurately identified Mark each example as one of: – Happy – Sad – Fear – Anger – Surprise – Disgust 21

22 89 Happy Sad Fear Anger Surprise Disgust 22

23 56 Happy Sad Fear Anger Surprise Disgust 23

24 79 Happy Sad Fear Anger Surprise Disgust 24

25 102 Happy Sad Fear Anger Surprise Disgust 25

26 54 Happy Sad Fear Anger Surprise Disgust 26

27 76 Happy Sad Fear Anger Surprise Disgust 27

28 52 Happy Sad Fear Anger Surprise Disgust 28

29 34 Happy Sad Fear Anger Surprise Disgust 29

30 94 Happy Sad Fear Anger Surprise Disgust 30

31 108 Happy Sad Fear Anger Surprise Disgust 31

32 Tally your results! Should match most of the time 32

33 89 Happy Sad Fear Anger Surprise Disgust 33

34 56 Happy Sad Fear Anger Surprise Disgust 34

35 79 Happy Sad Fear Anger Surprise Disgust 35

36 102 Happy Sad Fear Anger Surprise Disgust 36

37 54 Happy Sad Fear Anger Surprise Disgust 37

38 76 Happy Sad Fear Anger Surprise Disgust 38

39 52 Happy Sad Fear Anger Surprise Disgust 39

40 34 Happy Sad Fear Anger Surprise Disgust 40

41 94 Happy Sad Fear Anger Surprise Disgust 41

42 108 Happy Sad Fear Anger Surprise Disgust 42

43 Ekman - Microexpressions Play video Impact on Darwinian, jamesian, etc? What is deception w/r emotion? 43

44 Poker Tells – The Body Language of Poker (1984) Mike Caro Poker players exhibit “tells” that reveal valuable information about their hands or intentions to bet Some tells are unconscious and directly related to the internal emotion (excitement at a good hand, disappointment at a bad one) Some tells are the result of “acting the opposite” to try to deceive the opponents. Tells involve facial expressions, gestures, hand positions, body posture, and other behaviors 44

45 Poker Tells – instant reaction The Body Language of Poker (1984) Mike Caro 45

46 Poker Tells – acting the opposite 46

47 Poker Tells – guarding the hand 47

48 Expressions in virtual 3D faces – FaceGen Neutral Pose 48

49 Expressions in virtual 3D faces – FaceGen Anger 49

50 Expressions in virtual 3D faces – FaceGen Disgust+smile(open) 50

51 Expressions in virtual 3D faces – FaceGen blend other expression, phonemes, etc. 51

52 Next Week Hand in either: – A list of possible topics for your course project, with a paragraph on each, or (if you have already discussed the project with me and decided on one) – A one page project specification with: Project goal What you will do to achieve What resources you will need What prior work you can draw on 52


Download ppt "Cues to Emotion: Face, Body, Brain Bob Coyne (cs6998) 1."

Similar presentations


Ads by Google