1. Multiple Information Pathways in the Expression of Conversational Affect S YNCHRONY & S YMMETRY S YMPOSIUM APS, M AY, 2016 A LLISON G RAY, T IMOTHY R. B RICK, S TEVEN M. B OKER

2. Outline Emotion in context, information pathways in a conversation Current Study: Parse apart information pathways Rater variability (no ground truth) How important are different information pathways?

3. What is the role of emotion? Emotion is… –“flexible, contextually bound” (Campos et al., 1994) Expressions of emotion are… –“directed at a receiver” (Russell et al., 2003) –“conceptual instances belonging to a single emotion category are highly variable” (Barrett, 2009)

4. How do we interpret emotional information? Visual: –Facial Expression –Gestures Audial: –Semantics (words, meaning of word) –Prosody (pitch inflection, cadence) Background information of conversation partner (if any) Banziger, Grandjean, & Scherer, 2009; Russell et al., 2003

5. How do we interpret emotional information? Visual: –Facial Expression –Gestures Audial: –Semantics (words, meaning of word) –Prosody (pitch inflection, cadence) Background information of conversation partner (if any) Banziger, Grandjean, & Scherer, 2009; Russell et al., 2003

6. Conditions 5 second video clips –selected & cut randomly from emotionally prompted conversations Audial: Visual: SoundGarbledSilent MovieFace, Semantic, and Prosody Face and ProsodyFace only StillSemantics and Prosody Prosody only

7. Hypotheses Informational Hypothesis: more information present will yield higher degree of consensus Assumption: All channels are in agreement Conflicting Information Hypothesis: When different pathways convey conflicting information, fewer pathways will lead more consensus.

8. Design X 31 words X 30 thin slice clips Randomly ordered clips and words Presented with PsychoPy Participants, N = 51, taken from UVA participant pool Peirce, 2007

9. Movie and Sound

10. Movie and Silent

11. Still and Sound

12. Movie and Garbled

13. Still and Garbled

14. Expression Rating Words surprise fear hidden excitement calm contempt relaxed compassion embarrassment sadness candid happiness joy guilt contentment anxiety stress open disappointment anger shame disguised hate peaceful pleasure despair irritation interest hurt shy disgust

15. Methods and Results SD Rating clip = b 0 + b 1 Video condition clip + b 2 Audio condition clip + e clip Removing person means Removing clip means

16. Results Still stimuli yielded more variability than Movie stimuli ( t = 39.41, p < 0.0001) sd = 1.50sd = 1.69

17. Results Garbled stimuli yielded more variability than Sound stimuli ( t = 18.16, p < 0.0001) sd = 1.64sd = 1.54

18. Results Silent stimuli yielded more variability than Sound stimuli ( t = 11.26, p < 0.0001) sd movie = 1.52sd movie = 1.49

19. Happy Words Positive words follow the pattern in the first hypothesis. Removing pathways increases variability and decreases consensus Face, Semantics, Prosody Face, Prosody Face Semantics, Prosody sd = 1.39 1.43 1.55 1.76 1.58

20. Happiness Ratings of ’Happiness’ roughly reflect the same pattern. More information leads to more consensus. Face, Semantics, Prosody Face, Prosody Face Semantics, Prosody sd = 1.46 1.35 1.59 1.73 1.57

21. Openness The breakdown of rater consensus is more exaggerated with openness, especially across the movie/still break. Face, Semantics, Prosody Face, Prosody Face Semantics, Prosody sd = 1.42 1.52 1.53 1.92 1.58

22. Interested Ratings of ‘Interested’ support the second hypothesis—when information is conflicting, more information leads to more variability. Prosody alone provides the most definitive information for ’Interested.’ Face, Semantics, Prosody Face, Prosody Face Semantics, Prosody sd = 1.54 1.68 1.66 1.71 1.66

23. Limitations Garbling process works better for some clips rather than others Not a diverse set of raters Long study so potential differences in ratings across course of study

24. Summary Removing pathways increases variability and decreases consensus Groups of words follow different patterns in breakdown of consensus –movie vs. still or sound vs. garbled vs. silent Some words do not follow this pattern at all Contact info: Allison Gray axg5562@psu.edu Thank you! –Funding: National Institute on Aging (R21-AG041035), National Science Foundation (BCS–1030806) and Max Planck Institute for Human Development. –Human Dynamics Lab @ UVa –Undergraduate Research Assistants in HDL –Steven Boker –Tim Brick