Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho

Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Experimental Setup  Electrodes attached to: Corrugator supercilii (“brow”) Zygomaticus major (“cheek”)  Grounded on arm

Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Problems Addressed  Can we sense valence from facial electromyographic (EMG) activity? Intensity?  Can we sense valence from mouse behavior (pressure in particular)?  In general, what types of activity can we sense with facial EMG?

Relevant Prior Work  Recent study by Larsen, Norris, Cacioppo  Change in mean “cheek” EMG activity correlated positively with valence  Change in mean “brow” EMG activity negatively correlated with valence (CS = “brow” muscle, ZM = “cheek” muscle)

Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Experimental Procedure  6 subjects  General procedure: 1.Subject watches a short film clip 2.Subject answers questions about film content and experienced affective state  Subject uses pressure sensitive mouse  EMG output is recorded during all stages

Film Clip Sequence  Each film clip was roughly 3 minutes 1.Golf Instruction (Neutral) 2.Clip from “The Champ” (Negative)  Sadness in 94.2% (Gross & Levenson) 3.Golf Instruction (Neutral) 4.Clip from “Robin Williams Live” (Pos)  Amusement in 84.1% (Gross & Levenson)  Did not vary order

Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Self-reported Valence Consistent with Expectations  Neutral Valence: 4 subjects indicated “neutral” affect, 2 subjects indicated low intensity positive affect  Negative Valence: All 6 reported sadness  Positive Valence: 5 of 6 reported amusement The sixth subject did not like Robin Williams

Example of Output  Output normalized due to large scale differences

Mean Output Consistent with Prior Work

Individual Results Varied

Other Observations  For some subjects, brow activity was a good indicator of web form activity

Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Discussion  Highly sensitive to individual subject differences and electrode positioning  Mean output consistent with Larsen, Norris, Cacioppo  Do results extend to other positive/negative states?  Unable to correlate EMG output with self-reported mood intensity  EMG output may be useful as an activity recognition sensor  Unable to correlate mouse pressure or velocity with valence  Dynamic model of mouse behavior may produce better results?

Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Outline I.Introduction and Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Problem Addressed  Does the facial expression ‘frowning’ represent the feeling of confusion?  Can we recognize the feeling of confusion with the EMG outputs?  Does the importance of understanding influence the feeling of confusion?

Outline I.Introduction and Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Experimental Procedure  6 subjects  General procedure: 1.Subject listens to an audio clip 2.Subject answers questions about audio content and experienced affective state  EMG output is recorded during all stages

Eliciting Confusion with Audio Recordings AB Mood control40 s classic music Controlspoken with standard American English Confused by accentspoken with accent Confused by meaning A brief paragraph with confusing content spoken with American English Two levels of the importance of understanding:  Low level importance of understanding.  High level importance of understanding. Audio clips to induce different levels of confusion:

Ratings and Measurements for feeling of Confusion  Subjective rating 5 scale Self-report confusion rating  Objective rating Test of understanding  Measurement EMG responses

Outline I.Introduction and Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Result: EMG responses Brow Cheek Mean Std Difference in importance of understanding NO DF=1, P=0.28 NO DF=1, P=0.24

Result: Average EMG responses For brow  More EMG activity when  filling the web form  listening to clips with accent  EMG in ‘web-filling’ parts increase with level of confusion in high importance of understanding For cheek  EMG activity increase with degree of confusion ->Subject started to laugh

Result: Self confusion rating and test score create the feeling of confusion consistently Self rating confusion Test score Mean Std Difference for 6 clips Yes DF=5, P < Yes DF=5, P < difference among subjects NO DF=5, P=0.42 NO DF=5, P=0.33 Average plot

Outline I.Introduction and Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Discussion  Huge individual difference in EMG responses -> normalize the data  More Brow activity when filling web forms -> It is more correlated to ‘the process of thinking about confusion thing’  More Brow activity when listening to clip with accent  Cheek activity increase with level of confusion -> So confused that subjects were giving up  Difference in importance of understanding

Outline I.Introduction and Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

Conclusions  Baseline difference in EMG for different muscles  Individual difference in EMG response  EMG response sensitive to electrode positioning  Brow EMG is negatively correlated with valence and positively correlated with feeling of confusion  Cheek EMG is positively correlated with valence  Facial EMG may be useful for activity recognition