Problems Addressed Can we sense valence from facial electromyographic (EMG) activity? Intensity? Can we sense valence from mouse behavior (pressure in particular)? In general, what types of activity can we sense with facial EMG?
Relevant Prior Work Recent study by Larsen, Norris, Cacioppo Change in mean “cheek” EMG activity correlated positively with valence Change in mean “brow” EMG activity negatively correlated with valence (CS = “brow” muscle, ZM = “cheek” muscle)
Experimental Procedure 6 subjects General procedure: 1.Subject watches a short film clip 2.Subject answers questions about film content and experienced affective state Subject uses pressure sensitive mouse EMG output is recorded during all stages
Film Clip Sequence Each film clip was roughly 3 minutes 1.Golf Instruction (Neutral) 2.Clip from “The Champ” (Negative) Sadness in 94.2% (Gross & Levenson) 3.Golf Instruction (Neutral) 4.Clip from “Robin Williams Live” (Pos) Amusement in 84.1% (Gross & Levenson) Did not vary order
Self-reported Valence Consistent with Expectations Neutral Valence: 4 subjects indicated “neutral” affect, 2 subjects indicated low intensity positive affect Negative Valence: All 6 reported sadness Positive Valence: 5 of 6 reported amusement The sixth subject did not like Robin Williams
Example of Output Output normalized due to large scale differences
Discussion Highly sensitive to individual subject differences and electrode positioning Mean output consistent with Larsen, Norris, Cacioppo Do results extend to other positive/negative states? Unable to correlate EMG output with self-reported mood intensity EMG output may be useful as an activity recognition sensor Unable to correlate mouse pressure or velocity with valence Dynamic model of mouse behavior may produce better results?
Problem Addressed Does the facial expression ‘frowning’ represent the feeling of confusion? Can we recognize the feeling of confusion with the EMG outputs? Does the importance of understanding influence the feeling of confusion?
Experimental Procedure 6 subjects General procedure: 1.Subject listens to an audio clip 2.Subject answers questions about audio content and experienced affective state EMG output is recorded during all stages
Eliciting Confusion with Audio Recordings AB Mood control40 s classic music Controlspoken with standard American English Confused by accentspoken with accent Confused by meaning A brief paragraph with confusing content spoken with American English Two levels of the importance of understanding: Low level importance of understanding. High level importance of understanding. Audio clips to induce different levels of confusion:
Ratings and Measurements for feeling of Confusion Subjective rating 5 scale Self-report confusion rating Objective rating Test of understanding Measurement EMG responses
Result: EMG responses Brow Cheek Mean210.142654 Std214.445876 Difference in importance of understanding NO DF=1, P=0.28 NO DF=1, P=0.24
Result: Average EMG responses For brow More EMG activity when filling the web form listening to clips with accent EMG in ‘web-filling’ parts increase with level of confusion in high importance of understanding For cheek EMG activity increase with degree of confusion ->Subject started to laugh
Result: Self confusion rating and test score create the feeling of confusion consistently Self rating confusion Test score Mean3.812.38 Std1.511.87 Difference for 6 clips Yes DF=5, P < 0.0001 Yes DF=5, P < 0.0001 difference among subjects NO DF=5, P=0.42 NO DF=5, P=0.33 Average plot
Discussion Huge individual difference in EMG responses -> normalize the data More Brow activity when filling web forms -> It is more correlated to ‘the process of thinking about confusion thing’ More Brow activity when listening to clip with accent Cheek activity increase with level of confusion -> So confused that subjects were giving up Difference in importance of understanding
Conclusions Baseline difference in EMG for different muscles Individual difference in EMG response EMG response sensitive to electrode positioning Brow EMG is negatively correlated with valence and positively correlated with feeling of confusion Cheek EMG is positively correlated with valence Facial EMG may be useful for activity recognition