1. + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida

2. Microexpression Spotting in Video Using Optical Strain Sridhar Godavarthy Examining Committee Dmitry B. Goldgof, Ph.D. – Major Professor Sudeep Sarkar, Ph.D. Rangachar Kasturi, Ph.D. July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida

3. + Minutes of the presentation Microexpressions - “micro” expressions. Goal: Detect “interesting” sequences containing μE. Approach: optical flow + strain thresholding. Result: True positive detection as high as 80%. Good performance on real time videos. Conclusion: Novel system. Scope for improvement. Need more datasets.

4. + Introduction

5. + Expressions Social emotion conveyance Non verbal Voluntary or involuntary 6 primary expressions 5 Anger Disgust Fear Happiness Sadness Surprise

6. + Microexpressions – What? Subtle movements of the human body Observable Insufficient to convey emotion Masking an expression 1/25 th to 1/5 th of a second Almost impossible to suppress 07/01/2010 6

7. + Why? Lie Detection Pain detection for autistic and anaesthetized patients Social signal processing( boredom/ concentration detection) Psychological counseling. 07/01/2010 7

8. + State of the art 07/01/2010 8 Microexpression Research Psychology Vision IntentFACS classification Optical Flow Gabor Filters ANNs Rule Based

9. + Objective Design a preprocessing system that Spots microexpressions. Handles small translational and rotational motion Improves performance of existing systems Greater weight to true positives. 07/01/2010 9

10. + Some Fundamentals Optic Flow : Vector representation of temporal changes Strain: Relative deformation of material (skin) Haar Classifier / Viola-Jones Face detector Cascade of weak classifiers OpenCV implementation Uses Haar rectangular features

11. + Brief overview of algorithm ( Main Idea) Skin deforms during an expression. Deformation peaks at peak of expression Duration of increased strain corresponds to microexpression 07/01/2010 11 ~22 frames ~5 frames Peak DetectionThresholding Frames  Strain Magnitude  Macro Expression Micro Expression

12. + Algorithm

13. + System Flow Split Frames Face Detection & Alignment Optical Flow Strain MapSplit into regions Strain patterns and Thresholding

14. + Face detection Viola-Jones face detector -OpenCV implementation

15. + Face Alignment: Rotation 07/01/2010 15

16. + System Flow Split Frames Face Detection & Alignment Optical Flow Strain MapSplit into regions Strain patterns and Thresholding

17. + Optical flow 07/01/2010 17 Black and Anandan Dense MJ Black’s Matlab imlementation of OF

18. + System Flow Split Frames Face Detection & Alignment Optical Flow Strain MapSplit into regions Strain patterns and Thresholding

19. + Facial strain 07/01/2010 19

20. + System Flow Split Frames Face Detection & Alignment Optical Flow Strain MapSplit into regions Strain patterns and Thresholding

21. + Region Splitting forehead(fh) right cheek (rc) left cheek (lc) right mouth(rm) left mouth(lm) right eye(re) below mouth(bm) AUs not covered Blink Close eyes Neck tightening Nostril flare Automated. M anual intervention if classifier fails

22. + Datasets & Results

23. + Datasets USF: IRBCanal9: EULAFound Videos: Fair use act USF (100) Canal9 (24) Found (4)

24. + Threshold Determination Sl. No. Threshold as percentage of peak strain % True positives% False positives 1 7531.80 2 50 0 3 3577.222.7 4 3054.536.4 5 2513.60 07/01/2010 24

25. + Thresholded Strain Maps – Sample 1 / 3 07/01/2010 25

26. + Thresholded Strain Maps – Sample 2 / 3 07/01/2010 26

27. + Thresholded Strain Maps – Sample 3 / 3 07/01/2010 27  False Positive ( Indicative only)

28. + Results: Microexpression Spotting Dataset Name Number of sequences Number of Microexpressions % True positive %False Positive USF enacted 129680.237.5 USF questioning 44750 Canal9 6245075 Found videos 34750 Total 2512874.2142.18 07/01/2010 28

29. + Microexpression Spotting 07/01/2010 29

30. + Negative Test Case – Rejects Expressions 07/01/2010 30

31. + Concluding Remarks

32. + Contributions and Conclusions Automated thresholding Automated alignment ( Partial ) Region wise detection Up to 80% true detection Microexpressions with expressions are detected.

33. + Constraints Constant illumination Neutral face Some expressions may be falsely detected Talking

34. + Future Work Dataset Collection Real time questioning videos Fully automated face alignment By matching optical flow vectors Automatic identification of neutral face Automatic portioning of faces Anthropomorphic landmark identification 07/01/2010 34

35. + Related Publications Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., Sarkar, S., "Towards macro- and micro-expression spotting in video using strain patterns," Workshop on Applications of Computer Vision, 2009 pp:1-6 07/01/2010 35

36. + Other Publications by author Candamo, J., Kasturi, R., Goldgof, D., Godavarthy, S., "Detecting Wires in Cluttered Urban Scenes Using a Gaussian Model, " to appear in Proceedings of International Conference on Pattern Recognition(ICPR 2010), Turkey, 2010 Godavarthy, S., Roomi, M. Md., “Adaptive Contrast Based Unsharp Masking,” in Proceedings of the National Workshop on Computer Vision, Graphics and Image Processing, Feb 2002 Godavarthy, S., Pandian, A., Roomi, M. Md., “Histogram Equalization by Measure of Enhancement,” in Proceedings of the National Workshop on Computer Vision, Graphics and Image Processing, Feb 2002 Godavarthy, S., Shankar, A., Roomi, M. Md., “Adaptive Watermarking-a FFT Approach”, Proceedings of International Conference on Advances in Telecommunication and Information Technology "Asia - Pacific Telecom 2000" (14th, 15th December 2000), Vellore 07/01/2010 36

37. + Dr. Ekman on A-Rod http://www.nytimes.com/2009/02/15/weekinreview/15marsh.html

38. + THANK YOU 38

39. + Index Presentation: Minutes System Flow Thresholding Sample Strain Maps Results Negative Test Case Conclusion and Future Work Additional Slides Evolutionary psychology Detailed Flow Chart Optical Flow Elasticity and Strain FACS OF Vs OS Dataset Details

40. + Additional Slides

41. + Study of everything we discussed until now The child of ONE man - Paul Ekman. Over thirty years of research One of the world’s leading experts on lying. About 2 dozen books and innumerable articles Developed FACS Scientific Advisor to “Lie to Me” Co creator of Microexpression Training Tool (METTx) Evolutionary Psychology

42. + Flow Chart 07/01/2010 42

43. + Add: OF Motion Estimation: Optical Flow Method  Reflects the changes in the image due to motion  Computation is based on the following assumptions:  observed brightness of any object point is constant over time  nearby points in the image plane move in a similar manner  Minimization problem: (brightness const.) (smoothness const.)  Robust estimation framework (Black and Anandan, 1996)  Recast the least squared formulations with a different error-norm function instead of quadratic  Coarse-to-fine strategy  Construct a pyramid of spatially filtered and sub-sampled images  Compute flow values at lowest resolution and project to next level in the pyramid 43 10/29/2009

44. + Def: Optical Flow is the apparent motion of brightness patterns in the image Ideally, same as the motion field Have to be careful: apparent motion can be caused by lighting changes without any actual motion Optical Flow Key assumptions Brightness constancy: projection of the same point looks the same in every frame Small motion: points do not move very far Spatial coherence: points move like their neighbors

45. Elasticity Different materials have different elasticity Elasticity can be modeled Known Calculate

46. + What is Facial Strain? Strain on soft tissue when expressions are made. Anatomical method Uses a pair of frames to measure deformation Facial Strain

47. + Finite Difference Method Compute spatial derivatives from discrete points. Forward Difference Method Central Difference Method Richardson extrapolation Strain Measurement

48. + Thresholding Threshold Strain Maps to segment out μE 07/01/2010 48

49. + The Facial Action Coding System (FACS) Coding of human expressions Observational and Anatomical 32 Action Units and 14 Action Descriptors Encode any possible [facial] expression. Also used for facial expression simulations

50. + NameMethodType Conte nt AdvantagesDisadvantages FAST OTS Early methodOnly negative emotions FACS OAC  All muscles  Allows for discovery - MAX OTS Faster performance Only pre defined configurations. EMG ObtAC Muscular activity invisible to naked eye Interference from nearby muscles EMFACS OTS Faster performance. Only certain emotion expressions. O - Observational Obt - Obtrusive A – Anatomical T – Theoretical S – Selective C – Comprehensive

51. + FACS examples 07/01/2010 51

52. + Why Optical Strain? 07/01/2010 52

53. + Datasets Dataset Name No. of Sequence s Approximat e Duration per sequence(s) Microexp ressions per sequenc e TotalResolution USF – feigned 12140896SD / HD USF – questioning 46514HD Canal9 dataset 6300-400424HD Found videos330-4014Very Low 07/01/2010 53

54. + Index Presentation: Minutes System Flow Thresholding Sample Strain Maps Results Negative Test Case Conclusion and Future Work Additional Slides Evolutionary psychology Detailed Flow Chart Optical Flow Elasticity and Strain FACS OF Vs OS Dataset Details