1. - Sridhar Godavarthy

2. Expressions Microexpressions FACS Evolutionary Psychology Proposed method Outline Video Face Detection, Alignment and Splitting Motion Field Optic Flow Optic Strain Datasets Current Work Future Work Results Initial WACV Optic Flow Thresholding

3. Expressions Microexpressions FACS Evolutionary Psychology

4. Primary means of social emotion conveyance Non verbal Conveys emotional state Voluntary or involuntary Minuscule differences in muscle movement

5. 6 primary expressions(Not all clearly distinguishable) Can you identify the ones in this picture?

6. What are microexpressions? Subtle movements of the human face Usually caused when attempting to mask a macro- expression Quick enough to be completed within the blink of an eye Last from 1/25 th to 1/5 th of a second Restricted to certain muscles(regions) of the face Almost impossible to fake

7. Examples: Raising an eyebrow Shrugging of shoulders Pout of lips Fast blinking of eye Non Examples: Talking Smiling Laughing Anger

8. For compartmentalization and categorization of human expressions 32 Action Units (with muscle involvement) 14 Action Descriptors ( ! -do-) Can be used to code any possible expression Widely used in pain recognition and facial expression simulations

9. 1000 page manual. Devised and written by one man. Requires extensive training. Some success in automating [ Bartlett et. al 1999].

11. Study of everything we discussed until now The child of ONE man - Paul Ekman. Over thirty years of research One of the world’s leading experts on lying. About 2 dozen books and innumerable articles Developed FACS Scientific Advisor to “Lie to Me” Co creator of Microexpression Training Tool (METTx)

12. Expressions Microexpressions FACS Evolutionary Psychology Proposed method Outline Video Face Detection, Alignment and Splitting Motion Field Optic Flow Optic Strain Datasets Current Work Future Work Results Initial WACV Optic Flow Thresholding

13. Face Detection & Translation Optical Strain Split into ROI Motion Field Estimation (Optical Flow) Eye Detection/Alignment Thresholding for period, strain Combine and count ROI

14. Face Detection & Translation Optical Strain Split into ROI Motion Field Estimation (Optical Flow) Eye Detection/Alignment Thresholding for period, strain Combine and count ROI

15. Video is a collection of individual images also known as frames In reality: spatial and temporal compression using properties of the scene. Any video can be decoded into a series of frames. 24/30 frames per second of video.

16. The science of encoding a video in a manner such that Minimum number of bits are used Motion compensated prediction can be performed from a previous frame.

17. Face Detection & Translation Optical Strain Split into ROI Motion Field Estimation (Optical Flow) Eye Detection/Alignment Thresholding for period, strain Combine and count ROI

20. Face Detection & Translation Optical Strain Split into ROI Motion Field Estimation (Optical Flow) Eye Detection/Alignment Thresholding for period, strain Combine and count ROI

21. 2D vector field of velocities of the image points induced by the relative motion.

22. Feature-based methods Extract visual features (corners, textured areas) and track them over multiple frames Sparse motion fields, but more robust tracking Suitable when image motion is large (10 s of pixels) Direct methods Directly recover image motion at each pixel from spatio- temporal image brightness variations Dense motion fields, but sensitive to appearance variations Suitable for video and when image motion is small

23. Def: Optical Flow is the apparent motion of brightness patterns in the image Ideally, same as the motion field Have to be careful: apparent motion can be caused by lighting changes without any actual motion

24. Brightness constancy Under most circumstance, the apparent brightness of moving objects remain constant Optical Flow Equation Relation of the apparent motion with the spatial and temporal derivatives of the image brightness Aperture problem Only the component of the motion field in the direction of the spatial image gradient can be determined The component in the direction perpendicular to the spatial gradient is not constrained by the optical flow equation Key assumptions Brightness constancy: projection of the same point looks the same in every frame Small motion: points do not move very far Spatial coherence: points move like their neighbors

25. Constant Flow Method Assumption: the motion field is well approximated by a constant vector within any small region of the image plane Solution: Least square of two variables (u,v) from NxN Equations – NxN (=5x5) planar patch Condition: A T A is NOT singular (null or parallel gradients) Weighted Least Square Method Assumption: the motion field is approximated by a constant vector within any small region, and the error made by the approximation increases with the distance from the center where optical flow is to be computed Solution: Weighted least square of two variables (u,v) from NxN Equations – NxN patch Affine Flow Method Assumption: the motion field is well approximated by a affine parametric model u T = Ap T +b (a plane patch with arbitrary orientation) Solution: Least square of 6 variables (A,b) from NxN Equations – NxN planar patch

26. Face Detection & Translation Optical Strain Split into ROI Motion Field Estimation (Optical Flow) Eye Detection/Alignment Thresholding for period, strain Combine and count ROI

27. Different materials have different elasticity Elasticity can be modeled Known Calculate

28. What is Facial Strain? Strain on soft tissue when expressions are made. Anatomical method Uses a pair of frames to measure deformation

29. Finite Element Method Forward modeling when Dirichlet condition is satisfied Good at handling irregular shapes Computationally expensive This method is an approximation to the solution Finite Difference Method Strain, a tensor, can be expressed derivatives of the displacement vector This can be approximated by a Finite Difference Method. Very efficient when carried out on a regular grid. This method is an approximation to the differential equation

30. Finite Difference Method Compute spatial derivatives from discrete points. Forward Difference Method Central Difference Method Richardson extrapolation

31. Optical FlowOptical Strain

32. Expressions Microexpressions FACS Evolutionary Psychology Proposed method Outline Video Face Detection, Alignment and Splitting Motion Field Optic Flow Optic Strain Datasets Current Work Future Work Results Initial WACV Optic Flow Thresholding

33. Name#videos~ #SequencesTotal Political10 7-980 USF(30)8-9~250 Found Videos4(+10)1-2~20 ASL1TBGt-

34. Eye detection/face alignment to accommodate head movement/rotation. Automatic thresholding Dataset collection Testing on “interesting” videos Trying a different Optical Flow( Black and Anandan) Run expression detection to remove Macro expressions first.

35. Micro expression recognition

36. Expressions Microexpressions FACS Evolutionary Psychology Proposed method Outline Video Face Detection, Alignment and Splitting Motion Field Optic Flow Optic Strain Datasets Current Work Future Work Results Initial WACV Optic Flow Thresholding

40. Images resized non-uniformly for presentation

41. ~22 frames ~5 frames

42. P. Ekman and W. Friesen. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, 1978 Malcolm Gladwell,” Blink: The Power of Thinking Without Thinking”, Back Bay Books (April 3, 2007) G. Donato, M. Bartlett, J. Hager, P. Ekman, and T. Sejnowski. Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10):974–989, 1999

43. Sridhar Godavarthy Dept. Of Computer Science and Engineering University of South Florida sgodavar@cse.usf.edu