1. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Computer vision approaches to identifying people and possible malfeasant behavior. Dimitris N. Metaxas Mark G. Frank UCSF Rutgers PittCMU UCSD Computer vision approaches to identifying people and possible malfeasant behavior. Dimitris N. Metaxas Mark G. Frank Director, Computational Biomedicine School of Communication, Imaging & Modeling Center Information & Library Studies with special thanks to: Paul Ekman, UCSF; Sinuk Kang, Amy Marie Keller, Anastacia Kurylo, Maggie Herbasz, Belida Uckun, Rutgers; Jeff Cohn, Pitt; Takeo Kanade, CMU; Javier Movellan & Marni Bartlett, UCSD. David Dinges, UPENN Also thanks to : Office of Naval Research, National Science Foundation (ITR program), AFOSR, DARPA

2. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003

3. Signals relevant to counter-terror. Identification of bad guysIdentification of bad guys Changes in gait with loads as small as 1 kgChanges in gait with loads as small as 1 kg Anger prior to imminent attackAnger prior to imminent attack Fear/distress when lyingFear/distress when lying

4. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Take a closer look… Kim Philby, 1960’s (Frank & Ekman, 2003)

5. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 How lies are betrayed. Lie How lies are betrayed. Lie Cognitive clues - Contradictory statements - Hesitations -Speech errors -Reduced illustrators -Contradictory emblems -Reduced detail -Etc. Emotional clues Lying about feelings Feelings about lying -Look for reliable signs of emotion -Duping delight -Guilt -Detection apprehension

6. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 The Facial Action Coding System (FACS) Ekman & Friesen, 1978 Action code: 1, 2, 4, 5, 7, 20, 26 1 Inner brow raise 2 Outer brow raise 4 Brow lower 5 Upper lid raise 7 Lid tighten 20 Lip stretch 26 Jaw drop 46 Action Units

7. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Challenges facing behavioral science: Advantages:Advantages: - reliably identify people & behaviors - non-obtrusive - non-inferential, allows for discovery Disadvantage:Disadvantage: - laborious - mistaken identity via cognitive capacity, disguise, etc SolutionSolution - automatic computer vision techniques

8. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Gait recognition Identify people from the way they walk Important for surveillance and intrusion detection What are good features for identifying a person? –i.e., what features are person-specific?

9. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Background Sagittal plane - divides body into left and right halves Limb segment - a vector between two sites on a particular limb

10. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Elevation Angles

11. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 The trajectories of the sagittal elevation angles are invariant across different subjects. As a consequence, person-independent gait recognition will require less training data. (Borghese et al., 1996)

12. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 The cyclogram Elevation angles trace curve in a 4D space Curve is called “cyclogram” Cyclogram lies in a 2D plane –Well, almost Hypothesis: deviation of cyclogram from plane is person-specific

13. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Cyclogram example Curve is cyclogram projected into best-fit plane Green points are real points of cyclogram Red lines trace the deviation of points from plane (exaggerated scale)

14. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Cyclogram sequence Deviation from cyclogram plane can be represented as a sequence e.g., CCCGTTTTATATTTTTAAAAGCCGGTAAATTAGGGG Compare sequences between people via longest common subsequence (LCS) matching –Well-known dynamic programming algorithm, used in computational biology

15. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Examples of People Detection

16. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Examples of Gait Analysis

17. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Face: Tracking: Stress Recognition Identify which Facial Features (space and time) are important to recognize stress Assymetries and Movements around the mouth and eyebrows

18. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Slope = Asymmetry A horizontal line would indicate no asymmetry. This facial expression, however, is generally slanted upward and to the left.

19. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Plots of high and low stress Expression of high stress in form of asymmetrical facial expression time asymmetry

20. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 In contrast, low stress time asymmetry

21. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Some more high stress from different subjects

22. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Face: Tracking: Stress Recognition

23. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Face: Tracking: Stress Recognition

24. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003

25. Subtle brow changes important.

26. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Technical Challenges. PosePose Head motionHead motion Occlusion from glasses, facial hair, rotation, handsOcclusion from glasses, facial hair, rotation, hands TalkingTalking Video qualityVideo quality Frame rate (blinks)Frame rate (blinks)

27. M. Frank & D. Metaxas - Rutgers HMS Symposium 2003 Conclusions. There are reliable means to identify people as well as behaviors associated with deception and hostile intent.There are reliable means to identify people as well as behaviors associated with deception and hostile intent. We can detect these behaviors.We can detect these behaviors. We can represent them digitally.We can represent them digitally. Can this make us more secure?Can this make us more secure?