1. Mark Frank - RU HMS Symposium 2003 Deception, digitization, & counter-terror. Mark G. Frank School of Communication, Information, & Library Studies UCSF Rutgers PittCMU UCSD Deception, digitization, & counter-terror. Mark G. Frank Associate Professor School of Communication, Information, & Library Studies with special thanks to: Paul Ekman, UCSF; Dimitris Metaxas, Sinuk Kang, Amy Marie Keller, Anastacia Kurylo, Maggie Herbasz, Belida Uckun, Rutgers; Jeff Cohn, Pitt; Takeo Kanade, CMU; Javier Movellan & Marni Bartlett, UCSD. Also thanks to : Office of Naval Research and the National Science Foundation

2. Mark Frank - RU HMS Symposium 2003

3. Signals relevant to counter-terror. Changes in gait with loads as small as 1 kgChanges in gait with loads as small as 1 kg Anger prior to imminent attackAnger prior to imminent attack Fear/distress when lyingFear/distress when lying

4. Mark Frank - RU HMS Symposium 2003

5. How lies are betrayed. Lie How lies are betrayed. Lie Cognitive clues - Contradictory statements - Hesitations -Speech errors -Reduced illustrators -Contradictory emblems -Reduced detail -Etc. Emotional clues Lying about feelings Feelings about lying -Look for reliable signs of emotion -Duping delight -Guilt -Detection apprehension

6. Mark Frank - RU HMS Symposium 2003 Facial expressions in humans.

7. Mark Frank - RU HMS Symposium 2003 Some reliable facial muscles. Reliable angerReliable anger - Orbicularis oris Reliable disgustReliable disgust - levator Reliable fear/distressReliable fear/distress - medial frontalis Reliable enjoymentReliable enjoyment - Orbicularis oculi

8. Mark Frank - RU HMS Symposium 2003 Anger.

9. Anger. Eyebrows pulled down Upper lids pulled up Lower lids pulled up Red margins of lips rolled in Sometimes lips tight

10. Mark Frank - RU HMS Symposium 2003 Fear.

11. Fear. Eyebrows pulled up and together Upper eyelids pulled up Mouth stretched

12. Mark Frank - RU HMS Symposium 2003 The Facial Action Coding System (FACS) Ekman & Friesen, 1978 Action code: 1, 2, 4, 5, 7, 20, 26 1 Inner brow raise 2 Outer brow raise 4 Brow lower 5 Upper lid raise 7 Lid tighten 20 Lip stretch 26 Jaw drop 46 Action Units

13. Mark Frank - RU HMS Symposium 2003 Examples (con’t). Brow Motion –Upward motion AU 1+2 –Contraction of the frontalis, pars medialis and lateralis –Downward motion AU 4 or 9 –Contraction of the corrugator supercilii, depressor supercilii (AU 4) –Contraction of the levator labii superioris alaeque nasi (AU 9)

14. Mark Frank - RU HMS Symposium 2003 Subtle brow changes important.

15. Mark Frank - RU HMS Symposium 2003 Using FACS to identify emotion. Advantages:Advantages: - reliably scored - non-obtrusive - non-inferential, allows for discovery Disadvantage:Disadvantage: - laborious SolutionSolution - automatic computer vision techniques

16. Mark Frank - RU HMS Symposium 2003 Technical Challenges. PosePose Head motionHead motion Occlusion from glasses, facial hair, rotation, handsOcclusion from glasses, facial hair, rotation, hands TalkingTalking Video qualityVideo quality Frame rate (blinks)Frame rate (blinks)

17. Mark Frank - RU HMS Symposium 2003 Conclusions. There are reliable signals associated with deception and hostile intent.There are reliable signals associated with deception and hostile intent. We can detect these behaviors.We can detect these behaviors. We can represent them digitally.We can represent them digitally. Can this make us more secure?Can this make us more secure?