1. On Facial Recognition in Video VIVA Seminar, U. of Ottawa, August 28, 2003 Dr. Dmitry Gorodnichy Computational Video Group Institute for Information Technology National Research Council Canada www.cv.iit.nrc.ca/~dmitry www.perceptual-vision.com

2. 2. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Internet, tendencias & tecnología La nariz utilizada como mouse En el Instituto de Tecnología de la Información, en Canadá, se desarrolló un sistema llamado Nouse que permite manejar softwares con movimientos del rostro. El creador de este programa, Dmitry Gorodnichy, explicó vía e-mail a LA NACION LINE cómo funciona y cuáles son sus utilidades Si desea acceder a más información, contenidos relacionados, material audiovisual y opiniones de nuestros lectores ingrese en : http://www.lanacion.com.ar/03/05/21/dg_497588.asp http://www.lanacion.com.ar/03/05/21/dg_497588.asp Copyright S. A. LA NACION 2003. Todos los derechos reservados.

3. 3. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Talk overview On vision: –computer and biological ones On variety & hierarchy of facial recognition tasks in video: –FD, FT, FL, FR, and other F* On face detection and tracking (FD, FT) [AVBPA03] - demo –Good colour spaces and skin models –Power and deficiency of wavelets On face localization (FL) - demo –Power of convex-shape tracking [FGR02]. Nouse TM On motion / change detection: –Non-linear change detection –Second-order change detection [VIIP03]. Eye blink detection Eye-centered canonical face representation [AVBPA03] On memorizing and recognizing faces using associative memory - demo Perceptual Vision System: applications, licenses and further development

4. 4. Facial Recognition in Video (Dr. Dmitry Gorodnichy) What makes video special ? Constraints: - Real-time processing is required. - Low resolution: 160x120 images or mpeg-decoded. - Low-quality: week exposure, blurriness, cheap lenses. Importance: - Video is becoming ubiquitous. Cameras are everywhere. - For security, computer–human interaction, video-conferencing, entertainment … Essence: - It is inherently dynamic! - It has parallels with biological vision! NB: Living organisms are very successful in tracking, detection and recognition.

5. 5. Facial Recognition in Video (Dr. Dmitry Gorodnichy) The way nature does it Try to recognize this face

6. 6. Facial Recognition in Video (Dr. Dmitry Gorodnichy) The way nature does it (cntd) What about this one?

7. 7. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Localization first. Then recognition What did you do? – -First you detected face-looking regions. -Then, if they were too small or badly orientated, you did nothing. -Otherwise, you turned your face – right? -…to align your eyes with the eyes in the picture. -…since this was the coordinate system in which you stored the face. This is what biological vision does. – Localization (and tracking) of the object precedes its recognition – These tasks are performed by two different parts of visual cortex. So, why computer vision should not do the same?

8. 8. Facial Recognition in Video (Dr. Dmitry Gorodnichy) These mesmerizing eyes Did you notice that youve started examining this Slide by looking at the eyes (or circles) at left? - These pictured are sold commercially to capture infants attention. Now imagine that the eyes blinked … - For sure youll be looking at them! No wonder, animals and humans look at each others eyes. - This is apart from psychological reasons. Besides, there two of them, which creates a hypnotic effect (which is due to the fact that the saliency of a pixel just attended is inhibited to avoid attending it again soon.)

9. 9. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Eyes are the reference Eyes are the most salient features on a face. Besides, there two of them, which (besides creating a hypnotic affect ) makes the excellent reference frame out of them Finally, they also the best (and the only) stable landmarks on a face which can be used a reference. Intra-ocular distance (IOD) makes a very convenient unit of measurement!

10. 10. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Lessons from biological vision Images are of very low resolution except in the fixation point. Brain process the sequences of images rather than one image. - Bad quality of images is compensated by the abundance of images. The eyes look at points which attract visual attention. disparity Saliency is: in a) motion, b) colour, c) disparity, d) intensity. These channels are processed independently in brain (Think of a frog catching a fly or a bull running on a torero) Bottom-up (image driven) visual attention is very fast and precedes top- down (goal-driven) attention: 25ms vs 1sec.

11. 11. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Lessons from biological vision (cntd) Intensity means: frequencies, orientation, gradient. Animals & humans perceive colour non-linearly. Colour & motion are used for segmentation. Intensity is used for recognition.

12. 12. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Hierarchy of Face Processing Tasks Something yellow moves Face Segmentation Facial Event Recognition Face Memorization Face Detection Face Tracking (crude) Face Classification Face Localization (precise) Face Identification Its a face Its at (x,y,z, Lets follow it! Its face of a childS/he smiles, blinks Face unknown. Store it! Its Mila! I look and see… …

13. 13. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Applicability of 160x120 video According to face anthropometrics Tested with Perceptual User interfaces Face size ½ image¼ image 1/8 image 1/16 image In pixels80x8040x4020x2010x10 Between eyes-IOD 4020105 Eye size 201052 Nose size 105-- FS b FD b- FT b- FL b-- FER b- FC b- FM / FI -- – good b – barely applicable - – not good

14. 14. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Face Detection and Tracking

15. 15. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Face Detection and Tracking (lights off)

16. 16. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Face Detection and Tracking (lights on)

17. 17. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Face Localization Global facial cues: skin colour, head shape, head motion –good for FD & FT –not good for FL Local features have to be used for FL. Features are conventionally thought of as visually distinctive (ie with large DI(f) ). –Hence, the commonly used features are edge-based: corners of brows, eyes, lips, nostrils etc. –They however are not robust not always visible Solution: Convex-shape features

18. 18. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Nouse TM (Nose as Mouse) Technology Just think of your nose as a chalk or a joystick handle! "It is a convincing demonstration of the potential uses of cameras as natural interfaces." - The Industrial Physicist, Feb. 2003 Based on tracking the convex-shape nose feature. Head motion- and scale- invariant & sub-pixel precision Enables precise hand-free 2D control Allows aiming and drawing with the nose.

19. 19. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Nouse TM : range of tracking & precision No motion Yes motion Robustness to rotation Robustness to scale Test: The user rotates his head only! (the shoulders do not move)

20. 20. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Nouse TM : range and speed of tracking

21. 21. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Nouse TM for Stereotracking

22. 22. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Second-order change detection Detecting change in a change [Gorodnichy03] Non-linear change detection deals with changes due illumination changes [ Durucan02]

23. 23. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Eye Blink Detection Previously very difficult in moving heads With second-order change detection became possible Is currently used to enable people with brain injury face-to-face communication [AAATE03]

24. 24. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Building eye-centered face models Surprised to see how nicely eyes divide a face in equal blocks? But its true - Tested with 1500 faces from BioID face database and multiple experiments with perceptual user interfaces [Nouse02, BlinkDet03]. 2. IOD. IOD 24 2. IOD Anthropometrics of face:

25. 25. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Which part of the face is the most informative? What is the minimal size of a recognizable face? 1.By studying previous work: [CMU, MIT, UIUC, Fraunhofer, MERL, …] 2.By examining averaged faces: 3.By computing statistical relationship between face pixels in 1500 faces from the BioID Face Database: 9x9 12x12 16x1624x24 Size 24 x 24 is sufficient for face memorization & recognition and is optimal for low-quality video and for fast processing.

26. 26. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Eye-centered face models Canonical face model suitable for on-line Face Memorization and Recognition in video d 24 2.. IOD Procedure: after the eyes are located, the face is extracted from video and resized to the canonical 24x24 form, in which it is memorized or recognized. Canonical face model suitable for Face Recognition in documents [Identix02]

27. 27. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Canonical eye-centered face model

28. 28. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Recognition with Associative Memory When the eyes are detected, and a face is converted to a canonical representation, it is easy to memorize to recognize We use Pseudo-Inverse Associative Memory for on-line memorization and storing of faces in video. Converting 24x24 face to binary feature vector: A) V i =I i - I ave, B ) V i,j =sign(I i - I j ), C ) V i,j =Viola(i,j,k,l ), D ) V i,j =Haar(i,j,k,l )

29. 29. Facial Recognition in Video (Dr. Dmitry Gorodnichy) What is Memory? The one that stores data: and retrieves it:

30. 30. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Neural Network - a tool to do that A fully connected network of N neurons Y i, which evolves in time according to the update rule: until it reaches a stable state (attractor). Patterns can be stored as attractors -> Non-iterative learning - fast learning -> Process only active neurons - fast retrieval How to find the best weight matrix C ?

31. 31. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Pseudo-Inverse Learning Rule Obtained from stability condition CV=eV C = VV + or (Widrow-Hoffs rule is its approximation) Its dynamics can be studied theoretically. C=VV+, Cii=D*Cii, D=0.15 Complete retrieval from 8% noise of M=0.5N patterns from 2% noise of M=0.7N patterns

32. 32. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Pseudo-Inversse Associative Memory What's in name or what it is These neural networks are referred to as pseudo-inverse networks - for using Moore-Penrose pseudoinverse V + in computing the synapces projection networks - for synaptic (weight) matrix C=VV + being the projection Hopfield-like networks - for being binary and fully-connected in the stage of learning recurrent networks - for evolving in time, based on external input and internal memory attractor-based networks - for storing patterns as attractors (i.e. stable states of the network) dynamic systems - for allowing the dynamic systems theory to be applied associative memory - for being able to memorize, recall and forget patterns, just as much as humans do.

33. 33. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Pseudo-Inversse Associative Memory Eight reasons to like PINN memory It evolved from the network of formal neurons as defined by Hebb in 1949 and has many parallels with biological memory mechanisms. It provides an analytical (close-form) solution, the approximation of which is a well- known Widrow-Hoff delta learning rule. The performance of the network can be very nicely analytically examined and improved. It can update the memory on fly in real time, storing patterns with a non-iterative learning rule. It can store continuous stream of data. All this makes the network very suitable for on-line memorization and recognition, and also for preprocessing binary patterns, as noise removing filter. Finally, there's a free code which you can compile and try yourself!free code At PINN website: www.cv.iit.nrc.ca/~dmitry/pinn

34. 34. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Perceptual Vision System Goal: To detect, track and recognize face and facial movements of the user. x y, z PUI monitor binary event ON OFF recognition / memorization Unknown User!

35. 35. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Multi-channel video processing framework colour calibration face identification nose tracking (precise) blink detection face tracking (crude) face detection click event x y ( z ) face classification face memorization

36. 36. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Procedure The face is detected: using motion at far range (non-linear change detection), using colour at close range (non- linear colour mapping to perceptual uniform space) than tracked until convenient for recognition: using blink detection and nose tracking than localized and transformed to the canonical 24x24 representation, than recognized using the PINN associative memory trained pixel differences. After each blink, eyes and nose positions are retrieved. If they form an equilateral triangle (i.e.face is parallel to image plane), then face is extracted and recognized / memorized.

37. 37. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Experiments E.g. images retrieved from blink (at left) are recognized as the right image In experiments: With 63 faces from BioID database and 9 faces of our lab users (all of which are shown) stored, the system recognizes our users after a single (or several) blinks. It also updates the face memory on-fly.

38. 38. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Demos

39. 39. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Memorization

40. 40. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Memorization (cntd)

41. 41. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Recognition (from video)

42. 42. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Further developments Research -wise. Very many –Combining with audio –With panaramic cameras –With multiple cameras –Perceptual face gesture learner for handicaped –… License/application –wise. Very many –Talk to me…

43. 43. Facial Recognition in Video (Dr. Dmitry Gorodnichy) Intelligent Vision: Goal & Examples Goal: To Look AND See Case study: Face in Video