1. UNIVERSITY OF MURCIA (SPAIN) ARTIFICIAL PERCEPTION AND PATTERN RECOGNITION GROUP REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos Dept. de Informática y Sistemas University of Murcia - SPAIN

2. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 2 Introduction Main objective: develop a new technique to track human faces and facial features: –Working in real-time: fast processing –Under realistic conditions: robust to facial expressions, lighting conditions, 3D head pose and movements –With high location accuracy: facial features location (eyes, nose, mouth)

3. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 3 Introduction Index of the presentation: –Face integral projections –Integral projection models –Alignment of projections –The tracking process –Experimental results –Conclusions

4. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 4 Face integral projections Definition. Let i(x,y) be an image, and R(i) a region in it – Vertical integral projection P VR : {y min,..., y max }  R P VR (y) = i(x,y);  (x,y)  R(i) – Horizontal integral projection P HR : {x min,..., x max }  R P HR (x) = i(x,y);  (x,y)  R(i) FACE P VFACE (y) y EYES P HEYES (x) x

5. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 5 Face integral projections Dimensionality reduction: 3D world  2D images  1D integral projections Advantages: –Fast to compute and to process Disadvantages: –Loss of information. Is it relevant for the problem? What happens when applied to human faces?

6. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 6 Face integral projections Different individuals

7. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 7 Face integral projections Different facial expressions

8. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 8 Face integral projections Different segmented regions

9. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 9 Integral project. models Face integral projec. is an interesting and robust feature for tracking It has been applied using heuristic analysis: max-min search, fuzzy logic, thresholding projections Proposal: define and work with adaptable projection models How to model a variety of projection patterns?

10. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 10 Integral project. models A projection model is a pair: M : {m min,..., m max }  R (Mean) V : {m min,..., m max }  R(Variance)

11. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 11 Integral project. models Advantages of working with explicit projection models: –The model is learnt from examples. In tracking, it is adapted to tracked faces –We can define a signal to model distance:

12. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 12 Integral project. models Advantages of working with explicit projection models: –The model can be reprojected Reprojection (by outer product) using 1 vertical IP and 2 horizontal IP

13. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 13 Alignment of projections Corresponding facial features should be projected on the same locations Before alignment After alignment

14. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 14 Alignment of projections Alignment with respect to a model Problem formulation: –Let S: {s min,..., s max }  R be a signal –Let M,V: {m min,..., m max }  R be a model –Let S’ be a family of scale and translations alignments of S: –Find parameters ( a,b,c,d,e ) which minimize:

15. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 15 The tracking process Tracking is based on the alignment of integral projections Main steps: 1. Prediction and segmentation 2. Vertical alignment 3. Horizontal alignment 4. Orientation estimation

16. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 16 The tracking process Features to track Input to the tracker –Bounding ellipse –Facial features: eyes and mouth –Face model –State of tracking in frame t-1 –Frame t

17. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 17 The tracking process 1. Prediction and segmentation Null predictor: locations in frame t-1 are used to extract the face in frame t Predicted location Wrapped Segmented

18. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 18 The tracking process 2. Vertical alignment Using the vertical projection of the face, and the model, align the face vertically Segmented Model P VFACE (y) Align P VFACE to M VFACE y Align using the obtained parameters (a,b,c,d,e)

19. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 19 The tracking process 3. Horizontal alignment Using the horizontal projection of the eyes’ region, align the face horizontally Segmented after step 2 Model P HEYES (x) Align P HEYES to M HEYES y Align using the obtained parameters (a,b,c,d,e) x

20. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 20 The tracking process 4. Orientation estimation Using vertical projections of each eye, estimate the orientation of the face Segmented after step 3 Model P VEYE1, P VEYE2 Align P VEYEi to M VEYEi y

21. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 21 The tracking process Global structure of the tracker 1. Prediction and segmentation 2. Vertical alignment 3. Horizontal alignment 4. Orientation est.

22. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 22 Experimental results Experiments: –Location accuracy –Execution time per frame –Robustness to facial expressions, 3D pose, lighting conditions Different sources: TV, video- conference camera and DVD Compared with CamShift algorithm

23. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 23 Experimental results FILE NAME: tl5-02.aviSOURCE: TV FORMAT: 640x480 (25 fps)LENGTH: 280 frames MODEL SIZE (pixels): 97x123 AVG/MAX ERROR (mm): 3.41 / 12.9 TIME/FRAME (ms): 4.02

24. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 24 Experimental results FILE NAME: a3-05.aviSOURCE: TV FORMAT: 640x480 (25 fps)LENGTH: 541 frames MODEL SIZE (pixels): 101x136 AVG/MAX ERROR (mm): 1.95 / 9.76 TIME/FRAME (ms): 4.62

25. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 25 Experimental results FILE NAME: a3-2.aviSOURCE: TV FORMAT: 320x240 (25 fps)LENGTH: 440 frames MODEL SIZE (pixels): 75x95 AVG/MAX ERROR (mm): - TIME/FRAME (ms): 3.74

26. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 26 Experimental results FILE NAME: ggm2.aviSOURCE: QuickCam FORMAT: 320x240 (25 fps)LENGTH: 655 frames MODEL SIZE (pixels): 70x91 AVG/MAX ERROR (mm): 1.83 / 9.29 TIME/FRAME (ms): 3.69

27. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 27 Experimental results FILE NAME: sw2-1.aviSOURCE: DVD FORMAT: 320x240 (30 fps)LENGTH: 427 frames MODEL SIZE (pixels): 94x115 AVG/MAX ERROR (mm): - TIME/FRAME (ms): 3.57

28. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 28 Experimental results Location accuracy: –Errors in mm (in the face plane) using a ground-truth location of facial features –Average error below 4 mm, maximum error 14 mm –With CamShift: average error over 10 mm, maximum error 30 mm

29. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 29 Experimental results Execution time, per frame: –Off-the-self PC: AMD Athlon at 1.2 GHz –Average time below 5 ms, with 640x480 resolution, face size 100x120 pixels –With CamShift: average time about 10 ms, unable to work in one video sequence

30. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 30 Conclusions The tracking problem is decomposed into three main independent steps: –Vertical alignment –Horizontal alignment –Orientation estimation The process is fast, accurate and robust in the tested conditions It is exclusively based on integral projections

31. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 31 Conclusions The tracker is not affected by background distractors It can be applied either in color and grey-scale images Main limitation: maximum allowed movement (approx. 1 m/s, at 25 fps) Future work: improve the prediction step, e.g. with Kalman filters

32. REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, 2003 32 Last This work has been supported by Spanish CICYT project DPI-2001- 0469-C03-01 Demo videos: http://dis.um.es/~ginesgm/fip Grupo PARP web page: http://dis.um.es/parp Thank you very much