1. LYU0603 A Generic Real-Time Facial Expression Modelling System Supervisor: Prof. Michael R. Lyu Group Member: Cheung Ka Shun (05521661) Wong Chi Kin (05524554)

2. Outline Previous Work Objectives Work in Semester Two Review of implementation tool Implementation –Virtual Camera –3D Face Generator –Face Animation Conclusion Q&A

3. Previous Work Face analysis –Detect the facial expression –Draw corresponding model

4. Objectives Enrich the functionality of web-cam Make net-meeting more interesting Users are not required to pay extra cost on specific hardware Extract human face and approximate face shape

5. Work in Semester Two Virtual Camera –Make Facial Expression Modelling to be available in net-meeting software Face Generator –Approximate face shape –Generate a 3D face texture Face Animation –Animate the generated 3D face –Convert into standard file format

6. Review - DirectShow Filter graph –Source  Transform  Renderer

7. Review - Direct3D Efficiently process and render 3-D scenes to a display, taking advantage of available hardware Fully Compatible with DirectShow

8. Virtual Camera Focus on MSN Messenger

9. Virtual Camera Two components –3D model as output –Face Mesh Preview

10. Virtual Camera Actually it is a source filter

11. Virtual Camera Inner filter graph in virtual camera

12. Virtual Camera

13. Demonstration We are going to play a movie clip which demonstrate Virtual Camera

15. 3D Face Generator Aims: To approximate the human face and shape Comprise two parts

17. FaceLab Adopted from the face analysis project of Zhu Jian Ke, CUHK CSE Ph.D. Student The analysis is decomposed into training and building part The whole training phase is made up of three steps

18. However, thousands of point are demanded to describe the complex structure of human face It can be acquired either by 3D scanner or computer vision algorithm FaceLab – Data Acquisition To acquire human face structure data In 2D face modelling, 100 feature points are sufficient to represent the face surface

19. FaceLab – Data Registration To normalize the 3D data into same scale with correspondences Problem –The most accurate way is to compute 3D optical flow –Commercial 3D scanners and 3D registration are computed with specific hardware

20. FaceLab To simplify the process, it is decided to use software to generate the human face data. Each has a set of 752 3D vertex data to describe the shape of face

21. FaceLab – Shape Model Building A shape is defined as a geometry data by removing the translational, rotational and scaling components. The object containing N vertex data is represented as a matrix below

22. FaceLab – Shape Model Building The set of P shapes will form a point cloud in 3N- dimensional space which is a huge domain. A conventional principle component analysis (PCA) is performed.

23. FaceLab – Shape Model Building PCA Implementation It performs an orthogonal linear transform A new coordinate system which points to the directions of maximum variation of the point cloud. In this Implementation, the covariance method is used.

24. FaceLab – Shape Model Building PCA Implementation Step 1: Compute the empirical mean which is the mean shape along each dimension

25. FaceLab – Shape Model Building PCA Implementation Step 2: Calculate the covariance matrix C The axes of the point cloud are collected from the eigenvectors of the covariance matrix.

26. FaceLab – Shape Model Building PCA Implementation Step 3: Compute the matrix of eigenvectors V where D is the eigenvalue matrix of C The eigenvalue represents the distribution of the objects data ’ s energy

27. FaceLab – Shape Model Building PCA Implementation Final Step: Represent the resulted shape model as where m s are the shape parameters Adjusting the value of the shape parameters can generate a new face model by computing

28. FaceLab – Shape Model Building PCA Implementation An extra step: Select a subset of the eigenvectors The eigenvalue represents the variation of the corresponding axis The first seven columns are used in the system and achieve a majority of the total variance.

29. FaceLab – Render the face model The resulted data set is a 3D face mesh data Use OpenGL to render it

31. System Overview of Face Texture Generator Facial Expression ModellingFace Texture Generator

32. Face texture extraction –Three Approaches Largest area triangle aggregation Human-defined triangles aggregation Single photo on effect face

33. Largest area triangle aggregation Left face Right face Front face

34. Largest area triangle aggregation

35. Result

36. Human-defined triangles aggregation Divide the face mesh into three parts Define the particular photo to be sampled in triangles in each region Reduce fragmentation

37. Human-defined triangles aggregation Redefine the face mesh – Effect Face

38. Human-defined triangles aggregation Result

39. Single photo on effect face Similar to Human-defined triangles aggregation Use a single photo for pixel sampling Use Effect Face as outline

40. Single photo on effect face Result

41. Face Generator Filter

42. Dynamic Texture Generation To get back the rendered data from the video display card

43. Dynamic Texture Generation Lock the video display buffer

44. Dynamic Texture Generation Common buffer content is changed Update the texture buffer to reflect the changes immediately

45. Dynamic Texture Generation From 2D face mesh to 3D face mesh

46. Completed 3D Face Generator

47. Demonstration We are going to play a movie clip which demonstrates Face Generator

49. Face Viewer

50. Generate simple animation Looking at the mouse cursor Feature points provide sufficient information to locate the eye The two eyes will form a triangle planar with the mouse cursor

51. Generate simple animation Eye blinking One of the natural movements on a human face Adjust the vertex geometry on the eyelids The eyebrow is also needed to move backward

52. Generate simple animation A lot of muscle needed to be modified The cheeks are being pushed up and wide The chin is being pulled down To change the shape of the lip Smiling

53. Convert into standard file format Save the mesh in.x file format Current selected face mesh data would be saved The Microsoft DirectX.x file format is not specific It can be used by any other application

54. Demonstration We are going to play a movie clip which demonstrate Face Animation

56. Conclusion We have achieved our goals –Enrich functionality of webcam –Make net-meeting become more interesting –Extract human face and approximate face shape –Discover the potential of face recognition technology

57. End Thank you! Q&A