1. Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie Mellon University 2 Texas A&M University 3 Microsoft Research Asia

2. Face Poser Inputs Generate new facial expressions with a simple and intuitive interface

3. Face Poser InputsOutput Generate new facial expressions with a simple and intuitive interface

4. Why Face Poser? Pre-defined controls Difficult to build and use Complex facial expressions

5. Applications Films, GamesVirtual Reality Educational

6. Related Work Sketched-based interfaces Zeleznik et al. 96 Igarashi et al. 99 Nealen et al. 05 Kho and Garland 05 Chang and Jenkins 06 Nealen et al. 05 Igarashi et al. 99

7. Related Work Example-based modeling Blanz and Vetter 99 Pighin et al. 99 Chai et al. 03 Zhang et al. 04 Grochow et al. 04 Sumner et al. 05 Grochow et al. 04

8. Overview Database Preprocessing Model Prior

9. Overview Database Preprocessing Model Prior Neutral Pose User Constraints Interface

10. Overview Database Preprocessing Model Prior Runtime Optimization Neutral Pose New Pose User Constraints Interface Textured Pose

11. Motion capture data Captured mesh animations of various facial expressions: anger, fear, surprise, sadness, joy, disgust, speaking, singing All meshes translated and rotated to a standard view:

12. Data: PCA representation x = v 1x v 1y v 1z v 2x. p is low-dimensional representation of x

13. Problem statement Find best p satisfying user-constraints c: Best p is: Given a face model, how well does it match user-constraints Likelihood of face model using knowledge of data

14. Point Constraints More detailed control User inputs: blue – 3D source vertex green – 2D target pixel Can select in any camera view

15. Point Constraints We optimize for best p For each p: compute whole mesh x take selected 3D source vertex project it to 2D screen space compare to target pixel

16. Point Constraints Optimization term: Jacobian term:

17. Point Constraints InputsSolution

18. Point Constraints – Results

19. Point Constraints – Dragging interface

20. Stroke Constraints Large-scale changes with minimal input User inputs: blue – 2D source stroke (selects 3D points on mesh) green – 2D target stroke Any curve, line, or freeform region

21. Stroke Constraints 2D source stroke  raytrace each pixel to mesh surface to get dark gray points These can be 3D points on mesh surface (not just original mesh vertices)

22. Stroke Constraints We optimize for best p For each p: compute whole mesh x take selected 3D points project them to 2D screen space compare to target stroke

23. Stroke Constraints Optimization term: Jacobian term:

24. Stroke Constraints InputsSolution

25. Stroke Constraints – Results

26. Stroke Constraints – Tablet interface

27. Stroke Constraints – Additional term If strokes are far away from each other, energy term will reach local minimum Need additional optimization term to minimize distance between “center” of source stroke and “center” of target stroke Without additional term

28. Stroke Constraints – Additional term Without additional term Optimization term: Jacobian term:

29. Stroke Constraints – Additional term Without additional termWith additional term

30. Stroke Constraints – Results

31. Problem statement Find best p satisfying user-constraints c: Best p is: Given a face model, how well does it match user-constraints Likelihood of face model using knowledge of data

32. Model Priors There can be many solutions satisfying user constraints. Some of them are not realistic. We add another optimization term to constrain the solution to the space defined by the motion capture data. Without model priors term

33. Model Priors Without model priors term Learn a Mixtures of Factor Analyzers (MFA) model Probability density function to measure naturalness of facial expression MFA has been applied to high-dimensional nonlinear data

34. Model Priors Without model priors term Optimization term: Jacobian term:

35. Model Priors – Result

36. increasing weight on Model Prior term

37. Model Priors – Result

38. Computation time Standard PC hardware (Pentium IV 2 GHz) Point constraints takes 0.18 seconds for 10 points time increases linearly with number of points Stroke constraints takes 0.4 seconds for source stroke of ~900 pixels (about size of eyebrow) time increases linearly with number of pixels faster if using intermediate spline representation

39. Cross validation New face expression samples for testing Use new samples to get target constraints Generate solution and compare with test sample

40. Cross validation Ground truth Interpolation Optimization

41. Comparison with other techniques Opt-blend: FaceIK [Zhang et al. 04] PCA: Morphable model [Blanz and Vetter 99; Blanz et al. 03] LWR: Locally weighted regression 3D errors

42. Comparison with other techniques Ground truth, Optimization with PCA, Optimization with MFA

43. Application: Trajectory Keyframing Green points – given 2D target pixels Blue points and mesh – solution

44. Application: Trajectory Keyframing Ground truth Result

45. Application: Trajectory Keyframing Ground truth Result

46. Application: Trajectory Keyframing Ground truth Result

47. Summary: Face Poser InputsOutput Users can learn to use our system within minutes and can create new facial expressions within seconds

48. Limitation Global control  changing mouth also changes eyes  this is natural, but difficult to control sometimes Local control  changing mouth without changing eyes  but this might lead to “fake smiles”

49. Extensions / Future work We have added different types of constraints within the same optimization framework More general: model face as separate regions, generate each region separately, and blend them back together