Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie Mellon University 2 Texas A&M University 3 Microsoft Research Asia

Face Poser Inputs Generate new facial expressions with a simple and intuitive interface

Face Poser InputsOutput Generate new facial expressions with a simple and intuitive interface

Why Face Poser? Pre-defined controls Difficult to build and use Complex facial expressions

Applications Films, GamesVirtual Reality Educational

Related Work Sketched-based interfaces Zeleznik et al. 96 Igarashi et al. 99 Nealen et al. 05 Kho and Garland 05 Chang and Jenkins 06 Nealen et al. 05 Igarashi et al. 99

Related Work Example-based modeling Blanz and Vetter 99 Pighin et al. 99 Chai et al. 03 Zhang et al. 04 Grochow et al. 04 Sumner et al. 05 Grochow et al. 04

Overview Database Preprocessing Model Prior

Overview Database Preprocessing Model Prior Neutral Pose User Constraints Interface

Overview Database Preprocessing Model Prior Runtime Optimization Neutral Pose New Pose User Constraints Interface Textured Pose

Motion capture data Captured mesh animations of various facial expressions: anger, fear, surprise, sadness, joy, disgust, speaking, singing All meshes translated and rotated to a standard view:

Data: PCA representation x = v 1x v 1y v 1z v 2x. p is low-dimensional representation of x

Problem statement Find best p satisfying user-constraints c: Best p is: Given a face model, how well does it match user-constraints Likelihood of face model using knowledge of data

Point Constraints More detailed control User inputs: blue – 3D source vertex green – 2D target pixel Can select in any camera view

Point Constraints We optimize for best p For each p: compute whole mesh x take selected 3D source vertex project it to 2D screen space compare to target pixel

Point Constraints Optimization term: Jacobian term:

Point Constraints InputsSolution

Point Constraints – Results

Point Constraints – Dragging interface

Stroke Constraints Large-scale changes with minimal input User inputs: blue – 2D source stroke (selects 3D points on mesh) green – 2D target stroke Any curve, line, or freeform region

Stroke Constraints 2D source stroke  raytrace each pixel to mesh surface to get dark gray points These can be 3D points on mesh surface (not just original mesh vertices)

Stroke Constraints We optimize for best p For each p: compute whole mesh x take selected 3D points project them to 2D screen space compare to target stroke

Stroke Constraints Optimization term: Jacobian term:

Stroke Constraints InputsSolution

Stroke Constraints – Results

Stroke Constraints – Tablet interface

Stroke Constraints – Additional term If strokes are far away from each other, energy term will reach local minimum Need additional optimization term to minimize distance between “center” of source stroke and “center” of target stroke Without additional term

Stroke Constraints – Additional term Without additional term Optimization term: Jacobian term:

Stroke Constraints – Additional term Without additional termWith additional term

Stroke Constraints – Results

Problem statement Find best p satisfying user-constraints c: Best p is: Given a face model, how well does it match user-constraints Likelihood of face model using knowledge of data

Model Priors There can be many solutions satisfying user constraints. Some of them are not realistic. We add another optimization term to constrain the solution to the space defined by the motion capture data. Without model priors term

Model Priors Without model priors term Learn a Mixtures of Factor Analyzers (MFA) model Probability density function to measure naturalness of facial expression MFA has been applied to high-dimensional nonlinear data

Model Priors Without model priors term Optimization term: Jacobian term:

Model Priors – Result

increasing weight on Model Prior term

Model Priors – Result

Computation time Standard PC hardware (Pentium IV 2 GHz) Point constraints takes 0.18 seconds for 10 points time increases linearly with number of points Stroke constraints takes 0.4 seconds for source stroke of ~900 pixels (about size of eyebrow) time increases linearly with number of pixels faster if using intermediate spline representation

Cross validation New face expression samples for testing Use new samples to get target constraints Generate solution and compare with test sample

Cross validation Ground truth Interpolation Optimization

Comparison with other techniques Opt-blend: FaceIK [Zhang et al. 04] PCA: Morphable model [Blanz and Vetter 99; Blanz et al. 03] LWR: Locally weighted regression 3D errors

Comparison with other techniques Ground truth, Optimization with PCA, Optimization with MFA

Application: Trajectory Keyframing Green points – given 2D target pixels Blue points and mesh – solution

Application: Trajectory Keyframing Ground truth Result

Application: Trajectory Keyframing Ground truth Result

Application: Trajectory Keyframing Ground truth Result

Summary: Face Poser InputsOutput Users can learn to use our system within minutes and can create new facial expressions within seconds

Limitation Global control  changing mouth also changes eyes  this is natural, but difficult to control sometimes Local control  changing mouth without changing eyes  but this might lead to “fake smiles”

Extensions / Future work We have added different types of constraints within the same optimization framework More general: model face as separate regions, generate each region separately, and blend them back together