Learning a correlated model of identity and pose-dependent body shape variation for real-time synthesis Brett Allen 1,2, Brian Curless 1, Zoran Popović 1, and Aaron Hertzmann 3 1 University of Washington 2 Industrial Light & Magic 3 University of Toronto

Motivation moviesgames telepresence design

Goal We would like to be able to generate body models of any individual in any pose. Identity + pose shape - want to synthesize models in real-time - model should be learnable from real data

Data CAESAR data set: 44 subjects in 2 poses Multi-pose data set: 5 subjects in 16 poses Dense-pose data set: 1 subject in 69 poses [Anguelov et al. 2005]

Related work

Anatomical methods Aubel 2002 Chadwick et al Turner and Thalmann 1993 Scheepers et al Wilhelms and Van Gelder 1997 …

Example-based methods v = shape vector p = example parameters  = function parameters Given: n examples v 0 …v n and n sets of parameters p 0 …p n (optional) find .

Scattered data interpolation Allen et al Lewis et al Sloan et al Kry et al columns of  = key shapes  i = reconstruction weights from applying k-nearest neighbors or RBFs on p 11 22 44 33 11 33 22 44

An aside on enveloping Enveloping + scattered data interpolation = “corrective enveloping”

Latent variable models x = latent variable (component weights) W = components in columns v = average shape Blanz & Vetter 1999 Allen et al 2003 Seo et al 2003 Anguelov et al 2005 …

Pose variation vs body variation Sloan et al. 2001

Pose variation vs body variation Anguelov et al. 2005

Our approach c = “character vector”: all information needed to put a character in any pose v = shape in a particular pose Intrinsic skeleton parameters: bone lengths and carrying angles

Two Problems 1. Scans might not be at “key” poses 2. Scans are not complete Maximize: p(c | {e (  ) }) …actually, we don’t know the pose or skinning weights either: Maximize: p(c, s, q {  } | {e (  ) })

Maximum a posteriori estimation

Results

Going to multiple characters One possibility: Learn several character vectors separately, then run PCA. Two problems: –the character vector contains values that have different scales (rest positions, offsets, bone lengths) –we don’t have enough data!

Identity variation ~~ c  is the character vector of the  th example person g(c ,s,q  ) applies skinning and pose space deformations

Alternating optimization We initialize the {x  } with the weights from running PCA on estimated skeleton parameters. We then optimize W, c, s, q . Then we optimize for {x  }. Repeat…

Results

(video)

Conclusions We present a flexible approach for learning body shape variation between individuals and between poses, including the interrelationship between the two. + very general: can handle irregular and incomplete sampling in regard to both the poses/identities scanned, and in the surfaces themselves + the learned model can generate body shapes very quickly (over 75 fps)

Limitations You need a lot of data! Our data set was too sparse in some areas. Some poses are hard to capture. It’s very hard to compensate for the skinning artifacts. The shape matching could be improved (high-frequency details are lost if the matching is poor).

Acknowledgements UW Animation Research Labs Washington Research Foundation National Science Foundation, NSERC, CFI Microsoft Research, Electronic Arts, Sony, Pixar Kathleen Robinette and the AFRL lab Dragomir Anguelov Domi Pitturo