1. Learning a correlated model of identity and pose-dependent body shape variation for real-time synthesis Brett Allen 1,2, Brian Curless 1, Zoran Popović 1, and Aaron Hertzmann 3 1 University of Washington 2 Industrial Light & Magic 3 University of Toronto

2. Motivation moviesgames telepresence design

3. Goal We would like to be able to generate body models of any individual in any pose. Identity + pose shape - want to synthesize models in real-time - model should be learnable from real data

4. Data CAESAR data set: 44 subjects in 2 poses Multi-pose data set: 5 subjects in 16 poses Dense-pose data set: 1 subject in 69 poses [Anguelov et al. 2005]

5. Related work

6. Anatomical methods Aubel 2002 Chadwick et al. 1989 Turner and Thalmann 1993 Scheepers et al. 1997 Wilhelms and Van Gelder 1997 …

7. Example-based methods v = shape vector p = example parameters  = function parameters Given: n examples v 0 …v n and n sets of parameters p 0 …p n (optional) find .

8. Scattered data interpolation Allen et al. 2002 Lewis et al. 2000 Sloan et al. 2001 Kry et al. 2002 columns of  = key shapes  i = reconstruction weights from applying k-nearest neighbors or RBFs on p 11 22 44 33 11 33 22 44

9. An aside on enveloping Enveloping + scattered data interpolation = “corrective enveloping”

10. Latent variable models x = latent variable (component weights) W = components in columns v = average shape Blanz & Vetter 1999 Allen et al 2003 Seo et al 2003 Anguelov et al 2005 …

11. Pose variation vs body variation Sloan et al. 2001

12. Pose variation vs body variation Anguelov et al. 2005

13. Our approach c = “character vector”: all information needed to put a character in any pose v = shape in a particular pose Intrinsic skeleton parameters: bone lengths and carrying angles

14. Two Problems 1. Scans might not be at “key” poses 2. Scans are not complete Maximize: p(c | {e (  ) }) …actually, we don’t know the pose or skinning weights either: Maximize: p(c, s, q {  } | {e (  ) })

15. Maximum a posteriori estimation

16. Results

17. Going to multiple characters One possibility: Learn several character vectors separately, then run PCA. Two problems: –the character vector contains values that have different scales (rest positions, offsets, bone lengths) –we don’t have enough data!

18. Identity variation ~~ c  is the character vector of the  th example person g(c ,s,q  ) applies skinning and pose space deformations

19. Alternating optimization We initialize the {x  } with the weights from running PCA on estimated skeleton parameters. We then optimize W, c, s, q . Then we optimize for {x  }. Repeat…

20. Results

21. (video)

22. Conclusions We present a flexible approach for learning body shape variation between individuals and between poses, including the interrelationship between the two. + very general: can handle irregular and incomplete sampling in regard to both the poses/identities scanned, and in the surfaces themselves + the learned model can generate body shapes very quickly (over 75 fps)

23. Limitations You need a lot of data! Our data set was too sparse in some areas. Some poses are hard to capture. It’s very hard to compensate for the skinning artifacts. The shape matching could be improved (high-frequency details are lost if the matching is poor).

24. Acknowledgements UW Animation Research Labs Washington Research Foundation National Science Foundation, NSERC, CFI Microsoft Research, Electronic Arts, Sony, Pixar Kathleen Robinette and the AFRL lab Dragomir Anguelov Domi Pitturo