1. University of Geneva www.miralab.ch MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans Parag Chaudhuri MIRALab University of Geneva CGI 2008 Istanbul, Turkey All images used in this presentation are copyrights of their respective authors and publishers, and are always in reference to their original works. Please credit the original authors if you use these images.

2. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 2 What do we mean by interaction? or Interactive Control of the Virtual Human Interaction with the Virtual Human 4 5 Here, we will only talk about this. We will only talk about animating motion of the human body.

3. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 3 Animating Virtual Humans Layered Deformation Chadwick et al., 1989 5 Boulic et al., 1990 6 Perlin, 1995 7 Hodgins et al., 1995 8 Kinematic Walking Controller Motion Blending Dynamics Controller Gleicher, 1998 9 Retargeting Motion Vertex-blend Skinning Need to represent motion Badler and Smoliar, 1979 1 Girard and Maciejewski, 1985 2 Armstrong and Green, 1985 3 Magnenat-Thalmann et al., 1988 4 Inverse Kinematics Dynamics Increasing interactive control 1980’s 1990’s

4. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 4 Trends today Data or Performance driven Dynamics Controllers Hybrid methods 6 7 8 9

5. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 5 Trends today Data or Performance driven Dynamics Controllers Hybrid methods 6 7 8 9

6. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 6 Motion Graphs Motion Graphs, L. Kovar, M. Gleicher and F. Pighin, SIGGRAPH 2002. A corpus of motion clips is analyzed. Pair-wise distances between all clips is calculated. Small valued local minima in the distance function form good transition points between the clips. Such clips form nodes with the transitions form the edges of a Motion Graph. Motion clips from the database Local Minima of the Distance Function

7. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 7 Motion Graphs Motion Graphs, L. Kovar, M. Gleicher and F. Pighin, SIGGRAPH 2002. Constrained linear blending is used to generate smooth transitions between motions represented by the motion graphs nodes. A query is given in the form of a path. A walk on the graph is generated by using a search based on depth first search. To stop exponential growth, the search is pruned by a branch and bound strategy based on an error function given by the user. This walk generates the motion desired by the user. Motion generated in response to a user defined path spelling the words “Motion Graph”

8. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 8 Motion Synthesis from Annotations Motion Synthesis from Annotations, O. Arikan, D. Forsyth and J O’Brien, SIGGRAPH 2003. A motion database is annotated with a user defined vocabulary. This clusters the input motions into classes. A query is represented by a timeline painted by these annotations. The system uses a combination of hierarchical dynamic programming and local optimization to find motion which satisfies the annotated timeline. User annotates the input databaseThe motion is generated in response to a timeline painted with annotations

9. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 9 Style-Based Inverse Kinematics Style-Based Inverse Kinematics, K. Grochow, S. Martin, A. Hertzmann and Z. Popovic, SIGGRAPH 2004. Motion is learned from a database using a statistical learning model represented as a probability distribution over the space of all possible poses. Animation created by IK uses the learnt poses to create the keyframes while satisfying user defined constraints. Training the model on different input data produces different styles of IK.

10. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 10 Motion Patches Motion Patches: Building Blocks for Virtual Environments Annotated with Motion Data, K. H. Lee, M. G. Choi and J. H. Lee, SIGGRAPH 2006. Data is captured and clustered by type of motion. Small patches are formed based on geometry of capture environment and are annotated by captured human motion data.

11. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 11 Motion Patches Motion Patches: Building Blocks for Virtual Environments Annotated with Motion Data, K. H. Lee, M. G. Choi and J. H. Lee, SIGGRAPH 2006. Transitions between patches are represented in the form of a graph. A number of such patches can be used to automatically animate multiple characters in a complex virtual environment like an office or a playground. Transitions between patches Virtual Office Environment

12. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 12 Near-optimal Control  Simple motion controller to stitch together motion.  A learnt value function, approximated by a linear combination of simple basis functions, acts as a controller for simple tasks.  Runtime transitions between the controllers generates the character animation. Near-optimal Character Animation with Continuous Control, A Treuille, Y. Lee and Z. Popovic, SIGGRAPH 2007.

13. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 13 Trends today Data or Performance driven Dynamics Controllers Hybrid methods 6 7 8 9

14. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 14 Controllers for physics-based character animation Composable Controllers for Physics-Based Character Animation, P. Faloutsos, M. van de Panne and D. Terzopoulos, SIGGRAPH 2001. A controller encapsulates the knowledge for a particular task. It can be built using bio-mechanical laws or as per specific animation requirements. Different controllers for acts like balancing, falling, rising from supine and positions and other actions. The controllers are composed in a sequence to generate an animation. Hybrid Control For Interactive Character Animation, A. Shapiro, F. Pighin and P. Faloutsos, Pacific Graphics 2003. A character standing up with a jump.

15. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 15 Layered dynamic control for swimming Layered Dynamic Control for Interactive Character Swimming, P. Yang, J. Laszlo and K. Singh, SCA 2004. Basic swim strokes generated by a pose control graph driving joint level proportional-derivative controllers. A per-cycle control layer applies stroke variations on a per cycle basis. A final perturbation layer adapts the swimmers pose based on changes in the fluid environment. Poses generated by the animation system Actual poses of a swimmer recorded underwater

16. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 16 Adaptive Dynamics of Articulated Bodies Adaptive Dynamics of Articulated Bodies, S. Redon, N. Galoppo and M. Lin, SIGGRAPH 2005. An algorithm for automatic simplification of articulated body dynamics. The algorithm predicts which joints contribute most to the articulated body motion, and simulates those joints only. The set of simulated joints is determined based on the current state of the articulated body, the external forces applied to it, the active joint forces, and customizable motion metrics.

17. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 17 Multiobjective Control Contact constraints are specified in terms of frictions cones, that are generalizations of the ZMP constraint. Multiple objectives are specified to achieve a certain task such as reach for a point and maintain balance. A quadratic program solves the multiobjective optimization subject to the contact constraints. Multiobjective Control with Frictional Contacts, Y. Abe, M. daSilva and J. Popovic, SCA 2007.

18. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 18 Trends today Data or Performance driven Dynamics Controllers Hybrid methods 6 7 8 9

19. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 19 Learning response Capture samples of the motion for the response. When a push occurs, transition to the beginning of a recorded response. Use offset vectors in joint position and velocity to account for the direction and place of the push. Decay the offset vectors to zero over time. Train an oracle by using extensive user input to label good and bad motions and then use it to choose good transitions to make. Pushing People Around, O. Arikan, D. Forsyth and J O’Brien, SCA 2005.

20. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 20 Learning physics Learn a vector off all forces and torques at all joints from motion data that minimizes an Energy function Provide new footprint constraints to get new motion. Learn a statistical model of motion from data. Use the learnt model as a prior with user defined constraints to solve a trajectory optimization for a new motion. Constraint-based motion optimization using a statistical dynamic model, J. Chai and J. Hodgins, SIGGRAPH 2008. Learning Physics-Based Motion Style with Nonlinear Inverse Optimization, C. K. Liu, A. Hertzmann and Z. Popovic, SIGGRAPH 2005.

21. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 21 Dynamic Response for MoCap An unexpected impact event occurs. Use simple ragdoll simulation to find a clip to transition-to after impact. Use short-interval dynamics to generate the interval motion and blend between the transition. Dynamic response for motion capture animation, V. Zordan, A. Majkowska, B. Chiu and M. Fast, SIGGRAPH 2005

22. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 22 What to use when Data or Performance driven Acquisition is not a problem. Motion to be animated is similar to the one being captured. Physics is not desired. Stable, long motion sequences are desired. Dynamics Controllers Physically-correct response is important. Many characters and the environment – all interact. Hybrid methods A combination of the above needs. 8

23. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 23 Controlling Style Segment motion into affine and local deformations. Specify key-shape deformations. Use blending in higher dimensions to compute a multi-way blend. Turning to the Masters: Motion Capturing Cartoons, C. Bregler, L. Loeb, E. Chuang and H. Deshpande, SIGGRAPH 2002. Filter the second derivative of the motion signal. Convolve it with a Gaussian and then subtract it from the original motion. Equivalent of an unsharp filter for images. The Cartoon Animation Filter, J. Wang, S. Drucker, M. Agarwala and M. Cohen, SIGGRAPH 2006.

24. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 24 Controlling Style A sketched curve is parsed and used to generate animation by blending animation sequences corresponding to each parsed segment. Motion Doodles, M. Thorne, D. Burke and M. van de Panne, SIGGRAPH 2004. Associate key frames with points in space. Move a spatial cursor and do a radial- basis blend between the key poses. Spatial Keyframing for Performance-driven Animation, T. Igarashi, T. Moscovich and J.F. Hughes, SCA 2005.

25. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 25 Other control mechanisms Sketches of poses. Videos. Six-axis controllers like the Wiimote. Cameras and HMDs : We start to move towards interacting with the virtual humans. Interactive control of the virtual humans leads to better interaction with the virtual humans.

26. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 26Interaction Interactive Control of the Virtual Human Interaction with the Virtual Human 4 5

27. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 27 Self Adaptive Animation More on this tomorrow – in the session on virtual humans… P Chaudhuri, G. Papagiannakis and N. Magnenat-Thalmann. Self Adaptive Animation based on User Perspective. CGI 2008.

28. University of Geneva www.miralab.ch MIRALab Where Research Means Creativity Questions?

29. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 29References 1.N. Badler and S.W. Smoliar. Digital Representation of Human Movement, ACM Computing Surveys, March issue, pp.19-38. 1979. 2.W. W. Armstrong and M. W. Green. The dynamics of articulated rigid bodies for purposes of animation. The Visual Computer, 1(4), pp 231-240, 1985. 3.M. Girard and A. A. Maciejewski, Computational modeling for the computer animation of legged figures, SIGGRAPH 1985, pp 263-270, 1985. 4.N. Magnenat-Thalmann, R. Laperriere and D. Thalmann. Joint-dependent local deformations for hand animation and object grasping. Proceedings of the Graphics Interface. pp: 26-33, 1988. 5.J.E. Chadwick, D. R. Haumann and R. Parent. Layered construction for deformable animated characters. SIGGRAPH 1989. pp 243-252, 1989. 6.R. Boulic, N. Magnenat-Thalmann and D. Thalmann. A global human walking model with real-time kinematic personification. The Visual Computer, 6(6), pp. 344-358, Special Issue on Computer Animation 89/90, 1990. 7.K. Perlin. Real time responsive animation with personality, IEEE TVCG, 1(1), pp. 5 -15, 1995. 8.J Hodgins, W. Wooten, D. Brogan and J. O’Brien, Animating Human Athletics, SIGGRAPH 1995, pp. 71-78, 1995. 9.M. Gleicher. Retargeting motion to new characters, SIGGRAPH 1998, pp. 33-42, 1998. 10.L. Kovar, M. Gleicher and F. Pighin. Motion Graphs. SIGGRAPH 2002. 11.O. Arikan, D. Forsyth and J O’Brien. Motion Synthesis from Annotations. SIGGRAPH 2002. 12.K. Grochow, S. Martin, A. Hertzmann and Z. Popovic. Style-Based Inverse Kinematics. SIGGRAPH 2004. 13.K. H. Lee, M. G. Choi and J. H. Lee. Motion Patches: Building Blocks for Virtual Environments Annotated with Motion Data. SIGGRAPH 2006. 14.A Treuille, Y. Lee and Z. Popovic. Near-optimal Character Animation with Continuous Control. SIGGRAPH 2007.

30. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 30References 15.P. Faloutsos, M. van de Panne and D. Terzopoulos. Composable Controllers for Physics-Based Character Animation. SIGGRAPH 2001 16.A. Shapiro, F. Pighin and P. Faloutsos. Hybrid Control For Interactive Character Animation. Pacific Graphics 2003. 17.P. Yang, J. Laszlo and K. Singh. Layered Dynamic Control for Interactive Character Swimming. SCA 2004. 18.S. Redon, N. Galoppo and M. Lin. Adaptive Dynamics of Articulated Bodies. SIGGRAPH 2005. 19.Y. Abe, M. daSilva and J. Popovic. Multiobjective Control with Frictional Contacts. SCA 2007. 20.C. K. Liu, A. Hertzmann and Z. Popovic. Learning Physics-Based Motion Style with Nonlinear Inverse Optimization. SIGGRAPH 2005. 21.J. Chai and J. Hodgins. Constraint-based motion optimization using a statistical dynamic model. SIGGRAPH 2008. 22.V. Zordan, A. Majkowska, B. Chiu and M. Fast. Dynamic response for motion capture animation. SIGGRAPH 2005 23.C. Bregler, L. Loeb, E. Chuang and H. Deshpande. Turning to the Masters: Motion Capturing Cartoons. SIGGRAPH 2002. 24.J. Wang, S. Drucker, M. Agarwala and M. Cohen. The Cartoon Animation Filter. SIGGRAPH 2006. 25.T. Igarashi, T. Moscovich and J.F. Hughes. Spatial Keyframing for Performance-driven Animation. SCA 2005. 26.M. Thorne, D. Burke and M. van de Panne. Motion Doodles. SIGGRAPH 2004. 27.P Chaudhuri, G. Papagiannakis and N. Magnenat-Thalmann. Self Adaptive Animation based on User Perspective. CGI 2008.

31. CGI 2008 MIRALab Where Research Means Creativity Animation of Interactive Virtual Humans 31 Image Credits 1.The Sims 3, Copyright Electronic Arts, 2008. 2.Okan Arikan, David A. Forsyth, James F. O'Brien. Pushing People Around. SCA '05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, pp.59--66. 3.Richard Williams. Animator’s Survival Kit: A Manual of Methods. Faber & Faber, 2002. 4.A. Treuille, Y. Lee and Z. Popović. Near-optimal Character Animation with Continuous Control. ACM Transactions on Graphics 26(3). SIGGRAPH 2007. 5.M. Thiebaux, A. Marshall, S. Marsella, and M. Kallmann. SmartBody: Behaviour Realization for Embodied Conversational Agents Proceedings of Autonomous Agents and Multi-Agent Systems (AAMAS), 2008. 6.L. Kovar, M. Gleicher and F. Pighin. Motion Graphs. ACM Transactions on Graphics 21(3). SIGGRAPH 2002. 7.P. Faloutsos, M. van de Panne and D. Terzopoulos. Composable controllers for physics-based character animation. ACM Transactions on Graphics 21(3). SIGGRAPH 2001. 8.C. K. Liu, A. Hertzmann and Z. Popovic. Learning physics-based motion style with nonlinear inverse optimization. ACM Transactions on Graphics 24(3). SIGGRAPH 2005. 9.V. Zordan, A. Majkowska, B. Chiu and M. Fast. Dynamic response for motion capture animation. ACM Transactions on Graphics 24(3). SIGGRAPH 2005.