1. Constraint-based Motion Optimization Using A Statistical Dynamic Model Jinxiang Chai Texas A&M University

2. Goal: generate natural human motion from spatial- temporal constraints Human Animation from User Constraints

3. Goal: generate natural human motion from spatial- temporal constraints Key framesResulting animation Human Animation from User Constraints

4. Goal: generate natural human motion from spatial- temporal constraints Key framesResulting animation Key trajectories Resulting animation Human Animation from User Constraints

5. Applications Shrek Xtreme Martial Arts (Dicsovery channel) The Lord of the Rings Polar Express Friday Night 3D Bowling

6. Works for various spatial-temporal constraints Challenges: Human Animation Control

7. Works for various spatial-temporal constraints Human motions are high-dimensional while constraints are not Challenges: Human Animation Control

8. Works for various spatial-temporal constraints Human motions are high-dimensional while constraints are not People are experts on natural human motion Challenges: Human Animation Control

9. Problem Formulation θ MAP = argmax θ p(θ|C) Find the most likely human motion θ MAP from user constraints C: (Maximum a posterior)

10. Problem Formulation θ MAP = argmax θ p(θ|C) Find the most likely human motion θ MAP from user constraints C: = argmax θ p(C|θ) × p(θ) (Bayes rule) (Maximum a posterior)

11. Problem Formulation θ MAP = argmax θ p(θ|C) Find the most likely human motion θ MAP from user constraints C: = argmax θ p(C|θ) × p(θ) (Bayes rule) How well motion matches Constraints? How natural motion is generated? (Maximum a posterior)

12. Generate natural human motion from user-defined constraints Algorithm Overview Human motion database Human motion analysis Trajectory optimization Human motion prior p(θ) User-defined constraints C θ MAP θ MAP = argmax θ p(C|θ)p(θ)

13. [Witkin and Kass SIG88] [Cohen SIG92][Liu et al SIG94] [Fang and Pollard SIG03] Related Work: Physically Based Trajectory Optimization [Safonova et al. SIG04] [Sulejmanpasic and Popovic SIG 05] [Liu and Popovic SIG02] [Popovic and Witkin SIG99]

14. Reordering motion clip [Lee et al. SIG02, Kovar et al. SIG02, Pullen & Bregler SIG02, Arikan et al. SIG03] Learning model from human motion [Brand & Hertzmann SIG00, Li et al. SIG02] Interpolating motions [Rose et al. CG&A98, Kovar et al. SIG04] Related Work: Animation from Mocap Data

15. Generate natural motion from a small set of user-defined constraints Algorithm Overview Human motion database Human motion analysis Motion optimization Human motion prior User-defined constraints

16. A prerecorded large and heterogeneous human motion database 12 MX-40 Vicon cameras (120 hz) Full-body movement ( 1 hour ) and facial movement (10 minutes) Human Motion Database http://mocap.cs.cmu.edu/

17. Generate natural motion from user-defined constraints Algorithm Overview Human motion database Human motion analysis Motion optimization Human motion prior User-defined constraints

18. User-defined Constraints Any kinematic constraints throughout the motion Position cons. Orientation cons. Distance cons. Joint angle cons.

19. User input: Keyframe constraints Body Animation: Example Constraints

20. User input: Keyframe constraints Continuous trajectory constraints Body Animation: Example Constraints

21. Startt= 6st=3s User input: Face Animation: Example Constraints

22. Startt= 6st=3s t=1st=1.5st=3s Face Animation: Example Constraints User input:

23. Startt= 6st=3s User input: t=1st=1.5st=3s StartWidth of the mouthEnd Face Animation: Example Constraints

24. Human motion database Human motion analysis Motion optimization Human motion prior User-defined constraints Generate natural human motion from user-defined constraints Algorithm Overview

25. Statistical Dynamic Model Character pose Low-dimensional pose space y t = C x t + D Human motion database Human motion analysis Statistical dynamic model x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction

26. Model Complexity Character pose Low-dimensional pose space y t = C x t + D Human motion database Human motion analysis Statistical dynamic model x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction Dim(u t ) Dim(x t )

27. Model Learning Human motion database Human motion analysis Character pose Low-dimensional pose space y t = C x t + D x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction Dynamic model matrices: A 1, …,A m, B, C, D

28. Model Learning Human motion database Human motion analysis Dynamic model matrices: A 1, …,A m, B, C, D Dynamic system order: m, dimensionality of x t and u t Character pose Low-dimensional pose space y t = C x t + D x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction

29. Reconstruction error Full-body walking data Facial data X t = A 1 X t-1 +…+A m X t-m +B u t Statistical dynamic model: m = 3, dim(u t ) = 4, error = 0.7 deg m = 2, dim(u t ) = 1, error = 0.1 mm Y t = C X t + D

30. Forward Simulation y t = C x t + D x t = A 1 x t-1 + A 2 x t-2 + A 3 x t-3 +B u t Dim(u t ) = 4 Full-body movement:

31. Forward Simulation y t = C x t + D x t = A 1 x t-1 + A 2 x t-2 + A 3 x t-3 +B u t utut ytyt Dim(u t ) = 4 Full-body movement:

32. x t = A 1 x t-1 + A 2 x t-2 + B u t Forward Simulation utut YtYt Dim(u t ) = 1 y t = C x t + D Facial movement:

33. Motion Optimization Human motion database Human motion analysis Human motion prior User-defined constraints Generate natural motion from a small set of user-defined constraints Motion optimization

34. Human Motion Prior The motion prior of generated motion sequence

35. Human Motion Prior The motion prior of generated motion sequence p(θ)

36. Human Motion Prior The motion prior of generated motion sequence p(θ) Assume u t is independent each other x t depends on the x t-1,…,x t-m and u t

37. Human Motion Prior The motion prior of generated motion sequence p(θ) (Chain rule)

38. Human Motion Prior The motion prior of generated motion sequence Control priorDynamics prior p(θ) (Chain rule)

39. Generate natural motion from a small set of user-defined constraints Algorithm Overview Human motion database Human motion analysis Motion optimization Human motion prior User-defined constraints

40. Motion Optimization Find the most likely motion θ MAP from user constraints C: θ MAP = argmin θ –ln p(C|θ) - ∑ln p(u t ) - ∑ ln p(x t |x t-1,…,x t-m,u t )

41. Motion Optimization Find the most likely motion θ MAP from user constraints C: θ MAP = argmin θ –ln p(C|θ) - ∑ln p(u t ) - ∑ ln p(x t |x t-1,…,x t-m,u t ) Prior for control input Prior from statistical dynamic model Constraint term

42. Motion Optimization Find the most likely motion θ MAP from user constraints C: θ MAP = argmin θ –ln p(C|θ) - ∑ln p(u t ) - ∑ ln p(x t |x t-1,…,x t-m,u t ) Prior for control input Prior from statistical dynamic model Constraint term - Trajectory optimization - Cubic B-splines for x and u - Sequential Quadratic programming - Random initial guess [0...1]

43. Motion Optimization Initial guess 1 Initial guess 2 0102030405060708090100 0 05 1 15 2 2.5 x10 4 Iteration number Objective function Initial guess 3

44. Results: Climbing over an Obstacle User input Output animation

45. Results: Mickey Walking User input Output animation

46. Results: Baby Walking User input Output animation

47. Results: Running User input Output animation

48. Results: Walking & Jumping User input Output animation

49. Motion Generalization I Source motion Retargeted motion

50. Motion Generalization II Source motion Modified motion

51. Facial Animation I Start t= 6s t=3s User input:

52. Facial Animation II StartEndt=1st=1.5st=3s User input:

53. Algorithm Evaluation I: The Importance of Motion Priors Input constraints Inverse Kinematics (IK) Our method IK in PCA space

54. Algorithm Evaluation II: Different Motion Priors Input key frames Running prior Walking prior Jumping prior

55. Algorithm Evaluation III: Different Numbers of Constraints Ground truth motion Four key frames (1.1 deg per joint) Two key frames (1.8 deg per joint) Six key frames (0.9 deg per joint) Walking prior

56. Discussion An alternative for physically based optimization √ does not need physical models √ low-dimensional model and faster convergence √ can generate slow even stylized motion x requires appropriate mocap data x cannot specify dynamic constraints (e.g., mass)

57. Discussion An alternative for physically based optimization √ does not need physical models √ low-dimensional model and faster convergence √ can generate slow even stylized motion x requires appropriate mocap data x cannot specify dynamic constraints (e.g., mass) Intuitive Interfaces for specifying spatial-temporal constraints?

58. Conclusions Sparse spatial-temporal constraints Human motion database Human motion analysis Motion optimization Human motion prior Realistic animation or

59. Realistic human animation Trajectory optimization with statistical models Conclusions Human motion database User-defined constraints Human Motion prior p(C|θ)p(θ) argmax θ p(θ|C)

60. Realistic human animation Trajectory optimization with statistical models A new compact spatial-temporal representations for human motion data: statistical dynamic models Conclusions Human motion database User-defined constraints Human Motion prior X t = A 1 X t-1 +…+A m X t-m +B u t Y t = C X t + D p(C|θ)p(θ) argmax θ p(θ|C)

61. Questions?