1. Definition and realisation of modeling methods and motion computation algorithms for virtual humans Nicolas Pronost Advisor :Bruno Arnaldi Co-advisor :Georges Dumont Team :IRISA - SIAMES Thursday 7 December 2006

2. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 2 Where are they found ? Animation Motion sciences Robotics Anthropology Simulation Video games Movies Biomechanics Medicine, Health Sports Bipedal robot Modern human Fossilised hominids Motion analysis Motion synthesis

3. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 3 Fondamental aspects Modeling the human Symbolical and controlable representation Kinematical and physical properties Modeling the motion Manipulable mathematical representation Dependant of the editing methods Editing methods Based on motions laws Looking for generiness

4. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 4 Outline Related works Overview and motivations Analysis / synthesis loop Kinematical adaptation Evaluation of the dynamics Forward dynamics synthesis Conclusion and future work

5. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 5 Outline Related works Overview and motivations Analysis / synthesis loop Kinematical adaptation Evaluation of the dynamics Forward dynamics synthesis Conclusion and future works

6. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 6 Modeling a virtual human Consensus between the anatomical reality and motion control Manipulation of the rotational degrees of freedom Simplification through a hierarchical representation of rigid bodies [H-Anim 06] Mechanical joints are perfect and the number of limbs is limited l_mid_foot r_hand_center vt1 l_hip l_knee l_ankle l_subtalar l_toe r_hip r_knee r_ankle r_subtalar r_mid_foot r_toe vl2 vl3 vt4 vt5 vt6 head_top r_clav l_hand_center

7. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 7 Biomechanical modeling The physical properties describe the movement capacity of the limbs At least the masses and inertias These data are avalaible from anthropometrical tables and computable from regression laws [Vaughan et al. 99] Cadaverical data [Dempster 55, Winter 90] Gamma radiography [Zatsiorsky 90] Various definitions of limbs [Chandler et al. 75, De Leva 96] Additionnal data Articular limits, muscular activations [Liu et al. 05], articular elasticities

8. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 8 Modeling the motion Motion is a sequence of postures Mareys work [Marey 1894] on motion decomposition Kinematical model Positions, velocities and acceleration Description of the possibility of motions Static and quasi-static (kinetical) models Usefull for slow motions Balance of internal and external forces Dynamical model Use of motor forces at joints

9. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 9 Methods of kinematics and inverse kinematics Representation by splines [Zeltzer 82, Bruderlin and Calvert 93] Local linearisation and secondary tasks [Boulic and Thalmann 92] [Tolani and Badler 96] Motion editing methods θ1θ1 θ2θ2 X

10. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 10 Motion editing methods Kinematics and control of the center of mass Important on quasi-static positions [Phillips 91] Projection of the center of mass on the sustentation polygon [Boulic et al. 94] Resolution using inverse kinematics with a priority formulation [Boulic et al. 97] Conservation of the Zero Moment Point Point of null result of linear momentum of the limbs [Tak et al. 00] Dynamics filtering of motions [Yamane and Nakamura 03]

11. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 11 Motion editing methods Methods of dynamics and inverse dynamics Animation engine using a system of secondary order differential equations [Hodgins 98] Newtons laws of motion and fondamental physical laws Virtual works and Lagrangian formalism [Rémion 00]

12. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 12 Manipulation of real movements Real movements intrinsically have all of the information of the motion Correct perception of the realism from few positions of caracteristical joints [Johansson 73] Good realism of animations Low generiness Usable database for generating new motions

13. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 13 Manipulation of real movements Adaptation to new characters: retargeting

14. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 14 Manipulation of real movements Adaptation to new characters: retargeting With different morphologies [Gleicher 98] thanks to spacetime constraints Use of intermediate skeletons [Monzani et al. 00, Ménardais 03] Decomposition of articular trajectories into hierarchical splines [Lee and Shin 99] Take account of muscular forces [Komura et al. 00]

15. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 15 Manipulation of real movements Modification of the motion Frequential description of the articular trajectories and deformation function [Bruderlin and Williams 95] displacement map, conservation of the high frequencies Description by key postures and interpolation on the parameters [Witkin and Popovic 95] Interpolation of the deformation function (scale and translation)

16. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 16 Manipulation of real movements Combining motions The motion graphs [Kovar et al. 02] node = key posture arc = possible transition Motion blending Synchronisation by dynamic time warping [Bruderlin and Williams 95] Blending by linear combinaison or weighted sum [Guo and Robergé 96, Park et al. 02, Rose et al. 98, Ménardais et al. 04]

17. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 17 Manipulation of real movements Motion database Efficient interpolation on simple motions [Wiley et Hahn 97] Behavioral or frequential decomposition Fourier space [Unuma et al. 95] Radial basis function [Rose et al. 96] Hidden Markov chains [Brand and Hertzmann 00] Static models using PCA [Bowden 00] Physical simulations [Zordan et al. 05, Arikan et al. 05, Tang et al. 06]

18. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 18 Bipedal locomotion The locomotion is a cyclic movement Decomposable into phases [Marrey 1894, Plat and Veil 83] A step is a half cycle walk = sequence of single and double support phases run = sequence of single support and flying phases Specific modeling of locomotion With cyclic state machines [Multon 98] With deformed hypertorus from PCA [Martineau 06] Large number of biomechanical data Articular trajectories [Alexander 84, Patla 91] Support phases [Girard 87]

19. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 19 Summary The analysis and the synthesis of virtual human motions [Gibet 02] answer to very differents constraints according to the application Compromise interactivity / realism / generiness Kinematical correctionsInteractivity of natural looking motions Dynamical constraintsPhysical realism on specific motions Spacetime resolutionOffline and kinematical high control Physics-based simulationGeneric offline production of motions Motions editingBalanced applications

20. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 20 Outline Related works Overview and motivations Analysis / synthesis loop Kinematical adaptation Evaluation of the dynamics Forward dynamics synthesis Conclusion and future works

21. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 21 Overview Motivation : to study and to realise a process of modeling tools and motion computation Application to locomotion Database of motions Adaptation algorithm Analysis of the dynamics Synthesis by forward dynamics adapted motion resulting forces synthetised motion

22. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 22 Outline Related works Overview and motivations Analysis / synthesis loop Kinematical adaptation Evaluation of the dynamics Forward dynamics synthesis Conclusion and future works

23. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 23 Kinematical adaptation Adaptation algorithm adapted motion Database of motions Analysis of the dynamics Synthesis by forward dynamics resulting forces synthetised motion Database of motions Analysis of the dynamics Synthesis by forward dynamics resulting forces synthetised motion Database of motions Speed profilesScaleAnatomical data Footprints Rest posture Articular limits Interpolations - transversal - frontal - sagital Time adjustement and Synchronisation Post-treatments Adapted motion Kinematical and Dimensionnal Interpolation

24. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 24 Modeling the human Definition of a kinematical chain with 11 dof Spherical joints at pelvis and hips Pin joints at knees right femur reference frame world pelvis reference frame left femur reference frame left tibia reference frame right tibia reference frame 3 rotations 1 rotation

25. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 25 Why use this model ? Application field in paleoanthropology Study of the bipedalism of fossilised hominids Australopithecus afarensis Lucy (A.L. 288-1) pictures – courtesy of G. Berillon

26. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 26 Modeling the locomotion Treatments on the motion Homogeneous reconstruction [Ménardais 03] Orientation of the locomotion Identification of the cycles Definition of the movement of the articular centers Computed from real landmarks Accurate positions of articular centers and virtual points 1 1 2 2 3 3 4 4

27. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 27 Modeling the locomotion A parametrical representation of the locomotion: the poulaine Modeled by a cubic curve using 4 characteristic points of the cycle 1 2 3 4 Definition: the Cartesian displacement of the ankle in the root reference frame

28. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 28 Method of computation The principle of dimensional interpolation in the database Definition of the step size on x Definition of the step shift on y Definition of the rest posture on z

29. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 29 Post-treatments Adding the temporal dimension Use of an average profile of speed, normalised by the time cycle and distance on the ground Representation of the profile by a polynomial function Integrating the function, computing the curvilinear x- coordinate and the parameters of the cubic curves Synchronisation of the left and right poulaines By minimisation of vertical differences By minimisation of ground sliding

30. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 30 Post-treatments Computation of the postures by an IK solver Proposition of secondary tasks [Nicolas et al. 04] (C1) Maximal distance from joint limits (C2) Minimisation of the kinematical energy of rotation (C3) Search of the closest posture to the rest posture Evaluation of these tasks by 3 criteria The total Jerk, third derivate of the angles The difference between the final and the initial posture The internal work

31. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 31 Post-treatments Preparation of the animation Construction of the foot and the ankle angle Feet lenghts from anthropometrical tables [De Leva 96] Trajectories of ankles computed by corrections of the ground penetrations To go to the global motion Global displacement minimising the sliding Upper body movement Adapted to the morphology and synchronised with the real motion

32. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 32 Results of the adaptation Validation of the interpolation x axis z axisAverage RMS (cm)2.741.221.241.73 S.D. (cm)1.181.120.760.82

33. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 33 Results of the adaptation Validation of the adaptation Comparison between real angular trajectories and adapted trajectories of 7 subjects in the database Pelv. incl.Pelv. obliq.Pelv. rot. int/extHip flex/ext Average (rad)0.030.01 0.05 S.D. (rad)0.030.01 0.03 Hip abd/addHip rot. int/extKnee flex/ext Average (rad)0.040.030.02 S.D. (rad)0.040.01

34. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 34 Results of the adaptation Simulations

35. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 35 Partial summary The method computes a plausible locomotion from biomechanical knowledge and rules [Pronost et al. 05] Controled by limbs sizes, bones configuration, physical parameters of the limbs, joints types, footprints, articular limits and the style of motion Applied to paleoanthropology [Pronost et al. 06] Future work Combining the method with a real time adaptation to the environment To drive the extrapolation by physical properties A global resolution of the IK to reduce discontinuity of pelvis angles Increase the size and the diversity of the database

36. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 36 Outline Related works Overview and motivations Analysis / synthesis loop Kinematical adaptation Evaluation of the dynamics Forward dynamics synthesis Conclusion and future works

37. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 37 Evaluation of the dynamics Analysis of the dynamics resulting forces Adaptation algorithm adapted motion Database of motions Synthesis by forward dynamics synthetised motion Database of motions Synthesis by forward dynamics synthetised motion Adaptation algorithm adapted motion Mechanical model Adapted motion Scaled anthropometrical table Biomechanical model Resulting forces and torques Analysis of the dynamics Mapping dimensional model angle-based motion Support phases recognition

38. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 38 Modeling the human Creation of a biomechanical model Description of Denavit-Hartenberg [Hartenberg and Denavit 55] Parameters of rotation: user Parameters of translation: auto Gender and nature of limbs Gender: user Nature: auto Using anthropometrical tables [deLeva 96] and regression laws [Vaughan et al. 99]

39. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 39 Modeling the motion The mapping issue An iterative method on kinematical chains from the root joint to the effectors Using a sequencing of the articular systems Treatment according to the number of degrees of freedom 1dof => pin joint, minimisation of the error 3 dof => spherical joint, an infinity of solutions with constraints => minimisation of the future error without constraints => minimal rotation

40. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 40 Method of computation In order to solve the inverse dynamics issue, we have to know the external forces applied to the system, for locomotion: the gravity constant value for any motion the aerodynamical forces supposed negligible the ground reaction forces When are they applied ? Support phase recognition

41. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 41 Support phase recognition Evaluation of 4 methods of ground contacts recognition hand-labeled, method of reference, accurate at the frequency of the motion capture system speed, evaluation of a speed threshold for the effectors height, evaluation of a height threshold for the effectors configuration, particular configurations of the effectors By four criteria the number of failures the average error its S.D. the normalised S.D. of the thresholds

42. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 42 Support phase recognition Results with 12 x 2 (left/right) x 2 (flex/ext) = 48 contacts Results of the thresholds estimation with heels and toes effectors Our algorithm chooses the best method according to the configuration of the effectors and the evaluation of the criteria Criteria/rank123 Number of failure(0) speed(0) config(16) Height Average error(2.3) speed(5.6) height(9.7) config S.D.(1.6) speed(2.1) config(2.4) height S.D. norm(-) config(0.41) height(0.5) speed Heel strikeToe off Height (%)13.013.37 Speed (m/s)0.730.39

43. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 43 Application of Newtons law Application of the FBD principle Free Body Diagram on each segment Study of forces and torques applied to the limbs Application of Newtons second law of motion on limbs s

44. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 44 Resolution of the equations The translation form Single support, from the free foot to the support foot Double support, global resolution No support, independent resolution The rotation form Iterative resolution of the equation:

45. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 45 Validation of the resolution Forces (in N) at left toe and torques (in N.m) at left knee of 6 adapted locomotions Forces (N) Torques (N.m) cycle %

46. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 46 Validation of the resolution Comparison between 3 real ground reaction forces (black plots) and analysed forces (blue plot) from characters with similar biomechanical properties GRF (N) support phase

47. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 47 Validation of the resolution Ground reaction forces of different styles of real locomotions run jump walk

48. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 48 Influence of the retargeting The global scale Most used parameter Large influence on the dynamics of the motion Linear relation (c.c. = 0.87) between the scale and the relative values of the GRF Experimental validation between [0.7, 1.2] scales GRF (N) cycle %global scale relative GRF

49. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 49 Influence of the retargeting The femur/tibia ratio To evaluate errors on articular centers (relative length of limbs) Experimental validation between [0.8, 1.2] ratios The GRF are not compensated by relative lenghts of the limbs RMS error cycle % femur/tibia ratio GRF (N)

50. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 50 Influence of the retargeting The structure of the skeleton Models with 33 and 21 dof (pin joints at knees, ankles and elbows) Kinematical influence: 1.4 cm per limbs Dynamical influence: mostly on fore-aft acceleration Here, corresponds to 2.5 kg reduction (4.5 % of the mass) GRF (N) cycle % RMS error

51. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 51 Influence of the kinematical interpolation The step size To study the kinematical correction of support phases Results are in agreement with the interpolation: stage of valid corrections (database of motions) To improve this database with more motions, with more step sizes GRF (N) cycle % step size factor RMS error

52. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 52 Influence of the kinematical interpolation The motion style Defined by the rest posture, i.e. the erect percentile More bent style: nonsignificant errors More erect style: increase of the error Large steps can not be done with a hight erect posture GRF (N) cycle % erect percentile factor RMS error

53. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 53 Influence of the kinematical interpolation The character velocity Used for generating new motions Velocity, error but physics OK Velocity, double hump amplitude Need to change the style of locomotion (ex: run) GRF (N) cycle % velocity factor RMS error

54. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 54 Partial summary Method of evaluation of the dynamics influences of a locomotion editing method [Pronost and Dumont 06 a] The analysis is automatic, generic and independent [Pronost and Dumont 06 b] Applied on a retargeting approach and a kinematical interpolation in a database The editing method is valid on the experimental range of validation Future work To validate the analysis using more experimental data To overcome standard limitation of inverse dynamics issue, such as complex articular systems and muscular activation To use the dynamics-based analysis to correct motions

55. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 55 Outline Related works Overview and motivations Analysis / synthesis loop Kinematical adaptation Evaluation of the dynamics Forward dynamics synthesis Conclusion and future work

56. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 56 Forward dynamics synthesis Analysis of the dynamics resulting forces Adaptation algorithm adapted motion Database of motions Synthesis by forward dynamics synthetised motion Database of motions Adaptation algorithm adapted motion Analysis of the dynamics resulting forces Symbolical equations of motion Resolution of the equations Synthesised motion Synthesis by forward dynamics Dof variations Initial state External forces Motor torques Mechanical system Velocities and accelerations of the dof Ground reaction forces Bodies: masses, inertias, matrix Joints: type, mechanical parameters Dof values

57. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 57 Modeling Of the human Representation of the rigid bodies masses, inertias and rotation matrix Representation of the mechanical joints type: free (6 dof), spherical (3 dof), pin (1 dof) mechanical parameters spring, damper, and limits Of the motion Values of the degrees of freedom of the mechanical system

58. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 58 Normalisation of the forces and torques Principle Preliminary study on 5 motions Method of computation Motion Inverse dynamics analysis Motion Morphology (mass, inertia) Normalised forces & torques Motion Forward dynamics synthesis Morphology { Subject 1Subject 2 NormalN 1,1,N 1,2 N2N2 BentF1F1 F2F2 N 1,1 N 1,2 F1F1 N2N2 F2F2 RMS(1)00.2340.6210.5121.001 M.E.00.0260.0680.0560.109 M.STDP (x10 -3 )08.713.93.523.0 C.C.10.9730.6810.9800.399 C.ACP10.9950.3220.3320.328 Deriv (x10 -4 )012.023.06.442.0 N 1,1 N 1,2 F1F1 N2N2 F2F2 RMS(1)00.1500.4260.1410.946 M.E.00.0250.0710.0260.148 M.STDP (x10 -2 )022.655.622.9129.3 C.C.10.9380.8260.9540.702 C.ACP10.9450.8480.9620.802 Deriv (x10 -3 )036.574.137.3162.2 Differences and similarities of root positions Differences and similarities of normalised GRF

59. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 59 Preliminary results The forward dynamics resolution using the NMECAM library [Arnaldi 89, Dumont 90] library defining a symbolic solver of motion equations Results on a virtual human without external forces and with independent limbs

60. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 60 Partial summary The whole of the principles are not yet implemented Apprehension of the phenomena of the forward dynamics issue Usefull approach of normalisation in many fields such as kinematics and kinetics synthesis To make choices according to physical properties of the synthetised motion Future work Simulation of a complete chain of a human with external forces To organise a database including physical properties such as normalised profiles of forces and torques

61. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 61 Outline Related works Overview and motivations Analysis / synthesis loop Kinematical adaptation Evaluation of the dynamics Forward dynamics synthesis Conclusion and future work

62. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 62 Conclusion To understand and to simulate human motions Analysis and synthesis approaches Many applicatives fields animation, biomechanics, mechanics, anthropology Kinematical adaptation of captured motions preserving the physical credibility of the synthetised motions Proposition of methods and algorithms combining kinematical and dynamical approaches Adapted to the morphology and the locomotion Normalisation of the data, of the morphological structures and the forces and torques Use of principles and data from biomechanics

63. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 63 Limitations and future work On the methods Study of other motions with external forces To increase the size and the diversity of the database and the experimental data for the interpolation and the physical validation To use the analysis to drive the kinematical synthesis thanks to physical simulations To do a unique method with the proposed approaches On the applications Motion editing, biomechanics, anthropology But also on the modeling and the use of data on virtual humans

64. Nicolas Pronost – Definition and realisation of modeling methods and motion computation algorithms for virtual humans 64 Thank you for your attention Definition and realisation of modeling methods and motion computation algorithms for virtual humans Nicolas Pronost Advisor :Bruno Arnaldi Co-advisor :Georges Dumont Team :IRISA - SIAMES