1. Advanced Programming for 3D Applications CE00383-3 Bob Hobbs Staffordshire university Motion Capture Lecture 2 Week 3

2. 2 Definition of Motion Capture Motion capture is the recording of human body movement (or other movement) for immediate or delayed analysis and playback. “MoCap”

3. 3 History Late 1970’s – Rebecca Allen created the first primitive form of motion capture. “Rotoscoping” Late 1970’s – Rebecca Allen created the first primitive form of motion capture. “Rotoscoping” 1980-1983- Tom Calvert invented “Geniometers” 1980-1983- Tom Calvert invented “Geniometers” 1982-1983- Ginsberg & Maxwell invented “Graphical Marionette” 1982-1983- Ginsberg & Maxwell invented “Graphical Marionette” 1988- deGraf/Wahrman invented “Mike the Talking Head” 1988- deGraf/Wahrman invented “Mike the Talking Head”

4. 4 1988- Pacific Data Images invented “Waldo C. Graphic” 1988- Pacific Data Images invented “Waldo C. Graphic” 1989- Kleiser- Walczak invented “Dozo” 1989- Kleiser- Walczak invented “Dozo” 1991- Videosystem invented “Mat the Ghost” 1991- Videosystem invented “Mat the Ghost” 1992- SimGraphics invented “Mario” 1992- SimGraphics invented “Mario” 1992- Brad deGraph invented “Alive” 1992- Brad deGraph invented “Alive” 1993- Acclaim invented two character animation motion capture 1993- Acclaim invented two character animation motion capture

5. 5 Applications Entertainment Entertainment Entertainment Medicine Medicine Medicine Arts / Education Arts / Education Arts / Education Arts / Education Science / Engineering Science / Engineering Science / Engineering Science / Engineering

6. 6 Entertainment: Live Action Films Computer generated characters in live action films (e.g. Battle Droids and many others in Star Wars Prequels, Gullum in The Lord of the Rings, King Kong in King Kong) Computer generated characters in live action films (e.g. Battle Droids and many others in Star Wars Prequels, Gullum in The Lord of the Rings, King Kong in King Kong)

7. 7 Entertainment: 3D computer animations Characters in computer animated files (e.g. Polar Express, Monster House) Characters in computer animated files (e.g. Polar Express, Monster House)

8. 8 Entertainment: Video Games Video games by Electronic Arts, Gremlin, id, RARE, Square, Konami, Namco, and others, (e.g. Enemy Territory) Video games by Electronic Arts, Gremlin, id, RARE, Square, Konami, Namco, and others, (e.g. Enemy Territory)

9. 9 Medicine Medicine (e.g., gait analysis, rehabilitation) Medicine (e.g., gait analysis, rehabilitation) Sports medicine (e.g. injury prevention, performance analyses, performance enhancement) Sports medicine (e.g. injury prevention, performance analyses, performance enhancement) Gait Analysis Service

10. 10 Arts / Education Dance and theatrical performances Dance and theatrical performances Archiving (e.g., Marcel Marceau) Archiving (e.g., Marcel Marceau) OSU/ACCAD

11. 11 Science / Engineering Computer Science (e.g., human motion database, recognitions) Computer Science (e.g., human motion database, recognitions) Engineering (e.g., Biped robot developments) Engineering (e.g., Biped robot developments) Ergonomic product design Ergonomic product design Military (e.g., field exercises, virtual instructors, and role-playing games) Military (e.g., field exercises, virtual instructors, and role-playing games)

12. 12 Types of mocap equipment Magnetic systems Magnetic systems Magnetic systems Magnetic systems Mechanical systems Mechanical systems Mechanical systems Mechanical systems Optical systems Optical systems Optical systems Optical systems

13. 13 Magnetic systems Utilize sensors placed on the body to measure the magnetic field generated by a transmitter source. Utilize sensors placed on the body to measure the magnetic field generated by a transmitter source.

14. 14 Magnetic systems Advantages Advantages –Require no special lighting condition. –Sensors are never occluded. Disadvantages Disadvantages –Require a metal-fee environment. –Electro-magnetic interference

15. 15 Mechanical systems Exoskeleton with angle sensors. Exoskeleton with angle sensors.

16. 16 Mechanical systems Advantages Advantages –Measure joint angles (no marker ID problems). –Sensors are never occluded. Disadvantages Disadvantages –Breakable! –Configuration of sensors is fixed. –Constrains on joints.

17. 17 Optical systems The cameras are equipped with infrared LED's and filters. (Filters enhance the contrast of the image.) The cameras are equipped with infrared LED's and filters. (Filters enhance the contrast of the image.) The cameras see reflector markers. The cameras see reflector markers.

18. 18 Optical systems Advantages Advantages –Higher sampling rate. –Larger capture space. Disadvantages Disadvantages –Markers are sometimes occluded -> marker ID problems. –Provide only positional data -> joint angles need to be computed.

19. 19 Typical Mocap system Vicon optical system - Best system in Academia! Vicon optical system - Best system in Academia! 8 high-speed MX 13 (up to 1000 fps) and 8 high-resolution MX 40 (4 million pixels) cameras. 8 high-speed MX 13 (up to 1000 fps) and 8 high-resolution MX 40 (4 million pixels) cameras. Capture up to 5 performers at once. Capture up to 5 performers at once.

20. 20 Mocap animation Motion capture animation is different from keyframe animation in terms of how motion is created. Motion capture animation is different from keyframe animation in terms of how motion is created. Same principles apply to mocap animation & keyframe animation! Same principles apply to mocap animation & keyframe animation! A combination of motion capture animation and keyframe animation is often used. A combination of motion capture animation and keyframe animation is often used.

21. 21 Keyframe animation A keyframe is a drawing of a key moment in an animated sequence, where the motion is at its extreme. A keyframe is a drawing of a key moment in an animated sequence, where the motion is at its extreme. Inbetweens fill the gaps between keyframes. Inbetweens fill the gaps between keyframes. Every motion is created by animators. Every motion is created by animators.

22. 22 Advantages of mocap animation Faster to create (only if an established production pipeline exists.) Faster to create (only if an established production pipeline exists.) Secondary motions and all the subtle motions are captured -> more realism. Secondary motions and all the subtle motions are captured -> more realism. Physical interactions between performers and props can be captured. Physical interactions between performers and props can be captured.

23. 23 Disadvantages of mocap animation Cost. Cost. Manipulating mocap data is often difficult - > Re-capturing or key framing a shot with bad data is often easier. Manipulating mocap data is often difficult - > Re-capturing or key framing a shot with bad data is often easier. Mapping mocap data of a performer to a character with a different proportion often causes problems. Mapping mocap data of a performer to a character with a different proportion often causes problems.

24. 24 Process of MoCap Data needs to be manipulated Data needs to be manipulated Transform data into file format Transform data into file format Build specific models Build specific models Attaching the mesh Attaching the mesh

25. 25 movement flowchart for games Planning and Directing Motion Capture For Games By Melianthe Kines Gamasutra January 19, 2000 URL: http://www.gamasutra.com/view/feature/3420/planning_and_directing_motion _.php http://www.gamasutra.com/view/feature/3420/planning_and_directing_motion _.php

26. 26 processing passive markers each camera records capture session each camera records capture session extraction: markers need to be identified in the image extraction: markers need to be identified in the image –determines 2d position –problem: occlusion, marker is not seen use more cameras use more cameras markers need to be labeled markers need to be labeled –which marker is which? –problem: crossover, markers exchange labels may require user intervention may require user intervention compute 3d position: if a marker is seen by at least 2 cameras then its position in 3d space can be determined compute 3d position: if a marker is seen by at least 2 cameras then its position in 3d space can be determined

27. 27 Direct Linear Transformation Collinearity Condition B=cA B A

28. 28 Distortion Effects The DLT model of Abdel-Aziz/Karara only accounts for errors in perpendicularity between axes Achieves accuracy of 1:2000 Other types of distortion  Pin Cushion  Barrel Tangential De-centering Radial

29. 29 16 parameter DLT Equations representing non- linear distortions  L12-L14 represent symmetrical lens distortion  L15 & 16 represent asymmetrical/ de-centering distortion

30. 30 The Problem The original DLT method contained 11 unknowns Uo,Vo Xo,Yo,Zo 3 rotation angles 2 scaling factors d Only 10 are actually independent d and the 2 scaling factors are mutually dependent variables

31. 31 Modified Direct Linear Transformation The Non-Linear constraint is added to the system to ensure orthogonality To solve: Compute the 11 parameters normally Use the value of one parameter to remove it from the equation and estimate the 10 independent parameters Solve for the value of the removed parameter using the values of the 10 independent parameters

32. 32 Non-Linear MDLT Adds correction for lens distortion to the MDLT method Adds correction for lens distortion to the MDLT method

33. 33 retargeting: the character the character is controlled by skeleton the character is controlled by skeleton to control the skeleton, need to specify joint rotations to control the skeleton, need to specify joint rotations muscles? muscles?

34. 34 retargeting capture motion on performer capture motion on performer –positions of markers are recorded retarget motion on a virtual character retarget motion on a virtual character –motion is usually applied to a skeleton –a skeleton is hierarchical linked joints linked joints –need rotation data! need to convert positions to rotations need to convert positions to rotations

35. 35 performer → actor → character the actor is used to convert marker positions to rotational data the actor is used to convert marker positions to rotational data –markers are handles on the actor –actor should have similar proportions as the performer joint rotations of the actor are applied to the character joint rotations of the actor are applied to the character there are still issues with proportions there are still issues with proportions Video example Video example Video example Video example Alias Motionbuilder: actor and markers

36. 36 retargeting problems

37. 37

38. 38 Production pipeline overview Calibrate the system. Calibrate the system. Fit a generic skeleton to the subject’s proportion (subject calibration). Fit a generic skeleton to the subject’s proportion (subject calibration). Capture shots & reconstruct 3D trajectories using the calibrated subject. Capture shots & reconstruct 3D trajectories using the calibrated subject. Link the subject specific skeleton to a CG character’s skeleton and edit motion (in MotionBuilder). Link the subject specific skeleton to a CG character’s skeleton and edit motion (in MotionBuilder). Add skin to the CG character, edit motion, and render (in Maya). http://atec.utdallas.edu/midori/Handouts/mocap_pip eline.htm Add skin to the CG character, edit motion, and render (in Maya). http://atec.utdallas.edu/midori/Handouts/mocap_pip eline.htm http://atec.utdallas.edu/midori/Handouts/mocap_pip eline.htm http://atec.utdallas.edu/midori/Handouts/mocap_pip eline.htm

39. 39 2D Image 2D Camera Data 3D Markers Positions Trajectories Capture Range of Motion (ROM)Capture Range of Motion (ROM) Reconstruct trajectories of ROMReconstruct trajectories of ROM Label markers VSK Subject Calibration Actor Setup Place markers on Actor ( intermediate skeleton ) Character Setup Correlate : Actor (C3D, HIK) & Character (FBX) Edit motion Bake motion data to the skeleton HIK FBX Character Setup Create a skeleton Bind skin to the skeleton Rig the character Marge the rigged character (MB) and the skeleton with motion data (FBX) Edit motion (IK/FK blend, Trax)IK/FK blendTrax Render MB FBX C3D Skeleton Only Markers Set Calibrated Subject Skeleton with motion data ( joint rotation angles ) Triangulation IQIQ MAYA MOTION BUILDER Maya Scene File Positional Data System CalibrationSystem Calibration and Capturing and Processing Data Rendering The processes that you go through for each character The processes that you repeat for each shot VST (subject template) Reconstruction Mocap Pipeline Flow Chart Circle fitting

40. 40 References Sturman, David. “A Brief History of Motion Capture for Computer Character Animation.” 13 March 1999. MEDIALAB. 25 Nov. 2005. http://www.siggraph.org/education/materials/HyperGraph/animation/character_anim ation Sturman, David. “A Brief History of Motion Capture for Computer Character Animation.” 13 March 1999. MEDIALAB. 25 Nov. 2005. http://www.siggraph.org/education/materials/HyperGraph/animation/character_anim ation http://www.siggraph.org/education/materials/HyperGraph/animation/character_anim ation http://www.siggraph.org/education/materials/HyperGraph/animation/character_anim ation “When Motion Capture Beats Keyframing.” Sept. 1997. Game Developer. http://www.gdmag.com “When Motion Capture Beats Keyframing.” Sept. 1997. Game Developer. http://www.gdmag.com http://www.gdmag.com http://www.kwon3d.com/theory/dlt/dlt.html http://www.kwon3d.com/theory/dlt/dlt.html http://www.kwon3d.com/theory/dlt/dlt.html http://www.kwon3d.com/theory/dlt/mdlt.html http://www.kwon3d.com/theory/dlt/mdlt.html http://www.kwon3d.com/theory/dlt/mdlt.html http://express.howstuffworks.com/gollum3.htm http://express.howstuffworks.com/gollum3.htm http://express.howstuffworks.com/gollum3.htm

41. 41 Related Work Motion capture search Motion capture search –Arikan and Forsyth 2002; Kovar et al. 2002; Lee et al. 2002; Li et al. 2002; Metoyer 2002 Physics and motion capture Physics and motion capture –Rose et al. 1996 –Popovic and Witkin 1999; Pollard 1999; Pollard and Behmaram-Mosavat 2000 –Shapiro et al. 2003 Physically based reactions Physically based reactions –Oshita and Makinouchi 2001; Zordan and Hodgins 2002 –Komura et al. 2004; Mandel 2004 Industry Software Industry Software –Natural Motion’s Endorphin