1. Character Animation and Control using Human Motion Data Jehee Lee Carnegie Mellon University http://www.cs.cmu.edu/~jehee

2. Character Animation Final Fantasy Movie Characters from www.finalfantasy.com Final Fantasy XNBA Courtside 2002NFL 2k2WWF Raw All game characters from www.gamespot.com

3. Motion Capture Record movements of live performers –Realistic, highly detailed data can be obtained Motion capture lab at CMU

4. Animation from Motion Capture Motion Database Preprocess On-line Controller Motion Editing Toolbox Motion Sensor Data Convincing Animation Controllable Responsive Characters High-Level User Interfaces The Art of Animation

5. Animation from Motion Capture Motion Database Preprocess On-line Controller Motion Editing Toolbox Motion Sensor Data Convincing Animation Controllable Responsive Characters Mapping Live Performance High-Level User Interfaces The Art of Animation Computer Puppetry

6. Interactive 3D Avatar Control How to organize data ? –Large collection of motion data How to control ? –User interfaces Motion Database Preprocess On-line Controller Motion Sensor Data Controllable Responsive Characters High-Level User Interfaces

7. Related Work (Motion Control) Rule-basedControl system [Bruderlin & Calvert 96] [Perlin & Goldberg 96] [Chi 2000] [Cassell et at 2001] [Hodgins et al 95] [Wooten and Hodgins 96] [Laszlo et al 96] [Faloutsos et al 2001] Example-basedStatistical Models [Popovic & Witkin 95] [Bruderlin & Willams 95] [Unuma et al 95] [Lamouret & van de Panne 96] [Rose et al 97] [Wiley & Hahn 97] [Gleicher 97, 98, 01] [Sun & Mataxas 2001] [Bradley & Stuart 97] [Pullen & Bregler 2000] [Tanco & Hilton 2000] [Brand & Hertzmann 2000] [Galata et al 2001] [Lee et al 02]

8. Related Work (User Interfaces) Graphical User Interfaces Performance (Motion capture devices) Performance (Vision-based) [Bruderlin & Calvert 96] [Laszlo et al 96] [Rose et al 97] [Chi 2000] [Badler et al 93] [Semwal et al 98] [Blumberg 98] [Molet et al 99] “Mocap Boxing” (Konami) [Blumberg & Galyean 95] [Brand 1999] [Rosales et al 2001] [Ben-Arie et al 2001]

9. Motion Database In computer games –Many short, carefully planned, labeled motion clips –Manual processing

10. Walk CycleStopStart Left Turn Right Turn

11. Motion Database Our approach –Extended, unlabeled sequences of motion –Automatic processing

13. Jehee Lee, Jinxiang Chai, Paul Reitsma, Jessica Hodgins, and Nancy Pollard, Interactive Control of Avatars Animated with Human Motion Data, submitted. Sketch Interface

14. Motion Data for Rough Terrain

16. Unstructured Input Data

17. Connecting Transitions

18. Local Search for Path Following

19. Comparison to Real Motion

21. User Interfaces

22. Choice-based Interface What is available in database ? –Provided with several options –Select from among available behaviors

23. Jehee Lee, Jinxiang Chai, Paul Reitsma, Jessica Hodgins, and Nancy Pollard, Interactive Control of Avatars Animated with Human Motion Data, submitted.

25. How to Create Choices ?

26. Clustering

27. Find Reachable Clusters A B C D E F G

28. Most Probable Paths

29. Cluster Forest B C D E F G B D E F G

30. Performance Interface Motion Database Search Engine Animate Avatars Vision-based Interface

31. Silhouette extraction and matching implemented by Jinxiang Chai

32. Database Search 3 sec

33. Animation from Motion Capture Motion Database Preprocess On-line Controller Motion Editing Toolbox Motion Sensor Data Convincing Animation Controllable Responsive Characters Mapping Live Performance High-Level User Interfaces The Art of Animation Computer Puppetry

34. The Art of Animation Animators need good tools –Modify, vary, blend, transition, filter, … Motion Database Motion Editing Toolbox Convincing Animation The Art of Animation

35. Challenges in Motion Editing Reusability and flexibility –Motion data is acquired For a specific performer Within a specific environment In a specific style/mood High dimensionality Inherent non-linearity of orientation data

36. Related Work Physically- based Signal processing/ Interpolation Optimization + Interpolation Stochastic Modify [Popovic& Witkin 99] [Unuma et al 95] [Bruderlin & Williams 95] [Sun&Metaxas 01] [Lee & Shin 01, 02] [Gleicher 97, 98, 01] [Lee & Shin 99] [Perlin 95] [Bradley&Stuart 97] [Pullen&Bregler 00] Transition/ Blend [Rose et al 96] [Lamouret & van de Panne 96] [Rose et al 97] [Sun&Metaxas 01] [Lee & Shin 01, 02] [Tanco&Hilton 00] [Brand & Hertzmann 00] [Galata et al 01]

37. Basic Techniques Multiresolution Analysis –Signal processing approach –Transition, blend, modify style/mood, and resequence Hierarchical displacement mapping –Constraint-based approach –Interactive editing –adaptation to different characters/environments.

38. Multiresolution Analysis Represent signals at multiple resolutions –give hierarchy of successively smoother signals –facilitate a variety of signal processing tasks

39. Decomposition Reduction: upsampling followed by smoothing Expansion: smoothing followed by downsampling ReductionExpansion

40. Decomposition Reconstruction

41. Enhance / Attenuate Jehee Lee and Sung Yong Shin, General Construction of Time- Domain Filters for Orientation Data, IEEE Transactions on Visualization and Computer Graphics, to appear. Jehee Lee and Sung Yong Shin, A Coordinate-Invariant Approach to Multiresolution Motion Analysis and Synthesis, Graphical Models (formerly GMIP), 2001.

42. Enhance / Attenuate

43. Walk Limp Turn ? Turn with a Limp

44. Walk Limp Turn Turn with a Limp

45. Analogy Low frequency (Content) Result = Limp + (Turn – Walk) High frequency (Style) Result = Turn + (Limp – Walk) WalkTurn Limp Turn with A limp

46. Walk Strut Run

47. Stub toesLimp Stitched

48. Re-sequence

49. Reconstruction

52. Orientation Representation Inherent non-linearity of orientation space

53. Filtering Orientation Data How to generalize convolution filters ?

54. Related Work Re-normalization Azuma and Bishop (94) Global linearization Johnstone and Williams (95) Local linearization Welch and Bishop (97) Fang et al. (98) Lee and Shin (2002) Multi-linear Shoemake (85) Optimization Lee and Shin (96) Hsieh et al. (98) Buss and Fillmore (2001)

55. Re-normalization

56. Linearization

57. Exponential and Logarithm

60. logexp

61. Global Linearization

62. Angular Displacement

65. Local Linearization

66. The Drifting problem

67. Our Approach

68. Filtering Orientation Data

69. Filter Properties Coordinate invariant Time invariant Symmetric

70. Coordinate Invariance Decomposition Reconstruction

71. Coordinate Invariance Independent to the choice of coordinate systems

72. Basic Techniques Multiresolution Analysis –Signal processing approach –Transition, blend, modify style/mood, smoothen, resequence Hierarchical displacement mapping –Constraint-based approach –Interactive editing and adaptation

73. Motion Editing through Optimization Constraints [Witkin & Kass 88] [Cohen 92] [Gleicher 98] –Features to be retained –New features to be accomplished Find a new motion –Satisfy given constraints –Preserve original characteristics

74. Jehee Lee and Sung Yong Shin, A Hierarchical Approach to Interactive Motion Editing for Human-Like Figures, Siggraph 99

75. Motion Representation Motion of articulated characters –Bundle of motion signals –Each signal describe positions / orientations / joint angles

76. Basic Idea Inter-frame relationship –Enforce constraints –By inverse kinematics Inter-frame relationship –Avoid jerkiness –By curve fitting

77. Displacement Mapping Displacement Map Original Motion Target Motion

78. Hierarchical Displacement Mapping Representation of displacement maps –An array of spline curves –Over a common knot sequence Flexibility in representation –Hard to determine knot density –Adaptive refinement is needed

79. Adaptive Refinement Multi-level or hierarchical B-splines [Lee, Wolberg, and Shin 97] [Forsey and Bartel 95] –Sum of uniform B-spline functions –Coarse-to-fine hierarchy of knot sequences

80. Multi-Level B-spline Fitting

81. Adaptation to Rough Terrain Jehee Lee and Sung Yong Shin, A Hierarchical Approach to Interactive Motion Editing for Human-Like Figures, Siggraph 99

82. Adaptation to New Characters

83. Character Morphing

84. Animation from Motion Capture Motion Database Preprocess On-line Controller Motion Editing Toolbox Motion Sensor Data Convincing Animation Controllable Responsive Characters Mapping Live Performance High-Level User Interfaces The Art of Animation Computer Puppetry

85. Hyun Joon Shin, Jehee Lee, Michael Gleicher, and Sung Yong Shin, Computer Puppetry: An Importance-based Approach, ACM Transactions on Graphics, 2001. The videos were made by Hyun Joon Shin, Tae Hoon Kim, Hye-Won Pyun, Seung-Hyup Shin, Jehee Lee, Sung Yong Shin, and many others at the Korea Broadcasting System.

86. Summary Motion data processing –Multiresolution analysis –Hierarchical displacement mapping Interactive control –Motion databases –User interfaces: Choice, sketch, performance

87. Future Work Autonomous virtual humans –Convincing appearance, movements –Reasonable level of intelligence Collect real world data –Motions, pictures, videos, voices, facial expressions, and physical properties

89. Computer Puppetry Immediate mapping from a performer to an animated character Motion Sensor Data Mapping Live Performance Computer Puppetry

90. Time Invariance Independent to the position in the signal Time

91. Statistical Model

93. Motion Representation Statistical Model Markov Process User Control Update Avatar Pose

94. Markov Process Raw data –Extended –Unstructured Processed data –Connected –Flexible

95. Cluster Forest