Character Animation and Control using Human Motion Data Jehee Lee Carnegie Mellon University

Character Animation Final Fantasy Movie Characters from Final Fantasy XNBA Courtside 2002NFL 2k2WWF Raw All game characters from

Motion Capture Record movements of live performers –Realistic, highly detailed data can be obtained Motion capture lab at CMU

Animation from Motion Capture Motion Database Preprocess On-line Controller Motion Editing Toolbox Motion Sensor Data Convincing Animation Controllable Responsive Characters High-Level User Interfaces The Art of Animation

Animation from Motion Capture Motion Database Preprocess On-line Controller Motion Editing Toolbox Motion Sensor Data Convincing Animation Controllable Responsive Characters Mapping Live Performance High-Level User Interfaces The Art of Animation Computer Puppetry

Interactive 3D Avatar Control How to organize data ? –Large collection of motion data How to control ? –User interfaces Motion Database Preprocess On-line Controller Motion Sensor Data Controllable Responsive Characters High-Level User Interfaces

Related Work (Motion Control) Rule-basedControl system [Bruderlin & Calvert 96] [Perlin & Goldberg 96] [Chi 2000] [Cassell et at 2001] [Hodgins et al 95] [Wooten and Hodgins 96] [Laszlo et al 96] [Faloutsos et al 2001] Example-basedStatistical Models [Popovic & Witkin 95] [Bruderlin & Willams 95] [Unuma et al 95] [Lamouret & van de Panne 96] [Rose et al 97] [Wiley & Hahn 97] [Gleicher 97, 98, 01] [Sun & Mataxas 2001] [Bradley & Stuart 97] [Pullen & Bregler 2000] [Tanco & Hilton 2000] [Brand & Hertzmann 2000] [Galata et al 2001] [Lee et al 02]

Related Work (User Interfaces) Graphical User Interfaces Performance (Motion capture devices) Performance (Vision-based) [Bruderlin & Calvert 96] [Laszlo et al 96] [Rose et al 97] [Chi 2000] [Badler et al 93] [Semwal et al 98] [Blumberg 98] [Molet et al 99] “Mocap Boxing” (Konami) [Blumberg & Galyean 95] [Brand 1999] [Rosales et al 2001] [Ben-Arie et al 2001]

Motion Database In computer games –Many short, carefully planned, labeled motion clips –Manual processing

Walk CycleStopStart Left Turn Right Turn

Motion Database Our approach –Extended, unlabeled sequences of motion –Automatic processing

Jehee Lee, Jinxiang Chai, Paul Reitsma, Jessica Hodgins, and Nancy Pollard, Interactive Control of Avatars Animated with Human Motion Data, submitted. Sketch Interface

Motion Data for Rough Terrain

Unstructured Input Data

Connecting Transitions

Local Search for Path Following

Comparison to Real Motion

User Interfaces

Choice-based Interface What is available in database ? –Provided with several options –Select from among available behaviors

Jehee Lee, Jinxiang Chai, Paul Reitsma, Jessica Hodgins, and Nancy Pollard, Interactive Control of Avatars Animated with Human Motion Data, submitted.

How to Create Choices ?

Clustering

Find Reachable Clusters A B C D E F G

Most Probable Paths

Cluster Forest B C D E F G B D E F G

Performance Interface Motion Database Search Engine Animate Avatars Vision-based Interface

Silhouette extraction and matching implemented by Jinxiang Chai

Database Search 3 sec

Animation from Motion Capture Motion Database Preprocess On-line Controller Motion Editing Toolbox Motion Sensor Data Convincing Animation Controllable Responsive Characters Mapping Live Performance High-Level User Interfaces The Art of Animation Computer Puppetry

The Art of Animation Animators need good tools –Modify, vary, blend, transition, filter, … Motion Database Motion Editing Toolbox Convincing Animation The Art of Animation

Challenges in Motion Editing Reusability and flexibility –Motion data is acquired For a specific performer Within a specific environment In a specific style/mood High dimensionality Inherent non-linearity of orientation data

Related Work Physically- based Signal processing/ Interpolation Optimization + Interpolation Stochastic Modify [Popovic& Witkin 99] [Unuma et al 95] [Bruderlin & Williams 95] [Sun&Metaxas 01] [Lee & Shin 01, 02] [Gleicher 97, 98, 01] [Lee & Shin 99] [Perlin 95] [Bradley&Stuart 97] [Pullen&Bregler 00] Transition/ Blend [Rose et al 96] [Lamouret & van de Panne 96] [Rose et al 97] [Sun&Metaxas 01] [Lee & Shin 01, 02] [Tanco&Hilton 00] [Brand & Hertzmann 00] [Galata et al 01]

Basic Techniques Multiresolution Analysis –Signal processing approach –Transition, blend, modify style/mood, and resequence Hierarchical displacement mapping –Constraint-based approach –Interactive editing –adaptation to different characters/environments.

Multiresolution Analysis Represent signals at multiple resolutions –give hierarchy of successively smoother signals –facilitate a variety of signal processing tasks

Decomposition Reduction: upsampling followed by smoothing Expansion: smoothing followed by downsampling ReductionExpansion

Decomposition Reconstruction

Enhance / Attenuate Jehee Lee and Sung Yong Shin, General Construction of Time- Domain Filters for Orientation Data, IEEE Transactions on Visualization and Computer Graphics, to appear. Jehee Lee and Sung Yong Shin, A Coordinate-Invariant Approach to Multiresolution Motion Analysis and Synthesis, Graphical Models (formerly GMIP), 2001.

Enhance / Attenuate

Walk Limp Turn ? Turn with a Limp

Walk Limp Turn Turn with a Limp

Analogy Low frequency (Content) Result = Limp + (Turn – Walk) High frequency (Style) Result = Turn + (Limp – Walk) WalkTurn Limp Turn with A limp

Walk Strut Run

Stub toesLimp Stitched

Re-sequence

Reconstruction

Orientation Representation Inherent non-linearity of orientation space

Filtering Orientation Data How to generalize convolution filters ?

Related Work Re-normalization Azuma and Bishop (94) Global linearization Johnstone and Williams (95) Local linearization Welch and Bishop (97) Fang et al. (98) Lee and Shin (2002) Multi-linear Shoemake (85) Optimization Lee and Shin (96) Hsieh et al. (98) Buss and Fillmore (2001)

Re-normalization

Linearization

Exponential and Logarithm

logexp

Global Linearization

Angular Displacement

Local Linearization

The Drifting problem

Our Approach

Filtering Orientation Data

Filter Properties Coordinate invariant Time invariant Symmetric

Coordinate Invariance Decomposition Reconstruction

Coordinate Invariance Independent to the choice of coordinate systems

Basic Techniques Multiresolution Analysis –Signal processing approach –Transition, blend, modify style/mood, smoothen, resequence Hierarchical displacement mapping –Constraint-based approach –Interactive editing and adaptation

Motion Editing through Optimization Constraints [Witkin & Kass 88] [Cohen 92] [Gleicher 98] –Features to be retained –New features to be accomplished Find a new motion –Satisfy given constraints –Preserve original characteristics

Jehee Lee and Sung Yong Shin, A Hierarchical Approach to Interactive Motion Editing for Human-Like Figures, Siggraph 99

Motion Representation Motion of articulated characters –Bundle of motion signals –Each signal describe positions / orientations / joint angles

Basic Idea Inter-frame relationship –Enforce constraints –By inverse kinematics Inter-frame relationship –Avoid jerkiness –By curve fitting

Displacement Mapping Displacement Map Original Motion Target Motion

Hierarchical Displacement Mapping Representation of displacement maps –An array of spline curves –Over a common knot sequence Flexibility in representation –Hard to determine knot density –Adaptive refinement is needed

Adaptive Refinement Multi-level or hierarchical B-splines [Lee, Wolberg, and Shin 97] [Forsey and Bartel 95] –Sum of uniform B-spline functions –Coarse-to-fine hierarchy of knot sequences

Multi-Level B-spline Fitting

Adaptation to Rough Terrain Jehee Lee and Sung Yong Shin, A Hierarchical Approach to Interactive Motion Editing for Human-Like Figures, Siggraph 99

Adaptation to New Characters

Character Morphing

Animation from Motion Capture Motion Database Preprocess On-line Controller Motion Editing Toolbox Motion Sensor Data Convincing Animation Controllable Responsive Characters Mapping Live Performance High-Level User Interfaces The Art of Animation Computer Puppetry

Hyun Joon Shin, Jehee Lee, Michael Gleicher, and Sung Yong Shin, Computer Puppetry: An Importance-based Approach, ACM Transactions on Graphics, The videos were made by Hyun Joon Shin, Tae Hoon Kim, Hye-Won Pyun, Seung-Hyup Shin, Jehee Lee, Sung Yong Shin, and many others at the Korea Broadcasting System.

Summary Motion data processing –Multiresolution analysis –Hierarchical displacement mapping Interactive control –Motion databases –User interfaces: Choice, sketch, performance

Future Work Autonomous virtual humans –Convincing appearance, movements –Reasonable level of intelligence Collect real world data –Motions, pictures, videos, voices, facial expressions, and physical properties

Computer Puppetry Immediate mapping from a performer to an animated character Motion Sensor Data Mapping Live Performance Computer Puppetry

Time Invariance Independent to the position in the signal Time

Statistical Model

Motion Representation Statistical Model Markov Process User Control Update Avatar Pose

Markov Process Raw data –Extended –Unstructured Processed data –Connected –Flexible

Cluster Forest