Simulation of Characters in Entertainment Virtual Reality

2

3

Role of characters Attractive and Interactive Characters Attractiveness : Express the view of world like a fairytale Interactivity : Enable viewers to feel the character as an real existence = What is needed for Game Characters Tell Story Express Emotion and Intention Physical Interactivity (c.f. Wiimote ©NINTENDO) 4

Role of characters Interactive motion Physics simulation of character body Attractive motion Law of character motion is not clear. In game and anime: hand crafts of animators In dramas and movie: actions of actors We tried two methods Authoring by artist Simulation of attention 5

Range of simulation Physics simulation + keyframe animation Physics simulation Automatic generation of various reaction correspond to various interactions. Key frame animation Designed by animators Physics simulation + simulation of perception, attention and motor control. 6

Physics simulation + keyframe animation 7

Practical use in game Hanging motion in 「 Shadow of the Colossus 」 (2005 Sony Computer Entertainment Inc.) Simulation of spring pendulum + keyframe animation 8

Our method 9 Realtime rigid body simulation Multi dimensional keyframe animation Displacement Slant Invert pendulum control force

Approximation to single rigid body Character as a mass system Preservation of Momentum ＝ (Angular)momentum changed by external forces only. External forces: gravity and contact force Character can changes contact forces only → invert pendulum control is enough 10 dp/dt = F + g dL/dt = r × F L p g F r

Invert pendulum control Control (angular) momentum with force Angular momentum ： L → 0 momentum ： p → mv （ v ： speed of walking ） Character can control friction force from the floor: f （ |f|<f max ） Invert pendulum control Simplest control model for legged locomotion 11 g N f mvmv

Multi-dimensional keyframe animation Set of poses by designer. Character pose depends on State of the rigid body (slant, displacement) Type of behavior (walk, run, push…) Multi dimensional keyframe animation Put keyframes in a multi dimensional space. A point in the space corresponds to a state of rigid body. Get current pose of the character from the keyframes by interpolation. 12

Authoring of multi-dimensional keyframes It consists of one dimensional (conventional) keyframe anmations ① Create an animation for each axis ② Add keyframes from a keyframe of ① ③ Add and adjust keyframe by checking reactions 13 ② displacement slant ①

Example of generated motion 14

Application range Good for Whole body motion, reaction Walk, run, jump Pushed away, push large object. Interaction with large objects Not good for Motion by external forces acting on some part of body Pull hand strongly Interaction with small objects 15

Physics simulation + simulation of perception, attention and motor control. 16

In robotics MIT AI Lab Cog current Virtual reality MIT Media Lab Synthetic Characters Group Researched in context of artificial intelligence A robot mimics sensory motor system of human for researches on human intelligence. Create virtual creatures to understand intelligence. 17 Mimic sensory motor system

Automatic generation of motion which represent intention For precise physical interaction For example basic interaction with animals: rub, pick, grasp… It is very tough to create such motions with multi-dimensional keyframe. → A system which achieves role of animator is required. 18

Acting method for animators actors understand intention and motivation. Then, actors represent them by their motions. Actors reproduce motivation of the role. Action of actor with such motivation realizes natural actions. Animators reproduces actions and fix it. Mechanism of human behavior + motivation + situtaion = action 19 （ Ed Hooks, 2003 ）

A basic structure of behaviors of animals Behavior decision is based on sensory systems Input information is limited. Selective attention A mechanism to decide target of sensory system. 20 sensors Selective attention Decision of purpose motion Target of interest Decision of behavior Selected information (e.g. position)

Virtual Creature with attention 21 Vision modelTactile model v Tactile attention ・ Amount of attention ・ positoin Tactile attention ・ Amount of attention ・ positoin Visual attention ・ amount of attention ・ possition Visual attention ・ amount of attention ・ possition Physics simulator v F v F Jv = ω  ω = J # v Motor control Behavior generation rules Sensor Amount position Reaching target ・ position ・ gaze position etc Max

Vision modelTactile model v Physics simulator v F v F Sensory information Vision target: objects in the field of view Tactilitytarget: objects contact to the body Intensity of attention Visual ∝ velocity of the object in retinal image tactile ∝ pressure of contact Sensory system and attention model 22 Max Target of attention ・ Intensity ・ position … Target of attention ・ Intensity ・ position … Top-down attention ・ Intensity ・ position … Top-down attention ・ Intensity ・ position … 触覚情報 ・位置・部位など 触覚情報 ・位置・部位など 視覚情報 ・位置・種類など 視覚情報 ・位置・種類など Tactile attention ・ intensity ・ position … Tactile attention ・ intensity ・ position … Visual atention ・ intensity ・ position … Visual atention ・ intensity ・ position …

Behavior decision and motor control 23 Rules for behavior decision Decision of reaching target Position of attention target ＋ type of behavior decides body part to reach and target position. 運動制御 Inverse kinematics and PD control of joints Physics simulator Jv = ω  ω = J # v Motor control Rule of behavior decision Decision of reacing target Target of attention ・ Intensity ・ Position Target of attention ・ Intensity ・ Position Track the target with eyes Touch the target with hand Intensity of attention Track target Touch target

Example of motion 24 Interact with 3DOF haptic interface (SPIDAR) Large Middle Small Bear of 40DOF Intensity of top-down attention

Example of motion 25

Example of interaction 26

Range of simulation Physics simulation with keyframe animation Physics simulation Automatic generation of various reaction correspond to various interactions. Key frame animation Designed by animators Physics simulation and simulation of perception, attention and motor control. Next issue: How a designer designs actions. 27