A Brief Look In Movies Motion capture has it’s roots from movies (Hollywood). Game developers as a consequence begin to implement some of these new ideas or technologies to enhance their games. There’s really no significant difference in translating motion capture technology from movies to games.To give the audience a better sense on how motion capture was incorporated into games, movie references using motion capture are given.
Motion Capture In Movies Snow White and the Seven Dwarfs (1937) The Lord of the Rings: The Fellowship of the Ring (2001) Polar Express (2004) King Kong (2005) Happy Feet (2006) Avatar (2009) District 9 (2009) Mars Needs Moms (2011) The Adventures of Tintin (2011) Rango (2011) Rise of the Planet of the Apes (2011) John Carter (2012) The Avengers (2012) Ted (2012)
Snow White and the Seven Dwarfs (1937): Actors and actresses would act out scenes and would be filmed. The animators would then use the individual frames as a guide to their drawings (early form of mocap technology).
Polar Express (2004) & Avatar (2009) Actors and actresses would use a mocap suit which translated their movement into a digital effect which was then animated into a 3D computer animation.
The Adventures of Tintin (2011) Uses a mix of real actors and props, mocap allows a more accurate blend of images and film that can all be pieced together in an amazing sight.
Ted (2012) Ted (the teddy bear) was actually animated by using motion capture on an actor, which is why Ted’s movements seemed so realistic.
More Movie Examples Rise of the Planet of the Apes (2011) District 9 (2009) John Carter (2012)
Games The term ‘performance capture’ is used to describe advanced motion capture In addition to body movements, more minute details such as emotion, finger movement, and facial detection can be captured and used.
Examples Since the movie Avatar, one of the earliest, most famous, examples of performance capture, it has also been used in games such as L.A. Noire and Heavy Rain
Facial Motion Capture Beginnings 100+ markers on an actor’s face to capture facial expressions. Very cumbersome for the actor - takes a long time to put all dots on, and the actor cannot touch their face until they are removed
Facial Motion Capture Markerless motion capture: http://www.youtube.com/watch?v=7bX0qp sLfpE http://www.youtube.com/watch?v=7bX0qp sLfpE Mixamo recently released Face Plus for Unity - capture face data with only a webcam.
Early motion detection in games Sega’s Heavyweight Champ (1976) for arcades Two boxing glove controllers you punch, one for each player. A 1987 remake gave one player control of two punch controllers. The cabinet can also turn from side to side, and the boxer moves too.
Sega Hang On (1985) The ‘world’s first full-body experience video game’. Three arcade cabinet designs. The one pictured enables turning by leaning on the motorcycle.
NES Power Glove 1989 for NES Move the glove controller around. The time taken for ultrasonic sound to travel between speakers on the glove and microphones on a receiver unit is used to triangulate position. Optical fibers can detect bending of the glove (light is obstructed when bent). Considered difficult to use
Sega Activator (1993) Octogonal device on the floor that had eight infrared light sensors beamed upwards for detecting movement. Like the Power Glove, also not well-received.
Dance Dance Revolution (1998) http://www.youtube.com/watch?v=ExjhGrVJ8Do Most game modes/variants feature pressure sensors in four directions. These sensors must be stepped on to match the arrows appearing on screen in time with music. First major ‘exergaming’ (exercise gaming) series.
Konami’s Police 911 (2000) Shooting game where players duck for cover and dodge… by actually ducking for cover and dodging. Uses infrared sensors.
Notable motion capture games Prince of Persia (1989) Rotoscoping - videos (like the one below) were used as reference. http://www.youtube.com/watch?v=WAjRNU3DbSY https://en.wikipedia.org/wiki/Prince_of_Persia_(1989_video_game) (for animated GIF)
Metal Gear Solid Used from the second entry in the series (2001) onwards. The upcoming Metal Gear Solid 5 uses facial capture. Actors had their face and expressions captured from many angles. http://www.youtube.com/watch?feature=player_embedded&v=DbPxvBbZAhs
Enter The Matrix (2003) The actors (and stunt doubles) from the original movies were the motion capture actors for the game. Motion capture took 3 months.
L.A. Noire https://www.youtube.com/watch?v=3FIviB90Ih8 Used MotionScan for performance capture, with 32 cameras for facial animation. The blooper reel above shows the tech better than the game does because it features actors breaking character and being more natural.
Other well-known games using motion capture Madden (from 1994 on) The Legend Of Zelda: Ocarina of Time (1998) Devil May Cry 3 and 4 (2005, 2008) Uncharted series (2007) Grand Theft Auto IV (2008) The Last of Us (2013)
Motion Capture Technologies Many types: Optical (most prevalent) o a series of cameras with strobing lights are set up around an area to be “captured.” Inertial o Body suit or sensors attached to a person that track gyroscopic movement and transmit to a receiver o inertial sensors have 3d gyroscopes, accelerometers, and magnetometers
Motion Capture Technologies Last type: Mechanical o Suit that subject wears that transmits motion data to a receiver o Uses joint sensors to determine movement
Similarities between technologies All are fairly expensive, need between $5000-$10000 initial investment All require interfacing with a computer running specialized software Each brand has its own proprietary software or plugins for industry standard software
Optical Motion Capture OptiTrack series from NaturalPoint Prime 41 ($4,999) o Resolution: 2048 × 2048 o Frame Rate: 30– 180 FPS (adjustable) o Infrared Strobe (won’t interfere with video cameras)
Inertial Motion Capture XSens MVN o Full body suit with 17 inertial sensors o Flexible, since there are no special harnesses to hold the sensors, rather they are built in to the suit Used in “Paul”, by Seth Rogen to play the alien
Motion Capture Software Autodesk MotionBuilder o Industry standard 3d character animation software o Can import motion capture data and map motions to imported characters from Maya, 3ds, etc
Hardware and software ●Hardware ○ Real time ○ High frame rate ○ High bandwidth ●Software ○ Image processing algorithms ○ Extract the target image from its background
New Methods ●IR Camera Based ○ Leap Motion ●Electrical signal based ○ MYO ●Wifi Based ○ WiSee ○ Wivi
Leap Motion ●Human computer interaction ●Hand and finger motion ●Smaller observation area and higher resolution ●2 IR cameras and 3 IR LEDs ●Provide SDK, C++ ●App store - new ●https://www.leapmotion.com/
MYO ●Wristband ●Electrical activity in muscles. ●https://www.youtube.com/watch?v=oWu9 TFJjHaMhttps://www.youtube.com/watch?v=oWu9 TFJjHaM ●Wirelessly control computer. ●Interact with digital world.
Wifi Based Methods ●Wisee ○ From Univ of Washington ○ Published in MobiCom13 ○ http://wisee.cs.washington.edu/ ○ Classify a set of 9 gestures ○ Achieve 94% accuracy.
Wifi Based Methods ●Wivi ○ From MIT ○ Published in SIGCOMM13 ○ Motion Capture Behind Walls ○ http://people.csail.mit.edu/fad el/wivi/