Mitsubishi Electric Research Laboratories August 2006 Mitsubishi Electric Research Labs (MERL) Cambridge, MA Instant Replay: Inexpensive High Speed Motion Capture Hideaki Nii, Jay Summet, Yong Zhao, Jonathan Westhues, Paul H. Dietz, Shree K. Nayar, John Barnwell, Michael Noland, Vlad Branzoi, Erich Bruns, Masahiko Inami, & Ramesh Raskar

Mitsubishi Electric Research Laboratories August 2006 Demo Video

Mitsubishi Electric Research Laboratories August 2006 Optical Capture Goal –Find 2D or 3D location of tags Approach –Each tag has a photosensor –Label space using zones created by special high speed projectors –Find zone in which the tag lies

Mitsubishi Electric Research Laboratories August 2006 Key Features 500 Hz Tracking Id for each Marker Tag Capture in Natural Environment –Visually imperceptible tags –Photosensing Tag can be hidden under clothes –Ambient lighting is ok Unlimited Number of Tags Allowed Base station and tags only a few 10’s $

Mitsubishi Electric Research Laboratories August 2006 Potential Applications High Speed Motion Capture & Tracking –Motion Capture Character Animation CG alignment –Industrial applications Crash Test Dummies Robot Guidance –Indoor Positioning for Ubiquitous Computing Locating users and devices 3D User Input

Mitsubishi Electric Research Laboratories August 2006 Existing Solutions Optical: Uses expensive cameras to locate tags in the environment. –Passive tags give location, but no identity, leading to reacquisition problems Magnetic –Environmental interference, expensive Inertial Tracking (Gyro/ Compass) –Drift, Calibration

Mitsubishi Electric Research Laboratories August 2006 Solution Overview Use inexpensive solid state projectors to “label space”. –Solid State lighting (LED’s) can be modulated extremely quickly (MHz to GHz). Each point in space receives a different code sequence. Inexpensive and small tags receive the codes, and can detect their own location.

Mitsubishi Electric Research Laboratories August 2006 Labeling Space Each tag receives a unique signal based upon it’s physical location.

Mitsubishi Electric Research Laboratories August 2006 How Labeling Works Light source GrayCode Mask Optics Screen Light source blink one by one and each position on the screen has different light pattern. 4 light make 4 bit position resolution pos=0pos=15

Mitsubishi Electric Research Laboratories August 2006 System Hardware One projector can send 10 bit one dimensional position data. No limitation on the number of tags.

Mitsubishi Electric Research Laboratories August 2006 Inside of Projector The Gray code pattern Focusing Optics Gray code Slide Condensing OpticsLight Source

Mitsubishi Electric Research Laboratories August 2006 Inside of Tag

Mitsubishi Electric Research Laboratories August 2006 Specification NameValueRemarks Measurement time 1 mSPer one dimension FOV26.7 deg. Resolution10 bit Operational Range 1m ~ 5mGood quality in 2m~4m area.

Mitsubishi Electric Research Laboratories August 2006 Low cost Proto type systemMass Produced System Projector Electrical parts:$50 Projector body:$10 Optics: $ hundreds (4 custom glass lenses) Tag Electrical parts:$50 Wireless unit:$50 Projector Electrical parts:$30 Projector body:$10 Optics:$10 (Plastic lens) Tag Electrical parts:$10 Wireless unit: Optional

Mitsubishi Electric Research Laboratories August 2006 Expand to 2D/3D System 2D measurement3D measurement Y data X data Y data X data X2 data

Mitsubishi Electric Research Laboratories August 2006 Result Videos

Mitsubishi Electric Research Laboratories August 2006 Tags Hidden Under Costume Tags can be hidden under an actors costume with only small sensors exposed. This enables motion capture during primary filming.

Mitsubishi Electric Research Laboratories August 2006 High Speed Tracking

Mitsubishi Electric Research Laboratories August 2006 Flying Tags!

Mitsubishi Electric Research Laboratories August 2006 E-Tech Demo

Mitsubishi Electric Research Laboratories August 2006 Key Features 500 Hz Tracking Id for each Marker Tag Capture in Natural Environment –Visually imperceptible tags –Photosensing Tag can be hidden under clothes –Ambient lighting is ok Unlimited Number of Tags Allowed Base station and tags only a few 10’s $

Mitsubishi Electric Research Laboratories August 2006 Take Away Points Very Inexpensive Projectors and Tags. Reasonable accuracy which will be improved with better optical design. Very fast sampling rates possible! We want to license the technology!

Mitsubishi Electric Research Laboratories August 2006 Thank you! Contact: Ramesh Raskar

Mitsubishi Electric Research Laboratories August 2006 New Smaller Tag