1. 3/5/2002Phillip Saltzman Video Motion Capture Christoph Bregler Jitendra Malik UC Berkley 1997

2. 3/5/2002Phillip Saltzman Overview ChallengesChallenges ReviewReview MethodMethod ResultsResults ConclusionsConclusions

3. 3/5/2002Phillip Saltzman Challenges High AccuracyHigh Accuracy Frequent Inter-part OcclusionFrequent Inter-part Occlusion Low ContrastLow Contrast

4. 3/5/2002Phillip Saltzman Review

5. 3/5/2002Phillip Saltzman Review Motion capture on synthetic imagesMotion capture on synthetic images –O’Rouke and Balder, 1980 1 DOF marker free tracking1 DOF marker free tracking –Hogg, 1983. Rohr, 1993 Higher DOF full body trackingHigher DOF full body tracking –Gravrila and Davis, 1995

6. 3/5/2002Phillip Saltzman Review About the previous work –All in controlled environments with high contrast and clear edge bounries –Most use skintight suits or markers –Camera calibration needed

7. 3/5/2002Phillip Saltzman Method

8. 3/5/2002Phillip Saltzman Method Basic Assumptions –From frame to frame, all intensity pixel intensity changes are local: –u is motion model and is written as a matrix equation:

9. 3/5/2002Phillip Saltzman Method Finding Gradients –Gradient form of the first equation: –Find a least squares solution to  –Warp image I(t+1) based on  –Find new gradients –Repeat to minimize

10. 3/5/2002Phillip Saltzman Method Motion as twists –Standard pose matrix to move from object space to camera space (3D) –Scaled orthographic projection moves to image space –Requires knowing something about the 3D model of the image. Approximated as ellipsoids.

11. 3/5/2002Phillip Saltzman Method Motion as twists –Any motion can be represented as a rotation about an axis, and a translation about that axis –For example, to make this motion:

12. 3/5/2002Phillip Saltzman Method Motion as twists You make this motion:

13. 3/5/2002Phillip Saltzman Method Motion as twists –Twists can be represented as small vector or matrix –Can be made to a pose by –Encode the motion of a pixel between two frames

14. 3/5/2002Phillip Saltzman Method Motion as twists –Linear algebra manipulation allows using the twist vector to write a motion equation for each pixel –Those equations are put in a vector and used to find a global  parameter for that object

15. 3/5/2002Phillip Saltzman Method Kinematic chains –Body parts represented as multiple connected objects –Each object can be found by the top pose and an angle and twist for each object down the chain –More linear algebra is used to find a  for each body part

16. 3/5/2002Phillip Saltzman Method Multiple cameras –Adds accuracy because change of fully occluded parts drop with each view –Normal motion equation is: –H is system of equations for each pixel –  is global parameter vector for each object –z is initial position of the pixel

17. 3/5/2002Phillip Saltzman Method Multiple cameras –Adding synchronized cameras: –H becomes a matrix where each column represents a view –The  vector gets a term  for each view that represents the pose seen from that view –The  vector gets a term  for each view that represents the pose seen from that view –z becomes a vector with an initial position for each view.

18. 3/5/2002Phillip Saltzman Method Support maps –Limits pixel search to area defined by map for speed –Value for each pixel in range [0,1], where 1 means pixel is in the region –Method for finding starts as an elliptical guess, but refining it is not described

19. 3/5/2002Phillip Saltzman Method Algorithm review Input: Image I(t), I(t+1), pose and IK angles Output: Pose and IK angles for I(t+1) Find 3D points for each pixel in image Compute support map for each segment Set poses and IK angles for I(t+1) = I(t) Iterate: Compute gradients Estimate  Update poses and IK angles Warp image based on the pose and support map

20. 3/5/2002Phillip Saltzman Method Initialization –Algorithm depends on known positions for the first frame –For multiple views, each first frame must be initialized –User clicks joint positions, and 3D estimations and joint angles are computed –Values like symmetry can be enforced

21. 3/5/2002Phillip Saltzman Results

22. 3/5/2002Phillip Saltzman Results – Single angle – 53 frames with decent results – Upper leg hard to track, so IK chain compensates with lower leg and torso In Lab Movie

23. 3/5/2002Phillip Saltzman Results – Oblique angle – Tracking over 45 frames – Algorithm could track change in scale due to perspective changes Oblique Lab Movie

24. 3/5/2002Phillip Saltzman Results – Oldest known “movie” – High noise and low contrast – Low framerate – Multiple views Digital Muybridge

25. 3/5/2002Phillip Saltzman Conclusions Future Work/Shortcomings –May break with large movements –Fixed camera only –Did not show tracking of back limbs –No timing data –Few results