1. A Bayesian Formulation For 3d Articulated Upper Body Segmentation And Tracking From Dense Disparity Maps Navin Goel Dr Ara V Nefian Dr George Bebis

2. Overview Introduction Gesture Recognition System 3D Upper Body Model Initialization Segmentation Tracking Results Future work

3. Introduction Applications: Human-Computer Interaction Systems, American Sign Language Recognition, Security-Monitoring Systems, Gesture Recognition. Requirements: Background and scene independent, Robust hand free initialization, Robust tracking.

4. Introduction Related Work: A. V. Nefian, R. Grzeszczuk, and V. Eruhimov. A statistical upper body model for 3D static and dynamic gesture recognition from stereo sequences. In IEEE International Conference on Image Processing, volume 3, pages 286-289, 2001. Success: 96% Accuracy.

5. Gesture Recognition System:

6. Assumptions: One user in the camera view, The torso is the largest visible component, The torso plane is approximately perpendicular to the camera, The user’s head is approximately in vertical position.

7. Observation Vectors: Observation Vectors = depth + color Upper Body Model Pixel OdOd OcOc Threshold is chosen in such a way that the foreground pixels consist mostly on the upper body pixels.

8. Head: Where, N is a Gaussian density function, µ H and C H are mean and covariance of gaussian density function. U is a uniform distribution function, H is Head & S H is the size of head. Upper Body Model

9. Torso: Where, Upper Body Model T is Torso & W T is Width of Torso

10. Arms – Linear PDF Where, R F is radius of Fore Arm, F l is Left fore arm, ρ ij & σ is mean and covariance of Gaussian. Upper Body Model

11. EM Algorithm The E Step Compute: P(O ij | J,C) The M step finds: J,C = arg max(P(O ij | J,C)) Where, J are Joints & C are Components of upper body model

12. Head & Torso Starting with an initial position of the neck, the parameters of the torso plane are estimated below the neck plane using the EM approach. Using the torso center and anthropological measures of the head, parameters of the head are estimated using the EM approach. Estimate the position of Neck: N(x) = μ He (x) N(y) = μ He (y) + S H /2 N(z) = aN(x) + bN(y) + c Initialization

13. Head & Torso Estimate the position of Shoulders: S l (x) = µ T (x) + W T /2 S l (y) = N(y) S l (z) = aS l (x) + bS l (y) + c & S r (x) = µ T (x) - W T /2 S r (y) = N(y) S l (z) = aS l (x) + bS l (y) + c Repeat the above steps for updated J B until convergence error falls under a threshold. Initialization

14. Arms For each possible arm joint configuration we estimate the mean of the linear pdfs corresponding to the upper and fore arms, and the mean of the normal pdf for hands. For each joint configuration of the arms, we determine the best state assignment of the observation vectors. Find the max likelihood over all joint configuration and determine the best set of joints and the corresponding best state assignment. Initialization

15. Head & Torso: Estimate head and torso parameters and position of joints using the algorithm in Initialization – 1. The value of neck is obtained from previous frame. TRACKING

16. Arms: Step 1: Find 4 points in sphere that are equidistant from original elbow position based on theta and phi values passed. These points constitute the new set of possible elbow joints to search in the current iteration. FOR(I = 0; I < 5; I++) Step 2: Find log likelihood of Probability Density Function (PDF) found by EM given 3d planar distance. Step 3: For each new elbow joint, find 4 new positions for wrist + original based on theta and phi values passed. FOR(J = 0; J < 5; J++) Step 4: Find log likelihood of Probability Density Function (PDF) found by EM given 3d planar distance. PDF is defined as start (elbow) and end (wrist) points of joints. TRACKING

17. Arms: Step 5: For each new wrist joint, find 4 new positions for hand center + original based on theta and phi values passed. FOR(K = 0; K < 5; K++) Step 6: Find log likelihood of Probability Density Function (PDF) found by EM given 3d planar distance. PDF is defined as start (wrist) and end (hand center) points of joints. Step 7: Calculate Max Tracking Log Likelihood of Single Arm. Calculate argmaxElWl{ P(Oij|{Ul, Fl, Hl},{El,Wl})} Convergence Error = Best Probability – Previous Probability Step 8: Repeat the above steps until convergence error. TRACKING

18. Success: Results

19. Success: Results

20. Failure: Results

21. Future Work: The system collapses when hand is adjacent to head. Integration with other systems. Learning anthropological measures of the person.

22. Acknowledgements: National Science Foundation University Of Nevada, Reno Intel Corporation, Santa Clara