A Bayesian Formulation For 3d Articulated Upper Body Segmentation And Tracking From Dense Disparity Maps Navin Goel Dr Ara V Nefian Dr George Bebis

Overview Introduction Gesture Recognition System 3D Upper Body Model Initialization Segmentation Tracking Results Future work

Introduction Applications: Human-Computer Interaction Systems, American Sign Language Recognition, Security-Monitoring Systems, Gesture Recognition. Requirements: Background and scene independent, Robust hand free initialization, Robust tracking.

Introduction Related Work: A. V. Nefian, R. Grzeszczuk, and V. Eruhimov. A statistical upper body model for 3D static and dynamic gesture recognition from stereo sequences. In IEEE International Conference on Image Processing, volume 3, pages , Success: 96% Accuracy.

Gesture Recognition System:

Assumptions: One user in the camera view, The torso is the largest visible component, The torso plane is approximately perpendicular to the camera, The user’s head is approximately in vertical position.

Observation Vectors: Observation Vectors = depth + color Upper Body Model Pixel OdOd OcOc Threshold is chosen in such a way that the foreground pixels consist mostly on the upper body pixels.

Head: Where, N is a Gaussian density function, µ H and C H are mean and covariance of gaussian density function. U is a uniform distribution function, H is Head & S H is the size of head. Upper Body Model

Torso: Where, Upper Body Model T is Torso & W T is Width of Torso

Arms – Linear PDF Where, R F is radius of Fore Arm, F l is Left fore arm, ρ ij & σ is mean and covariance of Gaussian. Upper Body Model

EM Algorithm The E Step Compute: P(O ij | J,C) The M step finds: J,C = arg max(P(O ij | J,C)) Where, J are Joints & C are Components of upper body model

Head & Torso Starting with an initial position of the neck, the parameters of the torso plane are estimated below the neck plane using the EM approach. Using the torso center and anthropological measures of the head, parameters of the head are estimated using the EM approach. Estimate the position of Neck: N(x) = μ He (x) N(y) = μ He (y) + S H /2 N(z) = aN(x) + bN(y) + c Initialization

Head & Torso Estimate the position of Shoulders: S l (x) = µ T (x) + W T /2 S l (y) = N(y) S l (z) = aS l (x) + bS l (y) + c & S r (x) = µ T (x) - W T /2 S r (y) = N(y) S l (z) = aS l (x) + bS l (y) + c Repeat the above steps for updated J B until convergence error falls under a threshold. Initialization

Arms For each possible arm joint configuration we estimate the mean of the linear pdfs corresponding to the upper and fore arms, and the mean of the normal pdf for hands. For each joint configuration of the arms, we determine the best state assignment of the observation vectors. Find the max likelihood over all joint configuration and determine the best set of joints and the corresponding best state assignment. Initialization

Head & Torso: Estimate head and torso parameters and position of joints using the algorithm in Initialization – 1. The value of neck is obtained from previous frame. TRACKING

Arms: Step 1: Find 4 points in sphere that are equidistant from original elbow position based on theta and phi values passed. These points constitute the new set of possible elbow joints to search in the current iteration. FOR(I = 0; I < 5; I++) Step 2: Find log likelihood of Probability Density Function (PDF) found by EM given 3d planar distance. Step 3: For each new elbow joint, find 4 new positions for wrist + original based on theta and phi values passed. FOR(J = 0; J < 5; J++) Step 4: Find log likelihood of Probability Density Function (PDF) found by EM given 3d planar distance. PDF is defined as start (elbow) and end (wrist) points of joints. TRACKING

Arms: Step 5: For each new wrist joint, find 4 new positions for hand center + original based on theta and phi values passed. FOR(K = 0; K < 5; K++) Step 6: Find log likelihood of Probability Density Function (PDF) found by EM given 3d planar distance. PDF is defined as start (wrist) and end (hand center) points of joints. Step 7: Calculate Max Tracking Log Likelihood of Single Arm. Calculate argmaxElWl{ P(Oij|{Ul, Fl, Hl},{El,Wl})} Convergence Error = Best Probability – Previous Probability Step 8: Repeat the above steps until convergence error. TRACKING

Success: Results

Success: Results

Failure: Results

Future Work: The system collapses when hand is adjacent to head. Integration with other systems. Learning anthropological measures of the person.

Acknowledgements: National Science Foundation University Of Nevada, Reno Intel Corporation, Santa Clara