Multiple Frame Motion Inference Using Belief Propagation Jiang Gao Jianbo Shi Presented By: Gilad Kapelushnik Visual Recognition, Spring 2005, Technion IIT.

Abstract Find “best fit” upper body joint configuration. Input is a 2D video Each joint is described by its location on a 2D grid. S1(X,Y) S2(X,Y) S4(X,Y) S6(X,Y) S5(X,Y) S3(X,Y) Let J be a joint configuration – {S1,S2,S3,S4,S5,S6} We would like to find:

Step 1: Subtract two sequential frames. Step 2: Apply threshold. Motion Energy Image

From #NrgPixels To Probability Sum the Energy Pixels in the Patch. Calculate probability using the following: S5(10,60) S6(40,30)

Find configuration J with the highest probability. Computing all possible probabilities is inefficient. a-Priori data give better and faster results. removing impossible configurations reduce inference time. Main Idea

a-Priori Data A probability table for Each P(Sx,Sy). Compute probability at grid crossing. Use nearest neighbor for the rest of the image. Example: For right arm - P(S2,S3) Red – Low probability Green – High probability Ns^2…21P(S2,S3) 0…001 0… ………… 0000Ns^2

Face is detected using face detection algorithm. Initial assumption of Shoulders from face and pose. Even using BP there are too many possible states to go through. Candidates for elbows from shoulders & Energy Map. Candidates for Wrists from skin color model. Detect Candidate states (1)

Detect Candidate states (2) Many states can be discarded. Remove close candidate states. Pros: Much faster inference. Cons: Less accurate. Note: This is only an option. Fits skin color and wrist location Pink for right wrist Red for left wrist Blue for elbow

The Markov Model Empty Circles - States - 2D positions of joints Full Circles - Observations - Computed from energy map. Each state correspond to an observation.

Belief Propagation (1) Solve inference problem using an algorithm with Linear complexity. Each joint has a vector with probabilities for each candidate. Shoulder Elbow Wrist

Belief Propagation (2) m23 m32 m21 m12 m14 m41 For each iteration: Each node sends a message to its neighbor nodes containing the “wanted” probability (for each state). Messages are computed according to: Sum over all candidates A-priori Data for each state. Normalize variable. Observation (# of Energy pixels in patch) for each state converted to a probability. Message from k to i (all messages from the neighbors). This is actually a vector with a probability for each state. Message from i to j.

Belief Propagation (3) - Example 21 Message from 1 to 2 4 states 2 states

Belief Propagation (4) BP converge after 2-4 iterations (giving the right a-Priori data). For every joint there is a probability vector for each candidate state.

Multiple Frame Probability Multiple frame (8) is proposed for smoother transition between configurations. Prevents joints changing their state to a different which is “far away” (Euclidian distance). Though BP was designed to work with loopy-free models, the author stated that it worked fine. And for those who really want to know:

2D to 3D 2D -> 3D by Taylor (2000). Assuming (u1,v1) and (u2,v2) are projections then depth can be retrieved using the following:

Results(1)

Results(2)

Results(3) Errors accrue when 2 joints intersect each other. On some occasions, even when limbs intersect, it was possible to infer correctly.

Q?