ISOMAP TRACKING WITH PARTICLE FILTERING NIKHIL RANE AND STAN BIRCHFIELD CLEMSON UNIVERSITY Abstract Isomap tracking with particle filtering algorithm Tracking results The problem of tracking involves challenges like in-plane and out-of-plane rotations, scaling, variations in ambient light and occlusions. In this paper we look at the problem of tracking a person’s head and also estimating its pose in each frame. Robust tracking can be achieved by reducing the dimensionality of high-dimensional training data and using the recovered low-dimensional structure to estimate the state of an object at every time-step with recursive Bayesian filtering. Isometric feature mapping, also known as Isomap, provides an unsupervised framework to find the true degrees of freedom in high-dimensional input data like a person’s head with varying poses. After the data has been reduced to lower dimensions a particle filter can be used to track and at the same time approximate the pose of a person’s head in any image sequence. Isomap tracking with particle filtering is capable of handling rapid translation and out-of-plane rotation of a person’s head with a relatively small amount of training data. The performance of the tracker is demonstrated on an image sequence with a person’s head undergoing translation and out-of-plane rotation. Algorithm : Create training set of a person’s face (off-line) Use Isomap to reduce dimensionality of the training set (off-line) Run particle filter (Condensation) on test sequence to track the person Condensation algorithm Behavior of the particle cloud over time 2D out-of-plane rotation (length of vector yields amount of rotation) closest training template Dimensionality reduction using Isomap Let xi be H-dimensional and yi be L-dimensional, where H>L Isomap is a non-linear dimensionality reduction technique that computes a mapping f : xi  yi Isomap algorithm: Construct neighborhood graph Compute shortest paths between points using geodesic distance Apply classical Multidimensional Scaling (MDS) Isomap of training data Geodesic distance – The length of the shortest curve between two points taken along the surface of a manifold. It reflects the true geometry of the manifold. Top 3 eigenvectors capture almost 99% of the variation in training data High-dimensional images reduced to 3 dimensions by Isomap The blue square indicates the tracker output (weighted average of particles) Conclusion Training data Faces (151 x 151 pixels) with up-down and left-right poses. In-plane rotation is approximated by rotating these templates. Isomap provides successful framework for pose estimation Algorithm can handle rapid translation and out-of-plane rotation Algorithm can track a person and estimate the pose simultaneously Future goals: On-line construction of Isomap model so that the particle filter tracker learns as it tracks Handle occlusions Block diagram of algorithm for a single particle Two-dimensional projections of the low-dimensional structure with training templates superimposed