1. Person Following with a Mobile Robot Using Binocular Feature-Based Tracking Zhichao Chen and Stanley T. Birchfield Dept. of Electrical and Computer Engineering Clemson University Clemson, South Carolina USA

2. Motivation Goal: Enable a mobile robot to follow a person in a cluttered indoor environment by vision. Previous approaches: Appearance properties: color, edges. [Sidenbladh et al. 1999, Tarokh and Ferrari 2003, Kwon et al. 2005]  Person has different color from background or faces camera.  Lighting changes. Optical flow. [Piaggio et al 1998, Chivilò et al. 2004]  Drift as the person moves with out-of-plane rotation Dense stereo and odometry. [Beymer and Konolige 2001]  difficult to predict the movement of the robot (uneven surfaces, slippage in the wheels).

3. Our approach Algorithm: Sparse stereo based on Lucas-Kanade feature tracking. Handles: Dynamic backgrounds. Out-of-plane rotation. Similar disparity between the person and background. Similar color between the person and background.

4. System overview

5. Detect 3D features of the scene Kanade-Lucas-Tomasi (KLT) feature tracker Automatically selects 2D features using eigenvalues of 2x2 gradient covariance matrix Automatically matching features by minimizing sum of squared differences (SSD) between left and right images. Augmented with gain and bias to handle lighting changes Open-source implementation [http://www.ces.clemson.edu/~stb/klt] unknown displacement gray-level images gradient of image

6. Detect 3D features of the scene ( Cont. ) Features are selected in the left image I L and matched in the right image I R. Left imageRight image The size of each square indicates the horizontal disparity of the feature.

7. System overview

8. Overview of Removing Background 2) using the estimated motion of the background. 3) using the estimated motion of the person 1) using the known disparity of the person in the previous image frame.

9. Discard features for which where is the known disparity of the person in the previous frame, and is the disparity of a feature at time t. Remove Background Step 1: Using the known disparity Original features Foreground features in step 1 Background features

10. Remove Background Step 2: Using background motion Estimate the motion of the background by computing a 4 × 4 projective transformation matrix H between two image frames at times t and t + 1: Random sample consensus (RANSAC) algorithm is used to yield dominant motion. Foreground features with similar disparity in step 1 Foreground features after step 2

11. Remove Background Step 3: Using person motion Similar to step 2, the motion model of the person is calculated. The size of the person group should be the biggest. The centroid of the person group should be proximate to the previous location of the person. Foreground features after step 2 Foreground features after step 3

12. System overview

13. Detect Face The Viola-Jones frontal face detector is applied. This detector is used both to initialize the system and to enhance robustness when the person is facing the camera. Note: The face detector is not necessary in our system.

14. System overview

15. Experimental Results

16. Video

17. Comparison with a Color Histogram-Based Algorithm Color-based (Camshift, Bradski 1998) Our approach

18. Conclusion Approach –detects and matches feature points between a stereo pair of images and between successive images. –RANSAC-based procedure to estimate the motion of each region. Advantages –does not require the person to wear a different color from the background. –track a person in an office environment, even through doorways, with clutter, and in the presence of other moving objects. Future work –fusing the information with additional appearance-based information ( template or edges). –integration with other modules (obstacle avoidance)