Autonomous Learning of Object Models on Mobile Robots Xiang Li Ph.D. student supervised by Dr. Mohan Sridharan Stochastic Estimation and Autonomous Robotics Lab (SEARL) Department of Computer Science Texas Tech University October 23,

Object Recognition 2 Learn Object Model color, texture or shape Test match Challenges: How to identify ROI in the image (Region Of Interest) ? What features to extract in ROI (Object model) ? Efficient Implementation. ROI

Related Work Object model O. Bjorn, PAMI12; R. Fergus, CVPR03; P. Felzenszwalb, IJCV04; N.E.K. Roman, AR10. Mobile robot C. Guo, ICRA11; M. Sridharan, IROS07; D. Parikh, PAMI12 ; J. Hoey, CVIU10. Local image feature D. Lowe, IJCV05; J. Matas, BMVC02; S. Park, IROS09; M. Calonder, ECCV10. 3

Motivation 4 Goal Learn and recognize objects autonomously in dynamic environments. Our Work  Identify ROIs autonomously based on a limited number of images with moving objects.  Build probabilistic object models using the complementary properties of different visual cues.  Fuse the information by generative model and energy minimization algorithm.

Autonomously Learning Supervised Learning  Images with the labeled regions Unsupervised Learning  Images without any labeled regions Track and cluster local image gradient features [128D vector]  A short sequence of images (motion cue) 5 ROI tt+1 ROI

Object Model Given ROI (autonomous or manual) Use the complementary properties of different visual cues 6

Motivation of SCV and Undirected Graph The difference between correct and incorrect match  The spatial arrangements of local features  The connection between the local features 7 Match by Nearest Neighbor algorithm(shortest Euclidean distance). correct Incorrect

SCV from gradient features The individual gradient features may not be unique. The spatial arrangement of local gradient features corresponding to a specific object is difficult to duplicate. 8

Connection Potentials Connection potential is computed as the color distribution of pixels between gradient features in the image ROI. 9 Build an undirected graph of connection potentials to model the neighborhood relationships between gradient features.

Parts from image segments Considers the arrangement of object parts made up of image segments. Pixels within a part have similar values, while pixels in neighboring parts have dissimilar values. 10

Color-based Representation Computes the distance between every pair of pdfs and models the distribution of distances as a Gaussian. 11 Second order color distribution statisticsColor histogram (pdf) …

Local context from image segments Probabilistic mixture models Relative positions (on, under, beside) 12

Information Fusion 13 Generative model  considers the relationship between the components Energy minimization algorithm  Identifies the ROI for recognizing the stationary objects

Robot Platforms: Erratic 1.6GHz Core2 Duo CPU 2 cameras (monocular & stereo) Laser range finder 640 × 480 Resolution Wi-Fi On board computation 14

Experimental Trial 15 Test Image Match Probabilities Net Match

Experimental Trial (Cont) 16 Test Image Match Probabilities Net Match

Object Categories 17

The Classification Accuracy 18

Conclusions Mobile robot can identify interesting objects based on motion cues, and autonomously and efficiently learn object models that exploit the complementary properties of appearance- based and contextual visual cues. Exploiting the complementary properties of these visual cues enables the robot to use generative model and energy minimization algorithms to reliably and efficiently recognize the learned. 19

Future Work Consider image sequences with multiple moving objects. Add Shape representation into object model. Extend to a team of robots collaborating in dynamic environments. 20

Q & A 21

Convex Hull The convex hull of a set of points is the smallest convex set that contains the points. Quickhull algorithm 1 computes the convex hull. 1. Barber, C.B., Dobkin, D.P., and Huhdanpaa, H.T., “The Quickhull algorithm for convex hulls”, ACM Trans. on Mathematical Software, 22(4): , Dec 1996,

Gamma distribution Context Probability Context Probability Using Gamma MatchedNot Matched context0.28± ±0.03 Other components MatchedNot Matched context0.5± ±0.10 Other components0.70.3

Experiment Example Learned robot model (Corridor) Testing Corridor(1) Prob = 0.89 Corridor(2) Prob = 0.77 Office Prob = 0.12

Generative Model Randomly generating observable data Typically given some hidden parameters Specifies a joint probability distribution over observation and label sequences. 25

Generative model (Cont) 26