CAMEO: Meeting Understanding Prof. Manuela M. Veloso, Prof. Takeo Kanade Dr. Paul E. Rybski, Dr. Fernando de la Torre, Dr. Brett Browning, Raju Patil, Carlos Vallespi, Scott Lenser, Betsy Ricker, Francesco Tamburrino, Colin McMillen, Sonia Chernova CALO: Physical Awareness Computer Science Department /The Robotics Institute School of Computer Science Carnegie Mellon University

CAMEO: Camera Assisted Meeting Event Observer

Robust multi-person PA capture device Contributions Mosaic generation Person tracking Face recognition Activity recognition Logging/modeling Must effectively operate in unstructured environments. Each camera is hand-calibrated only once to compensate for radial distortion

Video Mosaic

Person Tracking : Mean Shift Based Color Tracking Register New Person: Person ID, Face histogram Face Center (x,y), Face width, Face height Additional filtering based on shape and color templates “Omega” head and shoulder template Henry Schneiderman. “Feature-Centric Evaluation for Cascaded Object Detection. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Henry Schneiderman. “Learning on Restricted Bayesian Network for Object Detection.” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2004.

Face Recognition: Training 1) Capture visual data 2) Normalize for geometry and illumination 3) Cluster the most discriminating face examples Multiple face discrimination Real-time performance

Face Recognition: Non-Linear Oriented Discriminant Analysis Find transformation matrix B that MAXIMIZES the Kullback-Leibler divergence between clusters among classes Use Iterative Majorization to approximate B Project clustered images into a lower- dimensional subspace to speed recognition Research challenge: Face subspace is multi-modal

Face Recognition: Results Each new face is projected into subspace and compared against the trained examples Closest match, via Mahalanobis distance, determines class membership 95% recognition rate with training database of 11 subjects

Inferring Activity from Observation Person action sequences can be represented as a simple finite state machine. Face tracker captures the (x,y) positions of faces in the image over time. Global meeting state is inferred from aggregate of person activity. Inferred state from classifier State transitions are encoded as a dynamic Bayesian network in a HMM structure. Current person state is a function of observed human activity and previous state.

Logging/Replay/Towards Learning Tracked person data is recorded for off-line activity analysis and learning of dynamics. The recorded logs can be replayed back through CAMEO. Model-based simulation generates high-level state descriptions of group activies. Data-based simulation generates low-level “frame-by-frame” individual person activity state descriptions. bring [carlos, computer] bring [carlos, cameo] set_up [carlos, computer] use [carlos, computer] set_up [carlos, cameo] use [carlos, cameo] give_demo [carlos, face_recognition] ask_question [fernando, face_recognition] answer_question [carlos, fernando, face_recognition] give_demo [raju, tracking] ask_question [carlos, tracking] answer_question [raju, carlos, tracking] give_demo [carlos, face_detection] ask_question [raju, face_detection] answer_question [carlos, raju, face_detection] ask_question [fernando, face_detection] answer_question [raju, fernando, face_detection] remove [carlos, computer] remove [carlos, cameo] leave [jon] leave [raju] leave [fernando] leave [carlos] leave [daniel]