MIT AI Lab Paul Viola NTT Visit: Image Database Retrieval Variable Viewpoint Reality Professor Paul Viola Collaborators: Professor Eric Grimson, Jeremy De Bonet, John Winn, Owen Ozier, Chris Stauffer, John Fisher, Kinh Tieu, Dan Snow, Tom Rikert, Lily Lee, Raquel Romano, Janey Hshieh, Mike Ross, Nick Matsakis, Jeff Norris, Todd Atkins Mark Pipes

MIT AI Lab Paul Viola Overview of Visit Morning: Image Database Retrieval –Gatekeeper: Face detection and recognition –Complex Feature Image Database Retrieval (Tieu) –Flexible Template Retrieval (Yu) Interlude –Video/Audio Source Separation (Fisher) –Mathematical Expression Recognition (Matsakis) Lunch Visit Prof. Brooks lab

MIT AI Lab Paul Viola Overview of Visit - 2 Afternoon: Variable Viewpoint Reality –Real-time 3D reconstruction of people (Snow) –Automatic camera calibration (Snow + Lee) –Tracking of articulate human models (Lee + Winn) –Modeling of human dynamics (Viola + Fisher)

MIT AI Lab Paul Viola Gatekeeper: Receptionist & Security Greet guests Direct people to their destinations Recognize employees Turn back unauthorized visitors

MIT AI Lab Paul Viola Gatekeeper in action … Gatekeeper Movie

MIT AI Lab Paul Viola Gatekeeper is a constant observer… Professor Paul Viola

MIT AI Lab Paul Viola Detecting faces is very difficult

MIT AI Lab Paul Viola Detecting and Recognizing Faces Key Difficulty: Variation in Pose State of the art: generalized templates –Neural Networks / Deformable Templates / etc. Templates have difficulty with pose variation… –Rotation, scale, complex deformation Must reduce the dependence on relative pose. Approach: Detecting people as a statistical distribution of multi-scale features

MIT AI Lab Paul Viola Statistical Distribution of Multi-scale Features Wavelet Pyramid The distribution of multi-scale features determines appearance

MIT AI Lab Paul Viola Multi-scale Wavelet Features A multi-scale feature associates many values with each pixel in the image

MIT AI Lab Paul Viola I MODEL I TEST Cross Entropy Discrimination via Cross Entropy

MIT AI Lab Paul Viola BTR70-C71T Motivation: Finding vehicles in clutter Supported by Darpa: IU/ATR initially MSTAR Extension

MIT AI Lab Paul Viola Can also be used for segmentation…

MIT AI Lab Paul Viola Synthesis Results Original Texture The multi-scale statistical model can be used to generate new example textures

MIT AI Lab Paul Viola Synthesis Procedure Step 1: Build analysis pyramid Input Image Note: We are using only the Gaussian pyramid here! Normally we use an oriented pyramid... 64x64 2x2

MIT AI Lab Paul Viola Synthesis Procedure Step 2: Build synthesis pyramid

MIT AI Lab Paul Viola Synthesis Procedure Step 2a: Fill in the top... Pixels are generated by sampling from the analysis pyramid.

MIT AI Lab Paul Viola Synthesis Procedure Step 2b: Fill in subsequent levels Pixels are generated by conditional sampling (dependent on the parent).

MIT AI Lab Paul Viola Synthesis Procedure Finish the pyramid Decisions made at low resolutions generate discrete features in the final image.

MIT AI Lab Paul Viola

MIT AI Lab Paul Viola

MIT AI Lab Paul Viola

MIT AI Lab Paul Viola

MIT AI Lab Paul Viola Detection Results Non-face test images Web face test images

MIT AI Lab Paul Viola Pruning the density estimator Reduce the number of bins through clustering 1000 bins or less! Result: Detection/Classification is faster than template correlation

MIT AI Lab Paul Viola Key facial features - determined automatically - located automatically Multi-scale features which are come from the face model can be automatically detected for many individuals

MIT AI Lab Paul Viola Another key feature

MIT AI Lab Paul Viola New Face Recognition Algorithm Measure the occurrence and location of “key” facial features. Facial identity depends both on the types of features and their location. Relation to Active Search… –Match measure is a histogram of multiscale features –Like color histogram, Active Search can be used...

MIT AI Lab Paul Viola Presentation on Image Database Technology