1. MIT AI Lab Paul Viola NTT Visit: Image Database Retrieval Variable Viewpoint Reality Professor Paul Viola Collaborators: Professor Eric Grimson, Jeremy De Bonet, John Winn, Owen Ozier, Chris Stauffer, John Fisher, Kinh Tieu, Dan Snow, Tom Rikert, Lily Lee, Raquel Romano, Janey Hshieh, Mike Ross, Nick Matsakis, Jeff Norris, Todd Atkins Mark Pipes

2. MIT AI Lab Paul Viola Overview of Visit Morning: Image Database Retrieval –Gatekeeper: Face detection and recognition –Complex Feature Image Database Retrieval (Tieu) –Flexible Template Retrieval (Yu) Interlude –Video/Audio Source Separation (Fisher) –Mathematical Expression Recognition (Matsakis) Lunch Visit Prof. Brooks lab

3. MIT AI Lab Paul Viola Overview of Visit - 2 Afternoon: Variable Viewpoint Reality –Real-time 3D reconstruction of people (Snow) –Automatic camera calibration (Snow + Lee) –Tracking of articulate human models (Lee + Winn) –Modeling of human dynamics (Viola + Fisher)

4. MIT AI Lab Paul Viola Gatekeeper: Receptionist & Security Greet guests Direct people to their destinations Recognize employees Turn back unauthorized visitors

5. MIT AI Lab Paul Viola Gatekeeper in action … Gatekeeper Movie

6. MIT AI Lab Paul Viola Gatekeeper is a constant observer… Professor Paul Viola

7. MIT AI Lab Paul Viola Detecting faces is very difficult

8. MIT AI Lab Paul Viola Detecting and Recognizing Faces Key Difficulty: Variation in Pose State of the art: generalized templates –Neural Networks / Deformable Templates / etc. Templates have difficulty with pose variation… –Rotation, scale, complex deformation Must reduce the dependence on relative pose. Approach: Detecting people as a statistical distribution of multi-scale features

9. MIT AI Lab Paul Viola Statistical Distribution of Multi-scale Features Wavelet Pyramid The distribution of multi-scale features determines appearance

10. MIT AI Lab Paul Viola Multi-scale Wavelet Features A multi-scale feature associates many values with each pixel in the image

11. MIT AI Lab Paul Viola I MODEL I TEST Cross Entropy Discrimination via Cross Entropy

12. MIT AI Lab Paul Viola BTR70-C71T72-132 Motivation: Finding vehicles in clutter Supported by Darpa: IU/ATR initially MSTAR Extension

13. MIT AI Lab Paul Viola Can also be used for segmentation…

14. MIT AI Lab Paul Viola Synthesis Results Original Texture The multi-scale statistical model can be used to generate new example textures

15. MIT AI Lab Paul Viola Synthesis Procedure Step 1: Build analysis pyramid Input Image Note: We are using only the Gaussian pyramid here! Normally we use an oriented pyramid... 64x64 2x2

16. MIT AI Lab Paul Viola Synthesis Procedure Step 2: Build synthesis pyramid

17. MIT AI Lab Paul Viola Synthesis Procedure Step 2a: Fill in the top... Pixels are generated by sampling from the analysis pyramid.

18. MIT AI Lab Paul Viola Synthesis Procedure Step 2b: Fill in subsequent levels Pixels are generated by conditional sampling (dependent on the parent).

19. MIT AI Lab Paul Viola Synthesis Procedure Finish the pyramid Decisions made at low resolutions generate discrete features in the final image.

20. MIT AI Lab Paul Viola

21. MIT AI Lab Paul Viola

22. MIT AI Lab Paul Viola

23. MIT AI Lab Paul Viola

24. MIT AI Lab Paul Viola Detection Results Non-face test images Web face test images

25. MIT AI Lab Paul Viola Pruning the density estimator Reduce the number of bins through clustering 1000 bins or less! Result: Detection/Classification is faster than template correlation

26. MIT AI Lab Paul Viola Key facial features - determined automatically - located automatically Multi-scale features which are come from the face model can be automatically detected for many individuals

27. MIT AI Lab Paul Viola Another key feature

28. MIT AI Lab Paul Viola New Face Recognition Algorithm Measure the occurrence and location of “key” facial features. Facial identity depends both on the types of features and their location. Relation to Active Search… –Match measure is a histogram of multiscale features –Like color histogram, Active Search can be used...

29. MIT AI Lab Paul Viola Presentation on Image Database Technology