1. Object Recognition in Images Slides originally created by Bernd Heisele

2. Fall 2002Pattern Recognition for Vision Object Detection Object Identification Where is Jane? Where is a Face? Is there a face in the image? Who is it? Is it Jane or Erik? Object Recognition

3. Fall 2002Pattern Recognition for Vision Applicable to many classes of objects Invariance: “External parameters” Pose Illumination “Internal parameters” Person identity Facial expression General Problems of Recognition

4. Fall 2002Pattern Recognition for Vision Task Given an input image, determine if there are objects of a given class (e.g. faces, people, cars..) in the image and where they are located. Object Detection

5. Fall 2002Pattern Recognition for Vision Detection—Problems 1.Classifier must generalize over all exemplars of one class. 2.Negative class consists of everything else. 3.High accuracy (small FP rate) required for most applications.

6. Fall 2002Pattern Recognition for Vision Feature Extraction Search for faces at different resolutions and locations Feature vector (x 1, x 2,…, x n ) Classifier Off-line training Face examples Non-face examples Pixel pattern Classification Result Face Detection – basic scheme

7. Fall 2002Pattern Recognition for Vision Training Set Train Classifier Labeled Test Set False Positive Correct Classify Sensitivity Training and Testing

8. Fall 2002Pattern Recognition for Vision Gray Histogram Equalization Masking 19x19 283 [0, 1] Histogram Equalization Gradient x-y-Sobel Filtering 17x17 [0, 1] Haar Wavelet Transform 1740 [0, 1] Wavelets Normalization Image Features

9. Fall 2002Pattern Recognition for Vision ROC Image Features

10. Fall 2002Pattern Recognition for Vision Classifiers

11. Fall 2002Pattern Recognition for Vision Real 2900 faces + 2900 mirrored faces 3D Morphing: Synthetic (T. Vetter, Univ. of Freiburg) Illumination Rotation Identity + = Positive Training Data

12. Fall 2002Pattern Recognition for Vision Real vs. Synthetic

13. Fall 2002Pattern Recognition for Vision Problem: 1 face in 116.440 examined windows Initial set of 25.000 non- faces Retrain Classifier Add to training set Determine false- positives on large set of non-face images Bootstrapping Negative Training Data

14. Fall 2002Pattern Recognition for Vision 6.7 FP/image 0.9 FP/image Bootstrapping

15. Fall 2002Pattern Recognition for Vision Performance of Global Face Detectors

16. Fall 2002Pattern Recognition for Vision Rotation in the image plane Rotation invariant features Apply 2D rotation to image Rotation out of image plane Component-based classification Train on rotated faces Rotation

17. Fall 2002Pattern Recognition for Vision Single template Component templates Global vs. Components

18. Fall 2002Pattern Recognition for Vision Eyes Nose Mouth 1st Level: Components classify 2nd Level: Geometrical relation between components maximum response of each component classifier + x, y location classify Component-based Detection

19. Fall 2002Pattern Recognition for Vision Synthetic faces: 7 different 3-D head models 2,500 faces Rotation: -30 o to + 30 o 3-D correspondences for automatic location of components Components: discriminatory robust against changes in pose and illumination Learning Components

20. Fall 2002Pattern Recognition for Vision Start with small initial regions Expand into one of four directions Extract new components from images Train SVM classifiers Choose best expansion according to error bound of SVMs Learning Components—One Way To Do It

21. Fall 2002Pattern Recognition for Vision x*1x*1 x*2x*2 M R Feature Space M Bound on error E < c(R / M) 2 Margin, Radius and Expected Error Cross Validation might be better

22. Fall 2002Pattern Recognition for Vision Some Examples

23. Fall 2002Pattern Recognition for Vision About 40,000 faces 68 people 13 poses 43 illumination conditions 4 different expressions Faces have been manually labeled (only –45 o to 45 o of rotation) Test on CMU PIE Database

24. Fall 2002Pattern Recognition for Vision Heisele, B., T. Serre, M. Pontil, T. Vetter and T. Poggio. Categorization by Learning and Combining Object Parts. In: Advances in Neural Information Processing Systems (NIPS'01), Vancouver, Canada ROC Component vs. Global

25. Fall 2002Pattern Recognition for Vision (From C. Papageorgiou, 2000) Representation: dictionary of Haar wavelets; high dimensional feature space (>1300 features) SVM classifier Pedestrian Detection

26. Fall 2002Pattern Recognition for Vision Examples

27. Fall 2002Pattern Recognition for Vision Components Haar wavelets 5 components Can deal with partial occlusions (From A. Mohan, 1999)

28. Fall 2002Pattern Recognition for Vision Components are small, and prone to false detection, even within the face. Advances on Component-base Face Detection Stan Bileschi Stan Bileschi

29. Fall 2002Pattern Recognition for Vision Use the remainder of the face in the negative training set Positive Negative Training on Faces

30. Fall 2002Pattern Recognition for Vision Red: Trained only with faces. Training on Faces Only Blue: Trained on faces and non- faces.

31. Fall 2002Pattern Recognition for Vision Errors Often, many components classify correctly, with only a few errors

32. Fall 2002Pattern Recognition for Vision Using Models of Pair-wise Positions

33. Fall 2002Pattern Recognition for Vision Pair-wise Biasing Leads to Tightened Result Images

34. Fall 2002Pattern Recognition for Vision Pair-wise Biasing

35. Fall 2002Pattern Recognition for Vision Application: Eye Detection

36. Fall 2002Pattern Recognition for Vision Task: Given an image of an object of a particular class (e.g. face) identify which exemplar it is. B B D D C A C A Identification

37. Fall 2002Pattern Recognition for Vision Identification—Problems 1.Multi-class problem 2.Classifier must distinguish between exemplars that might look very similar. 3.Classifier has to reject exemplars that were not in the training database.

38. Fall 2002Pattern Recognition for Vision Limited information in a single face image Illumination Rotation Problems in Face Identification

39. Fall 2002Pattern Recognition for Vision Face Image Gray, Gradient, Wavelets, … Feature extraction Support Vector Machine, …. Classifier Training Data A Identification Result System Architecture

40. Fall 2002Pattern Recognition for Vision Multi-class Classification with SVM A Bottom-Up 1vs1 A or B or C or D CDB A or B C or D Training: L (L-1) / 2 Classification : L-1 A BA,C,D CA,B,D DA,B,C 1 vs. All B,C,D Training: L Classification : L

41. Fall 2002Pattern Recognition for Vision Global Approach Detect and extract face Feed gray values into N SVMs Classify based on maximum output SVM A SVM B SVM C SVM D Max Operation Identification result

42. Fall 2002Pattern Recognition for Vision Global Approach with Clustering SVM C SVM C SVM C SVM C Max Operation Identification result Partition training images of each person into viewpoint- specific clusters Train a linear SVM on each cluster Take maximum over all SVM outputs

43. Fall 2002Pattern Recognition for Vision Component-based Approach Detect and extract components Feed gray values of components to N SVMs Take max. over all SVM outputs SVM A SVM B SVM C SVM D Max Operation Identification Results

44. Fall 2002Pattern Recognition for Vision Component-based Face Recognition with 3D Morphable Models Feature 1 != Feature 2 Feature 1 = Feature 2 Why Components for Identification? Jennifer Huang

45. Fall 2002Pattern Recognition for Vision Heisele, B., P. Ho and T. Poggio. Face Recognition with Support Vector Machines: Global Versus Component-based Approach, International Conference on Computer Vision (ICCV'01), Vancouver, Canada, Vol. 2, 688-694, 2001. More ROC Curves

46. Fall 2002Pattern Recognition for Vision 3D morphable models A Morphable Model for the Synthesis of 3D Faces. Blanz, V. and Vetter, T. SIGGRAPH'99 Conference Proceedings, pp. 187-194 Training data for component-based face recognition Morphable Models for Face Identification, Jennifer Huang Jennifer Huang

47. Fall 2002Pattern Recognition for Vision Generation of 3D head model from two images See class on Morphable Models Morphable Model Jennifer Huang

48. Fall 2002Pattern Recognition for Vision Synthetic images are easily rendered from 3D head model under varying illuminations and rotations in depth Some Training Images Jennifer Huang

49. Fall 2002Pattern Recognition for Vision Preliminary Results on Synthetic Images Jennifer Huang

50. Fall 2002Pattern Recognition for Vision Problems Encountered: Detection Inaccurate Component detection Recognition Accuracy of 3D models Choice of Illumination and Pose Current Work – Testing on Real Images Jennifer Huang

51. Fall 2002Pattern Recognition for Vision Literature B. Heisele, A. Verri and T. Poggio: Learning and Vision Machines. Proceedings of the IEEE, Visual Perception: Technology and Tools, Vol. 90, No. 7, pp. 1164-1177, 2002. See also CBCL Web page