Presentation is loading. Please wait.

Presentation is loading. Please wait.

IPCV 2006 Budapest130/08/2006 Face Detection and Recognition Zoltán VÁMOSSY Budapest Tech, John von Neumann Faculty of Informatics.

Similar presentations

Presentation on theme: "IPCV 2006 Budapest130/08/2006 Face Detection and Recognition Zoltán VÁMOSSY Budapest Tech, John von Neumann Faculty of Informatics."— Presentation transcript:


2 IPCV 2006 Budapest130/08/2006 Face Detection and Recognition Zoltán VÁMOSSY Budapest Tech, John von Neumann Faculty of Informatics

3 IPCV 2006 Budapest230/08/2006 Face tasks in CV Face detection: Are there any face on the image (where)? –concerns with a category of object Face localization: –Aim to determine the image position of a single face –A simplified detection problem with the assumption that an input image contains only one face Facial feature extraction: –To detect the presence and location of features such as eyes, nose, nostrils, eyebrow, mouth, lips, ears, etc –Usually assume that there is only one face in an image Face recognition (identification): concerns with individual identity Facial expression recognition Human pose estimation and tracking

4 IPCV 2006 Budapest330/08/2006 Authenticate Typical Biometric Authentication Workflow Database Enrollment subsystem Biometric reader Feature Extractor Authentication subsystem Biometric reader Feature Extractor Biometric Matcher Enroll Match or No Match … Template … Template

5 IPCV 2006 Budapest430/08/2006 Identification (1:N) Biometric reader Biometric Matcher Identification vs. Verification Database Verification (1:1) Biometric reader Biometric Matcher ID Database This person is Emily Dawson Match I am Emily Dawson

6 IPCV 2006 Budapest530/08/2006 Applications Face Recognition Applications Automatic border control Samsung’s Magicgate: Sydney airport door lock control

7 IPCV 2006 Budapest630/08/2006 A Solved Problem? Excellent results and products: –Fast, multi pose, partial occlusion Is face recognition and detection a solved problem? Not quite Omron’s face detector [Liu et al. 04]

8 IPCV 2006 Budapest730/08/2006 OutlineOutline Relevant psychophysics/neuroscience issues Face Detection –Detection on gray scale images –Detection on color images –Research directions Face Recognition –Still image face recognition Face projects at Budapest Tech

9 IPCV 2006 Budapest830/08/2006 Psychophysics Issues Is face perception the results of holistic/configural or local feature analysis? Is it a dedicated process? –Build a special or generic system Ranking of significance of facial features –Different weights for features Movement and face recognition –Favor video based recognition Role of race/gender/familiarity –Suggest adaptive system

10 IPCV 2006 Budapest930/08/2006 Inverted Faces Prof. Eric H. Chudler Dept. of Anesthesiology University of Washington

11 IPCV 2006 Budapest1030/08/2006 Inverted Faces Prof. Eric H. Chudler Dept. of Anesthesiology University of Washington

12 IPCV 2006 Budapest1130/08/2006 More Inverted Faces Prof. Eric H. Chudler Dept. of Anesthesiology University of Washington

13 IPCV 2006 Budapest1230/08/2006 Negated Faces

14 IPCV 2006 Budapest1330/08/2006 Human Face Recognition Limitations Degraded images Harmon and Julesz, 1973 Human can handle low resolution [Sinha05]

15 IPCV 2006 Budapest1430/08/2006 Outline of Detection Part Objective –Survey major face detection works –Address “how” and “why” questions –Pros and cons of detection methods –Research directions This part of the presentation is based on the survey M-H. Yang: survey.html

16 IPCV 2006 Budapest1530/08/2006 Face Detection Identify and locate human faces in an image regardless of their –position –scale –in-plane rotation –orientation –pose (out-of plane rotation) –and illumination Face is a highly non-rigid object Where are the faces, if any?

17 IPCV 2006 Budapest1630/08/2006 Why Face Detection is Important? First step for any fully automatic face recognition system First step in many surveillance systems Lots of applications A step towards Automatic Target Recognition (ATR) or generic object detection/recognition

18 IPCV 2006 Budapest1730/08/2006 Why Face Detection Is Difficult? Pose (Out-of-Plane Rotation): frontal, 45 degree, profile, upside down Presence or absence of structural components: beard, mustache and glasses Facial expression: face appearance is directly affected by a person's facial expression Occlusion: faces may be partially occluded by other objects Orientation (In-Plane Rotation): face appearance directly vary for different rotations about the camera's optical axis Imaging conditions: lighting (spectra, source distribution and intensity) and camera characteristics (sensor response, gain control, lenses), resolution

19 IPCV 2006 Budapest1830/08/2006 In One Thumbnail Face Image Consider a thumbnail 19 × 19 face pattern possible combination of gray values – = 2 8×361 = Total world population 6,600,000,000 –6,600,000,000 ≅ times more than the world population! Extremely high dimensional space!

20 IPCV 2006 Budapest1930/08/2006 What is a Correct Detection? Different interpretation of “correct detect” Precision of face location Affect the reporting results: detection, false positive, false negative rates

21 IPCV 2006 Budapest2030/08/2006 Receiver Operator Characteristic Curve Useful for detailed performance assessment Plot true positive (TP) proportion against the false positive (FP) proportion for various possible settings False positive: Predict a face when there is actually none False negative: Predict a non-face where there is actually one

22 IPCV 2006 Budapest2130/08/2006 Receiver Operator Characteristic (ROC) Correct identification of nearly all objects will cause a large percentage of unknown objects to be incorrectly classified into some known class

23 IPCV 2006 Budapest2230/08/2006 Face Detector: Ingredients Target application domain: single image, video Representation: holistic, feature, etc Pre processing: histogram equalization, etc Cues: color, motion, depth, voice, etc Search strategy: exhaustive, greedy, focus of attention, etc Classifier design: ensemble, cascade Post processing: interpret detection results

24 IPCV 2006 Budapest2330/08/2006 In our Focus Focus on detecting –upright, frontal faces –in a single gray- scale image –with decent resolution –under good lighting conditions

25 IPCV 2006 Budapest2430/08/2006 Methods to Detect/Locate Faces Knowledge-based methods: –Encode human knowledge of what constitutes a typical face (usually, the relationships between facial features) Feature invariant approaches: –Aim to find structural features of a face that exist even when the pose, viewpoint, or lighting conditions vary Template matching methods: –Several standard patterns stored to describe the face as a whole or the facial features separately Appearance based methods: –The models (or templates) are learned from a set of training images which capture the representative variability of facial appearance

26 IPCV 2006 Budapest2530/08/2006 Agenda (Detection) Detecting faces in gray scale images –K–Knowledge-based –F–Feature-based –T–Template-based –A–Appearance-based Detecting faces in color images Detecting faces in video Performance evaluation Research direction and concluding remarks

27 IPCV 2006 Budapest2630/08/2006 Knowledge-Based Methods Top-down approach: Represent a face using a set of human-coded rules Examples: –The center part of face has uniform intensity values –The difference between the average intensity values of the center part and the upper part is significant –A face often appears with two eyes that are symmetric to each other, a nose and a mouth Use these rules to guide the search process

28 IPCV 2006 Budapest2730/08/2006 Knowledge-Based Method: [Yang and Huang 94] Multi-resolution focus-of- attention approach Level 1 (lowest resolution): apply the rule “the center part of the face has 4 cells with a basically uniform intensity” to search for candidates Level 2: local histogram equalization followed by edge detection Level 3: search for eye and mouth features for validation

29 IPCV 2006 Budapest2830/08/2006 Knowledge-Based Method: [Kotropoulos & Pitas 94] Horizontal/vertical projection to search for candidates Search eyebrow/eyes, nostrils/nose for validation Difficult to detect multiple people or in complex background

30 IPCV 2006 Budapest2930/08/2006 Knowledge-based Methods: Summary Pros: –Easy to find simple rules to describe the features of a face and their relationships –Based on the coded rules, facial features in an input image are extracted first and face candidates are identified –Work well for face localization in uncluttered background Cons: –Difficult to translate human knowledge into rules precisely: detailed rules fail to detect faces and general rules may find many false positives –Difficult to extend this approach to detect faces in different poses: implausible to enumerate all the possible cases

31 IPCV 2006 Budapest3030/08/2006 Agenda (Detection) Detecting faces in gray scale images –K–Knowledge-based –F–Feature-based –T–Template-based –A–Appearance-based Detecting faces in color images Detecting faces in video Performance evaluation Research direction and concluding remarks

32 IPCV 2006 Budapest3130/08/2006 Feature-based Methods Bottom-up approach: Detect facial features (eyes, nose, mouth, etc) first –How can we detect facial features: edge, intensity, shape, texture, color, etc –Aim to detect invariant features Group features into candidates and verify them

33 IPCV 2006 Budapest3230/08/2006 Feature-based Example [Yang 2003] Use the Canny operator to find an edge map Separate face from background by grouping edges Best fit an ellipse to the remaining edges

34 IPCV 2006 Budapest3330/08/2006 Feature Grouping [Yow and Cipolla 90] Apply a 2nd derivative Gaussian filter to search for interest points Group the edges near interest points into regions Each feature and grouping is evaluated within a Bayesian network Handle a few poses

35 IPCV 2006 Budapest3430/08/2006 Feature Grouping [Yow and Cipolla 90] Face model and component Model facial feature as pair of edges Apply interest point operator and edge detector to search for features Using Bayesian network to combine evidence

36 IPCV 2006 Budapest3530/08/2006 Random Graph Matching [Leung et al. 95] Formulate as a problem to find the correct geometric arrangement of facial features –Facial features are defined by the average responses of multi- orientation, multi-scale Gaussian derivative filters –Learn the configuration of features with Gaussian distribution of mutual distance between facial features Random graph matching among the candidates to locate faces

37 IPCV 2006 Budapest3630/08/2006 Feature-Based Methods: Summary Pros: –Features are invariant to pose and orientation change Cons: –Difficult to locate facial features due to several corruption (illumination, noise, occlusion) –Difficult to detect features in complex background

38 IPCV 2006 Budapest3730/08/2006 Agenda (Detection) Detecting faces in gray scale images –K–Knowledge-based –F–Feature-based –T–Template-based –A–Appearance-based Detecting faces in color images Detecting faces in video Performance evaluation Research direction and concluding remarks

39 IPCV 2006 Budapest3830/08/2006 Template Matching Methods Store a template –Predefined: based on edges or regions –Deformable: based on facial contours (e.g., Snakes) Templates are hand-coded (not learned) Use correlation to locate faces

40 IPCV 2006 Budapest3930/08/2006 Face Template Use relative pair-wise ratios of the brightness of facial regions (14 × 16 pixels): the eyes are usually darker than the surrounding face [Sinha 94] Use average area intensity values than absolute pixel values See also Point Distribution Model (PDM) [Lanitis et al. 95]

41 IPCV 2006 Budapest4030/08/2006 Template-Based Methods: Summary Pros: –Relatively simple Cons: –Templates needs to be initialized near the face images –Difficult to enumerate templates for different poses (similar to knowledge- based methods)

42 IPCV 2006 Budapest4130/08/2006 Agenda (Detection) Detecting faces in gray scale images –K–Knowledge-based –F–Feature-based –T–Template-based –A–Appearance-based Detecting faces in color images Detecting faces in video Performance evaluation Research direction and concluding remarks

43 IPCV 2006 Budapest4230/08/2006 Appearance-Based Methods Idea: train a classifier using positive (and usually negative) examples of faces Representation Pre processing Train a classifier Search strategy Post processing

44 IPCV 2006 Budapest4330/08/2006 Appearance-Based Methods: Classifiers Neural network: Multilayer Perceptrons Principal Component Analysis (PCA), Factor Analysis Support vector machine (SVM) Mixture of PCA, Mixture of factor analyzers Distribution-based method Naive Bayes classifier Hidden Markov model Sparse network of winnows (SNoW) Kullback relative information Inductive learning: C4.5 AdaBoost (Jones and Viola) …

45 IPCV 2006 Budapest4430/08/2006 Representation Holistic: Each image is raster scanned and represented by a vector of intensity values Block-based: Decompose each face image into a set of overlapping or non-overlapping blocks –At multiple scale –Further processed with vector quantization, Principal Component Analysis, etc.

46 IPCV 2006 Budapest4530/08/2006 Face and Non-Face Examples Positive examples: –Get as much variation as possible –Manually crop and normalize each face image into a standard size face (e.g., 19 × 19 pixels) –Creating virtual examples [Sung and Poggio 94] Negative examples: –Fuzzy idea –Any images that do not contain faces –A large image subspace

47 IPCV 2006 Budapest4630/08/2006 Distribution-Based Method [Sung & Poggio 94] Masking: reduce the unwanted noise in a face pattern Illumination gradient correction: find the best fit brightness plane and then subtracted from it to reduce heavy shadows caused by extreme lighting angles Histogram equalization: compensates the imaging effects due to changes in illumination and different camera input gains

48 IPCV 2006 Budapest4730/08/2006 Creating Virtual Positive Examples Simple and very effective method Randomly mirror, rotate, translate and scale face samples by small amounts Less sensitive to alignment error

49 IPCV 2006 Budapest4830/08/2006 Search over Space and Scale Scan an input image at one-pixel Downsample the input image by a increments horizontally and vertically factor of 1.2 and continue to search

50 IPCV 2006 Budapest4930/08/2006 Continue to Search over Space and Scale Continue to downsample the input image and search until the image size is too small

51 IPCV 2006 Budapest5030/08/2006 Detecting Faces in Different Pose Probabilistic Modeling of Local Appearance Extend to detect faces in different pose with multiple detectors Each detector specializes to a view: frontal, left pose and right pose [Schneiderman and Kanade 98]

52 IPCV 2006 Budapest5130/08/2006 Viola and Jones Goals Motivation Features (rectangle features & integral image) Learning classifier functions (AdaBoost) Cascade of classifiers Results Viola & Jones: Robust Real-time Face Detection

53 IPCV 2006 Budapest5230/08/2006 Goals of Viola and Jones Robust real-time face detection in images  Decide for each sub-window (starting at 24x24) of the image whether it is a face Viola & Jones: Robust Real-time Face Detection

54 IPCV 2006 Budapest5330/08/2006 MotivationMotivation Rapid face detection in images and image sequences Previous approaches too slow –They are using auxiliary information (image differences, colors) Only grey scale images needed Viola & Jones: Robust Real-time Face Detection

55 IPCV 2006 Budapest5430/08/2006 The Framework of the Method Haar-like Feature: A feature representation Integral Image: A new image representation and a fast Haar-like feature calculation method Perceptron: Classifier, uses the features Boost Algorithm: An efficient classifier generating algorithm Cascade: Method for combining classifiers Viola & Jones: Robust Real-time Face Detection

56 IPCV 2006 Budapest5530/08/2006 FeaturesFeatures Detection based on features, not pixels Why? –Features can encode domain knowledge that is difficult to learn –Faster than a pixel-based system –Scale independent Viola & Jones: Robust Real-time Face Detection

57 IPCV 2006 Budapest5630/08/2006 Rectangle Features (Sum of pixels in black rectangles) - (sum of pixels in white rectangles) Features are very simple  rapid computation How to compute them rapidly? Viola & Jones: Robust Real-time Face Detection

58 IPCV 2006 Budapest5730/08/2006 Integral Image Rectangle Features can be computed rapidly using an intermediate representation of the image: integral image Value at point (x,y) is the sum of all the pixel values above and to the left Viola & Jones: Robust Real-time Face Detection

59 IPCV 2006 Budapest5830/08/2006 Integral Image The integral image can be computed with the following recurrences: where s(x, y): cumulative column sum One pass over the image Viola & Jones: Robust Real-time Face Detection

60 IPCV 2006 Budapest5930/08/2006 Computation of Rectangular Sums Sum of rectangle D: ii(4) – ii(3) – ii(2) + ii(1) Viola & Jones: Robust Real-time Face Detection

61 IPCV 2006 Budapest6030/08/2006 Integral Image Rectangular sum can be computed in four array references Two-rectangle features need six array references Three-rectangle features need eight array references Four-rectangle features need nine array references Viola & Jones: Robust Real-time Face Detection

62 IPCV 2006 Budapest6130/08/2006 Feature properties Rough, but sensitive to simple image structures Only 2 orientations Extremely efficient Compensates the disadvantages Face detection for an entire image at every scale in real-time Viola & Jones: Robust Real-time Face Detection

63 IPCV 2006 Budapest6230/08/2006 Learning Classification Functions Given: feature set and training set of positive + negative face images –In principle any machine learning approach can be used (SVM, neural network, …) –But: rectangle features for each sub- window Experiments showed that only a few features needed for an efficient classifier How to find them? Viola & Jones: Robust Real-time Face Detection

64 IPCV 2006 Budapest6330/08/2006 AdaBoost-Based Detector [Viola and Jones 01] Main idea: –Feature selection: select important features –Focus of attention: focus on potential regions –Use an integral graph for fast feature evaluation Use AdaBoost to learn –A small set of important features (feature selection)  sort them in the order of importance  each feature can be used as a simple (weak) classifier –A cascade of classifiers that  combine all the weak classifiers to do a difficult task  filter out the regions that most likely do not contain faces Viola & Jones: Robust Real-time Face Detection

65 IPCV 2006 Budapest6430/08/2006 Feature Selection [Viola and Jones 01] Training: If x is a face, then x –most likely has feature 1 (easiest feature, and of greatest importance) –very likely to have feature 2 (easy feature) –…–… –likely to have feature n (more complex feature, and of less importance since it does not exist in all the faces in the training set) Testing: Given a test sub-image image x’ –if x’ has feature 1:  Test whether x’ has feature 2 –Test whether x’ has feature n –…–… – else …  else, it is not face –else, it is not a face Similar to decision tree Viola & Jones: Robust Real-time Face Detection

66 IPCV 2006 Budapest6530/08/2006 AdaBoostAdaBoost Variant of AdaBoost is used both to select features and train the classifier AdaBoost is used to boost classification performance of a simple (weak) learning algorithm It combines a collection of weak classifier to form a stronger one The weak learning algorithm is designed to select the single rectangle feature which best separates the positive and negative examples Viola & Jones: Robust Real-time Face Detection

67 IPCV 2006 Budapest6630/08/2006 AdaBoostAdaBoost Searches over the set of possible weak classifiers and selects those with the lowest classification error Greedy feature selection process Efficient for selecting a small number of “good” features (f j (x)) with significant variety Weak classifier Error rates between (early rounds) and 0.4 and 0.5 (later rounds) Viola & Jones: Robust Real-time Face Detection

68 IPCV 2006 Budapest6730/08/2006 AdaBoost in Detail Given example images y i = 0,1 pos./neg. Initialize weights for y i = 0, 1: where m and l number of negatives and positives For t = 1,…,T : 1.Normalize weights 2.For each feature j train a classifier h j which is restricted to using a single feature, the error 3.Choose the classifier h t with lowest error 4.Update weights: where e i = 0, if x i correctly classified, else e i = 1 Final strong classifier: Viola & Jones: Robust Real-time Face Detection

69 IPCV 2006 Budapest6830/08/2006 Feature Selection by AdaBoost First and second feature selected by AdaBoost 60 microprocessor instructions Viola & Jones: Robust Real-time Face Detection

70 IPCV 2006 Budapest6930/08/2006 AdaBoost – Learning Results PIII 700 MHz, feature-classifier Detection rate: 95% False positive rate: 1/14084 Takes 0.7 seconds to scan 384x288 image Adding features to the classifier increases computation time Viola & Jones: Robust Real-time Face Detection

71 IPCV 2006 Budapest7030/08/2006 Cascade of Classifiers 0.7 s for 1 image?  too slow => Idea: Cascade of Classifiers to reduce computation time Simple Classifiers to reject majority of negative sub-windows: many sub-windows have no faces Viola & Jones: Robust Real-time Face Detection all sub-windows 213 TT T Further processing rejected sub-windows F F F

72 IPCV 2006 Budapest7130/08/2006 Cascade of Classifiers - Experiment Monolithic 200-feature-classifier vs. cascade of ten 20-feature-classifier Trained with 5000 faces and non-face sub-windows Viola & Jones: Robust Real-time Face Detection

73 IPCV 2006 Budapest7230/08/2006 Results - Training Structure of the Cascade: –32 layers with 4297 features –First layer uses two features, 60% non-faces rejected, close to 100% faces detected –Second layer uses five features, rejects 80% non-faces –3rd 20-feature-classifier –4th 50-feature-classifier –5th 100-feature-classifier –20th 200-feature-classifier Viola & Jones: Robust Real-time Face Detection

74 IPCV 2006 Budapest7330/08/2006 Results - Speed Speed of the detector depends on the number of features evaluated On the MIT+CMU test set, an average of 8 features (out of 4297), on a 700 MHz Pentium III, 384x288 takes 0.067s (2001) Viola & Jones: Robust Real-time Face Detection

75 IPCV 2006 Budapest7430/08/2006 OutlookOutlook Viola & Jones improved their detector to detect not only frontal faces, but also non-upright and non-frontal faces Additional rectangle features Pose estimator Classifier for each rotation class (12) Viola & Jones: Robust Real-time Face Detection

76 IPCV 2006 Budapest7530/08/2006 Haar-like Features Extension 45º rotated rectangle feature by Lienhart Feature is represented by (type, x, y, w, h): type [1(a), 2(b), etc.]; (x, y) for location in detection window and (w, h) for feature size Number of features: the window size 24x24, there are 117,941 different features, a large number

77 IPCV 2006 Budapest7630/08/2006 Integral Image, Fast Feature Calculation Sum Area Table Rotated Sum Area Table Corresponding to the origin feature and 45º rotated feature, there are two types of Integral Image, called by Lienhart as Sum Area Table and Rotated Sum Area Table SAT(x,y) RSAT(x,y)

78 IPCV 2006 Budapest7730/08/2006 Integral Image, Fast Feature Calculation SAT and RSAT Calculation: Sum of Rect Calculation: Feature value: calculated as the weighted sum of the rectangles

79 IPCV 2006 Budapest7830/08/2006 Appearance-Based Methods: Summary Pros: –Use powerful machine learning algorithms –Has demonstrated good empirical results –Fast and fairly robust –Extended to detect faces in different pose and orientation Cons –Usually needs to search over space and scale –Need lots of positive and negative examples –Limited view-based approach

80 IPCV 2006 Budapest7930/08/2006 Agenda (Detection) Detecting faces in gray scale images –K–Knowledge-based –F–Feature-based –T–Template-based –A–Appearance-based Detecting faces in color images Detecting faces in video Performance evaluation Research direction and concluding remarks

81 IPCV 2006 Budapest8030/08/2006 Color-Based Face Detector Distribution of skin color across different ethnic groups –Under controlled illumination conditions: compact –Arbitrary conditions: less compact Color space –RGB, normalized RGB, HSV, HIS, YCrCb, YIQ, UES, CIE XYZ, CIE LIV, … Statistical analysis –Histogram, look-up table, Gaussian model, mixture model, …

82 IPCV 2006 Budapest8130/08/2006 Methods of Skin Detection Pixel-Based Methods –Classify each pixel as skin or non-skin individually, independently from its neighbors –Simpler, faster –Color Based Methods fall in this category Region Based Methods –Try to take the spatial arrangement of skin pixels into account during the detection stage to enhance the methods performance –More computation extensive –Additional knowledge in terms of texture etc. are required

83 IPCV 2006 Budapest8230/08/2006 Skin Color Based Methods - Advantages Allows fast processing Robust to geometric variations of the skin patterns Robust under partial occlusion Robust to resolution changes Experience suggests that human skin has a characteristic color, which is easily recognized by humans

84 IPCV 2006 Budapest8330/08/2006 Skin Modelling Explicitly defined skin region –Using simple rules Non-parametric skin distribution modeling - Estimate skin color distribution from the training data without deriving an explicit model of the skin –Look up table or Histogram Model –Bayes Classifier Parametric skin distribution modeling - Deriving a parametric model from the training set

85 IPCV 2006 Budapest8430/08/2006 Explicit Rule To define explicitly (through a number of rules) the boundaries skin cluster in some color space E.g. assuming in space (RGB, rgb, HSI, YUV, etc) –(R’,G’,B’) is classified as skin if:  R’ > 95 and G’ > 40 and B’ > 20,  Max{R’,G’,B’} – min{R’,G’,B’} > 15, and  |R’-G’| > 15 and R’ > G’ and R’ > B’ The simplicity of this method is attractive Simplicity of skin detection rules leads to construction of a very rapid classifier The main difficulty achieving high recognition rates need to find –Good color space –Adequate decision rules Peer (2003)

86 IPCV 2006 Budapest8530/08/2006 Example: YC r C b Space [Yoo and Kim] Skin-colors form a cluster in YC r C b color space We can approximate the skin color region with several lines or with an ellipse

87 IPCV 2006 Budapest8630/08/2006 Color Histogram If –h(r k, g k, b k )=n k, Then –p(r k, g k, b k ) = n k / N –p(r’ k, g’ k, b’ k ) = (N-n k ) / N That is: –If there are n k skin color (r k, g k, b k ) pixels in the image, the probability of any pixel which is a skin pixel is n k /N –and the probability of any pixel which is NOT a skin pixel is (N-n k )/N

88 IPCV 2006 Budapest8730/08/2006 Bayes Probability - Recall P(A,B) = P(A|B) x P(B) P(A,B) = P(B|A) x P(A) If P(B) + P(~B) = 1, and P(A) + P(~A) = 1, then –P(A) = P(A|B) x P(B) + P(A|~B) x P(~B) –P(B) = P(B|A) x P(A) + P(B|~A) x P(~A) P(A|B) x P(B) = P(B|A) x P(A) P(B|A) x P(A) P(A|B) = P(B) P(B|A) x P(A) P(A|B) = P(B|A) x P(A) + P(B|~A) x P(~A) )

89 IPCV 2006 Budapest8830/08/2006 Bayes Classifier (1) p(r,g,b|skin) is a conditional probability - a probability of observing color (red, or green, or blue), knowing that we see a skin pixel A more appropriate measure for skin detection would be p(skin|r,g,b) - a probability of observing skin, given a concrete (r,g,b) color value To compute this probability, the Bayes rule is used:

90 IPCV 2006 Budapest8930/08/2006 Bayes Classifier (2) p(r,g,b|skin) and p(r,g,b|non-skin) can be computed from skin and non-skin color histograms The prior probabilities p(skin) and p(non-skin) can also be estimated from the overall number of skin and non-skin samples in the training set An inequality p(skin|r,g,b) >= Q, where Q is a threshold value, can be used as a skin detection rule

91 IPCV 2006 Budapest9030/08/2006 Bayes Classifier (3) Consider normalised RGB space Histograms approximate probability densities –p(skin) can be estimated as the ratio of the skin size N skin to the size of the image N total, –p(non-skin) can be estimated as the ratio of the skin size N non-skin to the size of the image N total, –p((r,g,b)|skin) the probability density of (r,g,b) given the skin, –p((r,g,b)|non-skin) the probability density of (r,g,b) given the non-skin,

92 IPCV 2006 Budapest9130/08/2006 Bayes Classifier (4) Then we can compute for any color (r,g,b) the probability p(skin|(r,g,b)) → Probability Map of the image Skin Probability Map (SPM) That is: Ratio of histograms

93 IPCV 2006 Budapest9230/08/2006 Bayes Classifier from [Rehg & Jones] Skin photos: Non skin photos:

94 IPCV 2006 Budapest9330/08/2006 Results from [Rehg & Jones] Used images to build a general color model Density is concentrated around the gray line and is more sharply peaked at white than black Most colors fall on or near the gray line Black and white are by far the most frequent colors, with white occurring slightly more frequently There is a marked skew in the distribution toward the red corner of the color cube 77% of the possible 24 bit RGB colors are never encountered (i.e. the histogram is mostly empty) 52% of web images have people in them

95 IPCV 2006 Budapest9430/08/2006 Marginal Distributions [Rehg & Jones]

96 IPCV 2006 Budapest9530/08/2006 Skin model [Rehg & Jones]

97 IPCV 2006 Budapest9630/08/2006 Non Skin Model [Rehg & Jones]

98 IPCV 2006 Budapest9730/08/2006 Skin and Non-Skin Color Model Analyze 1 billion labeled pixels Skin and non-skin models A significant degree of separation between skin non-skin model Achieves 80% detection rate with 8.5% false positives Histogram method outperforms Gaussian mixture model

99 IPCV 2006 Budapest9830/08/2006 Experimental Results [Jones and Rehg 02] May have lots of false positives in the raw results Need further processing to eliminate false negatives and group color pixels for face detection

100 IPCV 2006 Budapest9930/08/2006 Results from [Zarit et al.] They compared 5 different color spaces CIELab, HSV, HS, Normalized RGB and YCrCb Four different metrics are used to evaluate the results of the skin detection algorithms –C %– Skin and Non Skin pixels identified correctly –S %– Skin pixels identified correctly –SE – Skin error – skin pixels identified as non skin –NSE – Non Skin error – non skin pixels identified as skin They compared the 5 color space with 2 color models – look up table and Bayes classifier

101 IPCV 2006 Budapest10030/08/2006 Look up table results HSV, HS gave the best results Normalized rg is not far behind CIELAB and YCrCb gave poor results

102 IPCV 2006 Budapest10130/08/2006 Color-Based Face Detector: Summary Pros: –Easy to implement –Effective and efficient in constrained environment –Insensitive to pose, expression, rotation variation Cons: –Sensitive to environment and lighting change –Noisy detection results (body parts, skin-tone line regions)

103 IPCV 2006 Budapest10230/08/2006 Agenda (Detection) Detecting faces in gray scale images –K–Knowledge-based –F–Feature-based –T–Template-based –A–Appearance-based Detecting faces in color images Detecting faces in video Performance evaluation Research direction and concluding remarks

104 IPCV 2006 Budapest10330/08/2006 Video-Based Face Detector Motion cues: –Frame differencing –Background modeling and subtraction Can also used depth cue (e.g., from stereo) when available Reduces the search space dramatically

105 IPCV 2006 Budapest10430/08/2006 HONDA Humanoid Robot: ASIMO Using motion and depth cue –Motion cue: from a gradient-based tracker –Depth cue: from stereo camera Dramatically reduce search space Cascade face detector

106 IPCV 2006 Budapest10530/08/2006 Video-Based Detectors: Summary Pros: –An easier problem than detection in still images –Use all available cues: motion, depth, voice, etc. to reduce search space Cons: –Need to efficient and effective methods to process the multimodal cues –Data fusion

107 IPCV 2006 Budapest10630/08/2006 Agenda (Detection) Detecting faces in gray scale images –K–Knowledge-based –F–Feature-based –T–Template-based –A–Appearance-based Detecting faces in color images Detecting faces in video Performance evaluation Research direction and concluding remarks

108 IPCV 2006 Budapest10730/08/2006 Performance Evaluation Need to set the evaluation criteria/protocol –Training set –Test set –What is a correct detect? –Detection rate: false positive/negative –Precision of face location –Speed: training/test stage

109 IPCV 2006 Budapest10830/08/2006 Training Sets Cropped and pre processed data set with face and non-face images provided by MIT CBCL: datasets/index.html

110 IPCV 2006 Budapest10930/08/2006 Standard Test Sets MIT test set: subsumed by CMU test set CMU test set ( (de facto benchmark): 130 gray scale images with a total of 507 frontal faces CMU profile face test set: 208 images with faces in profile views Kodak data set (Eastman Kodak Corp): faces of multiple size, pose and varying lighting conditions in color images

111 IPCV 2006 Budapest11030/08/2006 CMU Test Set I: Upright Frontal Faces 130 images with 507 frontal faces Collected by K. Collected by K.-K. Sung K. Sung and H. Rowley Including 23 images used in [Sung and Poggio 94] Some images have low resolution De facto benchmark

112 IPCV 2006 Budapest11130/08/2006 CMU Test Sets: Rotated and Profile Faces 50 images with 223 faces in-plane orientation, collected by H. Rowley 208 images with 441 faces, collected by H. Schneiderman

113 IPCV 2006 Budapest11230/08/2006 Agenda (Detection) Detecting faces in gray scale images –K–Knowledge-based –F–Feature-based –T–Template-based –A–Appearance-based Detecting faces in color images Detecting faces in video Performance evaluation Research direction and concluding remarks

114 IPCV 2006 Budapest11330/08/2006 Research Issues Representation: How to describe a typical face? Scale: How to deal with face of different size? Search strategy: How to spot these faces? Speed: How to speed up the process? Precision: How to locate the faces precisely? Post processing: How to combine detection results?

115 IPCV 2006 Budapest11430/08/2006 Face Detection: A Solved Problem? Not quite yet… Factors: –Shadows –Occlusions –Robustness –Resolution Lots of potential applications Can be applied to other domains

116 IPCV 2006 Budapest11530/08/2006 Outline of Recognition Part Objective –Categorization –Brief description of representative face recognition techniques –Open research issues This part of the presentation is based on the survey W. Zhao, R. Chellappa, J. Phillips and A. Rosenfeld: Face Recognition: A Literature Survey ACM Computing Survey Vol. 35, Dec. 2003/

117 IPCV 2006 Budapest11630/08/2006 Recognition from Intensity Images High-level Categorization 1. Holistic matching methods –Classification using whole face region 2. Feature-based (structural) matching methods –Structural classification using local features 3. Hybrid Methods –Using local features and whole face region

118 IPCV 2006 Budapest11730/08/2006 Holistic Approaches Principal Component Analysis (PCA) –Eigenface –Fisherface/Subspace LDA –SVM –ICA Other Representations –LDA/FLA –PDBNN

119 IPCV 2006 Budapest11830/08/2006 What is PCA? Principal component analysis (PCA) is a technique for extracting a smaller set of variables with less redundancy from high- dimensional data sets in order to retain as much of the information as possible A small number of principal components often give most of the structure in the data IPCV, Saint-Etienne, 2004, Prof. V. Bochko

120 IPCV 2006 Budapest11930/08/2006 Standard PCA Assume that an n-dimensional observed vector is The vector is centered where The covariance matrix is where is an expectation The eigendecomposition is The matrix of eigenvectors is where The eigenvectors corresponds to eigenvalues The eigenvalues are The eigenvalues are in descending order The first k principal components are Finally, the reconstruction is IPCV, Saint-Etienne, 2004, Prof. V. Bochko

121 IPCV 2006 Budapest12030/08/2006 Facial Recognition: PCA - Overview Create training set of faces and calculate the eigenfaces Project the new image onto the eigenfaces Check if image is close to “face space” Check closeness to one of the known faces Add unknown faces to the training set and re-calculate

122 IPCV 2006 Budapest12130/08/2006 Facial Recognition: PCA Training Find average of training images Subtract average face from each image Create covariance matrix Generate eigenfaces Each original image can be expressed as a linear combination of the eigenfaces – face space

123 IPCV 2006 Budapest12230/08/2006 Facial Recognition: PCA Recognition A new image is project into the “facespace” Create a vector of weights that describes this image The distance from the original image to this eigenface is compared If within certain thresholds then it is a recognized face

124 IPCV 2006 Budapest12330/08/2006 Feature Based Approaches Elastic bunch graph matching using Gabor wavelets

125 IPCV 2006 Budapest12430/08/2006 Detecting Feature Points Using of Gabor Wavelets Create 5 frequencies, 8 angles Spatial figureTop view

126 IPCV 2006 Budapest12530/08/2006 Gabor filters The responses of the filters 2D Gabor filters can be used to detect dimensions along a specific orientation Jet: set of 40 complex coefficients for the reference points OriginalFilteredResult

127 IPCV 2006 Budapest12630/08/2006 Face Graph Facial fiducial points –Pupil, tip of mouth, etc. Face graph –Nodes at fiducial pts. –Un-directed graph –The structure of graph is the same for each face –Fitting a face image to a face graph is done automatically –Some nodes may be undefined due to occlusion Bunch –A set of Jets assoc. with the same fiducial point –e.g. an eye Jet may consists of different types of eyes: open, closed, etc. Face bunch graph (FBG): –Same as a face graph, except each node consists of a jet bunch rather than a jet

128 IPCV 2006 Budapest12730/08/2006 Face Bunch Graph FBG has the same structure as individual face graph –Each node labeled with a bunch of jets –Each edge labeled with average distance between corresponding nodes in face samples

129 IPCV 2006 Budapest12830/08/2006 Still Face Recognition: Open Issues Addressing the issue of recognition being too sensitive to inaccurate facial feature localization Robustly recognizing faces –Small and/or noisy images –Images acquired years apart –Outdoor acquisition: lighting, pose What is the limit on the number of faces that can be distinguished? What is the principal and optimal way to arbitrate and combine local features and global features

130 IPCV 2006 Budapest12930/08/2006 Web Resources (detection) Face detection home page Henry Rowley’s home page Henry Schneiderman’s home page MIT CBCL web page datasets/index.html Face detection resources

131 IPCV 2006 Budapest13030/08/2006 Web Resources (recognition) Face Recognition Home Page Face Databases UT Dallas Notre Dame database MIT database Edelman CMU PIE Stirlingdatabase M2VTS multimodal Yale database Yale databaseB Harvard database Weizmanndatabase UMIST database Purdue\_face\_DB.html Olivetti database Ouluphysics-based

132 IPCV 2006 Budapest13130/08/2006 References (detection) M.-H. Yang, D. J. Kriegman, and N. Ahuja, “Detecting Faces in Images: A Survey” IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol. 24, no. 1, pp , ( detection-survey.html M.-H. Yang and N. Ahuja, Face Detection and Hand Gesture Recognition for Human Computer Interaction, Kluwer Academic Publishers, Paul Viola, Michael J. Jones, Robust Real- time Object Detection, Cambridge Research Laboratory Technology Reports Series, CRL 2001/01, Feb, 2001

133 IPCV 2006 Budapest13230/08/2006 Face and Some CV-based Projects at Budapest Tech

134 IPCV 2006 Budapest13330/08/2006 OriginalFilter responseFeatures Szűrő válasz Hangfelvétel HangmintaLP spektrum Neurális háló Arc maszk Combined verification

135 IPCV 2006 Budapest13430/08/2006 Morphing by Automatic Feature Detection

136 IPCV 2006 Budapest13530/08/2006 Eye and Lip Tracking Real time face detection Eye a mouth motion tracking Client-server internet communication

137 IPCV 2006 Budapest13630/08/2006 MYRAMYRA Face detection Skin tone based tracking Feature tracking Motion detection Recognition options: –Eigenface method –Gabor wavelet based bunch graph

138 IPCV 2006 Budapest13730/08/2006 Mobile robots – navigation using CV

139 IPCV 2006 Budapest13830/08/2006 Recognition and navigation

Download ppt "IPCV 2006 Budapest130/08/2006 Face Detection and Recognition Zoltán VÁMOSSY Budapest Tech, John von Neumann Faculty of Informatics."

Similar presentations

Ads by Google