1. Ethnicity and Gender Classification using Image Processing
Under the Guidance of
Dr. Brejesh Lall
Sai Bharadwaj Popuri
Sunil Kumar
Department of Electrical Engineering
Indian Institute of Technology, Delhi
7/5/2011

2. Motivation LCD advertisement panels in malls/shopping areas/ McDonalds
Presently, no information known as to who sees the advertisement
A system could be integrated into a LCD panel which provides statistics about the types of audience
Helps delivering targeted advertisements by analyzing the audience
Classification of audience according to Age, Gender and Ethnicity was initially proposed; We solved the problems of Gender and Race classification in our thesis
7/5/2011

3. Objectives Face Detection; Resizing them to 96*96 pixels
Creation of database and establishment of ground truth information
Compute color histogram and extract Gabor filter coefficients for use as features
Combine color histogram and Gabor features and use a modified version of KNN classifier for ethnicity classification
Use Gabor features along with SVM classifier for gender classification
Combine both the classification schemes into an automatic classification system, which takes an image containing multiple faces as input, classify them and give out results
7/5/2011

4. Outline Random images from FERET Database
Face Detection and database formation
Outline
Pre-processing and resizing
Skin color histogram in RGB plane
Gabor filtering
(coefficients)
Modified KNN Classifier
SVM classifier
Tests and results for Ethnicity class’n
Tests and results for Gender classification
7/5/2011

5. Face Detection
Implemented Viola Jones algorithm – de facto standard for face detection systems today
New image representation – integral image
Features – Haar like features (like haar basis functions)
Adaboost learning to select best features and train classifiers using these features;
Cascaded classifiers – combination of weak classifiers
All the images were resized to a common resolution of 96*96 pixels using the affine transform
7/5/2011

6. Image Database
240 test images were randomly selected from the Color FERET database
The database had around 45% people of White ethnicity, 30% people of Asian ethnicity and 25% are of Black ethnicity
The database had 55% males and 45% females
65 images were used for testing and 175 for training
Ground truth information of all the training images with respect to gender and ethnicity established
This information is used by classifiers for classification of test images
7/5/2011

7. Features – Skin color histogram and Gabor Features
Ethnicity can generally be identified by looking at skin color
Histograms are computed in the R,G & B planes separately and are concatenated; this can be used as features for classification
Gabor filter – linear filter used for edge detection
The Gabor filter we used takes scalar input
Gabor filter highlights and extracts local features (edges, texture etc..) of an image; has tolerance to shape, scale and light
7/5/2011

8. Contd… The parameters affecting the Gabor filter outputs are
Wavelength
Orientation
Phase Offset
Aspect Ratio
Bandwidth
The frequency and orientation representation of Gabor filter are similar to that of human visual system
Once the coefficients are calculated, raster scan performed to attain a one dimensional feature array
7/5/2011

9. Classifiers – KNN & modified KNN
KNN classifier classifies objects by counting the number (K parameter) of nearest neighbors in the feature space
Can be used to address multiclass problems; Ethnicity classification
Standard KNN classifier was then modified to do the following
5 nearest neighbors using color histogram and Gabor features were computed separately
These ten neighbors (5 +5) were then considered for classification and decision was made by counting the number of occurrences of each class and considering the majority
7/5/2011

10. Classifiers – SVM
Provides superior classification results compared to other classification methods
Performs binary classification; Gender classification
To be trained first by providing ground truth information
Maps input data non-linearly to some high dimensional space where data can be linearly separated
7/5/2011

11. Ethnicity classification – testing and results
Initial testing involved using color histograms as features; KNN classifier was used for classification of faces based on ethnicity
Various color spaces were used and tested upon; The results are as follows
Best case accuracies are around 66%; Color space did not make much difference in the accuracy
7/5/2011

12. Contd… Only Gabor features were then used for classification
The results are as follows
Here also, we can observe that the best case accuracies are around 69% obtained with lambda = 4, K = 5 and bw =1; Similar to when color histogram is used
7/5/2011

13. Combined Scheme
Since the accuracies are low, a novel method of combining the two features to form a hybrid feature set was proposed
The modified KNN classifier is used for classification using this hybrid feature set
The results are as follows
7/5/2011

14. Contd…
In this scheme, the K parameter is kept fixed at 5 and lamdba and bw of the Gabor filter are varied
Accuracies close to 77%; considerable increase over individual classification schemes
Considerable accuracy considering classification is a three class problem and an inconsistent database
7/5/2011

15. Gender Classification
Gabor features used for classification
Initial testing done using KNN classifier with a K parameter 5
Best case accuracies achieved are around 85%
7/5/2011

16. The results of the classification are as follows
A SVM classifier is first trained using the 175 training images; It is then used for the classification of the 65 test images
The results of the classification are as follows
Best case accuracies of around 89.1% are achieved
7/5/2011

17. Automatic Classification of faces
The different classification schemes for gender and race have been integrated into a single system
This system has to be trained before any classification to take place
This scheme takes in a single group photograph or a list of group photos, separates faces out of them, classify them according to ethnicity and gender and display results
Number of faces of each ethnic group and gender are separately counted and are displayed
These can be used as criterion if targeted advertisements(if) are to be displayed
7/5/2011

18. Contd…
Matlab files with names ‘training.m’ and ‘test.m’ are created which when run (training.m before test.m) would display the before mentioned parameters in the main display window of matlab
Sample test images used are shown below
In this collage, Asian males dominate and
it was correctly displayed by our classifier
In this collage, our classifier showed that white
males dominate (4 males to 2 females)
While the race result is true, gender result has
one error
7/5/2011

19. Contd…
The collages are made using random pictures from the FERET database
Modified KNN is used for ethnicity classification and SVM for gender classification
Since not many faces are present on the collages, percentage accuracies of classification cannot be rightly made
Having tested this scheme on various different collages, we found that the accuracies are similar and some times better to the ones obtained using individual classification schemes; This is because it is an average
7/5/2011

20. Sample tests
7/5/2011

21. Sample output
7/5/2011

22. Future Scope Proper database development and age classification
Localized training of the automatic classifier; Can be built into the LCD panel
3 or more modalities can be combined for classification
A weak but robust classifier like KNN is used; better classifiers can be used for the multiclass problem
7/5/2011

23. References
Viola, P. and Jones, M (revised 2003). Robust real time face detection. In Proceedings, IEEE Conference on Computer Vision and Pattern Recognition,2004
A. K. Noulas, B. J. A. Kröse, 2006, EM-detection of Common Origin of Multi-Modal Cues, International Conference Multimodal Interfaces, Banff, Canada.
OpenCV Viola Jones Face Detection in Matlab. In Mathworks. Retrieved on 23rd September, 2010 from
Javier R. Movellan. “ Tutorial on gabor filters”, available at
Gabor Filter. In Mathworks. Retrieved on 22nd January, 2011 from
7/5/2011

24. References
Hui Lin, Huchuan Lu and Lihe Zhang, A New Automatic Recognition System of Gender, Age and Ethnicity. In Proceedings, 6th World Congress on Intelligent Control and Automation, June , 2006
Shen L et al, Gabor feature selection for face recognition using improved AdaBoost learning. In: Advances in biometric person authentication, proceedings, vol Lecture notes in computer science, pp 39–49
Baluja, S. and Rowley, H. A. ,2005. Boosting sex identification performance. In Innovative Applications of artificial Intelligence, Pittsburgh, PA, USA
OpenCV. In Open Computer vision, retrieved on 22nd Oct, 2010 from
7/5/2011

25. It’s been an awesome journey…
Thank You !
It’s been an awesome journey…
7/5/2011