Presentation on theme: "KINSHIP CLASSIFICATION BY MODELING FACIAL FEATURE HEREDITY Ruogu Fang 1, Andrew C. Gallagher 1 Tsuhan Chen 1, Alexander Loui 2 1 Cornell University 2 Eastman."— Presentation transcript:
KINSHIP CLASSIFICATION BY MODELING FACIAL FEATURE HEREDITY Ruogu Fang 1, Andrew C. Gallagher 1 Tsuhan Chen 1, Alexander Loui 2 1 Cornell University 2 Eastman Kodak Company 1 IEEE International Conference on Image Processing 2013
2 KINSHIP CLASSIFICATION BY MODELING FACIAL FEATURE HEREDITY Problem Definition: Recognize the family that a query person belongs to from a set of families. Solution: Reconstruct the query face from a mixture of parts from a set of family members for the recognition. Motivation: Genetic model of reproduction using the mathematical tool of sparsity.
A GENETIC PERSPECTIVE Why do we look like the way that we do? –DNA How are our appearances affected by ancestors? –Inheritance and mutation Facial features are part of the appearance. 3 DNA The facial feature heredity also follow the model of genetics.
MENDEL’S LAWS I Law of Random Segregation: For every particular trait, one randomly selected allele from each parent is passed down to the offspring. 4 BbBb b Bb BbBb BBBb bb B: Brown eyes (dominant) b: blue eyes (recessive) bb Bb Each facial feature of an individual can be represented by a sparse combination of the relatives with this feature.
FAMILY SPARSITY 5 ? Few families are selected.
ENERGY FUNCTION For one family, given sufficient training samples of a family (m = feature length, n = number of training samples) A new sample from the same family Approximately lies in the linear span of the family member samples associated with this family For all unrelated families, 6 L 1 norm: Individual sparsity term (illumination, pose and expression) L 2,1 norm: Family sparsity term N= # families
MENDEL’S LAWS II Law of Independent Assortment: Genes of separate traits are passed down independently from parents to offspring. 7 Credit: Northeast Medical School The facial features should be analyzed independently.
INDEPENDENCE OF FACIAL PARTS 8 For each part, a part-based dictionary is built.
CLASSIFICATION 1.Choose three representative parts with smallest possible residues R. 2.Rank the normalized residues for all families on these three parts. 3.Sum the ranks and use the highest rank. 9 … # families … … Error … Reconstruction error for part p from family j Remove outliers due to recessive genes Byproduct: Find the three most distinguishable features
POTENTIAL APPLICATIONS Family Photo Album DistillationFamily Image Retrieval Social Websites: Auto Family Tagging Tag Your Family Members From Sara Lee’s family? From Kelly Ng’s family? Find Lost Relatives 10
FAMILY 101 DATABASE 101 Different Families 607 Individuals14,816 Images 11 Kennedy 27 (410) # people # images Download:
FAMILY DATABASE COLLECTION 12 Kennedy Family 27 Individuals 48 Images of Caroline Kennedy
Database # of Family # of People # of Images HighlightsReferences Labeled Faces in the Wild (LFW) ,233 Unconstrained, “natural” variability in pose, lighting, etc. [Hung et al. 2007] PubFig020058,797 Real world, deep and large, celebrities and politicians [Kumar et al. 2009] Cornell Kinship Verification Controlled parent-child pairs [Fang et al. 2010] UB KinFace Child with young parent and old parent faces[Xia et al. 2011] Family (206 Nuclear) 60714,816 Real world, family structure of 2-3 genenrations, variations of age, pose, illumination, expression, ethnicity, etc. Political, royal, wealthy and celebrity families. This Work RELATED DATABASES Facts about Family101 Database –Multiple generations –Every nuclear family has 6 family members on average –Every individual has 24 images on average 13
EXPERIMENT SETUP Feature: Dense SIFT 16x16 Baseline –K nearest neighbors (KNN) –Support vector machine (SVM) –Sparse representation based recognition (SRC) Unless specified in each scenario: –3 family members for training, 2 for testing. –20 families randomly selected for evaluation. –30 images/person for both training and testing. Evaluation metric: Mean per-family accuracy 14
EXP 1: NO. OF FAMILIES 15 3 family members for training 2 family members for testing 30 images/person for training/testing
EXP 2: NO. OF PEOPLE FOR TRAINING families randomly selected 30 images/person for training/testing
FACIAL FEATURE MATCHING Task: Find the people with similar facial features to the query person. 17 Training Images Test Images Martin Sheen High Low Hair Eyes Nose Mouth Martin Sheen coefficients
CONCLUSION Motivation: biological process of inheritance –Mendel’s laws of random segregation and independent assortment A new challenge: kinship classification A new framework: reconstruct the query face from a mixture of parts from a set of families A new dataset: Family
FUTURE WORK Use family tree structure Hallucination –Hallucinate what the appearance of the father might be, just by looking at the differences between a child and her mother. 19
Q & A Thank you! 20 KINSHIP CLASSIFICATION BY MODELING FACIAL FEATURE HEREDITY Ruogu FangAndrew C. Gallagher Tsuhan ChenAlexander Loui Project Page and Dataset Download:
FACE DETECTION & ALIGNMENT Active Shape Model: 82 Facial Points Face Detection Face Alignment: 6 Fiducial Points 21