Face Recognition from Face Motion Manifolds using Robust Kernel RAD Ognjen Arandjelović Roberto Cipolla Funded by Toshiba Corp. and Trinity College, Cambridge

Face Recognition Single-shot recognition – a popular area of research since 1970s Many methods have been developed Bad performance in presence of: –Illumination variation –Pose variation –Facial expression –Occlusions (glasses, hair etc.) Eigenfaces Wavelet methods 3D Morphable Models

Face Recognition from Video Face motion helps resolve ambiguities of single shot recognition – implicit 3D Video information often available (surveillance, authentication etc.) Recognition setupTraining streamNovel stream

Face Manifolds Face patterns describe manifolds which are: –Highly nonlinear, and –Noisy, but –Smooth Facial featuresFace pattern manifoldFace region

Limitations of Previous Work In this work we address 3 fundamental questions: –How to model nonlinear manifolds of face motion –How to choose the distance measure –How and what noise sources to model ?

Comparing Nonlinear Manifolds Closest-neighbour Majority vote + Eigen/Fisherfaces Mutual Subspace Method Principal angles Our method, KLD method of Shakhnarovich et al. Information-theoretic measures

RAD: How well can we distinguish between P(x) and Q(x)? KLD vs. RAD vs. … KLD: How well does P(x) explain Q(x)? P(x)Q(x) P(x)

Nonlinear RAD Use closed form expression for KLD between Gaussians in KPCA space Kernel PCA Approximately linear manifolds Highly nonlinear manifolds Input spaceKPCA spaceRBF Kernel

Registration Linear operations on images are highly nonlinear in the pattern space Translation/rotation and weak perspective can be easily corrected for directly from point correspondences –We use the locations of pupils and nostrils to robustly estimate the optimal affine registration parameters Translation manifold Skew manifold Rotation manifold

Registration Method Used Feature localization based on the combination of shape and pattern matching (Fukui et al. 1998) Detect features Crop & affine register faces

Feature Tracking Errors We recognize two sources of registration noise: –Low-energy noise due to the imprecision feature detector –High-energy noise due to incorrectly localized features 20 automatically cropped and registered faces from a video sequence Outliers – high energy noise Imperfect alignment of facial features – low energy noise

Low Energy Noise Estimate misregistration manifold noise energy Augment data with synthetically perturbed samples = thickening of the motion manifold Synthetic data explicitly models the variation Original dataOriginal + synthetic data

Outliers – High Energy Noise Outliers are due to incorrect feature localization High energy noise – far from the ‘correct’ data mean in KPCA space Outliers Manifold of correctly registered faces (+low energy noise)

RANSAC for Robust KPCA Minimal, random sample Kernel PCA projection Valid data count Iteration Outliers

Algorithm: The Big Picture Input frames Distance Original + synthetic data Valid data in KPCA space

Face Video Database No standard database exists – we collected our own data 160 people, 10 different lighting conditions (each condition twice i.e. 20 video sequences per person)

Evaluation Results Robust Kernel RAD outperformed other methods on all databases Average recognition rate of 98%

Method Limitations / Future Work Less pose sensitivity (why should input and reference distributions be similar?) Illumination invariance is not addressed Same person, different illumination Novel person See Arandjelović et al., BMVC 2004 For suggestions, questions etc. please contact me at: