Fusion of Face and Iris Biometrics from a Stand-Off Video Sensor Ryan Connaughton Kevin W. Bowyer Patrick Flynn April 16, 2011 Computer Vision Research Lab Department of Computer Science & Engineering University of Notre Dame

Biometrics and Multi-Biometrics Trait Biometric Sample Output Sensor Matcher Multi-Modal Multi-Sensor Multi-Sample Multi-Algorithm Redundancy at any stage is referred to as multi-biometrics 2

Fusion in Multi-Biometrics Fusion: Combining information from multiple sources Types of fusion: Signal Level Feature Level Score Level Rank Level Decision Level 3 3

Advantages and Disadvantages Potential advantages of multi-biometrics: Increased recognition accuracy Wider population coverage & lower failure-to-acquire rates More difficult to spoof Potential disadvantages: Increased computation time Increased acquisition time Increased sensor cost 4

Project Goal Investigate the feasibility of multi-biometrics based on a single sensor Specifically, combine multi-sample and multi-modal elements to create a system based on face and iris biometrics Compare performance of multi-biometric approach to single biometric approach 5

Sensors – Iris on the Move (IOM) Developed by Sarnoff Corp. [1] Designed for Iris recognition Stand-off and on-the-move Array of 3 frontal video cameras Each frame is 2048 x 2048 px Average iris diameter is ~120 px Synchronized NIR illumination Image from K. W. Bowyer, K. Hollingsworth, and P. J. Flynn. Image understanding for iris biometrics: A survey. In Computer Vision and Image Understanding, volume 110, pages 281-307. 2008. 6

IOM Frame Example 7

Sensors – LG IrisAccess 4000 (LG-4000) Developed by LG Iris [2] High-quality iris sensor Short-range, stationary subjects Average iris diameter is ~250 px Image from LG Iris Products and Solutions, 2010. URL http://www.lgiris.com/ps/products/irisaccess4000.htm 8

LG-4000 Image Example 9

Diagram of Approach 10

Preprocessing Stitch and perform histogram matching between corresponding frames Use template matching to determine translation required to align frames 11

Face Detection Performed on stitched frames OpenCV version Viola-Jones face detector used [3],[4] Trained on whole faces Faces are cropped according to face detector's estimation of size and location 12

Eye Detection Used for iris biometrics and for alignment during face matching Performed in two phases Phase 1: Detect eyes in upper quadrants of previously detected faces Phase 2: Detect eyes in frames where no faces were found Both phases use template matching approach to search for specular highlights 13

Face and Iris Matcher Face Matcher Colorado State University's implementation of eigenface [5],[6] Mahalanobis Cosine: -1 to 1, -1 is perfect match Iris Matcher Modified version of Daugman's algorithm [7] Normalized Hamming Distance: 0 to 1.0, 0 is perfect match 14

Fusion Summary Multi-modal and multi-sample scenario Test and compare multiple fusion approaches Score-level Rank-level Three approaches: Min rule Borda count Sum rule 15

Min Fusion Multi-sample, uni-modal, score-level fusion MinIris = Min{ Ii,j | i=1...n, j=1...G } MinFace = Min { Fi,j | i=1...m, j=1...G } Ii,j = HD between i-th probe iris and j-th gallery iris Fi,j = Mahalanobis distance between i-th probe face and j-th gallery face n,m = number of irises and faces detected G = number of gallery subjects 16

Borda Fusion Multi-sample, multi-modal or uni-modal, rank-level fusion For each probe biometric sample Sort gallery subjects by match score (best to worst) Cast votes for the top v-ranked gallery subjects BordaLinear: VoteWeightn = v + 2 – n BordaExp: VoteWeightn = 2v-n Gallery subject with the most votes is the best match for that probe video Three variations: BordaIris, BordaFace, and BordaBoth 17

Sum Fusion Multi-sample, multi-modal or uni-modal, score-level fusion Ii,k = HD between i-th probe iris and k-th gallery iris FNormi,k = Normalized Mahalanobis distance between i-th probe face and k-th gallery face n,m = number of irises and faces detected α,β = weights assigned to face and iris modalities 18

Dataset Collected 1,886 IOM video sets, spanning 363 subjects Ranged from 1 to 15 probe videos per subject Iris gallery consisted of one left eye and one right eye for each subject Acquired with an LG-4000 Face gallery consisted of one full face image for each subject Manually selected and annotated from stitched IOM frames Earliest IOM video with full face available was used to generate gallery image Videos used to generate gallery images were not included in probe set 19

Detection Results 20

Independent rank-one: Face Matching Results Mean match score: -0.281 (σ = 0.213) Mean non-match score: 0.000 (σ = 0.676) Independent rank-one: 51.6% (5073/9833) 21

Independent rank-one: Iris Matching Results Mean match score: 0.398 (σ = 0.053) Mean non-match score: 0.449 (σ = 0.013) Independent rank-one: 46.6% (13556/29112) 22

Rank-One Recognition Rates 23

Comparison Summary 24

Conclusions Investigated fusion of face and iris biometrics from a single sensor Conducted multi-modal experiments on a genuine dataset of 1886 videos of 363 subjects Combined multi-modal and multi-sample biometrics, as well as score-level and rank-level fusion Implemented the proposed multi-biometric workflow on a stand-off and on-the-move sensor Thus far, the best tested multi-modal approach yielded an increase of 5.4% in rank-one recognition over uni-modal approach 25

Acknowledgments & Questions [1] J. Matey, O. Naroditsky, K. Hanna, R. Kolczynski, D. LoIacono, S. Mangru, M. Tinker, T. Zappia, and W. Zhao. Iris on the Move: Acquisition of Images for Iris Recognition in Less Constrained Environments. In Proceedings of the IEEE, volume 94, pages 1936-1947. November 2006. [2] LG Iris. LG Iris Products and Solutions, 2010. URL http://www.lgiris.com/ps/products/irisaccess4000.htm. [3] G. Bradski and A. Kaehler. Learning OpenCV. O'Reilly Media, Inc., 2008. [4] P. Viola and M. Jones. Rapid Object Detection Using a Boosted Cascade of Simple Features. In 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), volume 1, pages 511-518, 2001. [5] Colorado State University. Evaluation of Face and Recognition Algorithms, 2010. URL http://www.cs.colostate.edu/evalfacerec/algorithms6.html. [6] M. Turk and A. Pentland. Face Recognition Using Eigenfaces. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 1991), volume 1, pages 586-591, June 1991. [7] J. Daugman. How Iris Recognition Works. In 2002 International Conference on Image Processing, volume 1, pages 33-36, 2002. Datasets used in this work were acquired under funding from the National Science Foundation under grant CNS01-30839, by the Central Intelligence Agency, and by the Technical Support Working Group under US Army Contract W91CRB-08-C-0093. Current funding is provided by a grant from the Intelligence Advanced Research Projects Activity. 26