ICASSP'06 1 S. Y. Kung 1 and M. W. Mak 2 1 Dept. of Electrical Engineering, Princeton University 2 Dept. of Electronic and Information Engineering, The Hong Kong Polytechnic University On Consistent Fusion of Multimodal Biometrics

ICASSP'06 2Outline Why Fusion for Audio-Visual Biometrics Consistent (vs. Catastrophic) Fusion Mixture-of-Expert Fusion Architecture Consistent fusion Linear fusion Nonlinear fusion Conclusion

ICASSP'06 3 Why Fusion for Audio-Visual Biometrics Voice biometrics can suffer severe performance degradation under noisy environment, but facial images are unaffected. Facial image quality can be severely affected in poor lighting conditions, but lighting has no effect on voice quality. Speech and faces provide complementary information sources that are ideal candidates for fusion – as verified by ROC(DET). Results based on 295 subjects from XM2VTSDB

ICASSP'06 4 Mixture-of-Expert Fusion Architecture The lower layer contains local experts, each produces a local score based on a single modality The upper layer contains a gating network

ICASSP'06 5ROC(DET) We may consider the audio and visual sources separately, i.e., we have two decision thresholds and two decision boundaries. By shifting the decision boundaries independently, we obtain two DET curves, one for each modality. False Acceptance Rate False Rejection Rate

ICASSP' users Imposters Regions of Consistent and Catastrophic Fusion Consistent Region Catastrophic Region

ICASSP'06 7 Consistent Fusion Yield a lower bound performance of consistent fusion (fusion that leads to performance equal to or better than any individual modalities) False Acceptance Rate False Rejection Rate Face Voice Imposters users

ICASSP'06 8 False Acceptance Rate False Rejection Rate Linear Fusion

ICASSP'06 9 Score distribution of multi-modalities Nonlinear Fusion

ICASSP'06 10 False Acceptance Rate False Rejection Rate Nonlinear Fusion

ICASSP' Face Voice Face+Voice (Nonlinear) Face+Voice (Linear) Linear Vs. Nonlinear Fusion

ICASSP'06 12 What if there are N (N >2) modalities: Which pair of modalities would be the best choice? Answer: DET (ROC) could provide a good indication on (1) how good and (2) how complementary. What guaranteed advantage to adopt N (N>2) modalities? False Acceptance Rate False Rejection Rate A B C

ICASSP'06 13 But there is a catch on statistical significance! This can be upheld only if the training data set, held-out set, and test set are assumed to have statistically the same distribution and provided in large volume.

ICASSP'06 14 Thank you

ICASSP'06 15Conclusions The notion of consistent fusion is proposed for multimodality fusion The consistent fusion framework leads to several adaptive fusion schemes, such as hard-switching, linear combination, and adaptive nonlinear SVM fusion. Results suggest that consistent fusion provides a valuable framework for choosing different modalities in multimodal biometric authentication.

ICASSP'06 16 For a single modality, a test sequence from a claimant is classified as coming from the true client if Decision threshold Score Distributions of Single Modality

ICASSP'06 17 DET Based on Single Modality Changing the threshold ηfrom small to large values, we obtain an ROC or DET False Acceptance Rate Large η Small η False Rejection Rate

ICASSP'06 18 Is Linear Fusion a good idea?

ICASSP'06 19 Classifier for Audio Channel Classifier for Visual Channel Adaptive Gating Network (e.g. hard-switch, linear combiner, and SVM) Fused Score Why Fusion for Audio-Visual Biometrics