1. ICASSP'06 1 S. Y. Kung 1 and M. W. Mak 2 1 Dept. of Electrical Engineering, Princeton University 2 Dept. of Electronic and Information Engineering, The Hong Kong Polytechnic University On Consistent Fusion of Multimodal Biometrics

2. ICASSP'06 2Outline Why Fusion for Audio-Visual Biometrics Consistent (vs. Catastrophic) Fusion Mixture-of-Expert Fusion Architecture Consistent fusion Linear fusion Nonlinear fusion Conclusion

3. ICASSP'06 3 Why Fusion for Audio-Visual Biometrics Voice biometrics can suffer severe performance degradation under noisy environment, but facial images are unaffected. Facial image quality can be severely affected in poor lighting conditions, but lighting has no effect on voice quality. Speech and faces provide complementary information sources that are ideal candidates for fusion – as verified by ROC(DET). Results based on 295 subjects from XM2VTSDB

4. ICASSP'06 4 Mixture-of-Expert Fusion Architecture The lower layer contains local experts, each produces a local score based on a single modality The upper layer contains a gating network

5. ICASSP'06 5ROC(DET) We may consider the audio and visual sources separately, i.e., we have two decision thresholds and two decision boundaries. By shifting the decision boundaries independently, we obtain two DET curves, one for each modality. False Acceptance Rate False Rejection Rate

6. ICASSP'06 6 9 1 2 6 7 8 5 4 3 9 8 123 6 4 7 5 users Imposters Regions of Consistent and Catastrophic Fusion Consistent Region Catastrophic Region

7. ICASSP'06 7 Consistent Fusion Yield a lower bound performance of consistent fusion (fusion that leads to performance equal to or better than any individual modalities) False Acceptance Rate False Rejection Rate 1 3 2 4 5 6 8 9 7 Face Voice 9 8 12345 7 6 5 Imposters users

8. ICASSP'06 8 False Acceptance Rate False Rejection Rate Linear Fusion

9. ICASSP'06 9 Score distribution of multi-modalities Nonlinear Fusion

10. ICASSP'06 10 False Acceptance Rate False Rejection Rate Nonlinear Fusion

11. ICASSP'06 11 1 3 2 4 5 6 8 9 7 Face Voice Face+Voice (Nonlinear) Face+Voice (Linear) Linear Vs. Nonlinear Fusion

12. ICASSP'06 12 What if there are N (N >2) modalities: Which pair of modalities would be the best choice? Answer: DET (ROC) could provide a good indication on (1) how good and (2) how complementary. What guaranteed advantage to adopt N (N>2) modalities? False Acceptance Rate False Rejection Rate A B C

13. ICASSP'06 13 But there is a catch on statistical significance! This can be upheld only if the training data set, held-out set, and test set are assumed to have statistically the same distribution and provided in large volume.

14. ICASSP'06 14 Thank you

15. ICASSP'06 15Conclusions The notion of consistent fusion is proposed for multimodality fusion The consistent fusion framework leads to several adaptive fusion schemes, such as hard-switching, linear combination, and adaptive nonlinear SVM fusion. Results suggest that consistent fusion provides a valuable framework for choosing different modalities in multimodal biometric authentication.

16. ICASSP'06 16 For a single modality, a test sequence from a claimant is classified as coming from the true client if Decision threshold Score Distributions of Single Modality

17. ICASSP'06 17 DET Based on Single Modality Changing the threshold ηfrom small to large values, we obtain an ROC or DET False Acceptance Rate Large η Small η False Rejection Rate

18. ICASSP'06 18 Is Linear Fusion a good idea?

19. ICASSP'06 19 Classifier for Audio Channel Classifier for Visual Channel Adaptive Gating Network (e.g. hard-switch, linear combiner, and SVM) Fused Score Why Fusion for Audio-Visual Biometrics