June 28th, 2004 BioSecure, SecurePhone 1 Automatic Speaker Verification : Technologies, Evaluations and Possible Future Gérard CHOLLET CNRS-LTCI, GET-ENST Biometrics in Current Security Environments Biometrics in Current Security Environments

June 28th, 2004BioSecure, SecurePhone2Outline  State of affairs (tasks, security, forensic,…)  Speaker characteristics in the speech signal  Automatic Speaker Verification :  Decision theory  Text dependent / Text independent  Imposture (occasional, dedicated)  Voice transformations  Audio-visual speaker verification  Evaluations (algorithms, field tests, ergonomy,…)  Conclusions, Perspectives

June 28th, 2004BioSecure, SecurePhone3 Why should a computer recognize who is speaking ?  Protection of individual property (habitation, bank account, personal data, messages, mobile phone, PDA,...)  Limited access (secured areas, data bases)  Personalization (only respond to its master’s voice)  Locate a particular person in an audio-visual document (information retrieval)  Who is speaking in a meeting ?  Is a suspect the criminal ? (forensic applications)

June 28th, 2004BioSecure, SecurePhone4 Tasks in Automatic Speaker Recognition  Speaker verification (Voice Biometric)  Are you really who you claim to be ?  Identification (Speaker ID) :  Is this speech segment coming from a known speaker ?  How large is the set of speakers (population of the world) ?  Speaker detection, segmentation, indexing, retrieval, tracking :  Looking for recordings of a particular speaker  Combining Speech and Speaker Recognition  Adaptation to a new speaker, speaker typology  Personalization in dialogue systems

June 28th, 2004BioSecure, SecurePhone5 Applications  Access Control  Physical facilities, Computer networks, Websites  Transaction Authentication  Telephone banking, e-Commerce  Speech data Management  Voice messaging, Search engines  Law Enforcement  Forensics, Home incarceration

June 28th, 2004BioSecure, SecurePhone6 Voice Biometric  Avantages  Often the only modality over the telephone,  Low cost (microphone, A/D), Ubiquity  Possible integration on a smart (SIM) card  Natural bimodal fusion : speaking face  Disadvantages  Lack of discretion  Possibility of imitation and electronic imposture  Lack of robustness to noise, distortion,…  Temporal drift

June 28th, 2004BioSecure, SecurePhone7 Speaker Identity in Speech  Differences in  Vocal tract shapes and muscular control  Fundamental frequency (typical values)  100 Hz (Male), 200 Hz (Female), 300 Hz (Child)  Glottal waveform  Phonotactics  Lexical usage  The differences between Voices of Twins is a limit case  Voices can also be imitated or disguised

June 28th, 2004BioSecure, SecurePhone8 spectral envelope of / i: / f A Speaker A Speaker B Speaker Identity  segmental factors (~30ms)  glottal excitation: fundamental frequency, amplitude, voice quality (e.g., breathiness)  vocal tract: characterized by its transfer function and represented by MFCCs (Mel Freq. Cepstral Coef)  suprasegmental factors  speaking speed (timing and rhythm of speech units)  intonation patterns  dialect, accent, pronunciation habits

June 28th, 2004BioSecure, SecurePhone9 Acoutic features  Short term spectral analysis

June 28th, 2004BioSecure, SecurePhone10 Intra- and Inter-speaker variability

June 28th, 2004BioSecure, SecurePhone11 Speaker Verification Typology of approaches (EAGLES Handbook)  Text dependent  Public password  Private password  Customized password  Text prompted  Text independent Incremental enrolment Evaluation

June 28th, 2004BioSecure, SecurePhone12 History of Speaker Recognition

June 28th, 2004BioSecure, SecurePhone13 Current approaches

June 28th, 2004BioSecure, SecurePhone14 HMM structure depends on the application

June 28th, 2004BioSecure, SecurePhone15 Gaussian Mixture Model  Parametric representation of the probability distribution of observations:

June 28th, 2004BioSecure, SecurePhone16 Gaussian Mixture Models 8 Gaussians per mixture

June 28th, 2004BioSecure, SecurePhone17  Two types of errors :  False rejection (a client is rejected)  False acceptation (an impostor is accepted)  Decision theory : given an observation O and a claimed identity  H 0 hypothesis : it comes from an impostor  H 1 hypothesis : it comes from our client  H 1 is chosen if and only if P(H 1 |O) > P(H 0 |O) which could be rewritten (using Bayes law) as Decision theory for identity verification

June 28th, 2004BioSecure, SecurePhone18 Signal detection theory

June 28th, 2004BioSecure, SecurePhone19 Decision

June 28th, 2004BioSecure, SecurePhone20 Distribution of scores

June 28th, 2004BioSecure, SecurePhone21 Detection Error Tradeoff (DET) Curve

June 28th, 2004BioSecure, SecurePhone22 Evaluation  Decision cost (FA, FR, priors, costs,…)  Receiver Operating Characteristic Curve  Reference systems (open software)  Evaluations (algorithms, field trials, ergonomy,…)

June 28th, 2004BioSecure, SecurePhone23 National Institute of Standards & Technology (NIST) Speaker Verification Evaluations Annual evaluation since 1995 Common paradigm for comparing technologies

June 28th, 2004BioSecure, SecurePhone24 NIST evaluations : Results

June 28th, 2004BioSecure, SecurePhone25 Combining Speech Recognition and Speaker Verification.  Speaker independent phone HMMs  Selection of segments or segment classes which are speaker specific  Preliminary evaluations are performed on the NIST extended data set (one hour of training data per speaker)

June 28th, 2004BioSecure, SecurePhone26 ALISP data-driven speech segmentation

June 28th, 2004BioSecure, SecurePhone27 Searching in client and world speech dictionaries for speaker verification purposes

June 28th, 2004BioSecure, SecurePhone28 Fusion

June 28th, 2004BioSecure, SecurePhone29 Fusion results

June 28th, 2004BioSecure, SecurePhone30 Speaking Faces : Motivations  A person speaking in front of a camera offers 2 modalities for identity verification (speech and face).  The sequence of face images and the synchronisation of speech and lip movements could be exploited.  Imposture is much more difficult than with single modalities. Many PCs, PDAs, mobile phones are equiped with a camera. Audio-Visual Identity Verification will offer non-intrusive security for e-commerce, e- banking,…

June 28th, 2004BioSecure, SecurePhone31 Talking Face Recognition (hybrid verification)

June 28th, 2004BioSecure, SecurePhone32 Lip features  Tracking lip movements

June 28th, 2004BioSecure, SecurePhone33 A talking face model  Using Hidden Markov Models (HMMs) Acoustic parameters Visual parameters

June 28th, 2004BioSecure, SecurePhone34 Morphing, avatars

June 28th, 2004BioSecure, SecurePhone35 Conclusions, Perspectives   Deliberate imposture is a challenge for speech only systems   Verification of identity based on features extracted from talking faces should be developped   Common databases and evaluation protocols are necessary   Free access to reference systems will facilitate future developments