1. “Articulatory Talking Head” Showcase Project, INRIA, KTH. Articulatory Talking Head driven by Automatic Speech Recognition INRIA, Parole Team KTH, Centre for Speech Technology

2. “Articulatory Talking Head” Showcase Project, INRIA, KTH. Aim of the INRIA-KTH collaboration recreate in real-time the articulatory movements of a speaker with an talking head, using the speech signal only. Applications: – Communication help for HOH people – Second language learning – Speech therapy

3. “Articulatory Talking Head” Showcase Project, INRIA, KTH. Articulation Articulation display is important to understand English pronunciation as well as a support for perception. In this demonstration: 1.Voice Activation Detection is achieved (separation of dpeech and and non-speech) 2.English phoneme recognition is performed 3.English articulation is analysed 4.Articulation is displayed

4. “Articulatory Talking Head” Showcase Project, INRIA, KTH. New VAD is a combination GMM and automaton 64 gaussiennes Type de mfccalgorithme % reconnaissa nce globale % de trame de parole correctement reconnu % de trame reconnu comme parole au lieu de non parole % de trame de non parole correctement reconnu % de trame reconnu comme non parole au lieu de parole energy 79,6298,6424,5550,024,042 esperegmm_direct91,3495,879,51984,297,072 esperemicro_viterbi74,7784,4223,4559,7428,86 esperemicro_viterbi286,1487,0710,1584,6819,19 htkgmm_direct87,9897,9915,3172,414,131 htkmicro_viterbi83,1994,6819,0565,3111,24 htkmicro_viterbi287,3996,8115,3172,746,382

5. “Articulatory Talking Head” Showcase Project, INRIA, KTH. 3D Reconstruction The reconstruction was made using a semi-polar grid of 20 gridlines. 9 12 15 18 21 24 27 A polygon mesh of 420 vertices and about 800 polygons was constructed. One contour per image.

6. “Articulatory Talking Head” Showcase Project, INRIA, KTH. Models have been adapted to English Movetrack Electromagnetic Articulograph: 6 coils; upper lip, upper & lower incisors, three tongue coils: 8, 20 and 52 mm from the tip. Qualisys optical motion tracking: 4 IR cameras 28 reflectors 3 reference reflectors on headmount CC C CC R Audio & video recorders V VRf

7. “Articulatory Talking Head” Showcase Project, INRIA, KTH.

8. Prosody Prosody is important for message understanding. It is present both in speech sound and in facial expressions and gestures. Some prosody information is extracted from the signal as: –Fundamental frequency (F0) –Energy –Speech rate and displayed with the talking head.

9. “Articulatory Talking Head” Showcase Project, INRIA, KTH. Pitch(F0): Comb filters estimation

10. “Articulatory Talking Head” Showcase Project, INRIA, KTH. F0 comparison between French and Native Speaker Please note that the F0 and narrow band spectogram scales are different

11. “Articulatory Talking Head” Showcase Project, INRIA, KTH. Speech rate Speech rate can be computed as the average number of phonemes produced by second. We define it as a ratio between: the average duration of the produced phonems The average duration of the same phonems in the phonem recognizer trainning database.

12. “Articulatory Talking Head” Showcase Project, INRIA, KTH. Speech rate

13. “Articulatory Talking Head” Showcase Project, INRIA, KTH.

14. Usage Scenario 1.The teacher, the learner or the speech therapist speaks 2.The talking head reproduces what has been uttered showing articulators 3.The talking head shows what should have been articulated. This is a first step towards an interactive learning loop.

15. “Articulatory Talking Head” Showcase Project, INRIA, KTH. A French student pronounces an English sentence…

16. “Articulatory Talking Head” Showcase Project, INRIA, KTH.

17. The student and the teacher can have a closer look at the articulation and prosody…

18. “Articulatory Talking Head” Showcase Project, INRIA, KTH.

19. The teacher can pronouce the sentence as it should be…

20. “Articulatory Talking Head” Showcase Project, INRIA, KTH.

21. The student and the teacher can watch together the correct articulation and prosody…

22. “Articulatory Talking Head” Showcase Project, INRIA, KTH.

23. And of course the teacher can give more detailed explanations and advices…

24. “Articulatory Talking Head” Showcase Project, INRIA, KTH.

25. Thank you.