Teaching Tool For French Speech Pronunciation Capstone Design Project 2008 Joseph Ciaburri Advisor: Professor Catravas

Motivation Use feedback that allows for self diagnosis Make tool as simple as possible for student Improve French pronunciation through the repetition of visual and aural aids Tony Blair Congratulating Nicolas Sarkozy on Election Win Interview with Domnique Villepin

USER Window 1 Native Speaker Audio SpeechSpeech MicrophoneWebcam VideoVideo Data Acquisition Window 2 Audio and Video of User Speaking Data DataData Window 3 Diagnostics Video Audio Video Audio Audio/Visual Databank Proposed Learning System

Design Specifications Goals Read in audio and video at the same time Play back audio and video at the same time within 1 second Minimize system requirements Implement diagnostics that are sensitive to pronunciation differences Provide pronunciation feedback via bulls eye Simplicity

Microphone and Webcam to Data Acquisition USER Webcam VIDEOVIDEO Data Acquisition Data Auto light compensation Average Frame Rate 15 frames per second Large file stored as a variable Length of video is short ~5 Seconds Camera lights up when recording USER Microphone Data Acquisition SpeechSpeech Data Webcam Microphone Sampling rate used is adjustable up to 44.1khz Saved as a variable Reads in simultaneously with video Microphone Built into webcam Auto noise cancellation Power comes from computer Able to crop to only speech

Repetition of User USER Window 2 Video of User Data Acquisition Data AUDIOAUDIO VIDEOVIDEO Play back from variable Allows for a quicker load time Less than 3 seconds to load video Audio and Video do not play in sync Play length ~ 5 Seconds Keeps memory requirements low

Diagnostics Data Acquisition DataData Window 3 Diagnostics USER AudioAudioVideoVideo Can create and graph spectrogram data Allows for determination of vowels using the formants and consonants using the transitions of the formants Can create and graph cepstrum data Inverse Fourier Transform of the log of the Fourier Transform Can find fundamental frequency Can find Zero Crossings Zero Crossings show silence versus speech Bulls eye allows for two inputs, along the x and y, graphed as a percent distance from the center

Results Time Domain: Non Native SpeakerTime Domain: Native Speaker Spectrogram: Non Native SpeakerSpectrogram: Native Speaker

Results Continued Cepstrum: Non Native SpeakerCepstrum: Native Speaker Zero : Non Native Speaker Zero Crossings: Native Speaker

Design Specifications Goals Read in audio and video at the same time Play back audio and video at the same time within 1 second Implement diagnostics that are sensitive to pronunciation differences Provide pronunciation feedback via bulls eye Simplicity Minimize system requirements Accomplished Can read synchronized audio and video into MATLAB Can play back audio or video separately, or unsynchronized audio and video in MATLAB Can plot diagnostics and find fundamental frequency Can plot on bulls eye All in one webcam as well as keeping the whole program in MATLAB

In Progress Identifying specific components of speech that specific to French –Vowels –Consonants Quantifying these components and using them on the bulls eye Creating a GUI Gather more video samples

Future Research Integrating other languages Evaluation –Use of non-native speakers –Use of native speakers Testing in the use of facial communication in oral communication Basis for comparison of other audible signals

Acknowledgements Professor Rudko Professor Hanson Professor Streignitz Professor Cotter Professor Catravas Professor Chilcoat Professor Pickering Professor Spallholz