1. Speech Assessment 語音評測 J.-S. Roger Jang ( 張智星 ) jang@cs.nthu.edu.tw http://www.cs.nthu.edu.tw/~jang Multimedia Information Retrieval Lab CS Dept, Tsing Hua Univ, Taiwan

2. -2- Outline zIntroduction zMethods zProblems to be solved zDemos

3. -3- Speech Assessment zSpeech assessment: How to assess an utterance for the purpose of learning a spoken language? yAssessment levels: syllables, words, sentences, paragraphs yAssessment criteria: timbre, tone, energy, rhythm, co-articulation, … yFeedbacks: High-level correction and suggestions

4. -4- Related Disciplines zRelated disciplines for speech assessment: yLanguage learning: xCALL: Computer Assisted Language Learning xCAPT: Computer Assisted Pronunciation Training ySpeech technology: xUV: Utterance Verification

5. -5- Our Approach zBasic approach to timbre assessment yLexicon net construction (Usually a sausage net) yForced alignment to identify phone boundaries yPhone scoring based on several criteria, such as ranking, histograms, posterior prob., etc. yWeighted average to get syllable score yWeighted average to get sentence score

6. -6- Basic Assessment Criteria zTimber yBased on acoustic models zTone yBased on tone recognition (for tonal language) yBased on pitch similarity with the target utterance zEnergy yBased on energy comparison with the target utterance zRhythm yBased on duration comparison with the target utterance zFluency

7. -7- Additional Assessment Criteria zEnglish yStress xLevels (word or sentence) xMeanings yIntonation xDeclarative sentence xInterrogative sentence yCo-articulation xA red apple. xDid you call me? xHit and run zMandarin yTone yRetroflex or not yCo-articulation x 兒化音

8. -8- Problems to be Solved zScore related yOptimization yConsistency yInterpretability zConfusing phone id. ( 日本人的發音 ) zSlightly adaptation zParagraph-level assessment zContents construction

9. -9- Demo: Practice of Mandarin Idioms of Length 4 ( 一語中的 ) yLevel (difficulty) of an idiom is based on it’s freq. via Google search: x 孤掌難鳴 ===> 260,000 x 鶼鰈情深 ===> 43,300 x 亡鈇意鄰 ===> 22,700 x 舉案齊眉 ===> 235,000 yCan be adapted for English learning yNext step: multi- threading, fast decoding via FSM

10. -10- Demo: Recitation Machine （唸唸不 忘） zSupport Mandarin & English zSupport user-defined recitation script zNext step: multithreading for recording & recognition

11. -11- Demo: Dialog Practice via Videos zDialog-based practice and evaluation

12. -12- Demos on PC and PMP zPC 軟體 yLucy’s Café: Speech and Score zPMP y 華語練習機

13. -13- Demo: Embedded Systems yChicken run ( 落跑雞 )Chicken run ( 落跑雞 yPenguin for Tang Poetry ( 唐詩企鵝 )Penguin for Tang Poetry ( 唐詩企鵝 ) yRobot Fighter ( 蘿蔔戰士 )Robot Fighter ( 蘿蔔戰士 ) ySinging Bass & Dog ( 大 嘴鱸魚和唱歌狗 )Singing Bass & Dog ( 大 嘴鱸魚和唱歌狗 )

14. -14- On-going Work zOn-going work: yTone recognition and assessment yRetroflex & nonretroflex recognition yDetection of “ 兒化音 ” zDemo page: yhttp://mirlab.org/mir_main/demo.htmhttp://mirlab.org/mir_main/demo.htm