2. STARDUST – Speech Training And Recognition for Dysarthric Users of Assistive Technology Mark Hawley et al Barnsley District General Hospital and University of Sheffield

3. STARDUST To develop speech-driven environmental control and voice output communication devices for people with dysarthria –To develop a reliable small vocabulary speech recogniser for dysarthric speakers –To develop a computer training program to help to stabilise the speech of dysarthric speakers

4. Research Team Department of Medical Physics and Clinical Engineering, Barnsley District General Hospital (Mark Hawley, Simon Brownsell, Stuart Cunningham) Institute of General Practice and Primary Care, University of Sheffield (Pam Enderby, Mark Parker, Rebecca Palmer) Department of Computer Science, University of Sheffield (Phil Green, Nassos Hatzis, James Carmichael) Project funded by Dept of Health New and Emerging Applications of Technology (NEAT) programme

5. Dysarthria  A neurological motor speech impairment  characterised by slow, weak, imprecise and/or uncoordinated movements of the speech musculature.  Speech is often difficult to understand (unintelligible) and variable (inconsistent)  Frequently associated with other physical disabilities  Severe = <40% intelligible

6. Speech recognition systems Large vocabulary, speaker adaptive –Speech-input writing programmes (eg Dragon) Small vocabulary, speaker independent –telephone automatic voice response Small vocabulary, speaker dependent –Speech-input control systems Environmental control systems Mobile phones Smart homes

7. Speech-input writing programmes Normal speech - with recognition training can get >90% recognition rates (Rose and Galdo, 1999) Mild dysarthric speech - 10-15% lower recognition rates (Ferrier, 1992 ) Recognition declines as speech deteriorates - by 30-50% for single words (Thomas-Stonell, 1998, Hawley 2002)

8. Performance of a commercial speaker- dependent recogniser (in ‘ideal’ conditions)

9. Difficult speech recognition problem Dysarthric speech –different to ‘normal’ models –more variable than ‘normal’ speech, both between and within speakers –difficult to collect large corpus of speech

10. STARDUST To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria To develop a reliable small vocabulary speech recogniser for dysarthric speakers To develop a computer training program to help to stabilise the speech of dysarthric speakers (ie improve consistency) and improve recognition

11. Intelligibility and Consistency ‘ Normal’ speech will be almost 100% intelligible and with few articulatory differences over time (consistent). ‘Severe’ dysarthria may be completely unintelligible to the naïve listener and will show high variability (inconsistent) –but may show consistency of key elements which will make it more intelligible to the familiar listener. STARDUST is concerned with consistency

12. Training program Visual feedback to improve consistency at word level –Quantitative –Real time To be used by the client alone or with carer or therapist Training tool records speech - used to build recogniser

13. Training program set-up Record 10 examples of each word to be trained Program builds models of words based on examples For each word, program selects example that best matches its model (the best-fit recording) Program feeds back a measure of the match between last utterance and model

14. Recogniser Score calculation Microphone PC Switch

15. Outcome of speech training (preliminary data) In a group of 5 users, 3 showed an upward trend, 2 showed no upward trend

16. STARDUST To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria To develop a reliable small vocabulary speech recogniser for dysarthric speakers To develop a computer training program to help to stabilise the speech of dysarthric speakers

17. Recognition technology Small vocabulary Speaker dependent uses hidden Markov models based on HTK (University of Cambridge)

18. STARDUST recogniser performance (N=number of words used for training)

19. Confusion matrix

20. STARDUST To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria To develop a reliable small vocabulary speech recogniser for dysarthric speakers To develop a computer training program to help to stabilise the speech of dysarthric speakers

21. Recogniser Look-up table Speech synthesiser or recording Microphone PC Switch

22. Vocabulary mapping One to one (word to phrase) mapping –‘Want’ = I need something, could you help me? Pseudo-grammatical combinations –‘Want... drink’ = Could I have a drink, please? Coding –‘3…6…4’ = I went to Spain for my holidays –n m possible combinations, where n is no of words in vocab, m is length of vocabulary string

23. Recogniser Look-up table Microphone PC Infra-red Switch

24. Work in progress Test systems in home-based field trials –acceptability –usability (eg speed of access) –accuracy –reliability –practicality

25. Work in progress Remove switch activation of recogniser Increase vocabularies to test limits of recogniser Develop tools for clinicians to build and test individual configurations

26. STARDUST - conclusions Recogniser that recognises severely dysarthric speech Computer-based training program to improve recognition and consistency –word level (and sub-word level in future) –collects lots of speech data for recogniser Developed demonstrators of environmental control and voice-output device –next step to test in real usage