STARDUST – Speech Training And Recognition for Dysarthric Users of Assistive Technology Mark Hawley et al Barnsley District General Hospital and University of Sheffield

STARDUST To develop speech-driven environmental control and voice output communication devices for people with dysarthria –To develop a reliable small vocabulary speech recogniser for dysarthric speakers –To develop a computer training program to help to stabilise the speech of dysarthric speakers

Research Team Department of Medical Physics and Clinical Engineering, Barnsley District General Hospital (Mark Hawley, Simon Brownsell, Stuart Cunningham) Institute of General Practice and Primary Care, University of Sheffield (Pam Enderby, Mark Parker, Rebecca Palmer) Department of Computer Science, University of Sheffield (Phil Green, Nassos Hatzis, James Carmichael) Project funded by Dept of Health New and Emerging Applications of Technology (NEAT) programme

Dysarthria  A neurological motor speech impairment  characterised by slow, weak, imprecise and/or uncoordinated movements of the speech musculature.  Speech is often difficult to understand (unintelligible) and variable (inconsistent)  Frequently associated with other physical disabilities  Severe = <40% intelligible

Speech recognition systems Large vocabulary, speaker adaptive –Speech-input writing programmes (eg Dragon) Small vocabulary, speaker independent –telephone automatic voice response Small vocabulary, speaker dependent –Speech-input control systems Environmental control systems Mobile phones Smart homes

Speech-input writing programmes Normal speech - with recognition training can get >90% recognition rates (Rose and Galdo, 1999) Mild dysarthric speech % lower recognition rates (Ferrier, 1992 ) Recognition declines as speech deteriorates - by 30-50% for single words (Thomas-Stonell, 1998, Hawley 2002)

Performance of a commercial speaker- dependent recogniser (in ‘ideal’ conditions)

Difficult speech recognition problem Dysarthric speech –different to ‘normal’ models –more variable than ‘normal’ speech, both between and within speakers –difficult to collect large corpus of speech

STARDUST To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria To develop a reliable small vocabulary speech recogniser for dysarthric speakers To develop a computer training program to help to stabilise the speech of dysarthric speakers (ie improve consistency) and improve recognition

Intelligibility and Consistency ‘ Normal’ speech will be almost 100% intelligible and with few articulatory differences over time (consistent). ‘Severe’ dysarthria may be completely unintelligible to the naïve listener and will show high variability (inconsistent) –but may show consistency of key elements which will make it more intelligible to the familiar listener. STARDUST is concerned with consistency

Training program Visual feedback to improve consistency at word level –Quantitative –Real time To be used by the client alone or with carer or therapist Training tool records speech - used to build recogniser

Training program set-up Record 10 examples of each word to be trained Program builds models of words based on examples For each word, program selects example that best matches its model (the best-fit recording) Program feeds back a measure of the match between last utterance and model

Recogniser Score calculation Microphone PC Switch

Outcome of speech training (preliminary data) In a group of 5 users, 3 showed an upward trend, 2 showed no upward trend

STARDUST To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria To develop a reliable small vocabulary speech recogniser for dysarthric speakers To develop a computer training program to help to stabilise the speech of dysarthric speakers

Recognition technology Small vocabulary Speaker dependent uses hidden Markov models based on HTK (University of Cambridge)

STARDUST recogniser performance (N=number of words used for training)

Confusion matrix

STARDUST To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria To develop a reliable small vocabulary speech recogniser for dysarthric speakers To develop a computer training program to help to stabilise the speech of dysarthric speakers

Recogniser Look-up table Speech synthesiser or recording Microphone PC Switch

Vocabulary mapping One to one (word to phrase) mapping –‘Want’ = I need something, could you help me? Pseudo-grammatical combinations –‘Want... drink’ = Could I have a drink, please? Coding –‘3…6…4’ = I went to Spain for my holidays –n m possible combinations, where n is no of words in vocab, m is length of vocabulary string

Recogniser Look-up table Microphone PC Infra-red Switch

Work in progress Test systems in home-based field trials –acceptability –usability (eg speed of access) –accuracy –reliability –practicality

Work in progress Remove switch activation of recogniser Increase vocabularies to test limits of recogniser Develop tools for clinicians to build and test individual configurations

STARDUST - conclusions Recogniser that recognises severely dysarthric speech Computer-based training program to improve recognition and consistency –word level (and sub-word level in future) –collects lots of speech data for recogniser Developed demonstrators of environmental control and voice-output device –next step to test in real usage