1. ViSiCAST Consortium Meeting, January 2002 1 WP3 Report, September-December 2001

2. ViSiCAST Consortium Meeting, January 2002 2 Main events q Blue Peter appearance q Evaluations of unconstrained system (RNID+Consignia) q Deliverable 3-2, video + report q Discussions on TESSA trials with Consignia q Discussions on sign recognition strategy with members of vision group (Milestone 3-10)

3. ViSiCAST Consortium Meeting, January 2002 3 Overview of speech/language processing evaluation Speech Recogniser Language Processor Retrieved phrase Word error-rate Phrase error-rate ConceptText

4. ViSiCAST Consortium Meeting, January 2002 4 Testing the Phrase Retrieval Accuracy of the Unconstrained System q 10 speakers q For each speaker and each of 200 phrases: è record one utterance of the “constrained” phrase è ask speaker to write down another way of expressing the phrase è record speaker saying this phrase q Training of recogniser not possible for 10 different speakers q Hence: è measure phrase retrieval accuracy on text of unconstrained phrases only è use high-quality speaker-independent recogniser (HTK) to produce recognised text, test on this text

5. ViSiCAST Consortium Meeting, January 2002 5 Results on 5 subjects using their own phrases (text only) Trained on “canonical” phrases + phrases from the other subjects Av error (1-best) = 9.1% Av error (3-best) = 3.6% Av error (5-best) = 2.7%

6. ViSiCAST Consortium Meeting, January 2002 6 Speech Recognition Experiments q HTK speech recogniser è speaker-independent (no speaker adaptation) è trained on 90 speakers, about 2.5 hours of speech è “triphone” acoustic models è different language models tested: l finite-state network (49% words correct) l word bigram (75.1% words correct)

7. ViSiCAST Consortium Meeting, January 2002 7 Relationship between scale factor and % correct/accurate

8. ViSiCAST Consortium Meeting, January 2002 8 Recognition results using best scale-factor

9. ViSiCAST Consortium Meeting, January 2002 9 Relationship between word and phrase accuracy

10. ViSiCAST Consortium Meeting, January 2002 10 Speech recognition errors that lead to phrase recognition errors Target phrase: is it valuable Spoke: is the item valuable Recognised: that available RESULT: i'm afraid we don't have that available Target phrase: i'm afraid you can't send that by post Spoke:this cant be posted Recognised: that’s card to posted RESULT:could i have a look at your cheque guarantee card please Target phrase:what cc is the car Spoke:what cc is the vehicle Recognised: what’s use do that available RESULT:i’m afraid we don’t have that available

11. ViSiCAST Consortium Meeting, January 2002 11 TESSA trials q Five trial sites: Derby, Bristol, Wolverhampton, Liverpool, London(?) q 3-5 month trial q Consignia to do training, support, staff interviews etc. q RNID to help with evaluation and also design of questionnaires, deaf awareness, etc. q Evaluation criteria to be clarified