1. PG 1 Caroline V Nelson Speech Solutions Team Technical Lead Nortel Enterprise Multimedia Professional Services August 20, 2007 Success Criteria for Speech Applications

2. NORTEL CONFIDENTIAL PG 2 Zen and the Art of Speech Applications >We all want to deploy “successful” speech systems >But what exactly is the definition of “success”? >That depends on who you ask…

3. NORTEL CONFIDENTIAL PG 3 “Success” depends on who you ask… Three different perspectives: Purchasing Customer Speech Scientist End User (Caller)

4. NORTEL CONFIDENTIAL PG 4 Speech Scientist (designs/develops the app) >The Micro View: Recognition Accuracy In/Out of Grammar Utterances Per-state “success” (retries, etc) Host database interaction Caller transactions (tasks across multiple dialog states)

5. NORTEL CONFIDENTIAL PG 5 “Micro” can’t see the forest for the trees… >The recognition accuracy numbers look great! (but there’s a very high rate of callers speaking out-of- grammar…) >The overall speech numbers look nice! (but broken out into states shows a hotspot down the flow path…) >The speech is working like a charm! (but the back-end host is experiencing some serious issues…)

6. NORTEL CONFIDENTIAL PG 6 End Use Caller (calls into the app) >The “Middle-Management” View: Task Completion Quick Easy “usability”

7. NORTEL CONFIDENTIAL PG 7 Now that we have the foundation… >Need speech scientist metrics to achieve caller goals Callers prompted to speak in- grammar Utterances recognized Error recovery through effective strategies Reliable back-end host data >Caller “task” start & end points must be defined and agreed upon in advance to be properly logged. >But if it’s awkward and cumbersome to use… Which is why we do usability testing – right?

8. NORTEL CONFIDENTIAL PG 8 Customer (purchases the app for business) >The Macro View: Self-service call containment Cost reduction Customer (caller) satisfaction = loyalty/retention >In the end it’s all about the bottom line $$$

9. NORTEL CONFIDENTIAL PG 9 Comparison is Key to “Macro” >Usability testing for caller satisfaction Benchmarking against “industry standards” >Need metrics from previous system (dtmf?) for comparison To demonstrate level of improvement >If no previous system… Then each self service completed task is automation gain Compare automation level against benchmarks per industry

10. NORTEL CONFIDENTIAL PG 10 It’s all connected… Macro Micro Mid-level >Recognition accuracy >In/Out of Grammar >State success (retries, etc) >Reliable host data >Task completion >Quick and easy >“Usability” >Customer satisfaction >Self-service automation >Call containment >Bottom line $$$

11. NORTEL CONFIDENTIAL PG 11 Be careful what you measure… >…because what you measure is what you’ll get. >Metrics need to be identified, agreed upon and planned in advance – including any comparison measures.