1. Beyond Usability: Measuring Speech Application Success Silke Witt-Ehsani, PhD VP, VUI Design Center TuVox

2. S P E E C H W I T H I N R E A C H Outline What is Success? Success Criteria Success Metrics Putting it all together: A health check methodology Success vs Design How they effect each other Case studies

3. S P E E C H W I T H I N R E A C H Success Criteria: i.e. What is “success”? Common criteria: Are callers transferred to the correct destination? How many callers are being helped? How do callers like my speech applications? What is the system recognition accuracy? Different questions (Success Criteria) require different answers (Success Metrics) How do we do that?

4. S P E E C H W I T H I N R E A C H Success Metrics: Subjective vs Objective Subjective Usability study Whole call recordings Individual caller feedback Objective = Application Statistics Automation rates Containment rates Non-cooperative caller rate

5. S P E E C H W I T H I N R E A C H Success Metrics: Business vs Technical Business Metrics for Business User: Routing Accuracy Agent Transfers Customer Satisfaction Technical Users: need detailed application performance on dialog state level grammar coverage NoMatch, NoInput need ability to drill down More Transfers out of application = higher call center cost Higher Routing Accuracy = Less Agent-to- agent transfers Business stakeholders care about the bottom line impact of several application and speech events

6. S P E E C H W I T H I N R E A C H Common Business Metrics Containment rate = “keep caller hostage in the system” Automation rate = “offer complete functionality…” Successful routing = “get the caller to the right expert” Average call duration And many, many more ….

7. S P E E C H W I T H I N R E A C H Application Health Check - Business 3 main elements of a Business Health Check are 1.Custom defined success rate 2.Non co-operative Caller rate 3.Agent Transfer rate  Transfer due to explicit caller request  Transfer due to errors (both speech and system)  Transfer by design (i.e. correctly routed calls)

8. S P E E C H W I T H I N R E A C H Example Success Metric: Routing Accuracy Definition: Confirmed routed calls (calls reaching an end destination) over all calls Useful metric when using: Skills-based routing Routing application with N routing points 68.3% 77% % Routing Accuracy ~150 routing points ~ 50 routing points 4 routing points 85%

9. S P E E C H W I T H I N R E A C H Example: Non Co-operative Callers Possible reasons: Degree of caller acceptance of system Non application related, such as wrong number, child crying etc. Definition: Non-cooperative callers is the percentage of all callers that immediately hang-up or request an agent but never interact with the application Expected range: 5-10% of call volume 6.3% 8.6% % Non-cooperative Callers Open-ended Router Directed Dialog Technical Support

10. S P E E C H W I T H I N R E A C H Example: Agent Transfers Applications tend to have many different types of agent transfers. Main categories: Customer zero-ing out Routing to an agent based on caller information is a “Designed Transfer” Routing due to some logic in the application is a “Necessary Transfer” Agent Transfers have immediately impact on call center cost 45% 4.7% % Agent Requests Definition: % Agent transfers of all calls Example from a Telecommunications Company

11. S P E E C H W I T H I N R E A C H Baseline and Trending Numbers are relative, they only have meaning in a context When defining success metrics, 1.create a baseline 2.then compare to that. Potential Baselines: previous IVR touch-tone application Go-live Performance 52% 66% Customers finding speech easier or much easier than IVR 76% UsabilityGo-liveTuning 1

12. S P E E C H W I T H I N R E A C H Application Health check = Technical Purpose of hotspot analysis Identify areas where application is performing sub-optimal Hotspot analysis should be done for each dialog state Important: Hotspot analysis gives the “ where ” of issues, not the “ why ”!

13. S P E E C H W I T H I N R E A C H Framework for Technical Health Check TuVox Hotspot analysis = Integrated view of: Hang-up ( %H ) % Final NoInput ( %NI) % Final NoMatch ( %NM) Transfer Requests ( %TR ) State Exit Count = # of calls * ( %H + %NI + %NM + %TR) Rule of Thumb : These numbers are a first order of approximation: Sort by highest state exit count Review one by one in context, i.e. high hang-up because it is a logical end point

14. S P E E C H W I T H I N R E A C H Hotspot Analysis Example Prompt IDPrompt Text# Hits # 2nd No Match # 2nd No Input # User Hangup Total Exit Number STTransferTS#124 Would you like to hear that website again899318120560916477 STTransferSS#11 Please hold while I get someone who can help you.289400 NTGetQueue#302Please say yes or no.215731800143323 NTDisBilling#9 Which do you need help with a bill a service charge, a purchase or something else.22112172581323 NTFinder#301Please say yes or no.37111210102223

15. Success Criteria and Design

16. S P E E C H W I T H I N R E A C H Success and Design are tightly linked Success Metric Authentication Look up all loans for this callers Does caller has a line of credit? no yes no Loan Menu:  Balance  More loan details  Make loan payment Caller selects from list of loans Does caller have more than 1loan? yes Design Success determines the design Design influences success

17. S P E E C H W I T H I N R E A C H Case Study 1: Airline application Customer requirement: 64% Success Success definition: “For 64% of the callers entering the application, their ticket reservation record has to be retrieved from the back-end Design consequences: Ensure via prompting that callers have their record identifier number before entering the application Make it hard to get to an agent, i.e. multiple retries Explain what the record identifier was Design tailored to success criteria but at the expense of ease of use and caller experience

18. S P E E C H W I T H I N R E A C H Case Study 2: Travel Application Impact on Application Performance Turn failure rate = Decreased by 39% Opt-out rate to the call center = Decreased by 44% Hotspot analysis identifies a too high number of exists at a main menu Observation: One menu option is much more common than other 5 choices Old Design: Menu with 6 options New Design: Yes/no question followed by a menu

19. S P E E C H W I T H I N R E A C H Case Study 3: HighTech Routing Application 3 success criteria: Average call handling less than 30 secs High customer satisfaction 4 queues to route to, but many different call reasons Influence of these criteria on the design: Only 1 reprompt instead to standard 2 attempts No traditional error prompting a la ‘sorry I didn’t get that’ Natural language open ended prompting with high coverage grammar

20. S P E E C H W I T H I N R E A C H Summary Define Application Success Criteria Based on that, define success metrics Use trending and baseline to put data in context Success Criteria and Design are highly interlinked, i.e. success criteria determine the design The design influences how targeted success metrics can be met