1. Beyond Usability: It Ain’t The Only Outcome That Matters! Melanie D. Polkosky, Ph.D., CCC-SLP Social/Cognitive Psychologist Melanie D. Polkosky, Ph.D., CCC-SLP Social/Cognitive Psychologist

2. Presentation Goals 1.To identify 4 factors that are critically important to successful speech implementations 2.To describe the role of users’ expectations in creating positive outcomes with speech 3.To identify specific expectation types that are handled by effective applications 4.To identify items to prioritize in your next VUI design

3. What is Usability?  Ease of use  The design and deliverance of useful, usable, desirable, easy-to-learn solutions that have consistent functions that allow people to do what they want to do and are well-liked (Gould)  Nielson (1993) criteria: learnability, efficiency, memorability, errors, satisfaction  ISO 9241-11 - [Usability refers to] the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use."

4. Speech Usability must…  Account for broad perspectives on communication (communication, social psychology, cognitive psychology, services marketing, speech science, human factors, linguistics, esp. pragmatics)  Address common practice issues (e.g., customer satisfaction)  Create viable measurement techniques  Factor analysis = statistical method of investigating variables  Reliable and valid  Explain standard user behavior (e.g., user politeness)

5. So what?  Without a clear understanding of usability for speech, we could:  Prioritize incorrectly in our designs  De-emphasize or overlook important issues  Get distracted by irrelevant topics  Not measure the right outcomes  Practical VUI design will not be based on knowledge of human communication behavior  Our field will lack scientific rigor (ie., credibility)

6. Researching Speech Outcomes Polkosky (2005)  862 individuals listen to 6 applications and rate 76 characteristics  Factor analysis and reliability analysis to eliminate poor items  Whittle to 25 items that group into 4 factors  What are the important outcomes of speech?  User Goal Orientation  Speech Characteristics  Verbosity  Customer Service Behavior

7. User Goal Orientation  The system made me feel like I was in control.  I could find what I needed without any difficulty.  I could trust this system to work correctly.  I felt confident using this system.  This system would help me be productive.  The quality of this system made me want to remain a customer of this business.  I would be likely to use this system again.

8. Speech Characteristics  The system’s voice was pleasant.  The system’s voice sounded like people I hear on the radio or television.  The system’s voice sounded natural.  The system’s voice sounded enthusiastic or full of energy.  The system’s voice sounded like a regular person.

9. Verbosity  The messages were repetitive.  The system gave me more details than I needed.  The system was too talkative.  I felt like I have to wait too long for the system to stop talking so I could say something.

10. Customer Service Behavior  The system used terms I am familiar with.  The system used everyday words.  The system was organized and logical.  The system spoke at a pace that was easy to follow.  The system seemed polite.  The system seemed courteous.  The system seemed friendly.  The system seemed professional in its speaking style.

11. Speech Outcome Factors and Customer Satisfaction As this rating goes UP…Customer Sat goes… User Goal Orientation** (best predictor) up Speech Characteristics** (strongest effect size) up Verbositydown Customer Service Behavior up

12. Adhering to user EXPECTATIONS is the most important thing to do to insure a high-quality outcome with speech The bottom line…

13. What expectations?  Conversational expectations - ingrained, unconscious knowledge of how conversation works  Informative, but not more informative  True statements, clear, unambiguous and brief  Relevant  Customer service expectations - ingrained, unconscious knowledge about how service providers should act  Deferential, polite, friendly, facilitative  Service quality - the gap between expected and actual service  Other expectations - media, sequencing, task priority  Professional voice (media expectations)  User task-based prompt sequencing and organization

14. What else matters (beyond usability)?  Adhering to users’ expectations about conversation and customer service  User personality characteristics  Age and gender  Inherent novelty seeking  Social skills (sensitivity)  Need for interaction with a service provider  Providing a pleasurable listening environment

15. How to design based on this research…  Highest priority on understanding users’ goals with the system  Does sequencing/organization match what users want?  Question, question, question directives from The Business about sequence, what users know  Pre-design (concept phase) user analysis and needs assessment  Cognitive walkthrough or usability testing a must  Next highest priority on finding a high-quality, natural voice talent and coaching him/her effectively  Avoid task irrelevant messaging or “over-explaining”  Don’t design a ‘persona,’ design a customer service provider  Learn more about conversation and social expectations - critical knowledge  Do not underestimate the impact of non-user knowledge and assumptions by the design team - it has the power to break your application