1. Telephony Speech Recognition Application Testing Presentation for IEEE SCV Signal Processing Society March 8, 2004 Copyright CoAssure, Inc., 2004

2. 3/8/2004 1 Telephony Applications Automated Telephony Applications: –DTMF (Touchtone) –Speech Recognition –Speech With DTMF Support Advantages: –24/7 Self-service –Access to Real-time Information –Cost Effective, Reduces Need for Live Agents –Convenient for Wireless Users

3. Copyright CoAssure, Inc., 20043/8/2004 2 Need for Test Why Test Thoroughly: –Mission Critical Applications –Problems Cause Customer Dissatisfaction –Degradation Can Eliminate Cost Savings When to Test: –Prior to New Deployment –Before Hardware and Software Upgrades –Periodic, Ongoing Monitoring

4. Copyright CoAssure, Inc., 20043/8/2004 3 Types of Testing Wizard of Oz Initial Prototype Usability Dialog Traversal (Call Flow) Backend Integration and Data Access Usability Load Testing Recognition Performance Ongoing Monitoring

5. Copyright CoAssure, Inc., 20043/8/2004 4 State Machine Model Applications are Built as State Machines State: –Context or Location Within the Application –Prompt –Caller Given List of Choices to Select From –Active Vocabulary and Grammar –Transition from State to State Upon User Input

6. Copyright CoAssure, Inc., 20043/8/2004 5 Dialog Traversal Process Each State is Tested As Follows: –All Valid Speech Inputs –All Valid DTMF Inputs –Synonyms/Aliases –Global Commands –Recoverable Silence –Recoverable Invalid Speech Input –Recoverable Invalid DTMF Input –Consecutive Silence –Consecutive Invalid Speech Input –Consecutive Invalid DTMF Input

7. Copyright CoAssure, Inc., 20043/8/2004 6 Manual Dialog Traversal Development Engineers Place Calls Problems: –Time Consuming –Wasteful of Engineering Resources –Not Repeatable No Log of Test Steps No Call Recordings –Not Consistent (Timing and Speech Variation) –Not Accurate –Prone to Missing Paths/Incomplete Coverage

8. Copyright CoAssure, Inc., 20043/8/2004 7 Benefits of Automation Speeds Time-to-market Frees Engineering Resources Reduces Testing Cost Ensures Comprehensive Test Coverage Repeatable and Consistent Process Accurate Timing Precise Measurements and Metrics Call Logs, Reports and Statistics Call Recordings Suitable for Regression Testing

9. Copyright CoAssure, Inc., 20043/8/2004 8 Automation Elements Test Generation Test Execution Results Generation

10. Copyright CoAssure, Inc., 20043/8/2004 9 Automated Test Generation Generates Test Set Automatically Adapts to Spec Changes Easily Ensures Complete Test Coverage Creates Efficient Test Calls

11. Copyright CoAssure, Inc., 20043/8/2004 10 Automated Test Execution Places Test Calls to System Under Test Eliminates Human Inconsistency Accurate and Repeatable Stores Log and Results in Database Records All Calls

12. Copyright CoAssure, Inc., 20043/8/2004 11 Automated Results Generation Extracts Reports from Log Database Presents Variety of Reports Makes Call Recordings Available Web Accessible

13. Copyright CoAssure, Inc., 20043/8/2004 12 Conclusion Automated Speech and DTMF Telephony Applications Allow Access to Real-time Data and Reduce Cost. They Must Be Tested Thoroughly. Test Automation Can Improve Quality and Customer Satisfaction, While Reducing Cost and Speeding Time-to-market.