1. © 2008 The MathWorks, Inc. ® ® Using Instant Messaging to Usability Test an API Rachel Cobleigh Donna Cooper

2. 2 ® ® Our task: evaluate the usability of an API API = Application Programming Interface Used for writing software code Programmers (our users) have different expectations than point-and-click users This API was in the design phase, so no live prototype was available

3. 3 ® ® Our product: MATLAB ®

4. 4 ® ® Example command-line interaction

5. 5 ® ® Challenge #1: No GUI

6. 6 ® ® Partial Solution: Use the documentation ?

7. 7 ® ® Another solution: Language Norming Describe a task and ask the participant to suggest an API name Provide a list of API names and ask the participant to choose the one that best describes a task Show the participant an API name and ask what they think it does Provide help documentation and observe adoption of API names Describe a task and ask the participant to suggest an API name Provide a list of API names and ask the participant to choose the one that best describes a task Show the participant an API name and ask what they think it does

8. 8 ® ® How does API testing compare to GUI testing? What's different Users will need to rely more heavily on documentation Prepare at least a first pass at error and warning messages and bare-bones help documentation Whats the same Recruit as usual Prepare a scenario and user tasks to structure the test session Have a developer act as the computer Do walkthroughs before the first real usability test

9. 9 ® ® Challenge #2: Realistic API interaction when theres no live prototype Paper prototype? Awkward participant experience Difficult to manage test results

10. 10 ® ® Challenge #2 continued... Word processor? Participant experience still awkward Pre-written computer responses could bias test results Difficult to manage test results ?? Undefined function or variable c. Warning: site vector should be arrow Vector with integer elements ??? Error using ==> imshow>preParseInputs at 340 IMSHOW expected at least 1 input argument but was called instead with 0 input arguments. >> d = zeros(0.5) Warning: site vector should be arrow vector with integer elements d = [ ] d = zeros(c) ?? Undefined function or variable c.

11. 11 ® ® Solution: Use Instant Messaging (IM) to simulate the command-line interface! IM interaction closely resembles command-line interaction Blank screen interface Text-based input and responses Immediate interactive responses to input

12. 12 ® ® Example of an IM prototype interaction

13. 13 ® ® Set up with a local participant

14. 14 ® ® Set up with a remote participant

15. 15 ® ® Mechanics of a successful test session Customize Use copy-and-paste Have responses ready Agree on a protocol Provide help and/or documentation Ensure that computer realistically simulates the command-line [>>] try looking at other methods[>>] command mostly worked

16. 16 ® ® Observations & tips for success IM interactions closely resemble command-line interactions Participants quickly understood what was expected of them because IM conversations felt like the command- line interface Pre-defined text made the IM session flow quickly so that the illusion of really interacting with the software could be maintained Delays while computer is typing are good opportunities to ask participants questions

17. 17 ® ® Benefits of using IM Improved participant experience Easier to manage test results Transcripts are easy to save and disseminate

18. 18 ® ® Limitations of using IM It's not a perfect simulation It can have a slightly different look-and-feel There can be security issues if testing proprietary designs over the internet Avoid using personal IM accounts: minimize outside interruptions Hey hon, can u stop for milk on the way home?

19. 19 ® ® Conclusions We improved the usability of our API The techniques were simple and effective

20. 20 ® ® Acknowledgements We are grateful for the input given by the following people at The MathWorks, without whom this work would not have been possible: Mary Beth Rettger Amy Kidd Jennifer Lymneos Jeff Goodwin Ellen Mangan

21. 21 ® ® Related work Steven Clarke has published work on API usability testing Clarke, Steven, Measuring API Usability, Dr. Dobbs Journal Special Windows/.NET Supplement, May 2004, pp. S6-S9 Focus is put on analyzing test data in terms of the Cognitive Dimensions framework http://blogs.msdn.com/stevencl/ Testing was done on a live prototype, in an integrated development environment (i.e., a specialized text editor) Laura Chessman introduced Language Norming to The MathWorks