1. Usability Testing Of Interaction Components: Taking the Message Exchange as a Measure of Usability Willem-Paul Brinkman Brunel University, London Reinder Haakma Philips Research Laboratories Eindhoven Don Bouwhuis Eindhoven University of Technology

2. Topics  Introduction  Evaluation Method  Experimental validation  Comparison with other evaluation methods

3. Introduction Component-Based Software Engineering Empirical Usability Testing Single device with Multiple components

4. Layered communication

5. Layered Protocol Theory 15 + 23 = 15+23= 01111 10111 Add 100110 38 ProcessorEditor Control results Control equation UserCalculator 15 15 + 15 + 23 38

6. Evaluation Method Aim to evaluate the usability of a component based on the message exchange between a user and a specific component

7. Evaluation Method: Test Procedure  Normal procedures of a usability test  User task which requires interaction with components under investigation  Users must complete the task successfully

8. Evaluation Method Component specific objective performance measure: 1.Messages received + Weight factor A common currency 2.Compare with ideal user A common point of reference Usability of individual components in a single device can be compared with each other and prioritized on potential improvements

9. Click {1} Click {1} Call <>{2} Set <Fill colour red, no border> {7} Right Mouse Button Menu Properties Assigning weight factors to represent the user’s effort in the case of ideal user

10. Total effort value Total effort =  MR i.W MR i.W : Message received. Weight factor Click {1} Click {1} Call <>{2} Right Mouse Button Menu Properties 5 2 = 7 + 2

11. Assigning weight factors in case of real user Correction for inefficiency of higher and lower components Visual Drawing Objects Properties Right Mouse Button Menu

12. Assigning weight factors in case of real user Assign weight factors as if lower components operate optimal Visual Drawing Objects Properties Right Mouse Button Menu Inefficiency of lower level components: need more messages to pass on a message upwards than ideally required

13. Assigning weight factors in case of real user Visual Drawing Objects Properties Right Mouse Button Menu Inefficiency of higher level components: more messages are requested than ideally required UE: User effort MR i.W : Message received. Weight factor #MSU real :Number of messages sent upward by real user #MSU ideal :Number of messages sent upward by ideal user  MR i.W #MSU real  #MSU ideal UE =

14. Ideal User versus Real User Extra User Effort = User Effort - Total effort The total effort an ideal user would make The total effort a real user made The extra effort a real user made Calculate for each component: Prioritize

15. Experimental validation 40 users 4 mobile telephones 2 components were manipulated according to Cognitive Complexity Theory  Function Selector: Broad/shallow versus Narrow/deep  Short Text Messages: Simple versus Complex

16. Architecture Mobile telephone Send Text Message Send Text Message Function Selector Function Selector

17. Experimental validation Functions Selector Broad/shallow Narrow/deep

18. Experimental validation Send Text Message Simple Complex

19. Results Mobile phones Extra User Effort

20. Results MeasureFunction Selector Send Text Message Objective Extra keystrokes0.64**0.44** Task duration0.63**0.39** Perceived Overall ease-of-use-0.43**-0.26* Overall satisfaction-0.25*-0.22 Component-specific ease-of-use-0.55**-0.34** Component-specific satisfaction-0.41**-0.37** Partial correlation between extra user effort regarding the two components and other usability measures *p. <.05. **p. <.01.

21. Comparison with other evaluation methods Overall measures Sequential Data analysis GOMS Thinking-aloud, Cognitive Walkthrough and heuristic evaluation Example: Keystrokes, task duration, overall perceived usability Relatively easy to obtain Unsuitable to evaluate components

22. Overall measures Sequential Data analysis GOMS Thinking-aloud, Cognitive Walkthrough and heuristic evaluation Based only on lower-level events Pre-processing: selection, abstraction, and re-coding Relation between higher- level component and compound message less direct Components’ status not recorded Comparison with other evaluation methods

23. Help to understand the problem Only looking at error-free task execution Considers the system only at the lowest-level layer Overall measures Sequential Data analysis GOMS Thinking-aloud, Cognitive Walkthrough and heuristic evaluation Comparison with other evaluation methods

24. Quicker Evaluator effect (reliability) Overall measures Sequential Data analysis GOMS Thinking-aloud, Cognitive Walkthrough and heuristic evaluation Comparison with other evaluation methods

25. Summary  The usability of individual components can be tested  Taking the message exchange and assigning weight factor to them (common currency)  Comparison between ideal user and real users (common reference point)

26. Questions Thanks for your attention