1. Jesper Kjeldskov & Jan Stage Department of Computer Science Aalborg University Denmark New Techniques for Usability Evaluation of Mobile Systems

2. 2 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Background Mobile technologies and systems… PDAs, wearables, mobile phones, tablet computers … challenge usability testing methods Users are physically mobile during use Use involve activities in physical surroundings Use context can be difficult to recreate realistically This paper explore new techniques for usability testing mobile systems in laboratory settings

3. 3 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Two Evaluation Approaches Field experiments Realistic use context Difficult to control Complicated data collection Complex and time consuming Safety and ethical issues Laboratory experiments Experimental control High quality data collection Lack of realism

4. 4 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Laboratory vs. Field Most usability evaluations of mobile systems are currently conducted in laboratory settings A recent literature study revealed that… 41% of mobile HCI research involve evaluation 71% of this is done in laboratory settings It is a widely adopted point of view that mobile systems require field evaluations, but… It is difficult to conduct field evaluations The added value of testing in the field is unknown Additional problems come at a high cost (time & effort)

5. 5 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Overall Research Question How can new techniques for usability tests of mobile systems increase realism of use in a laboratory setting? … while facilitating systematic data collection in a controlled environment Previous studies: Specialized use contexts This study: Physical mobility during use

6. 6 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Two Experiments Two experiments comparing techniques for lab- and field- based usability testing of mobile systems were conducted Experiments explored different techniques requiring… 1. Different levels of physical movement 2. Divided cognitive attention Example application: use of Short Message Service (SMS) on PDAs and mobile phones Narrow focus on interaction rather than broad focus on use

7. 7 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Five Laboratory Techniques… Attention needed to navigate NoneConscious Body Motion None1. Sitting at a table or standing n/a Constant2. Walking on a treadmill with constant speed 4. Walking at constant speed on a changing track Varying3. Walking on a treadmill with varying speed 5. Walking at varying speed on a changing track The experiment: measuring the relative strengths and weaknesses with reference to field testing

8. 8 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems The Laboratory Experiments 5 conditions (6 test subjects per condition) Number of usability problems Performance (task completion time) Subjective workload (NASA TLX)

9. 9 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems The Field Experiment 1 condition: walking in a pedestrian street (6 test subjects) Number of usability problems Performance (task completion time) Subjective workload (NASA TLX)

10. 10 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Findings (1) More problems found on average when seated at a table Statistical significance Lab techniques with physical movement comparable to field evaluation Mean number of usability problems identified by each technique 1.Sitting at a table 2.Walking on a treadmill with constant speed 3.Walking on a treadmill with varying speed 4.Walking at constant speed on a changing track 5.Walking at varying speed on a changing track 6.walking in a pedestrian street

11. 11 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Findings (2) TechniquesTotal Lab 1Lab 2Lab 3Lab 4Lab 5Field Critical4434334 Serious11 999817 Cosmetic1988861232 Total34232021182353 No technique identified all problems Most problems found when seated at table (34) Comparable numbers of critical problems found (3-4) Number of identified usability problems categorized by severity

12. 12 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Findings (2) TechniquesTotal Lab 1Lab 2Lab 3Lab 4Lab 5Field Critical4434334 Serious11 999817 Cosmetic1988861232 Total34232021182353 More than double the number of cosmetic problems were found while seated compared to the other lab techniques Number of identified usability problems categorized by severity

13. 13 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Findings (2) TechniquesTotal Lab 1Lab 2Lab 3Lab 4Lab 5Field Critical4434334 Serious11 999817 Cosmetic1988861232 Total34232021182353 Only 3 out of 4 critical usability problems were identified on basis of the field technique Number of identified usability problems categorized by severity

14. 14 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Findings (3) Techniques Lab 1Lab 2Lab 3Lab 4Lab 5Field Mental demands2975204126185148 Physical demands92117112118127194 Effort52163106228178186 Overall workload273548554854 Sitting at a table (lab 1) required significantly less mental activity compared to all other techniques but lab 2 Overall, sitting or walking at constant speed is experienced significantly less demanding than any other technique Subjective experience of workload with the different techniques

15. 15 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Discussion (1) Seating test subjects at a table… superior? Supported the identification of most usability problems Difference mostly accounted for by cosmetic problems Less workload facilitated more thinking-aloud Relevance of cosmetic problems can be questioned Increasing workload in lab Helped approximating the field condition but resulted in fewer problems identified compared to when being seated Better focus missing vital problems??

16. 16 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Discussion (2) Usability problems and mobility Physical motion in the lab triggered unique interaction problems also found in the field Added value in relation to e.g. layout and button sizes Data collection in the field All field tests were recorded with a camcorder Difficult to capture good images of screen The “bodyguard” effect Changing the role of the test monitors? Mounting small cameras on test subject and device?

17. 17 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems Future work Cost/benefit of different techniques and settings Time and effort spent per problem found When should lab or field tests ideally be applied? How can field test techniques be improved? Improving data collection? Enforcing more experimental control? Doing something completely different?