A Usability Study of a Language Centre Web Site By Andy Morrall CITERS2003 Presentation 5 July 2003

Why, in general? Teachers write educational web pages Students use them outside the classroom Teachers don’t always see them being used Hard to see if the pages work properly Students rarely give detailed feedback

Why, specifically? I am in charge of an educational website Very little feedback from users Need to write reports about quality control

How? Need a tried-and-tested, reliable and valid way to test the site Can’t test if it is good for learning, as there are too many other ways users could be learning English Users opinions of the user-friendliness of the site may be too nice and non-specific So, a need to watch students using the site and analyse the results systematically

What is Usability? Rubin (1994, p.22) says that usability testing “employs techniques to collect empirical data while observing representative end users using the product to perform representative tasks”. He describes an approach to usability testing, which “employs an iterative cycle of tests intended to expose usability deficiencies and gradually shape or mould the product in question.”

What?

Research Questions Do the improvements to the site made during the usability study result in improvements in test participants’ success rate in completing tasks on the CILL site? Do the improvements to the site made during the usability study result in participants being able to carry out these tasks in a shorter time? Do the participants say that they believe that the site is easier to use?

Who and what? To learn if the site is usable, I needed to know what it was used for. I asked 5,100 CILL members to take part in the survey by e-mail 215 replied filled in the questionnaire at http://elc.polyu.edu.hk/CILL/research/cill_site_improvement_questionnaire.htm The survey asked users to: Identify the right site Say how long they had been using the Internet When they last used the CILL site What did they use the site for If they had any suggestions for improving the site

Survey Results 78 users correctly identified the right site Analysis of Responses to Question 7 of the Initial Questionnaire Grammar 22 Pronunciation 12 Listening 9 Vocabulary Exercises 7 Dictionary 6 Reference Machine 4 Job Application Letter Reading 3 Writing Oral 2 Referencing Idioms Newspapers Academic Writing English in the Workplace Reports Interviews Finding Materials Testing Level CILL Information Research 1 Purdue On-line Writing Lab. CILL teachers’ e-mail Example Assignment Staff information Opening Hours Games Resumes/CV Discussion Exit Test Memos Speaking

Tasks Find a grammar exercise on the difference between ‘staff’ and ‘staffs’. Find out how to pronounce ‘thorough’. Find a page where you can learn vocabulary about the news. Find a page where you can learn English by listening to pop music. Find a page with some common job interview questions and advice and an exercise about how to answer them. Look up the meaning of the word ‘discharge’. Make a reference (Details provided) Find a page which helps you to write a job application letter. Find a page where you can read about e-mail pen-pals you can write to. Find a page that helps you improve your newspaper reading skills.

Participants 2 groups: Both groups included: Re-test group – did the test tasks both before and after the changes New group – only did the tests after the changes Both groups included: CILL students and teachers native speakers of English and non-natives participants with varying degrees of computer experience and skills

How to do a usability test Welcome the participant and check their identity and profile Explain the purpose and duration of the test (1 hour in total, including post-test questionnaire). Elicit any questions. Ask for consent for the participant to take part. Ask the participant to sign the consent form. Participant chooses the browser and version they usually use to access the CILL web site. The participant attempts to carry out the tasks, with the tester watching and taking notes. The participant either completes the final task or runs out of time. Give the participant a short break. Ask the participant to complete a post-test questionnaire. Chat about the test and the site Thank the participant for taking part. Participant departs.

Task 1 Find a grammar exercise on the difference between ‘staff’ and ‘staffs’. Easy task first to give users confidence.

Task 1

Task 1 Changed

Results of Task 1

Summary of Improvements   Task Number Task Success Mean Task Duration Improvement or remained at maximum Improvement in percent Improvement or remained the same Improvement in seconds Re-test group New group 1.  20% 52 15 2. 0% 10% 26 10 3. 71 76 4. 37 12 5.  40% -5% 118 111 6. 21 7. 5% 139 -12 8. 3 9. -20% 11 24 10. 42 40 Total / Mean Total = 9 Total = 9 Mean = 4% Mean = 50 30

Results – Research Question 1 Do the improvements to the site made during the usability study result in improvements in test participants’ success rate in completing tasks on the CILL site? Re-test Group: +4% New Group: +5%

Results – Research Question 2 2. Do the improvements to the site made during the usability study result in participants being able to carry out these tasks in a shorter time? Re-test Group: 50 seconds faster New Group: 30 seconds faster

Results – Research Question 3 3. Do the participants say that they believe that the site is easier to use? Re-test group only: Strongly agree: 1 Agree: 1 Neither agree nor disagree: 3

Limitations of the Study The Small Number of Tasks in Relation to the Size of the Site (10 tasks, 234 pages) The Small Number of Design Iterations (I did 3, iterative design principles recommend repeated iterations) Debriefing Participants – it’s embarrassing to talk about why they didn’t do or see things - the worse the performance of a participant is, the more valuable their debriefing feedback and further test participation is, but the more difficult they are to get.

Recommendations for Further Study Further study could be done into: using more authentic tasks such as a text correction task, with the answers on the site investigating the effects of testers and participants reading the task instructions aloud, because some participants didn’t read all the instructions the effects of participants’ levels of computing skills on their level of task success and task duration the optimum number of tests to run between site design iterations

Usability Studies Discount Usability Testing Nielsen, J. (2000). Designing web usability: The practice of usability. Indianapolis: New Riders. Nielsen, J. (1994). Guerilla HCI: Using discount usability engineering to penetrate the intimidation barrier. In Bias, R.G. & Mayhew, D. (Eds.) Cost-justifying usability. Boston: Academic Press. Nielsen, J. (1993). Usability engineering. Boston: AP Professional. http://www.useit.com Rubin, J. (1994). Handbook of Usability Testing. New York, NY. John Wiley & Sons.