1. A Usability Study of a Language Centre Web Site
By Andy Morrall
CITERS2003 Presentation 5 July 2003

2. Why, in general? Teachers write educational web pages
Students use them outside the classroom
Teachers don’t always see them being used
Hard to see if the pages work properly
Students rarely give detailed feedback

3. Why, specifically? I am in charge of an educational website
Very little feedback from users
Need to write reports about quality control

4. How? Need a tried-and-tested, reliable and valid way to test the site
Can’t test if it is good for learning, as there are too many other ways users could be learning English
Users opinions of the user-friendliness of the site may be too nice and non-specific
So, a need to watch students using the site and analyse the results systematically

5. What is Usability?
Rubin (1994, p.22) says that usability testing “employs techniques to collect empirical data while observing representative end users using the product to perform representative tasks”. He describes an approach to usability testing, which “employs an iterative cycle of tests intended to expose usability deficiencies and gradually shape or mould the product in question.”

6. What?

7. Research Questions
Do the improvements to the site made during the usability study result in improvements in test participants’ success rate in completing tasks on the CILL site?
Do the improvements to the site made during the usability study result in participants being able to carry out these tasks in a shorter time?
Do the participants say that they believe that the site is easier to use?

8. Who and what?
To learn if the site is usable, I needed to know what it was used for.
I asked 5,100 CILL members to take part in the survey by
215 replied filled in the questionnaire at
The survey asked users to:
Identify the right site
Say how long they had been using the Internet
When they last used the CILL site
What did they use the site for
If they had any suggestions for improving the site

9. Survey Results 78 users correctly identified the right site
Analysis of Responses to Question 7 of the Initial Questionnaire
Grammar 22
Pronunciation 12
Listening 9
Vocabulary
Exercises 7
Dictionary 6
Reference Machine 4
Job Application Letter
Reading 3
Writing
Oral 2
Referencing
Idioms
Newspapers
Academic Writing
English in the Workplace
Reports
Interviews
Finding Materials
Testing Level
CILL Information
Research 1
Purdue On-line Writing Lab.
CILL teachers’
Example Assignment
Staff information
Opening Hours
Games
Resumes/CV
Discussion
Exit Test
Memos
Speaking

10. Tasks
Find a grammar exercise on the difference between ‘staff’ and ‘staffs’.
Find out how to pronounce ‘thorough’.
Find a page where you can learn vocabulary about the news.
Find a page where you can learn English by listening to pop music.
Find a page with some common job interview questions and advice and an exercise about how to answer them.
Look up the meaning of the word ‘discharge’.
Make a reference (Details provided)
Find a page which helps you to write a job application letter.
Find a page where you can read about pen-pals you can write to.
Find a page that helps you improve your newspaper reading skills.

11. Participants 2 groups: Both groups included:
Re-test group – did the test tasks both before and after the changes
New group – only did the tests after the changes
Both groups included:
CILL students and teachers
native speakers of English and non-natives
participants with varying degrees of computer experience and skills

12. How to do a usability test
Welcome the participant and check their identity and profile
Explain the purpose and duration of the test (1 hour in total, including post-test questionnaire).
Elicit any questions.
Ask for consent for the participant to take part.
Ask the participant to sign the consent form.
Participant chooses the browser and version they usually use to access the CILL web site.
The participant attempts to carry out the tasks, with the tester watching and taking notes.
The participant either completes the final task or runs out of time.
Give the participant a short break.
Ask the participant to complete a post-test questionnaire.
Chat about the test and the site
Thank the participant for taking part.
Participant departs.

13. Task 1
Find a grammar exercise on the difference between ‘staff’ and ‘staffs’. Easy task first to give users confidence.

14. Task 1

15. Task 1 Changed

16. Results of Task 1

17. Summary of Improvements
Task Number
Task Success
Mean Task Duration
Improvement or remained at maximum
Improvement in percent
Improvement or remained the same
Improvement in seconds
Re-test group
New group 1. 
20% 52 15 2. 0%
10% 26 10 3. 71 76 4. 37 12 5. 
40%
-5%
118
111 6. 21 7. 5%
139
-12 8. 3 9.
-20% 11 24
10. 42 40
Total / Mean
Total = 9
Total = 9
Mean = 4%
Mean = 50 30

18. Results – Research Question 1
Do the improvements to the site made during the usability study result in improvements in test participants’ success rate in completing tasks on the CILL site?
Re-test Group: +4% New Group: +5%

19. Results – Research Question 2
2. Do the improvements to the site made during the usability study result in participants being able to carry out these tasks in a shorter time?
Re-test Group: 50 seconds faster New Group: 30 seconds faster

20. Results – Research Question 3
3. Do the participants say that they believe that the site is easier to use? Re-test group only: Strongly agree: 1 Agree: 1 Neither agree nor disagree: 3

21. Limitations of the Study
The Small Number of Tasks in Relation to the Size of the Site (10 tasks, 234 pages)
The Small Number of Design Iterations (I did 3, iterative design principles recommend repeated iterations)
Debriefing Participants – it’s embarrassing to talk about why they didn’t do or see things - the worse the performance of a participant is, the more valuable their debriefing feedback and further test participation is, but the more difficult they are to get.

22. Recommendations for Further Study
Further study could be done into:
using more authentic tasks such as a text correction task, with the answers on the site
investigating the effects of testers and participants reading the task instructions aloud, because some participants didn’t read all the instructions
the effects of participants’ levels of computing skills on their level of task success and task duration
the optimum number of tests to run between site design iterations

23. Usability Studies Discount Usability Testing
Nielsen, J. (2000). Designing web usability: The practice of usability. Indianapolis: New Riders.
Nielsen, J. (1994). Guerilla HCI: Using discount usability engineering to penetrate the intimidation barrier. In Bias, R.G. & Mayhew, D. (Eds.) Cost-justifying usability. Boston: Academic Press.
Nielsen, J. (1993). Usability engineering. Boston: AP Professional.
Rubin, J. (1994). Handbook of Usability Testing. New York, NY. John Wiley & Sons.