271. CS 321: Human-Computer Interaction Design
November 12, 2015
Questionnaires
Usability Questionnaires
Post-Study & Post-Task Questionnaires
Triangulation

272. Usability Questionnaires
Using standardized questionnaires for usability studies offers several advantages.
Objectivity
Usability practitioners are able to independently verify the measurement statements of others.
Replicability
Studies can easily be replicated, improving their reliability.
Quantification
Results can be reported in finer detail and more objectivity.
Communication
Standardized measures facilitate communication between practitioners.
Economy
Developing standardized measures takes work, but reusing them is inexpensive.
CS 321
November 12, 2015
Questionnaires
Page 272

273. Post-Study Usability Questionnaires
The PSSUQ is a 16-item survey that measures users’ perceived satisfaction with a product or system.
The Post-Study System Usability Questionnaire (Version 3)
Strongly Agree
Strongly Disagree 1 2 3 4 5 6 7 NA 1.
Overall, I am satisfied with how easy it is to use this system.  2.
It was simple to use this system. 3.
I was able to complete the tasks and scenarios quickly using this system. 4.
I felt comfortable using this system. 5.
It was easy to learn to use this system. 6.
I believe I could become productive quickly using this system. 7.
The system gave error messages that clearly told me how to fix problems. 8.
Whenever I made a mistake using the system, I could recover easily and quickly. 9.
The information (such as on-line help, on-screen messages, and other documentation) provided with this system was clear.
10.
It was easy to find the information I needed.
11.
The information was effective in helping me complete the tasks and scenarios.
12.
The organization of information on the system screens was clear.
13.
The interface of this system was pleasant.
14.
I liked using the interface of this system.
15.
This system has all the functions and capabilities I expect it to have.
16.
Overall, I am satisfied with this system.
An overall satisfaction score is obtained by averaging the sub-scales of System Quality (items 1-6), Information Quality (items 7-12), and Interface Quality (items 13-16).
The PSSUQ is susceptible to “acquiesce bias”, the fact that people are more likely to agree with a statement than to disagree with it.
CS 321
November 12, 2015
Questionnaires
Page 273

274. Interpreting Questionnaire Results
Psychometric analysis of usability questionnaires is conducted to determine their reliability, validity, and sensitivity.
For example, the PSSUQ-3 norms here show that most items have means that fall below the scale midpoint of 4, indicating that the scale midpoint should not be used exclusively as a reference from which to judge participants’ perceptions on usability.
PSSUQ-3 Norms (Means and 99% Confidence Intervals)
Lower Limit
Mean
Upper Limit 1.
Overall, I am satisfied with how easy it is to use this system.
2.60
2.85
3.09 2.
It was simple to use this system.
2.45
2.69
2.93 3.
I was able to complete the tasks and scenarios quickly using this system.
2.86
3.16
3.45 4.
I felt comfortable using this system.
2.40
2.66
2.91 5.
It was easy to learn to use this system.
2.07
2.27
2.48 6.
I believe I could become productive quickly using this system.
2.54
3.17 7.
The system gave error messages that clearly told me how to fix problems.
3.36
3.70
4.05 8.
Whenever I made a mistake using the system, I could recover easily and quickly.
3.21
3.49 9.
The information (such as on-line help, on-screen messages, and other documentation) provided with this system was clear.
2.65
2.96
3.27
10.
It was easy to find the information I needed.
2.79
3.38
11.
The information was effective in helping me complete the tasks and scenarios.
2.46
2.74
3.01
12.
The organization of information on the system screens was clear.
2.41
2.92
13.
The interface of this system was pleasant.
2.06
2.28
2.49
14.
I liked using the interface of this system.
2.18
2.42
15.
This system has all the functions and capabilities I expect it to have.
2.51
3.07
16.
Overall, I am satisfied with this system.
2.55
2.82
Also note the relatively poor ratings associated with Item 7, which reflect the difficulty of providing usable error messages in a software product, as well as the overall dissatisfaction that such errors cause in users.
CS 321
November 12, 2015
Questionnaires
Page 274

275. Post-Task Usability Questionnaires
While post-study surveys provide information regarding the general satisfaction of users with an interface, brief mini-surveys of user reaction to specific tasks in specific scenarios are often more useful when attempting to diagnose more focused problems.
The After-Scenario Questionnaire (Version 1)
Strongly
Agree
Strongly Disagree 1 2 3 4 5 6 7 NA 1.
Overall, I am satisfied with the ease of completing the tasks in this scenario.  2.
Overall, I am satisfied with the amount of time it took to complete the tasks in this scenario. 3.
Overall, I am satisfied with the support information (online help, messages, documentation) when completing the tasks.
Example scenarios and tasks for office software systems:
Mail Scenario #1
Open a note
Send reply
Delete note
Mail Scenario #2
Open a note
Forward w/reply
Save response
Delete original
Address Scenario
Create new listing
Modify old listing
Delete unmodified listing
File Scenario
Rename file
Copy file
Delete file
Editor Scenario
Locate document
Edit document
Open note
Copy note’s text into document
Save document
Print document
CS 321
November 12, 2015
Questionnaires
Page 275

276. Any given research method has advantages and limitations.
Triangulation
Any given research method has advantages and limitations.
Lab experiments are abstract and obtrusive, and may not be representative of the real world.
Field studies cannot be controlled, so it’s hard to make strong, precise claims regarding comparative usability.
Self-reporting (via questionnaires) is often biased by reactivity (e.g., the subjects try to be polite or to say what they think they should say, instead of the truth).
One way to deal with this problem is via triangulation, using multiple methods to tackle the same research question.
If they all support your claim, then you have much stronger evidence, without as many biases.
CS 321
November 12, 2015
Questionnaires
Page 276