1. Human-Computer Interaction: Overview of User Studies
CSCE 315 – Programming Studio
Spring 2017
Project 3, Lecture 2

2. User Studies At the heart of HCI: Getting feedback from users
User experience is just as (sometimes more) important than issues of efficiency, robustness, etc.
Wide variety of ways to get feedback from users
Goal from this lecture: give overview of methods, including some specific data gathering methods

3. Principles (Study Design)
The user evaluation is core part of iterative design process
Each iteration should bring in user feedback
Multiple ways to collect data
Some material from Kerne’s slides

4. The User Study Questions to Address
How to get participants?
Where to gather data?
What data to gather?
How to gather that data?
How to analyze that data?

5. Some material from Kerne’s slides
The Pilot Study
Can think of the study process itself as an iterative one: pilot is the first iteration
Run very small number of subjects
Sometimes just 1 or 2
Tune tasks, times, instructions, what data is gathered
Perform preliminary analysis
Consider:
Re-framing Questions
Re-defining Tasks
The goal of the pilot study is much less on analyzing the artifact, but rather in analyzing the study itself
Some material from Kerne’s slides

6. How to Get Participants? Ethical Issues
Privacy
Confidentiality
Anonymity
Informed Consent
Tell participants their rights
Including how to opt-out or stop
Give clear description of what they are agreeing to
Time involved, possible dangers, goals
Get permission to use results
IRB: Institutional Review Board
At organizations like universities, IRB reviews human-oriented studies to ensure appropriate measures are followed
Some material from Kerne’s slides

7. How to Get Participants? Questions to Answer
How many participants needed?
Depends on study:
usabililty: at least 5
Eye tracking: 39 for stable heat maps
What type of participant?
Expert, novice, or in between?
What other characteristics might need to be addressed?
Do you need a diverse or homogeneous group?
Gender, ethnicity, experience, physical characteristics (e.g. colorblindness, handedness), nationality, etc.
Affects how you go about finding people and who you allow in to the study
What is motivation for participating?
Guaranteed compensation?
Chance at compensation?
No tangible compensation?
Coerced participation?
Required by employer, required for class, etc.
Ensure that any compensation is not tied to their feedback

8. Where to Gather Data? Options
Laboratory studies
Artificial environment set up to study the artifact
Field studies
In the natural environment
Living labs
Concurrent research and new implementations
Crowdsourced
Micro-task workers (e.g. Mechanical Turk)
Twitter, etc.
Some material from Kerne’s slides

9. What Data to Gather? Experiment Design
Hypotheses
Independent Variables (to vary)
Comparative experimental conditions
e.g. with/without some feature/apparatus
Task/activities
Dependent Variables (to measure)
What is being measured
How?
Subjects:
Within-subjects (same participant)
Between-subjects (different participants
Counterbalancing
Eliminating Ordering Effects
Some material from Kerne’s slides

10. What Data to Gather? Data Paradigms
Qualitative Experience Data
Ethnography
Role of the ethnographer: takes the view of the subject of the study
Methods:
Writing culture down
Thick description of all observed behavior
Layers of signification and meaning
Observation
Stories
Context
Interpretation
Grounded Theory (for analysis)
Take data and code it based on observation
Use coding to inductively generate hypotheses
Some material from Kerne’s slides

11. What Data to Gather? Data Paradigms
Quantitatve Data
Time & Errors
Number and type of errors
Time to complete task
Ideation metrics
Fluency: number of answers
Flexibility: number of categories of answers
Novelty of answers
Experience Measures
Likert scales, semantic differential scales
Not as good of a source of data
Some material from Kerne’s slides

12. What Data to Gather? Data Paradigms
Predictive Evaluation (experts)
Cognitive Walkthrough
Task analysis
As if performing task, go through step-by-step
Heuristic Evaluation
Have expert evaluate the usability
Apply known principles
Some material from Kerne’s slides

13. How to Gather Data? Data Gathering Processes
Interview
Pre-questionnaire
Interactive System Use
Observation!
Recording: photos, audio, video
Logging
Answers found/products produced
Think-aloud
Post-questionnaire
Some material from Kerne’s slides

14. How to Analyze Data? Depends on type of data and how it was collected
Qualitative Data
Quantitative Data
Expert Data

15. Upcoming Talks Will give details of several types of user studies
How to conduct and analyze results from several different user studies
Will look at some details of how to design good user interfaces

16. References Some topics taken from:
Interaction Design, 4th edition, by Rogers, Sharp, and Preece, Wiley, 2014.
See chapters 7-8 in particular
Some slides taken from Andruid Kerne’s slides:
Noted in footers
Material is copyrighted
Can be used and cited freely for non-commercial education and scholarship without profit provided that credit is given to the author and this notice is included