1. Data analysis and interpretation

2. Quantitative and qualitative
Quantitative data – expressed as numbers
Qualitative data – difficult to measure sensibly as numbers, e.g. count number of words to measure dissatisfaction
Quantitative analysis – numerical methods to ascertain size, magnitude, amount
Qualitative analysis – expresses the nature of elements and is represented as themes, patterns, stories
Be careful how you manipulate data and numbers!

3. Descriptive Statistics
For all variables, get a feel for results:
Total scores, times, ratings, etc.
Minimum, maximum
Mean, median, ranges, etc.
e.g. “Twenty participants completed both sessions (10 males, 10 females; mean age 22.4, range years).”
e.g. “The median time to complete the task in the mouse-input group was 34.5 s (min=19.2, max=305 s).”

4. Simple quantitative analysis
Averages
Mean: add up values and divide by number of data points
Median: middle value of data when ranked
Mode: figure that appears most often in the data
Percentages versus numbers
Graphical representations give overview of data

5. Subgroup Stats
Look at descriptive stats (means, medians, ranges, etc.) for any subgroups
e.g. “The mean error rate for the mouse-input group was 3.4%. The mean error rate for the keyboard group was 5.6%.”
e.g. “The median completion time (in seconds) for the three groups were: novices: 4.4, moderate users: 4.6, and experts: 2.6.”

6. Plot the Data
Look for the trends graphically

7. Other Presentation Methods
Scatter plot
Box plot
Middle 50%
low
high
Age
Mean 20
Time in secs.

8. Visualizing log data Interaction profiles of players in online game
Log of web page activity

9. Simple qualitative analysis
Recurring patterns or themes
Emergent from data
Categorizing data
Categorization scheme may be emergent or pre-specified
Looking for critical incidents
Helps to focus in on key events

10. Presenting the findings
Only make claims that your data can support
The best way to present your findings depends on the audience, the purpose, and the data gathering and analysis undertaken
Graphical representations may be appropriate for presentation
Other techniques are:
Using stories, e.g. to create scenarios based on the data
Summarizing the findings

11. Interviews Raw data: Initial processing Qualitative processing
Audio or video recordings, interviewer notes
Initial processing
Transcribe audio, or expand upon notes
Qualitative processing
Group answers to same question (small # of questions and people)
Label interesting phrases or words
Put labels on post-its or in software and group labels
Quantitative processing
Gather quantitative responses such as age, etc.
Categorize and count responses (5 liked, 3 disliked, etc.)
Presentation
Summarize responses, tell stories and patterns
Use descriptive quotes

12. Questionnaire Raw data: Quantitative processing Qualitative processing
Tables of questions and numbers or text answers
Quantitative processing
Calculate descriptive stats (means, percentages, etc.) for each question
Can break into subgroups or use statistics to look for relationships between items (does age correlate to stronger preferences?)
Qualitative processing
Group answers to same question
Presentation
Present tables & charts of means, percentages, etc.
Explain overall meaning of all the responses

13. Observation Raw data: Initial processing: Quantitative processing
Audio or video recording, log files, notes
Initial processing:
Transcribe audio, expand notes or take more based on video, synchronize logs with recordings
Quantitative processing
Record metrics such as errors, times, clicks, etc.
Produce descriptive stats and charts of those metrics
Qualitative processing
Note places where problems occurred, interesting behaviors, common behaviors
Presentation
Descriptions of common or interesting problems
Videos demonstrating issues, or descriptive quotes
Charts describing quantitative data

14. Sample Think-aloud categorization
Interface problems
Verbalizations show evidence of dissatisfaction about an aspect of the interface.
Verbalizations show evidence of confusion/uncertainty about an aspect of the interface.
Verbalizations show evidence of confusion/surprise at the outcome of an action.
Verbalizations show evidence that they are having problems achieving a goal.
Verbalizations show evidence that the user has made an error.
The participant I unable to recover from error without external help from the experimenter.
The participant makes a suggestion for redesign of the interface.
See pg 380 for more complete example

15. Experimental Results
How does one know if an experiment’s results mean anything or confirm any beliefs?
Example: 40 people participated, 28 preferred interface 1, 12 preferred interface 2
What do you conclude?

16. Goal of analysis Get >95% confidence in significance of result
that is, null hypothesis disproved
Ho: Timecolor = Timeb/w
OR, there is an influence
ORR, only 1 in 20 chance that difference occurred due to random chance

17. Means Not Always Perfect
Experiment 1
Group Group 2
Mean: Mean: 10
1,10, ,6,21
Experiment 2
Group Group 2
Mean: Mean: 10
6,7, ,11,11

18. Inferential Stats and the Data
Are these really different? What would that mean?

19. Hypothesis Testing Tests to determine differences
t-test to compare two means
ANOVA (Analysis of Variance) to compare several means
Need to determine “statistical significance”
“Significance level” (p):
The probability that your null hypothesis was wrong, simply by chance
p (“alpha” level) is often set at 0.05, or 5% of the time you’ll get the result you saw, just by chance

20. Errors Errors in analysis do occur Main Types:
Type I/False positive - You conclude there is a difference, when in fact there isn’t
Type II/False negative - You conclude there is no difference when there is
And then there’s the True Negative…

21. Drawing Conclusions
Make your conclusions based on the descriptive stats, but back them up with inferential stats
e.g., “The expert group performed faster than the novice group t(1,34) = 4.6, p > .01.”
Translate the stats into words that regular people can understand
e.g., “Thus, those who have computer experience will be able to perform better, right from the beginning…”

22. Tools to support data analysis
Spreadsheet – simple to use, basic graphs
Can even do basic statistical analysis
Statistical packages, e.g. SPSS
Qualitative data analysis tools
Categorization and theme-based analysis, e.g. NVivo, Atlas.ti
Quantitative analysis of text-based data

23. Analysis and Presentation for Part 3
List of problems from HE with severity ratings
List of problems found in CW
Basic quantitative analysis from your observation
Basic qualitative analysis from your observation
Places where problems occur, general story of what and how people did, etc.
Basic quantitative and qualitative analysis from the questionnaire or interview
Tables of responses, averages, etc. as appropriate

24. Interpreting your results
Go through each usability criteria – do results demonstrate support for meeting this criteria or not? How do they?
Discuss any other problems with aspects of the design that your results demonstrate.
Discuss how you would modify the design based on these results.

25. Reminder: Wednesday’s plan
3:30-4:05 Cognitive Walkthrough
1 member of design team (D) will facilitate 4 members of the evaluating team
D walks through each task, evaluating team asks and answers the 4 questions for every action
D makes sure all feedback is written down and takes back to rest of design team

26. Cognitive Walkthrough pairs
Rabid Bunnies – User One
The Team - Satisfaction
Inventive Innovators – No Spoon
Awesome - NoName

27. Monday’s plan 4:10-4:45 Heuristic evaluation
1 member of the Design team (D2) demos the prototype to the evaluating team
D2 provides the set of heuristics
Evaluating team individually writes down all problems they see and gives it to D2

28. Heuristic evaluation teams
Rabid Bunnies – The Team
User Won - Satisfaction
Inventive Innovators - Awesome
NoSpoon – NoName

29. Come Prepared! Bring your prototype, have it ready to go at 3:30
Choose facilitator (D1 & D2) for both
Bring task & action lists for cognitive walkthrough
Bring heuristics for heuristic evaluation
As an evaluator – be detailed, thorough, and constructive