1. Department of Industrial Management Engineering 1.Introduction ○Usability evaluation primarily summative ○Informal intuitive evaluations by designers even users unable to provide sufficiently accurate feedback ○Why formative evaluation seldom practiced? Egocentric intuition fallacy ○Illusion about knowing the determinants of one’s own behavior ○Underestimate variability in performance and preference Behavioral Research

2. Department of Industrial Management Engineering 2.Goals Behavioral Research New Technology Existing Technology New Domain Guidelines & Standards Expand the applicability Existing Domain Evaluation and Comparison More general understanding of the determinants

3. Department of Industrial Management Engineering 4.Research Design 1.Bias ○Because of the fundamental nature of human perception and thinking, human judgment not by the evidence at hand but by personal interpretation of events ○Experimenter bias – subtle and insidious; it often works by simple, unintentional, unconscious shifts in attention or interpretations ○To avoid bias  double-blind procedures; always have a comparison system Behavioral Research

4. Department of Industrial Management Engineering 4.Research Design 2.Statistical Significance ○Not so important ○More interested in knowing how much difference there probably is to guide cost-benefit strategy ○Science decisions can be postponed without enough evidence while engineering decisions can be made on whatever evidence is at hand Behavioral Research

5. Department of Industrial Management Engineering 4.Research Design 3.Generality and Transfer ○Participants who are similar in background, occupation, activity, age, and variability ○Systems that are easy for first half hour with novices, but not very effective thereafter  usability testing was done only in “first-use” settings ○Robustness in variation  generality Behavioral Research

6. Department of Industrial Management Engineering 4.Research Design 6.Invention and Specification Oriented Method ○task analysis – what task to design and how it should be performed ○Thoughtful observation and analysis of the whole task in its naturalistic context – ethnographic method Human-problem Oriented Invention 1.Identify a goal 2.Find what causes the difficulty 3.If possible, invent a way to help people around the difficulty 4.Incorporate it in a system Behavioral Research

7. Department of Industrial Management Engineering 4.Research Design 6.Invention and Specification Oriented Method Failure Analysis ○ Find out where people go wrong or slowly Individual Difference Analysis Time Profiling ○ Measuring the amount of time spent on all isolatable components ○ Components occupying large amount of time are obvious candidates for new method ○ Variability of time Behavioral Research

8. Department of Industrial Management Engineering 4.Research Design 7.Design-Oriented Research Method ○Full-scale Evaluation Studies ○Simply compare the overall effectiveness of two or more systems – fairly standard and obvious ○The biggest problems in the cost and in the selection of the sample of participants ○Already experienced users of each of the different kinds of systems ○What to measure – a set of benchmark tests ○Study strengths and deficiencies of systems already long in use Behavioral Research

9. Department of Industrial Management Engineering 4.Research Design 7.Design-Oriented Research Method ○Formative Evaluation ○The best strategy for good design is to try various options, test them, redesign, and iterate ○Effective and economical ○2-3 iterations, each requiring less than a dozen hours of human testing and an equivalent amount of reprogramming Behavioral Research

10. Department of Industrial Management Engineering 5.Measurement and Analysis 1.What to Measure and How Many ○Power ○The ability of the experiment to detect differences of a certain size, or to make measurements at a certain level of precision ○Determined by the accuracy of the measurements, the amount of variability in the performance, and the size of sample ○What Measurements? ○Completion time, number of errors, subjective satisfaction ○Individual differences (cognitive ability measures such as spatial memory, logical reasoning) ○Measures during the marketing or use of a system Behavioral Research

11. Department of Industrial Management Engineering 5.Measurement and Analysis 2.Data Quality ○To prevent inadvertent bias, data recording and analysis should be fully automated ○Take excessive caution 3.Reliability ○Random as well as bias errors ○Standard approach in Psychology – to have observations made by two different people working independently for a sizable random or representative subsample of the occasions ○Increase reliability – revise the way the judgments are made, more explicit rules, train more thoroughly, more practice and feedback Behavioral Research

12. Department of Industrial Management Engineering 5.Measurement and Analysis 3.Reliability ○Statistical Analysis ○Summarizing Results Accurately ○Descriptive statistics – central tendency measures ○Robust statistical methods ○Trimming – don’t use a certain proportion ○Measure of variability ○Probability distribution ○Confidence interval Behavioral Research

13. Department of Industrial Management Engineering 5.Measurement and Analysis 3.Reliability ○Estimating Effect Size: The Most Important Step ○Magnitude of the effect of a difference ○Critical ratio, odds ratio ○Instead of significance test, effect sizes, CI, and odds ratios are the proper tools Behavioral Research

14. Department of Industrial Management Engineering 5.Measurement and Analysis 3.Reliability ○Exploration ○Exploratory data analysis -- plotting of data – scaled differently ○Regression, Structural Analysis (factor analysis, multi- dimensional scaling, cluster analysis) ○Modeling Behavioral Research