Conducting a User Study Human-Computer Interaction

Overview Why run a study? Why run a study? Evaluate if a statement is true Evaluate if a statement is true Ex. The heavier a person weighs, the higher their blood pressure Ex. The heavier a person weighs, the higher their blood pressure Many ways to do this: Many ways to do this: Look at data from a doctor’s office Look at data from a doctor’s office What’s the pros and cons? What’s the pros and cons? Get a group of people to get weighed and measure their BP Get a group of people to get weighed and measure their BP What’s the pros and cons? What’s the pros and cons? Ideal solution: have everyone in the world get weighed and BP Ideal solution: have everyone in the world get weighed and BP Participants are a sample of the population Participants are a sample of the population You should immediately question this! You should immediately question this! Restrict population Restrict population

Population Design Identify the statement to be evaluated Identify the statement to be evaluated Ex. The heavier a person weighs, the higher their blood pressure Ex. The heavier a person weighs, the higher their blood pressure Create a hypothesis Create a hypothesis Ex. Weight is directly proportional to blood pressure Ex. Weight is directly proportional to blood pressure Identify Independent and Dependent Variables Identify Independent and Dependent Variables Independent Variable – the variable that is being manipulated by the experimenter (weight) Independent Variable – the variable that is being manipulated by the experimenter (weight) Dependent Variable – the variable that is caused by the independent variable. (blood pressure) Dependent Variable – the variable that is caused by the independent variable. (blood pressure) Design Study Design Study Invite 100 people Invite 100 people Weigh them and take their BP Weigh them and take their BP Graph Graph See if there is a trend See if there is a trend

Two Group Design Identify the statement to be evaluated Identify the statement to be evaluated Ex. Shorter people are smarter than taller people Ex. Shorter people are smarter than taller people Create a hypothesis Create a hypothesis Ex. IQ of people shorter than 5’9” > IQ of people 5’9” or taller Ex. IQ of people shorter than 5’9” > IQ of people 5’9” or taller Design Study Design Study Two groups called conditions Two groups called conditions How many people? How many people? What’s your design? What’s your design? What is the independent and dependent variables? What is the independent and dependent variables? Confounding factors – factors that affect outcomes, but are not related to the study Confounding factors – factors that affect outcomes, but are not related to the study

Design External validity – do your results mean anything? External validity – do your results mean anything? Results should be similar to other similar studies Results should be similar to other similar studies Use accepted questionnaires, methods Use accepted questionnaires, methods Power – how much meaning do your results have? Power – how much meaning do your results have? The more people the more you can say that the participants are a sample of the population The more people the more you can say that the participants are a sample of the population Generalization – how much do your results apply to the true state of things Generalization – how much do your results apply to the true state of things

Design People who use a mouse and keyboard will be faster to fill out a form than keyboard alone. People who use a mouse and keyboard will be faster to fill out a form than keyboard alone. Let’s create a study design Let’s create a study design Two types: Two types: Between Subjects Between Subjects Across Subjects Across Subjects Everyone do this now for your study Everyone do this now for your study

Procedure Formally have all participants sign up for a time slot (if individual testing is needed) Formally have all participants sign up for a time slot (if individual testing is needed) Informed Consent (let’s look at one) Informed Consent (let’s look at one) Execute study Execute study Questionnaires/Debriefing (let’s look at one) Questionnaires/Debriefing (let’s look at one)

Hypothesis Proving Hypothesis: Hypothesis: People who use a mouse and keyboard will be faster to fill out a form than keyboard alone. People who use a mouse and keyboard will be faster to fill out a form than keyboard alone. US Court system: Innocent until proven guilty US Court system: Innocent until proven guilty NULL Hypothesis: Assume people who use a mouse and keyboard will fill out a form than keyboard alone in the same amount of time NULL Hypothesis: Assume people who use a mouse and keyboard will fill out a form than keyboard alone in the same amount of time Your job to prove differently! Your job to prove differently! Alternate Hypothesis 1: People who use a mouse and keyboard will fill out a form than keyboard alone, either faster or slower. Alternate Hypothesis 1: People who use a mouse and keyboard will fill out a form than keyboard alone, either faster or slower. Alternate Hypothesis 2: People who use a mouse and keyboard will fill out a form than keyboard alone, faster. Alternate Hypothesis 2: People who use a mouse and keyboard will fill out a form than keyboard alone, faster.

Analysis Most of what we do involves: Most of what we do involves: Normal Distributed Results Normal Distributed Results Independent Testing Independent Testing Homogenous Population Homogenous Population

Raw Data What does the mean (average) tell us? Is that enough? What does the mean (average) tell us? Is that enough?

Variances standard deviation – measure of dispersion (square root of the sum of squares divided by N) standard deviation – measure of dispersion (square root of the sum of squares divided by N) Small Pattern (seconds)Large Pattern (seconds) MeanS.D.MinMaxMeanS.D.MinMax Real Space (n=41) Purely Virtual (n=13) Hybrid (n=13) Vis Faith Hybrid (n=14)

Hypothesis We assumed the means are “equal” We assumed the means are “equal” But are they? Or is the difference due to chance? But are they? Or is the difference due to chance? Small Pattern (seconds)Large Pattern (seconds) MeanS.D.MinMaxMeanS.D.MinMax Real Space (n=41) Purely Virtual (n=13) Hybrid (n=13) Vis Faith Hybrid (n=14)

T - test T – test – statistical test used to determine whether two observed means are statistically different T – test – statistical test used to determine whether two observed means are statistically different

T – test (rule of thumb) Good values of t > 1.96 (rule of thumb) Good values of t > 1.96 Look at what contributes to t Look at what contributes to t htm htm

F statistic, p values F statistic – assesses the extent to which the means of the experimental conditions differ more than would be expected by chance F statistic – assesses the extent to which the means of the experimental conditions differ more than would be expected by chance t is related to F statistic t is related to F statistic Look up a table, get the p value. Compare to α Look up a table, get the p value. Compare to α α value – probability of making a Type I error (rejecting null hypothesis when really true) α value – probability of making a Type I error (rejecting null hypothesis when really true) p value – statistical likelihood of an observed pattern of data, calculated on the basis of the sampling distribution of the statistic. (% chance it was due to chance) p value – statistical likelihood of an observed pattern of data, calculated on the basis of the sampling distribution of the statistic. (% chance it was due to chance)

Small PatternLarge Pattern t – test with unequal variance p – value t – test with unequal variance p - value PVE – RSE vs. VFHE – RSE ** *** PVE – RSE vs. HE – RSE ** * VFHE – RSE vs. HE – RSE

Significance What does it mean to be significant? What does it mean to be significant? You have some confidence it was not due to chance. You have some confidence it was not due to chance. But difference between statistical significance and meaningful significance But difference between statistical significance and meaningful significance Always know: Always know: samples (n) samples (n) p value p value variance/standard deviation variance/standard deviation means means

IRB Let’s look at a completed one Let’s look at a completed one You MUST turn one in by October 28 th to the TA! You MUST turn one in by October 28 th to the TA! Must have OKed before running study Must have OKed before running study