Putting Things Together 12/3

What are we doing? In science, we run lots of experiments In some cases, there's an "effect”; in others there's not Some drugs work; some don't Some measures help predict outcome; some don't Goal: Tell these situations apart, using the sample data If there is an effect, reject the null hypothesis If there’s no effect, retain the null hypothesis Challenge Sample is imperfect reflection of population, so we will make mistakes How to minimize those mistakes?

Analogy: Prosecution Think of cases where there is an effect as "guilty” No effect: "innocent" experiments Scientists are prosecutors We want to prove guilt Hope we can reject null hypothesis Can't be overzealous Minimize convicting innocent people Can only reject null if we have enough evidence How much evidence to require to convict someone? Tradeoff Low standard: Too many innocent people get convicted (Type I Error) High standard: Too many guilty people get off (Type II Error)

Example: Drug Testing 1995 2005 Clean Cheaters Cheaters Testosterone Strategy: Select cutoff by choosing a Accept a certain fraction of wrong convictions, and hope we catch most cheaters For each individual above cutoff Don’t know they cheated; don’t even know probability that they cheated Only know, if innocent, would have had only 5% chance of scoring so high Also don’t know what fraction of cheaters get caught All we know is by following this rule, 5% of innocent people will be convicted How to decide whether someone cheated? Measure testosterone level Where to put the cutoff? Complication: Don't know distribution of cheaters Can only control percentage of innocent people unfairly convicted a, Type I Error Rate 1995 2005 Clean Cheaters Cheaters Get away (Type II Error) Wrongly convicted (Type I Error) a (5%) Testosterone Probably cheated clean

From 1 Score to a Sample Scientific studies use a set of data, not just one score Same question: Is nature “innocent” (null hypothesis – nothing happening) or “guilty” (alternative hypothesis – real effect) Boil sample down to a single measure of effect size If effect size is large enough, reject null hypothesis Same challenge: How large is large enough?

Binomial Test Example: Halloween candy Sack holds boxes of raisins and boxes of jellybeans Each kid blindly grabs 10 boxes Some kids cheat by peeking Want to make cheaters forfeit candy How many jellybeans before we call kid a cheater?

Binomial Test ? How many jellybeans before we call kid a cheater? Work out probabilities for honest kids Binomial distribution Don’t know distribution of cheaters Make a rule so only 5% of honest kids lose their candy For each individual above cutoff Don’t know for sure they cheated Only know that if they were honest, chance would have been less than 5% that they’d have so many jellybeans Therefore, our rule catches as many cheaters as possible while limiting Type I Errors to 5% Honest Cheaters ? Jellybeans 0 1 2 3 4 5 6 7 8 9 10

Testing the Mean ? Example: IQs at various universities Some schools have average IQs (mean = 100) Some schools are above average Want to decide based on n students at each school Need a cutoff for the sample mean Find distribution of sample means for Average schools Set cutoff so that only 5% of Average schools will be mistaken as Smart 100 Average Smart ? p(M)

Unknown Variance: t-test Want to know whether population mean equals m0 H0: m = m0 H1: m ≠ m0 Don’t know variance Don’t know distribution of M under H0 Don’t know Type I Error rate for any cutoff Divide by sample variance to get t We know distribution of t exactly p(t) Distribution of t Values for Experiments when H0 is True m0 p(M) Critical value Critical Region

Steps of Hypothesis Testing State clearly the two hypotheses Determine which is H0 and which is H1 Compute a test statistic from the sample Measures deviation from prediction of H0 Known distribution when H0 is true Find cutoff (critical value) for test statistic Determined by choice of a (usually .05) Among experiments where there is no real effect, 5% lead to Type I Errors If test statistic is beyond critical value, reject H0 H0 H1 ? Critical value a (5%) Critical Region: Reject H0 Retain H0

p-value p Alternative approach Instead of comparing test statistic to critical value Find probability beyond test statistic and compare to a If p < a, then test statistic must be beyond critical value Critical value Test statistic 1 a p Critical value a p Test statistic Test statistic

Two-tailed Tests Often we want to catch effects on either side Split Type I Errors into two critical regions Each must have probability a/2 H0 H1 ? Test statistic Critical value a/2 1 a p

Confidence Intervals p Values of m0 that don’t lead to rejecting H0 Three levels Effect size / descriptive statistic Test statistic / inferential statistic p-value and a Can answer hypothesis test at any level Effect size predicted by H0 vs. confidence interval Test statistic vs. critical value p vs.  m0 m0 m0 m0 Test statistic Critical value 1 a p

3 Levels for All Tests Test Effect Size Test Statistic Type I Error rate Binomial Count a, p Goodness of Fit Frequencies c2 t-tests M, M – m0 MA – MB t ANOVAs SSfactor SSinteraction F Regression: One predictor Regression coefficient Regression: All predictors SSregression R2