1. Putting Things Together
12/3

2. What are we doing? In science, we run lots of experiments
In some cases, there's an "effect”; in others there's not
Some drugs work; some don't
Some measures help predict outcome; some don't
Goal: Tell these situations apart, using the sample data
If there is an effect, reject the null hypothesis
If there’s no effect, retain the null hypothesis
Challenge
Sample is imperfect reflection of population, so we will make mistakes
How to minimize those mistakes?

3. Analogy: Prosecution
Think of cases where there is an effect as "guilty”
No effect: "innocent" experiments
Scientists are prosecutors
We want to prove guilt
Hope we can reject null hypothesis
Can't be overzealous
Minimize convicting innocent people
Can only reject null if we have enough evidence
How much evidence to require to convict someone?
Tradeoff
Low standard: Too many innocent people get convicted (Type I Error)
High standard: Too many guilty people get off (Type II Error)

4. Example: Drug Testing 1995 2005 Clean Cheaters Cheaters Testosterone
Strategy: Select cutoff by choosing a
Accept a certain fraction of wrong convictions, and hope we catch most cheaters
For each individual above cutoff
Don’t know they cheated; don’t even know probability that they cheated
Only know, if innocent, would have had only 5% chance of scoring so high
Also don’t know what fraction of cheaters get caught
All we know is by following this rule, 5% of innocent people will be convicted
How to decide whether someone cheated?
Measure testosterone level
Where to put the cutoff?
Complication: Don't know distribution of cheaters
Can only control percentage of innocent people unfairly convicted
a, Type I Error Rate
1995
2005
Clean
Cheaters
Cheaters
Get away (Type II Error)
Wrongly convicted (Type I Error)
a (5%)
Testosterone
Probably
cheated
clean

5. From 1 Score to a Sample
Scientific studies use a set of data, not just one score
Same question:
Is nature “innocent” (null hypothesis – nothing happening)
or “guilty” (alternative hypothesis – real effect)
Boil sample down to a single measure of effect size
If effect size is large enough, reject null hypothesis
Same challenge: How large is large enough?

6. Binomial Test Example: Halloween candy
Sack holds boxes of raisins and boxes of jellybeans
Each kid blindly grabs 10 boxes
Some kids cheat by peeking
Want to make cheaters forfeit candy
How many jellybeans before we call kid a cheater?

7. Binomial Test ? How many jellybeans before we call kid a cheater?
Work out probabilities for honest kids
Binomial distribution
Don’t know distribution of cheaters
Make a rule so only 5% of honest kids lose their candy
For each individual above cutoff
Don’t know for sure they cheated
Only know that if they were honest, chance would have been less than 5% that they’d have so many jellybeans
Therefore, our rule catches as many cheaters as possible while limiting Type I Errors to 5%
Honest
Cheaters ?
Jellybeans

8. Testing the Mean ? Example: IQs at various universities
Some schools have average IQs (mean = 100)
Some schools are above average
Want to decide based on n students at each school
Need a cutoff for the sample mean
Find distribution of sample means for Average schools
Set cutoff so that only 5% of Average schools will be mistaken as Smart
100
Average
Smart ?
p(M)

9. Unknown Variance: t-test
Want to know whether population mean equals m0
H0: m = m0 H1: m ≠ m0
Don’t know variance
Don’t know distribution of M under H0
Don’t know Type I Error rate for any cutoff
Divide by sample variance to get t
We know distribution of t exactly
p(t)
Distribution of t Values for Experiments when H0 is True m0
p(M)
Critical value
Critical Region

10. Steps of Hypothesis Testing
State clearly the two hypotheses
Determine which is H0 and which is H1
Compute a test statistic from the sample
Measures deviation from prediction of H0
Known distribution when H0 is true
Find cutoff (critical value) for test statistic
Determined by choice of a (usually .05)
Among experiments where there is no real effect, 5% lead to Type I Errors
If test statistic is beyond critical value, reject H0 H0 H1 ?
Critical value
a (5%)
Critical Region: Reject H0
Retain H0

11. p-value p Alternative approach
Instead of comparing test statistic to critical value
Find probability beyond test statistic and compare to a
If p < a, then test statistic must be beyond critical value
Critical value
Test statistic 1 a p
Critical value a p
Test statistic
Test statistic

12. Two-tailed Tests
Often we want to catch effects on either side
Split Type I Errors into two critical regions
Each must have probability a/2 H0 H1 ?
Test statistic
Critical value
a/2 1 a p

13. Confidence Intervals p Values of m0 that don’t lead to rejecting H0
Three levels
Effect size / descriptive statistic
Test statistic / inferential statistic
p-value and a
Can answer hypothesis test at any level
Effect size predicted by H0 vs. confidence interval
Test statistic vs. critical value
p vs.  m0 m0 m0 m0
Test statistic
Critical value 1 a p

14. 3 Levels for All Tests Test Effect Size Test Statistic
Type I Error rate
Binomial
Count
a, p
Goodness of Fit
Frequencies c2
t-tests
M, M – m0 MA – MB t
ANOVAs
SSfactor SSinteraction F
Regression: One predictor
Regression coefficient
Regression:
All predictors
SSregression R2