1. The Earth is Round (p<.05)
Good Afternoon,
My name is Sonya and this is Monika, we will be presenting to you today the paper by Jakob Cohen: The Earth is round (p<0.05)
Jakob Cohen
Presented by: Sonya Pastran
Monika Rogowska
This Photo by Unknown Author is licensed under CC BY-NC-ND

2. Outline NHST: what we want VS what it gives The Permanent Illusion
P(D|H0) is not P(H0|D)
Bayesian Statistics
The Nil Hypothesis
P-Value or Effect Size
Alternatives to NHST
Just a brief outline of what we will be covering today…

3. Option 1: Chi-Square with Yates correction?
Wanted to test belief that rare disease doesn’t exist at all in a given population
Null Hypothesis: P=0
In random sample of 30 there is one person with disease. Ps= How to test Hnull?
Option 1: Chi-Square with Yates correction?
Option 2: Fisher exact test?
We are first given an example about his colleagues dilemma in trying to put a significance test on a problem with only a single population.
This is an example hoe NHST can impede science.

4. What’s Wrong with Null Hypothesis Significance testing (NHST) ?
What we want
Given these data, what is the probability H0 is true?
What it actually tells us
Given that H0 is true.. What is the probability of these (or more extreme) data?
The misinterpretation begins when we interpret H0 as what we want it to mean vs what it really means
When we look at H0 we want to be able to answer.. Based on the data… what is probability H0 is true…
In reality we can only answer… Given we got a significant null… What is the probability of the data we tested (or more extreme data).
This at first glance is confusing as they appear to almost be saying the same thing.

5. The Permanent Illusion
The invalid Bayesian interpretation:
The Belief that rejecting H0 makes HA TRUE or at least H0 unlikely
If the null hypothesis is correct, then this datum cannot occur
It HAS occurred… therefore the null is false
Our attempt in using deducing reasoning.. has lead to what Cohen called “The Permanent Illusion”.
The belief that rejecting H0 makes HA true or at least H0 unlikely.
Also known as:
The invalid Bayesian interpretation
illusion of attaining improbability
Bayesian Ids wishful thinking
The first type of misinterpretation is when we attempt to make the reasoning formally correct
If the null is correct datum cannot occur, datum has occurred therefore null is false.
NHST is not formally correct …. So the next step is to make reasoning probabilistic to better suite the reasoning behind NHST
If this reasoning stood.. Would be then formally correct.

6. Conditional inferences do not work with probabilistic relations
The Permanent Illusion
Make reasoning probabilistic…
Conditional inferences do not work with probabilistic relations
Attempt to make sensible
**Making erroneous inference that there is a symmetrical relation
If a person is an American, then he is not a member of Congress
This person is a member of Congress. Therefore, he is not an American.
If the null is correct, then these data are highly unlikely
These data HAVE occurred… therefore, the null is highly unlikely
If a person is American, then he is probably not a member of Congress
Person is a member of Congress, Therefore, he is probably not an American.
Here we have an example of changing our reasoning from absolute to probabilistic
The problem is that conditional inferences do not work with probabilistic relations.
So when we say:
If the null is correct, then these data are highly unlikely … these data have occurred, therefore the null is highly unlikely
We are claiming they have a symmetrical relation… Which is incorrect.

7. P(D|H0) P(H0|D) H0 True Data Arose Probability Probability If H0 True
The probability of the data arising if the null is true doesn’t equal the probability of H0 being true given the data
We can only make inferences about how likely or unlikely H0 is if we actually have the probability of H0 before the experiment begins.
Probability
If H0 True
Given the data

8. Bayesian Statistics
Bayesians Statistics post a prior probability or distribution of probabilities of H0
Example of solving problem knowing H0:
Screening Test:
95% accurate in positive (Sensitivity)
97% accuracy in declaring normality (Specificity)
H0= Case is normal
HA= Case is Schizophrenic
D= Test result (data) is positive for schizophrenia.
𝑃(𝐻0)∗𝑃(𝑡𝑒𝑠𝑡 𝑤𝑟𝑜𝑛𝑔|𝐻0) 𝑃 𝐻0 ∗𝑃 𝑡𝑒𝑠𝑡 𝑤𝑟𝑜𝑛𝑔 𝐻0 +𝑃 𝐻1 ∗𝑃(𝑡𝑒𝑠𝑡 𝑐𝑜𝑟𝑟𝑒𝑐𝑡|𝐻1)
For Bayesian statistics the prior probability of H0 is stated before the experiment in order to calculate the actual probability of H0 given the data.
To showcase this… We use the example of diagnosing schizophrenia
-In our problem the null is set to when a case is normal
-The Alternative is having schizophrenia
If case is diagnosed schizophrenic… We want to know the probability of a case being normal – so probability of H0 being true.
First it tempting to look at the 95% accuracy and say there is a 5% chance of patient being normal. THIS IS NOT THE CASE
If you actually plug in the numbers…. We see that its actually a 60% chance of patient being normal
Inverse Probability 0.60 NOT 0.05

9. If you set p to 0.01 and reject H0 … doesn’t mean 99% of replicated studies will reject H0
The last type of misinterpretation that occurs from Bayesians wishful thinking is that the result of the null dictates the results of the replicated experiments.
In a mathematical psychological association meeting… 42 of the 70 psychologists believed that if you set the p to 0.01 and reject the null.. 99% of replicated studies should have the same result.
Repeatability of H0 results cannot be inferred from just the p-value of H0, the power of your study will very much influence how likely your results will be replicated.

10. The Nil Hypothesis Effect size of 0
Null Hypothesis is a hypothesis to be nullified
-> RIDICULOUS BECAUSE: to be nullified it has to have
Effect size of 0
then population mean difference is 0
Correlation is 0
The H0 can only be used for “true experiments” with randomization (like clinical trials)

11. The “Crud” Factor
“Everything is related to everything else.” - Meehl, 1990
Study in 1966: 15 variables were cross tabulated with a sample size of school children.
All 105 cross tabulations were significant, of which 101 were significant with a p=
Nobody can know how large the “crud factor” is in a given research domain!

12. P-value or Effect Size?
P-Value: It’s the probability that sample means are THE SAME
Effect size: HOW DIFFERENT are the sample means
What happens to the P-value and the Effect Size if the sample size is dramatically increased?

13. P-value or Effect Size?
What happens to the P-value and the Effect Size if the sample size is dramatically increased?
p-value will artificially become significant (the probability that means are the same is LOW)
Effect size will also increase, statistical power decrease

14. Problematic p-value
If taken literally, the null hypothesis is always false (Meehl, 1990)
With ideal sample, one could reject every null hypothesis
PROBLEM:
every experiment with p>.05 will become a Type II
error!
-> because sample size was not big enough to
detect effect

15. Type I and Type II Errors
“False negative finding”
to falsely infer the absence of something that is.
Type I error:
“false positive finding”
to falsely infer the existence of something that is not there

16. Should we use Correlation Coefficients instead?
“Correlations are subject to vary with selections as researchers change populations”
Correlations cannot give useful information on causal strength
Causality operates on single instances
Thus cannot be applied to populations with varying members!

17. Alternatives to NHST? Short answer: NONE EXIST!
BUT: Exploratory Data Analysis can be performed!
REPORT the effect sizes in the form of confidence limits more!
Confidence intervals reveal the status of the trivial nil hypothesis
remind researchers of the “Crud factor”
the author also states that the confidence intervals are almost never reported, because they are so embarrassingly large