1. Hypothesis Testing

2. A decision-making process for evaluating claims about a population
Based on information obtained from samples
Hypotheses are made a priori
before the experiment was conducted
Usually tests a prediction that some kind of effect exists in the population
A method for scientists to test the scientific questions they generate

3. Usually conjecture about a population parameter
A numerical value that characterizes some aspect of a population
Can never be completely sure that a hypothesis is true; instead, we work with probabilities
Generally, we will calculate the probability that results we have obtained in an experiment occurred by chance
We are studying whether the results we see are a random occurrence or due to some mechanism

4. Testing a Statistical Hypothesis
Steps:
Define the population to study
State the hypothesis that will be investigated
Give a statistical level
Select a sample from the population
Collect the data
Calculate statistical test based on sample data
Draw a conclusion
Testing a Statistical Hypothesis

5. Two types of hypothesis
Null Hypothesis
Usually written as H0 (H-knot)
A statement that there is no difference between a population parameter and a specific value (or no difference between two parameters)
Alternative Hypothesis (Research Hypothesis)
Usually written as HA
A statement that there is a difference between a population parameter and a specific value (or there is a difference between two parameters)
Two types of hypothesis

6. One and Two Tailed Tests
A hypothesis test that is directional is called a one-tailed test
Examples:
Older than age 21
H0: μ = 21 versus HA: μ > 25
Treatment A will do better than treatment B
H0: μA = μB versus HA: μA > μB
One and Two Tailed Tests

7. One and Two Tailed Tests
A hypothesis test that is non-directional is called a two-tailed test
Examples:
The population mean age is 21
H0: μ = 21 versus HA: μ ≠ 25
Treatment A has the same effect as treatment B
H0: μA = μB versus HA: μA ≠ μB
One and Two Tailed Tests

8. One and Two Tailed Tests
Notice the statement of equality – a statement of no effect – is written as the null hypothesis
Preview: If we reject the null hypothesis, this is the same thing as saying have detected an effect
One and Two Tailed Tests

9. Significance Level of a Statistical Test
Significance level is the probability we use as our criterion for an unlikely outcome, supposing that H0 is true
Denoted by α (alpha)
Must be decided a priori, this means that α must be determined before conducting a statistical test
Choosing α after the data as been analyzed lacks objectivity
Significance Level of a Statistical Test

10. Significance Level of a Statistical Test
α is our criterion for rejecting the null hypothesis
Reflects how careful the researcher wishes to be
The smaller the α that is specified, the stronger the evidence needed to reject H0
Scientific and medical literature commonly test hypotheses with α levels of 0.05 or 0.01
Significance Level of a Statistical Test

11. Calculating a Statistical Test
Remember, statistical inference uses sample statistics to make inferences about population parameters
We use sample data to calculate a test statistic
A test statistic is a univariate statistic (a number)
Sample data is used to calculate the test statistic
Test statistic is used in hypothesis testing to make decisions
To use the test statistic, we have to determine its sampling distribution under the null hypothesis (assuming the null hypothesis is true)
Calculating a Statistical Test

12. Calculating a Statistical Test
Once we know the distribution of the test statistic, we use this to calculate its p-value
Each time we calculate a test statistic, we will couple it with a corresponding p-value
In the literature, statistical test results are often presented in the form of (test statistic, p-value)
Examples: (t=-1.98, p=0.04) (t=2.5, p<0.013) (χ(2)=12.32, p<0.001) (F=6.17, p=0.0001)
Calculating a Statistical Test

13. Calculating a Statistical Test
Examples: (t=-1.98, p=0.04) (t (28)=2.5, p<0.013) (χ(2)=12.32, p<0.001) (F=6.17, p=0.0001)
In these examples, the letter denotes the sampling distribution of the test statistic
For example:
t denotes a t-distribution
t(28) denotes a t-distribution with 28 degrees of freedom
χ(2) denotes a Chi-squared distribution with 2 degrees of freedom
F denotes a F-distribution
Calculating a Statistical Test

14. Calculating a Statistical Test
The p-value is calculated using the test statistic and its sampling distribution
To calculate the p-value, we calculate the area under the curve for specific values of the test statistic and its sampling distribution
We will focus on how to use a p-value to make decisions about statistical significance
Calculating a Statistical Test

15. p - values On the slides that follow, we:
Define a p-value
Describe its properties
Clarify what it does and does not tell us
Following this, we will then go on to Step 7 of Hypothesis Testing
In Step 7, we will talk about Significance Testing
Significance Testing describes how the p-value is used to make decision
p - values

16. p-values Let’s not confuse The definition of a p-value
The process of using a p-value in making a decision
p-values

17. A p-value is the probability of obtaining the same sample statistic (mean value) or a more extreme value if the null hypothesis is true
The p-value is the most commonly reported result of a significance test
It enables us to judge the extent of the evidence against H0
Definition of p-value

18. Properties of a p-value
Ranges from 0 to 1
Summarizes the evidence in the data about H0
A large p-value (e.g., 0.58) indicates the observed data would not be unusual if H0 were true
A small p-value (e.g., ) indicates the observed data would be very doubtful if H0 were true
Properties of a p-value

19. P (A | B) = “the probability of A given B”No
Is P(A|B) = P(B|A) ?
Example; A = dramatic overdose of medication
B = death
P(A|B): Given a person dies, what is the probability of death due to overdose? (this probability will likely be very low)
P(B|A): Given a person overdoses, what is the probability of this person dying? (this probability will likely be very high)
Is P(A|B) = P(B|A) here?
p-value

20. Properties of p-values
A p-value does not give information about trend, direction, strength of an association, size of an effect, or magnitude of a difference
p=0.04 is not less significant than p=
Statistical significance is influenced by the sample size
With a large enough sample size anything can be statistically significant
When reporting p-values, footnotes such as
*p<.05, **p<0.01
suggest a trend and are misleading
Properties of p-values

21. Properties of p-values
A smaller p-value does not indicate a more important result
Magnitude of the p-value is not a guide to clinical significance
p-values do not take Into account the size of the effect
A small effect in a large study can have the same p-value as a large effect In a small one
Conclusions should not be based only on p-values
p=0.06 is not evidence of 'marginal significance' or proof of a 'trend towards significance'; these types of conclusions are untrue
Properties of p-values

22. Properties of p-values
p-value does give the probability the result observed is due to chance
The p-value does give the chance of obtaining the effect we have observed, assuming the null hypothesis is true (in other words, assuming there is no real effect)
Whenever a p-value is reported, it is recommended to also report a measure of effect size
Confidence Interval
Correlation coefficient
Regression parameter
Etc.
Properties of p-values

23. We have not discussed how the p-value is used in making a decision
Significance Testing describes the process for using the p-value to make a decision
Next, we talk about Step 7; drawing conclusions and making inferences
We need to discuss the decision we are making, and possible errors in making that decision
p-values

24. Errors of Inference In hypothesis testing, we make a decision
Only two possible outcomes
We decide to either
Reject H0
Fail to reject H0
Errors in inference describes each of these two possible incorrect decisions
Errors of Inference

25. Hypothesis Testing Language
"Accepting the null hypothesis" is NOT the same conclusion as "Failing to reject the null hypothesis"
Hypothesis Testing Language

26. Hypothesis Testing Language
If we do not reject the null hypothesis
We do not conclude it is true
We can only recognize the null hypothesis is a possibility
Showing the null hypothesis is true is not the same thing as failing to reject it
Failing to find an effect is not the same thing as showing there is no effect
Hypothesis Testing Language

27. Errors of Inference These two errors are given specific names:
Type I Error
We draw a conclusion that H0 is false when it is in fact true
Occurs when we conclude there is an effect in our population, when in fact there is not
Type II Error
We draw a conclusion that H0 is true when it is in fact false
The probability of a type II error is denoted by the Greek letter β ('beta’)
In other words, P(Type II Error)= β
Errors of Inference

28. We draw a conclusion that H0 is false when it is in fact true
Occurs when we conclude there is an effect in our population, when in fact there is not
Example 1: Incorrectly conclude an HIV prevention intervention is effective (Reject H0, conclude HA is true) in preventing HIV infections through behavioral changes when, in fact, it is not (H0 Is actually true)
Example 2: Incorrectly conclude a new chemo agent is effective (Reject H0, conclude HA is true) in reducing tumor size when, in fact, it is not (H0 Is actually true)
Type I Error

29. We draw a conclusion that H0 is true when it is In fact false
The probability of a type II error Is denoted by the Greek letter β ('beta')
Example 1: Incorrectly conclude an HIV prevention intervention is not effective (Fall to reject H0, conclude HA is false) in preventing HIV infections through behavioral changes when, in fact, it is (HA is actually true)
Example 2: Incorrectly conclude a new chemo agent is not effective (Fall to reject H0, conclude HA is false) in reducing tumor size when, in fact, It is (HA is actually true)
Type II Error

30. Type I and Type II Errors
Decision
Null Hypothesis
True
False
Fail to Reject H0
Correct
Type II Error
(False Negative)
Reject H0
Type I Error
(False Positive)
False Positive: We reject H0 and conclude there is an effect (Positive; Alternative Hypothesis) when, in fact, there is not an effect (False Conclusion)
False Negative: We fail to reject H0 and conclude there is not an effect (Negative; Null Hypothesis) when, in fact, there is an effect (False Conclusion)
Type I and Type II Errors

31. This describes the process of using the p-value to make a decision
We compare the calculated p-value from Step 6 to the predetermined significance level (α) from Step 3
If p < α, we reject H0 and conclude the result is statistically significant
If p > α, we fail to reject H0, and conclude we have failed to find statistical significance
Significance Testing

32. Conclusion: Significance
Essential to understand the difference between statistical and clinical significance
Statistical Significance - the likelihood that the difference could have occurred by chance alone
Clinical Significance - the smallest clinically beneficial and harmful values of the effect; in other words, the smallest values that matter to the patient
Conclusion: Significance

33. A research article gives a p-value of. 001 in the analysis section
A research article gives a p-value of .001 in the analysis section. Which definition of a p-value is the most accurate?
the probability that the observed outcome will occur again.
the probability of observing an outcome as extreme or more extreme than the one observed if the null hypothesis is true.
the value that an observed outcome must reach in order to be considered significant under the null hypothesis.
the probability that the null hypothesis is true.
Question

34. Reference: Dr. Matt Hayat Statistics Lecture – Rutgers University 2013 Plichta SB, Kelvin E. (2012) Munro’s Statistical Methods for Healthcare Research, 6th Edition