CIVE2602 - Engineering Mathematics 2.2 (20 credits) Statistics and Probability Lecture 9 Hypothesis testing –Examples Undertaking experiments t-test for.

Presentation on theme: "CIVE2602 - Engineering Mathematics 2.2 (20 credits) Statistics and Probability Lecture 9 Hypothesis testing –Examples Undertaking experiments t-test for."— Presentation transcript:

CIVE2602 - Engineering Mathematics 2.2 (20 credits) Statistics and Probability Lecture 9 Hypothesis testing –Examples Undertaking experiments t-test for two sample means P-values F-tests Dr D Borman ©Claudio Nunez 2010, sourced from http://commons.wikimedia.org/wiki/File:2010_Chile_earthquake_- _Building_destroyed_in_Concepci%C3%B3n.jpg?uselang=en-gb Available under creative commons license

1) Testing a single sample mean Does one distribution come from a certain population? How certain are we that it comes from a different population? population sample ?? 2) Two sample tests on the mean Are two distributions different? Do they come from the same population? How different? How certain are we that they are different?

Is there a difference in the strength of our concrete depending on if we use chemical X or chemical Y? (test sample of each type) Does our concrete get stronger if we leave it an additional week to set? (e.g. sample 1 this week, sample 2 week after) Has pollution level in a river changed since a new factory was built? (before /after sample) Has there been a reduction in vibrations to a bridge after modifications Hypotheses Testing – 2 sample tests ?? We might want to set up an experiment to test for… ©Terraplanner 2007, sourced from http://www.flickr.com/photos/halonfury/1977813099/ Available under creative commons license

Standard Deviation (σ) Testing X against Y (where Y is population) Testing X against Y (where Y is another sample) σ x ~ σ y σ is known or large sample z - test use σ (or S x ) = population variance 2 Sample t-test or 2 Sample z-test σ x ≠ σ y or σ is unknown, small samples t - test use σ =S x 2 Sample t-test Summary of when to use z-test and t-test population sample ??

No

‘Student’ t test The t-test was developed by W. S. Gossett at around 1899 Employed at the Guinness brewery in Dublin He was responsible for developing procedures for ensuring the similarity of batches of Guinness Had to publish anonymously so did so under the name ‘Student’ ©Sami Keinanen 2005, sourced from http://commons.wikimedia.org/wiki/File:Guinness.jpg Available under creative commons license

Ronald Fisher (famous Statistician), what he says rings even more true today than when he uttered it, back in 1938: "To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of." Designing a statistical experiment

http://www.youtube.com/watch?v=BX9iMIC6mcg Hypotheses Testing (2 sample t-test) Setting up and performing an experiment

1 )( 2 2     x x n XX S Formula B: - not equal variances Formula A: - assumes equal variances P-value –value from t-tables (We double it for 2TT so that it can be compared to significance level, that’s because value from tables only shows value for one tail)

1 )( 2 2     x x n XX S Formula B: - not equal variances Formula A: - assumes equal variances Rejection region for H 0 Acceptance region for H 0 Critical Level (from tables) (t crit ) P-value –value from t-tables (We double it for 2TT so that it can be compared to significance level, that’s because value from tables only shows value for one tail)

3 ways to check if H 0 should be rejected (all effectively the same thing) Rejection region for H 0 Acceptance region for H 0 Critical Level (from tables) t crit -If P-value< significance level (<5%) (p-value found from tables, but doubled for 2TT) -If value from tables < ½ significance level (2TT) -If t statistic > t crit t crit =t-value for a particular significance level e.g. 5% (depends on if 1TT or 2TT) http://www.youtube.com/watch?v=ZFXy_UdlQJg&NR=1

A simple example The breaking stress of 66 Oak beams to be used in houses around Yorkshire were measured We want to –Describe the data –Test for differences between Red Oak and Yellow Oak

Good to look at your data before analysis Box plot Scatter plot median IQR Red yellow Oak Type Breaking strength Red yellow Oak Type Breaking strength

What is the distribution of the data?

The frequency distribution of the breaking stress of Oak beams in 66 Yorkshire houses Approximately normal distribution +/- symmetrical (skewness = 0) Breaking stress

How do I analyse my data? Think about analysis before you collect the data What is the form of the data? How do I describe the data? Hypothesis testingHypothesis testing What test do I need?What test do I need? What do the results mean?What do the results mean?

Hypothesis testing Phrase as a hypothesis: e.g. Strength of beams vary with Oak type Rephrase as a Null hypothesis, H 0 e.g. “beam strength does not vary with Oak type” Test whether data are consistent with H 0 Think of an interesting/appropriate question

Types of null hypothesis data follow a given distribution a and b are not associated means do not differ x does not influence y

What test do we need? Breaking strength of Oak Beam -Single continuous dependent variable Oak Type (Red/Yellow) -Single categorical predictor variable => t-test (or ANOVA if more than two)

Could draw a graph show 95% Confidence Intervals of sample mean Graph should reflect analysis e.g. if analysing means present means, Figure legend should be understandable without reference to text Type of Oak Red Yellow 95% CI Breaking strength

Hypothesis testing Choose the null hypothesis –H 0 : breaking strengths of Red and Yellow are equal Calculate a test statistic –Mean for Red (x 1 )= 181, Yellow (x 2 )= 170 –Calculate t = (x 1 - x 2 )/SE = 11/4.6 (use formula) –t =2.37 - Null hypothesis allows us to predict the distribution of the test statistic -Test the observed value against the predicted distribution assuming the null hypothesis is true -Is this is significant i.e. larger or smaller than we would expect by chance?

If H 0 is true, predicted distribution of t for 64 df is t crit =-2.00 t crit =2.00 There is a probability of 2.1% that H 0 is true (p-value= 0.021), 5% of the time we see t>2.00 or t<-2.00 (P-value =0.021, found from tables using t-statistic= 2.37) t=2.37 Remember n=66, df=v=n-2=64 t crit -value for 5% significance (from tables) Therefore, reject H0 and take alternative Hypothesis=> There is a difference between red and yellow oak

Data presentation (what do we need to report) The frequency distribution of Oak breaking strength did not differ significantly from normal distribution. –Make it clear that your data meets any assumptions of the tests used The mean (±SD) Breaking strength was 181 kN (±20) for Red Oak (n=30), and 170 kN (±18) for Yellow Oak (n=36) –Present appropriate descriptive statistics There was a significant difference in Breaking strength between Red Oak and Yellow Oak (t=2.37, df=64, P=0.021)… –Present the test statistic, degrees of freedom and exact P-value

P-value measures strength of evidence against null hypothesis, not magnitude of effect P<0.0001 P=0.021 Point to note

Interpreting Non-Significant results Non-significant result does not mean no effect (could be good to take larger samples!) Very hard to prove no effect Other techniques can be used

F-test – test on the variance

Example- Comparing Two means with small sample Five cubes can be made from one mix of concrete. They are tested after 5 days and give the following compressive strengths: 23 17 22 33 25 (say, X) while another 5 cubes from the same mix are tested after 6 days and give 26 19 29 37 24 (say, Y) Is there a significant difference between the means for the two samples? 1) –Test variances (not done this yet) – assume underlying variances are the same Solution We have small samples with an unknown σ (population standard deviation) Use the t-test n x =5 n y =5 X =24,Y =27Find sample means: Find sample standard deviation: Formula B: Formula A: Degrees of freedom, v = n – 2 = 5+5 -2 = 8 (assume underlying population is Normally distributed)

F-test – test on the variance So far we have only considered testing a sample mean against a population mean (μ). Here we look at tests on the sample variance to see if two samples have significantly different variances (normally done before comparing means). Distribution of Variances If X and Y are normally distributed with identical variances and if samples of size n x and n y respectively are drawn from each, then the ratio satisfies an F distribution with V 1 = n x -1 and V 2 = n y -1 degrees of freedom. Remember and are the unbiased estimators of the X and Y population variances given by:

F-test – test on the variance So far we have only considered testing a sample mean against a population mean (μ). Here we look at tests on the sample variance to see if two samples have significantly different variances. Distribution of Variances If X and Y are normally distributed with identical variances and if samples of size n x and n y respectively are drawn from each, then the ratio satisfies an F distribution with V 1 = n x -1 and V 2 = n y -1 degrees of freedom. Remember and are the unbiased estimators of the X and Y population variances given by: F-distribution

F-test Example – we have two small samples, X and Y Calculate Since our calculated value of F=4 exceeds the value at the 5% significance, we conclude that the two sample variances are different Degrees of freedom: V 1 = n x -1=17 and V 2 = n y -1=15 2.373 Find F at 5% significance

F – Test-summary The F-test is used to test for differences among sample variance. Like the Student's t, one calculates an F and compares this to a table value. The formula for F is simply The variance are arranged so that F>1. That is; s 1 2 >s 2 2. We use the F-test as the Student's t test, only we are testing for significant differences in the variances. 1)Invoke the null hypothesis that states that the two variances we are comparing are from the same population. (i.e., they are not statistically different) 2) Calculate the F value (the ratio of the two variances) 3) Look up the table value of F for the degrees of freedom used to calculate both variances and for a given confidence level. 4) If the calculated F is greater than the table value, then the null hypothesis is not correct. Else, the two could have come from the same population of measurements.

T-statistic Difference in the means ‘Sum’ of the standard errors T-statistic =

Assumptions about the T-statistic You have normally distributed data –You might not have. –You can’t use this test if you don’t. (there are ways to investigate if a population is likely to be normal- but beyond this course) - Important to state assumptions you are making

Confidence Need something to compare it with How certain do you need to be that the means are the same? 90% - 1 in 10 you’ll be wrong 95% - 1 in 20 you’ll be wrong 99%– 1 in 100 you’ll be wrong

What about if more than 2 samples? - ANOVA The ANalysis Of VAriance (or ANOVA) is a powerful and common statistical procedure The t-test tells us if the variation between two groups is "significant". Why not just do t-tests for all the pairs of samples. Multiple t-tests are not the answer because as the number of groups grows, the number of needed pair comparisons grows quickly. For 7 groups there are 21 pairs. ANOVA puts all the data into one number and gives us one P for the null hypothesis.

http://www.youtube.com/watch?v=BX9iMIC6mcg Hypotheses Testing (2 sample t-test) Setting up and performing an experiment - maybe need explanation of p-value 9perhaps cover here!)

Hypothesis testing considerations Do two samples come from the same population? Z-test (for very large samples or when σ known) t-test (for small samples when σ not known) Population should be Normally distributed

Testing if strength of concrete changes if kept in an humidity chamber for a week We want to find out if leaving concrete in a humidity chamber for a week changes the breaking strength of the concrete. We have 5 samples to put in the chamber and 6 to leave out in normal drying conditions. Following the week of drying the 11 samples are strength tested. 23 17 22 33 25 (say, X)26 19 29 37 24 32 (say, Y) The breaking strengths (in KN) are as follows Is there a significant difference between the means for the two samples? (assume underlying population is Normally distributed) Example

21 17 21 33 23 (say, X) 27 25 29 37 24 29 (say, Y) The breaking strengths are as follows Is there a significant difference between the means for the two samples? (don’t assume same variances) Solution We have small samples with an unknown σ (population standard deviation) =>Use the t-test n x =5 n y =6 X =23,Y =271) Find sample means: 2) Find sample standard deviation: Use Formula B (i.e. don’t assume same variances) 3) Degrees of freedom, v = n – 2 = 5+6 -2 = 9 Which test statistic to use?

Rejection region for H 0 Acceptance region for H 0 Critical Level (from tables) Use the t-test n x =5 n y =6 X =23,Y =27 Degrees of freedom, v = 9 H 0 : μ x = μ y H 1 : μ x ≠ μ y H 0 : the population means are the same H 1 : population mean for X is not the same as the population mean for Y. = 0 :H 0 Assumes means are equal 2TT we are looking for any difference (at 5% significance 2.5% in each tail) So P-value =~ 0.2 x2 =40% As P-value is higher than 5% significance (i.e.40%>5%) do not reject H 0 ->conclude that there is no significant difference in strength between the concrete that has been in the chamber (although might want to try with bigger samples!) Is there a significant difference between the means for the two samples? (don’t assume same variances) From t tables (t-stat =0.84): Could have also compared the probability of 0.2 from tables with 2.5% (i.e 20% > 2.5) and again would have rejected the H0 (this is just the same)

CIVE2602 - Engineering Mathematics 2.2 Lecture 9- Summary Hypothesis testing very important technique –used very widely We have looked at testing to see if there is a difference between 2 means – can test other properties as well When testing the means- perform an F-test on the variance first For more than 2 samples we use ANOVA

If X and Y are normally distributed with means μ x, and μ y, and population standard deviations σ x and σ y, and independent random samples of size n x and n y are drawn from X and Y, then the difference between the sample means is normally distributed with mean: μ x - μ y Differences of Sample Means: known population Variances (or estimated using very large samples) and variance or standard deviation Compare with:

Choosing the null hypothesis “x does not affect y” “distribution is not different from normal” “means of 2 groups do not differ” could be “means of 2 groups differ by 3.1” NOT “means of 2 groups are different” Allows a prediction to be made about expected value of a statistic

Summary of Tables in Notes P101 of notes - Normal tables -finds probabilities from z-values P102 of notes - Reverse Normal tables -finds z-statistics from probabilities. P103 of notes - t-tables (reverse) -finds t-statistic from probabilities (can use to look up probabilities from t-stat as well ). P104 – F-tables, used for F-test on variances (not covered this yet) Find probability for z=2.12. prob= 0.0170

What are the assumptions of an independent samples t test? Assumptions. For the equal-variance t test, -the observations should be independent, random samples from normal distributions with the same population variance. Formula A For the unequal-variance t test, -the observations should be independent, random samples from normal distributions. Formula B

Hypothesis test of two sample means is also referred to as the ‘Student’ T test The t-test was developed by W. S. Gossett at around 1899 Employed at the Guinness brewery in Dublin He was responsible for developing procedures for ensuring the similarity of batches of Guinness Had to publish anonymously so did so under the name ‘Student’ T-test is sometimes referred to as "Student's” t- test. http://www.youtube.com/watch?v=snrajeiZLK0&feature=related ©Sami Keinanen 2005, sourced from http://commons.wikimedia.org/wiki/File:Guinness.jpg Available under creative commons license

We want to find if a new factory being built near the river Aire, has changed the level of pollution in the river. Before the factory is built we take 30 samples and find that the mean pollution level of these samples to be 21.51 and the standard deviation to be 4.3. After the factory has been built a further 40 samples are taken from the river and the mean level is found to be 19.51 and sample standard distribution is 3.10. Q Can we say at 5% significance level that the level of pollution in the river has changed. i.e. Is there a significant difference between the underlying pollution levels of the river on these two occasions WORK IN GROUPS OF 2-3

Summary of Procedure Compare your t-statistic with the one in the table Need to decide on the –degrees of freedom and the –percentile you are trying to check within –Whether it is a 1 tail or a 2 tail comparison –Usually use 2 tails

T-Statistic Tables

3 variants of t-test Comparing 2 independent datasets –Equal variances (use Formula A) –Unequal variances (use Formula B) Matched pair comparison

Comparing 2 independent samples You have two different datasets looking at the same thing. Are the means significantly different? e.g. Is patient recovery rate faster with drug A or drug B e.g. Is the fuel economy better with Car Type A or Car Type B e.g. Is Concrete with Add

Matched pair comparison You have two datasets looking at the same thing with pairs that are links. Are the means significantly different? e.g. Do patients temperatures change after you give the drug A

Procedure for independent samples Find the mean of each group Find the number of observations in each group Find the standard deviation in each group Put the numbers into the formula Calculate the t-statistic Compare the t-statics with the ones in the table Is your value higher than the critical one (you can disprove the null) or lower (you can’t)

Analysis of lots of data Measure size of leaves on maple trees from 2 different sites Are the means the same at all the sites? Perform a t-test

Analysis of lots of data Measure size of leaves on maple trees from 10 different sites Could perform t-tests between all the combinations 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8, 1-9, 1-10 2-3, 2-4, 2-5, 2-6,2-7….. 3-4, 3-5, 3-6, 3-7,3-8,

Analysis of lots of data 45 different combinations Testing at 95% confidence, will find significant differences between the samples just because of the number of tests you are doing. Not the best way of doing this!!!!

Fisher Sir Ronald Aylmer Fisher Geneticist ‘Fisher was a genius who almost single- handedly created the foundations for modern statistical science’ Hald, Anders (1998). A History of Mathematical Statistics. New York: Wiley.Hald, Anders "I occasionally meet geneticists who ask me whether it is true that the great geneticist R.A. Fisher was also an important statistician" (Annals of Statistics, 1976). Fisher didn’t believe that smoking caused lung cancer.smoking caused lung cancer

Fisher’s analysis If we have several samples taken from identical populations we can estimate the standard deviation in 2 ways 1.The spread of values within the samples 2.The spread of the mean of the samples Providing the samples are from identical populations the two ways should give approximately the same result