Statistical tests for replicated experiments Normal probability plots are a less formal diagnostic tool for detecting effects Normal probability plots are a less formal diagnostic tool for detecting effects F-tests and t-tests provide a statistical test of factor effects F-tests and t-tests provide a statistical test of factor effects

Statistical tests for replicated experiments Statistical tests are possible for unreplicated designs (unreplicated pilot studies are essential tools in sample size calculations) Statistical tests are possible for unreplicated designs (unreplicated pilot studies are essential tools in sample size calculations) We will first focus on statistical tests for replicated designs We will first focus on statistical tests for replicated designs

Statistical tests for replicated experiments--Example Response--Pulse rate of subject Response--Pulse rate of subject Factors Factors –Treatment (Energy Drink, Placebo) –Setting (Moderate, Difficult) –Machine (Stair climber, Recumbent bike)

Statistical tests for replicated experiments--Example

Statistical tests for replicated experiments Effect sizes depend on the measurement scale Effect sizes depend on the measurement scale Statistical tests are based on standardized effects Statistical tests are based on standardized effects To compute standardized effects, start with an estimate of experimental error To compute standardized effects, start with an estimate of experimental error

Statistical tests for replicated experiments Experimental error can be summarized by the square root of the variance of the background noise (the standard deviation) Experimental error can be summarized by the square root of the variance of the background noise (the standard deviation) The experimental error measures variation in a single observation The experimental error measures variation in a single observation

Statistical tests for replicated experiments The variance is best estimated by the Mean Square for Pure Error (MSPE) The variance is best estimated by the Mean Square for Pure Error (MSPE)

Statistical tests for replicated experiments--Example The standard deviation for each run is ~3 beats per minute The standard deviation for each run is ~3 beats per minute

Statistical tests for replicated experiments While the standard deviation for a single response is the square root of MSPE, the standard deviation of an effect (its standard error) is: While the standard deviation for a single response is the square root of MSPE, the standard deviation of an effect (its standard error) is:

Statistical tests for replicated experiments We divide an effect in a k-factor experiment with n replications (e.g., A) by its standard error to compute a t-test statistic : We divide an effect in a k-factor experiment with n replications (e.g., A) by its standard error to compute a t-test statistic :

Statistical tests for replicated experiments Test statistics for other effects are computed similarly Test statistics for other effects are computed similarly U-do-it: Calculate the T-statistics of all effects for the Exercise data U-do-it: Calculate the T-statistics of all effects for the Exercise data

Statistical tests for replicated experiments When an effect is negligible, T has a t- distribution When an effect is negligible, T has a t- distribution The shape of the t-distribution curve depends on the number of replicates (degrees of freedom=2 k (n-1)) The shape of the t-distribution curve depends on the number of replicates (degrees of freedom=2 k (n-1)) The t-distribution curves have slightly more spread than the bell-shaped (normal) curve The t-distribution curves have slightly more spread than the bell-shaped (normal) curve

Statistical tests for replicated experiments--t curve for 3- factor design

Statistics tests for replicated experiments If |T| is larger than the 99.5 th or 97.5 th percentile of the t distribution, an effect is significant If |T| is larger than the 99.5 th or 97.5 th percentile of the t distribution, an effect is significant These percentiles are commonly found in textbooks (but please use a computer package instead) These percentiles are commonly found in textbooks (but please use a computer package instead)

Statistical tests for replicated experiments--t critical value for 3-factor design (n=4)

Statistical tests for replicated experiments Sometimes, twice the area to the right of |T| is reported as a p-value. Small p- values suggest that a standardized effect is distinguishable from background noise Sometimes, twice the area to the right of |T| is reported as a p-value. Small p- values suggest that a standardized effect is distinguishable from background noise You definitely need a computer to compute p-values--in the following example, the p-value for the M effect is 2*.122=.244 You definitely need a computer to compute p-values--in the following example, the p-value for the M effect is 2*.122=.244

Statistical test for replicated experiments--Example

Statistical tests for replicated experiments--Example U-do-it: Compute p-values for the remaining effects. Which effects are significant? Are these the same effects that the probability plot detected? U-do-it: Compute p-values for the remaining effects. Which effects are significant? Are these the same effects that the probability plot detected?

Statistical tests for replicated experiments F tests for individual effects are equivalent to t-tests F tests for individual effects are equivalent to t-tests F tests allow several comparisons to be tested simultaneously F tests allow several comparisons to be tested simultaneously The t-test can be used to help in computing the number of replications needed in a factorial experiment The t-test can be used to help in computing the number of replications needed in a factorial experiment

Statistical tests for replicated experiments Hypothesis testing can be extended to combine estimates of error from both pure error and negligible effects Hypothesis testing can be extended to combine estimates of error from both pure error and negligible effects –Negligible effects can be selected a priori or from effects plots –Degrees of freedom for t-tests and F-tests should be adjusted accordingly

Statistical tests for replicated experiments Error estimated from negiglible terms (Lack of Fit) is similar to MSPE Error estimated from negiglible terms (Lack of Fit) is similar to MSPE Source DF SS MS Residual Error Lack of Fit Lack of Fit Pure Error Pure Error