Topic 20: Single Factor Analysis of Variance

Outline Analysis of Variance –One set of treatments (i.e., single factor) Cell means model Factor effects model –Link to linear regression using indicator explanatory variables

One-Way ANOVA The response variable Y is continuous The explanatory variable is categorical –We call it a factor –The possible values are called levels This approach is a generalization of the independent two-sample pooled t-test In other words, it can be used when there are more than two treatments

Data for One-Way ANOVA Y is the response variable X is the factor (it is qualitative/discrete) –r is the number of levels –often refer to these levels as groups or treatments Y i,j is the j th observation in the i th group

Notation For Y i,j we use –i to denote the level of the factor –j to denote the j th observation at factor level i i = 1,..., r levels of factor X j = 1,..., n i observations for level i of factor X –n i does not need to be the same in each group

KNNL Example (p 685) Y is the number of cases of cereal sold X is the design of the cereal package –there are 4 levels for X because there are 4 different package designs i =1 to 4 levels j =1 to n i stores with design i (n i =5,5,4,5) Will use n if n i the same across groups

Data for one-way ANOVA data a1; infile 'c:../data/ch16ta01.txt'; input cases design store; proc print data=a1; run;

The data Obscasesdesignstore

Plot the data symbol1 v=circle i=none; proc gplot data=a1; plot cases*design; run;

The plot

Plot the means proc means data=a1; var cases; by design; output out=a2 mean=avcases; proc print data=a2; symbol1 v=circle i=join; proc gplot data=a2; plot avcases*design; run;

New Data Set Obsdesign_TYPE__FREQ_avcases

Plot of the means

The Model We assume that the response variable is –Normally distributed with a 1.mean that may depend on the level of the factor 2.constant variance All observations assumed independent NOTE: Same assumptions as linear regression except there is no assumed linear relationship between X and E(Y|X)

Cell Means Model A “cell” refers to a level of the factor Y ij = μ i + ε ij –where μ i is the theoretical mean or expected value of all observations at level (or cell) i –the ε ij are iid N(0, σ 2 ) which means –Y ij ~N(μ i, σ 2 ) and independent –This is called the cell means model

Parameters The parameters of the model are – μ 1, μ 2, …, μ r –σ 2 Question (Version 1) – Does our explanatory variable help explain Y? Question (Version 2) – Do the μ i vary? H 0 : μ 1 = μ 2 = … = μ r = μ (a constant) H a : not all μ’s are the same

Estimates Estimate μ i by the mean of the observations at level i, (sample mean) û i = = ΣY i,j /n i For each level i, also get an estimate of the variance = Σ(Y ij - ) 2 /(n i -1) (sample variance) We combine these to get an overall estimate of σ 2 Same approach as pooled t-test

Pooled estimate of σ 2 If the n i were all the same we would average the –Do not average the s i In general we pool the, giving weights proportional to the df, n i -1 The pooled estimate is

Running proc glm proc glm data=a1; class design; model cases=design; means design; lsmeans design run; Difference 1: Need to specify factor variables Difference 2: Ask for mean estimates

Output Class Level Information ClassLevelsValues design Number of Observations Read19 Number of Observations Used19 Important summaries to check these summaries!!!

SAS 9.3 default output for MEANS statement

MEANS statement output Level of designN cases MeanStd Dev Table of sample means and sample variances

SAS 9.3 default output for LSMEANS statement

LSMEANS statement output designcases LSMEAN Standard ErrorPr > |t| < < < <.0001 Provides estimates based on model (i.e., constant variance)

Notation

ANOVA Table Source df SS MS Model r-1 Σ ij ( - ) 2 SSR/df R Error n T -r Σ ij (Y ij - ) 2 SSE/df E Total n T -1 Σ ij (Y ij - ) 2 SST/df T

ANOVA SAS Output SourceDF Sum of Squares Mean SquareF ValuePr > F Model <.0001 Error Corrected Total R-SquareCoeff VarRoot MSEcases Mean

Expected Mean Squares E(MSR) > E(MSE) when the group means are different See KNNL p 694 – 698 for more details In more complicated models, these tell us how to construct the F test

F test F = MSR/MSE H 0 : μ 1 = μ 2 = … = μ r H a : not all of the μ i are equal Under H 0, F ~ F(r-1, n T -r) Reject H 0 when F is large Report the P-value

Maximum Likelihood Approach proc glimmix data=a1; class design; model cases=design / dist=normal; lsmeans design; run;

GLIMMIX Output Model Information Data SetWORK.A1 Response Variablecases Response DistributionGaussian Link FunctionIdentity Variance FunctionDefault Variance MatrixDiagonal Estimation Technique Restricted Maximum Likelihood Degrees of Freedom MethodResidual

GLIMMIX Output Fit Statistics -2 Res Log Likelihood84.12 AIC (smaller is better)94.12 AICC (smaller is better) BIC (smaller is better)97.66 CAIC (smaller is better) HQIC (smaller is better)94.08 Pearson Chi-Square Pearson Chi-Square / DF10.55

GLIMMIX Output Type III Tests of Fixed Effects Effect Num DF Den DFF ValuePr > F design <.0001 design Least Squares Means designEstimate Standard ErrorDFt ValuePr > |t| < < < <.0001

Factor Effects Model A reparameterization of the cell means model Useful way at looking at more complicated models Null hypotheses are easier to state Y ij = μ +  i + ε ij –the ε ij are iid N(0, σ 2 )

Parameters The parameters of the model are – μ,  1,  2, …,  r –σ 2 The cell means model had r + 1 parameters –r μ’s and σ 2 The factor effects model has r + 2 parameters –μ, the r  ’s, and σ 2 –Cannot uniquely estimate all parameters

An example Suppose r=3; μ 1 = 10, μ 2 = 20, μ 3 = 30 What is an equivalent set of parameters for the factor effects model? We need to have μ +  i = μ i μ = 0,  1 = 10,  2 = 20,  3 = 30 μ = 20,  1 = -10,  2 = 0,  3 = 10 μ = 5000,  1 = -4990,  2 = -4980,  3 = -4970

Problem with factor effects? These parameters are not estimable or not well defined (i.e., unique) There are many solutions to the least squares problem There is an X΄X matrix for this parameterization that does not have an inverse (perfect multicollinearity) The parameter estimators here are biased (SAS proc glm)

Factor effects solution Put a constraint on the  i Common to assume Σ i  i = 0 This effectively reduces the number of parameters by 1 Numerous other constraints possible

Consequences Regardless of constraint, we always have μ i = μ +  i The constraint Σ i  i = 0 implies –μ = (Σ i μ i )/r (unweighted grand mean) –  i = μ i – μ (group effect) The “unweighted” complicates things when the n i are not all equal; see KNNL p

Hypotheses H 0 : μ 1 = μ 2 = … = μ r H 1 : not all of the μ i are equal are translated into H 0 :  1 =  2 = … =  r = 0 H 1 : at least one  i is not 0

Estimates of parameters With the constraint Σ i  i = 0

Solution used by SAS Recall, X΄X does not have an inverse We can use a generalized inverse in its place (X΄X) - is the standard notation There are many generalized inverses, each corresponding to a different constraint

Solution used by SAS (X΄X) - used in proc glm corresponds to the constraint  r = 0 Recall that μ and the  i are not estimable But the linear combinations μ +  i are estimable These are estimated by the cell means

Cereal package example Y is the number of cases of cereal sold X is the design of the cereal package i =1 to 4 levels j =1 to n i stores with design i

SAS coding for X Class statement generates r explanatory variables The i th explanatory variable is equal to 1 if the observation is from the i th group In other words, the rows of X are for design= for design= for design= for design=4

Some options proc glm data=a1; class design; model cases=design /xpx inverse solution; run;

Output The X'X Matrix Int d1 d2 d3 d4 cases Int d d d d cases Also contains X’Y

Output X'X Generalized Inverse (g2) Int d1 d2 d3 d4 cases Int d d d d cases

Output matrix Actually, this matrix is (X΄X) - (X΄X) - X΄Y Y΄X(X΄X) - Y΄Y-Y΄X(X΄X) - X΄Y Parameter estimates are in upper right corner, SSE is lower right corner (last column on previous page)

Parameter estimates St Par Est Err t P Int 27.2 B <.0001 d B <.0001 d B <.0001 d B d4 0.0 B...

Caution Message NOTE: The X'X matrix has been found to be singular, and a generalized inverse was used to solve the normal equations. Terms whose estimates are followed by the letter 'B' are not uniquely estimable.

Interpretation If  r = 0 (in our case,  4 = 0), then the corresponding estimate should be zero the intercept μ is estimated by the mean of the observations in group 4 since μ +  i is the mean of group i, the  i are the differences between the mean of group i and the mean of group 4

Recall the means output Level of design N Mean Std Dev

Parameter estimates based on means Level of design Mean = 27.2 = = = = = = = = = 0

Last slide Read KNNL Chapter 16 up to We used programs topic20.sas to generate the output for today Will focus more on the relationship between regression and one-way ANOVA in next topic