# Proactive Monte Carlo Analysis in Structural Equation Modeling James H. Steiger Vanderbilt University.

## Presentation on theme: "Proactive Monte Carlo Analysis in Structural Equation Modeling James H. Steiger Vanderbilt University."— Presentation transcript:

Proactive Monte Carlo Analysis in Structural Equation Modeling James H. Steiger Vanderbilt University

Some Unhappy Scenarios A Confirmatory Factor Analysis – You fit a 3 factor model to 9 variables with N=150 – You obtain a Heywood Case Comparing Two Correlation Matrices – You wish to test whether two population matrices are equivalent, using ML estimation – You obtain an unexpected rejection

Some Unhappy Scenarios Fitting a Trait-State Model – You fit the Kenny-Zautra TSE model to 4 waves of panel data with N=200. You obtain a variance estimate of zero. Writing a Program Manual – You include an example analysis in your widely distributed computer manual – The analysis remains in your manuals for more than a decade – The analysis is fundamentally flawed, and gives incorrect results

Some Common Elements Models of covariance or correlation structure Potential problems could have been identified before data were ever gathered, using “proactive Monte Carlo analysis”

Confirmatory Factor Analysis VariableFactor 1Factor 2Factor 3 VIS_PERCX CUBESX LOZENGESX PAR_COMPX SEN_COMPX WRD_MNGX ADDITIONX CNT_DOTX ST_CURVEX

Confirmatory Factor Analysis VariableFactor 1Factor 2Factor 3 Unique Var. VIS_PERC0.460.79 CUBES0.650.58 LOZENGES0.250.94 PAR_COMP1.000.00 SEN_COMP0.410.84 WRD_MNG0.220.95 ADDITION0.380.85 CNT_DOT1.000.00 ST_CURVE0.300.91

Confirmatory Factor Analysis VariableFactor 1Factor 2Factor 3 Unique Var. VIS_PERC0.600.64 CUBES0.600.64 LOZENGES0.600.64 PAR_COMP0.600.64 SEN_COMP0.600.64 WRD_MNG0.600.64 ADDITION0.600.64 CNT_DOT0.600.64 ST_CURVE0.600.64

Proactive Monte Carlo Analysis Take the model you anticipate fitting Insert reasonable parameter values Generate a population covariance or correlation matrix and fit this matrix, to assess identification problems Examine Monte Carlo performance over a range of sample sizes that you are considering Assess convergence problems, frequency of improper estimates, Type I Error, accuracy of fit indices Preliminary investigations may take only a few hours

Confirmatory Factor Analysis (Speed)-1{.3}->[VIS_PERC] (Speed)-2{.4}->[CUBES] (Speed)-3{.5}->[LOZENGES] (Verbal)-4{.6}->[PAR_COMP] (Verbal)-5{.3}->[SEN_COMP] (Verbal)-6{.4}->[WRD_MNG] (Visual)-7{.5}->[ADDITION] (Visual)-8{.6}->[CNT_DOT] (Visual)-9{.3}->[ST_CURVE]

Confirmatory Factor Analysis

(Speed)-1{.53}->[VIS_PERC] (Speed)-2{.54}->[CUBES] (Speed)-3{.55}->[LOZENGES] (Verbal)-4{.6}->[PAR_COMP] (Verbal)-5{.3}->[SEN_COMP] (Verbal)-6{.4}->[WRD_MNG] (Visual)-7{.5}->[ADDITION] (Visual)-8{.6}->[CNT_DOT] (Visual)-9{.3}->[ST_CURVE]

Confirmatory Factor Analysis

VariableFactor 1Factor 2Factor 3 Unique Var. VIS_PERC0.600.64 CUBES0.600.64 LOZENGES0.600.64 PAR_COMP0.600.64 SEN_COMP0.600.64 WRD_MNG0.600.64 ADDITION0.600.64 CNT_DOT0.600.64 ST_CURVE0.600.64

Proactive Monte Carlo Analysis

Standard Errors

Distribution of Estimates

Standard Errors (N =300)

Distribution of Estimates

Correlational Pattern Hypotheses “Pattern Hypothesis” – A statistical hypothesis that specifies that parameters or groups of parameters are equal to each other, and/or to specified numerical values Advantages of Pattern Hypotheses – Only about equality, so they are invariant under nonlinear monotonic transformations (e.g., Fisher Transform).

Correlational Pattern Hypotheses Caution! You cannot use the Fisher transform to construct confidence intervals for differences of correlations – For an example of this error, see Glass and Stanley (1970, p. 311-312).

Comparing Two Correlation Matrices in Two Independent Samples Jennrich (1970) – Method of Maximum Likelihood (ML) – Method of Generalized Least Squares (GLS) – Example Two 11x11 matrices Sample sizes of 40 and 89

Comparing Two Correlation Matrices in Two Independent Samples ML Approach Minimizes ML discrepancy function Can be programmed with standard SEM software packages that have multi-sample capability

Comparing Two Correlation Matrices in Two Independent Samples Generalized Least Squares Approach Minimizes GLS discrepancy function SEM programs will iterate the solution Freeware (Steiger, 2005, in press) will perform direct analytic solution

Monte Carlo Results – Chi-Square Statistic MeanS.D. Observed75.813.2 Expected6611.5

Monte Carlo Results – Distribution of p-Values

Monte Carlo Results – Distribution of Chi-Square Statistics

Monte Carlo Results (ML) – Empirical vs. Nominal Type I Error Rate Nominal .010.050 Empirical .076.208

Monte Carlo Results (ML) Empirical vs. Nominal Type I Error Rate N = 250 per Group Nominal .010.050 Empirical .011.068

Monte Carlo Results – Chi-Square Statistic, N = 250 per Group MeanS.D. Observed67.711.6 Expected6611.5

Kenny-Zautra TSE Model T Y1Y1 11 Y2Y2 22 Y3Y3 33 YJYJ JJ O2O2 O1O1 O3O3 OJOJ … 22 33 JJ    J  TSE model

Likelihood of Improper Values in the TSE Model

Constraint Interaction Steiger, J.H. (2002). When constraints interact: A caution about reference variables, identification constraints, and scale dependencies in structural equation modeling. Psychological Methods, 7, 210- 227.

Constraint Interaction

Constraint Interaction – Model without ULI Constraints (Constrained Estimation) (XI1)-1->[X1] (XI1)-2->[X2] (XI1)-{1}-(XI1) (DELTA1)-->[X1] (DELTA2)-->[X2] (DELTA1)-3-(DELTA1) (DELTA2)-4-(DELTA2) (ETA1)-98->[Y1] (ETA1)-5->[Y2] (ETA2)-99->[Y3] (ETA2)-6->[Y4] (EPSILON1)-->[Y1] (EPSILON2)-->[Y2] (EPSILON3)-->[Y3] (EPSILON4)-->[Y4] (EPSILON1)-7-(EPSILON1) (EPSILON2)-8-(EPSILON2) (EPSILON3)-9-(EPSILON3) (EPSILON4)-10-(EPSILON4) (ZETA1)-->(ETA1) (ZETA2)-->(ETA2) (ZETA1)-11-(ZETA1) (ZETA2)-12-(ZETA2) (XI1)-13->(ETA1) (XI1)-13->(ETA2) (ETA1)-15->(ETA2)

Constraint Interaction

Constraint Interaction – Model With ULI Constraints (XI1)-->[X1] (XI1)-2->[X2] (XI1)-1-(XI1) (DELTA1)-->[X1] (DELTA2)-->[X2] (DELTA1)-3-(DELTA1) (DELTA2)-4-(DELTA2) (ETA1)-->[Y1] (ETA1)-5->[Y2] (ETA2)-->[Y3] (ETA2)-6->[Y4] (EPSILON1)-->[Y1] (EPSILON2)-->[Y2] (EPSILON3)-->[Y3] (EPSILON4)-->[Y4] (EPSILON1)-7-(EPSILON1) (EPSILON2)-8-(EPSILON2) (EPSILON3)-9-(EPSILON3) (EPSILON4)-10-(EPSILON4) (ZETA1)-->(ETA1) (ZETA2)-->(ETA2) (ZETA1)-11-(ZETA1) (ZETA2)-12-(ZETA2) (XI1)-13->(ETA1) (XI1)-13->(ETA2) (ETA1)-15->(ETA2)

Constraint Interaction – Model With ULI Constraints

Typical Characteristics of Statistical Computing Cycles Back-loaded – Occur late in the research cycle, after data are gathered Reactive – Often occur in support of analytic activities that are reactions to previous analysis results

Traditional Statistical World-View Data come first Analyses come second Analyses are well-understood and will work Before the data arrive, there is nothing to analyze and no reason to start analyzing

Modern Statistical World View Planning comes first – Power Analysis, Precision Analysis, etc. Planning may require some substantial computing – Goal is to estimate required sample size Data analysis must wait for data

Proactive SEM Statistical World View SEM involves interaction between specific model(s) and data. – Some models may not “work” with many data sets Planning involves: – Power Analysis – Precision Analysis – Confirming Identification – Proactive Analysis of Model Performance Without proper proactive analysis, research can be stopped cold with an “unhappy surprise.”

Barriers Software – Design – Availability Education

Download ppt "Proactive Monte Carlo Analysis in Structural Equation Modeling James H. Steiger Vanderbilt University."

Similar presentations