Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Sample Size and Power for CFA: A Monte Carlo Approach Myers, N.D., Ahn, S., & Jin, Y.

Similar presentations


Presentation on theme: "1 Sample Size and Power for CFA: A Monte Carlo Approach Myers, N.D., Ahn, S., & Jin, Y."— Presentation transcript:

1 1 Sample Size and Power for CFA: A Monte Carlo Approach Myers, N.D., Ahn, S., & Jin, Y.

2 2 I. Purpose Demonstrate how Monte Carlo methods can be used to make empirically-based decisions: 1. N for a fixed level of π, and, (i.e., N?) 2. π for a fixed N (π?) under a CFA model with model-data conditions commonly encountered in sport & exercise Generality of MC methods has allowed for two types of applications in statistics 1 1.MC studies of statistical methods 2.MC methods in data analysis

3 3 I. Purpose Why is the additional complexity worthwhile? Rules of thumb are known to be of limited utility 2 N 200 N 200 N/p 10 N/p 10 N/q 5 N/q 5 construct reliability and adequate N construct reliability and adequate N Adequate N for CFA depends on many factors that typically vary across any two studies that use real data and inexact theoretical models These factors can be directly modeled using MC methods These factors can be directly modeled using MC methods

4 4 I. Purpose Informal review of RQES (and others) Some observed trends in practice: single population models single population models many p and multiple many p and multiple ordinal data ordinal data a sig 2 value and.05 a sig 2 value and.05 variable and (parameters of interest, i ) variable and (parameters of interest, i ) few vc > 1 and/or off-diagonal 0 few vc > 1 and/or off-diagonal 0 A priori plan for N, for a desired level of ? A priori plan for N, for a desired level of ? An estimate of, for a fixed N? An estimate of, for a fixed N?....must have knowns....must have knowns Model the messiness of practice… N? Model the messiness of practice… N?

5 II. A Conceptual Demonstration Coaching Efficacy: coachs belief in his/her ability to influence the learning and performance of his/her athletes 4 Psych: Bandura (1997) 5 ; Educ: Denham & Michael (1981) 6 Psych: Bandura (1997) 5 ; Educ: Denham & Michael (1981) 6 Coaching Effectiveness: Horn (2002) 7 5

6 II. A Conceptual Demonstration Coaching Efficacy Scale II-HST 8 6

7 7 III. Methods/Results A.Design Stage: Conceptualize experiment - Intro 9 1.Research Question(s) Smallest N necessary to achieve at least.80 for each i Smallest N necessary to achieve at least.80 for each i i i Given a particular N, what is the estimate for each i Given a particular N, what is the estimate for each i 2. Derive theoretical model & population model theoretical model & population modeltheoretical model & population model 3. Design Experiment Model mis-specification (MODEL POPULATION versus MODEL) Model mis-specification (MODEL POPULATION versus MODEL) N = 800, 200, 300, 400, 500; N R = 10,000 N = 800, 200, 300, 400, 500; N R = 10,000 Ordinal data Ordinal data Ordinal data Ordinal data 4.Choose values of population parameters range of parameter estimates range of parameter estimatesparameter estimatesparameter estimates 5.Choosing Software Mplus 5.2 (code is available by request) 10,11 Mplus 5.2 (code is available by request) 10,11code

8 8 III. Methods/Results B.Generating Data Stage: Performing experiment Data collection 6.Executing the simulations Same code varied N Same code varied N 7.File storage Saved where input file is located (stuff happens…) Saved where input file is located (stuff happens…) 8. Trouble shooting Data generation problem (e.g., no obs in a particular category) Data generation problem (e.g., no obs in a particular category) Data analytic problem (e.g., non-convergent, improper solution) Data analytic problem (e.g., non-convergent, improper solution)

9 9 III. Methods/Results C.Interpreting Results Stage: Findings from experiment Results/Discussion 9.Summarizing Results Summarizing ResultsSummarizing Results Theoretical ModelTheoretical Model fit to data generated from Population Model Theoretical Model Q1 (N?): A relatively small N (200) provides ample π (> 98%) Q1 (N?): A relatively small N (200) provides ample π (> 98%) Q2 ( ?): common N (300, 400, 500) is similar to Q1 (π > 99%) Q2 ( ?): common N (300, 400, 500) is similar to Q1 (π > 99%) Problematic bias values (> |10%|) and coverage values ( |10%|) and coverage values (< 91%), for a few parameters (~20%) are observed for all sample sizes Population ModelPopulation Model fit to data generated from Population Model Population Model To what degree might the relatively minor model mis-specifications be responsible for these problems? Q1 (N?): and Q2 ( ?) similar to findings from Theoretical Model except no problems with bias values and coverage values Q1 (N?): and Q2 ( ?) similar to findings from Theoretical Model except no problems with bias values and coverage values

10 10 IV. Importance 1.MC methods have long been used to advance statistical theory. There have been several recent calls to use Monte Carlo methods as a tool to improve applications of quantitative methods in substantive research. 2.Mplus code is available and can easily be altered 3.CES II – HST: N 200 for the theoretical model N 300 for the population model N 300 for the population model 4.a level of misfit that may be regarded as trivial in practice may have troubling effects on parameters of conceptual interest encourage sustained efforts toward generating closer approximations of population models encourage sustained efforts toward generating closer approximations of population models Thank you. Ms is in press at RQES.

11 11 1.Gentle, J. E. (2003). Random number generation and Monte Carlo methods (2 nd ed.). New York: Springer. 2.Marsh, H. W., Hau, K.-T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research, 33, MacCallum, R. (2003). Working with imperfect models. Multivariate Behavioral Research, 38, Feltz, D.L., Chase, M.A., Moritz, S.E., & Sullivan, P.J. (1999). A conceptual model of coaching efficacy: Preliminary investigation and instrument development. Journal of Educational Psychology, 91, Bandura, A. (1997). Self-efficacy: The exercise of control. New York: Freeman. 6.Denham, C.H., & Michael, J.J. (1981). Teacher sense of efficacy: A definition and a model for further research. Educational Research Quarterly, 5, Horn, T.S. (2002). Coaching effectiveness in the sport domain. In T.S. Horn (Ed.), Advances in sport psychology (2nd ed., pp ). Champaign, IL: Human Kinetics. 8.Myers, N. D., Feltz, D. L., Chase, M. A., Reckase, M. D., & Hancock, G. R. (2008). The Coaching Efficacy Scale II - High School Teams. Educational and Psychological Measurement, 68, Paxton, P., Curran, P. J., Bollen, K. A., Kirby, J., & Chen, F. (2001). Monte Carlo experiments: Design and implementation. Structural Equation Modeling, 8, Muthén, L. K., & Muthén, B. O. (2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, 9, Muthén, L.K. and Muthén, B.O. ( ). Mplus Users Guide (6th ed.). Los Angeles, CA: Muthén & Muthén.

12 12

13 13

14 14

15 15

16 II. A Conceptual Demonstration Coaching Efficacy Scale II-HST 8 16

17 17


Download ppt "1 Sample Size and Power for CFA: A Monte Carlo Approach Myers, N.D., Ahn, S., & Jin, Y."

Similar presentations


Ads by Google