Presentation on theme: "Sample Size and Power for CFA: A Monte Carlo Approach"— Presentation transcript:
1 Sample Size and Power for CFA: A Monte Carlo Approach Myers, N.D., Ahn, S., & Jin, Y.
2 I. PurposeDemonstrate how Monte Carlo methods can be used to make empirically-based decisions:N for a fixed level of π, and, (i.e., N?)π for a fixed N (π?)under a CFA model with model-data conditions commonly encountered in sport & exerciseGenerality of MC methods has allowed for two types of applications in statistics1MC studies of statistical methodsMC methods in data analysis
3 Why is the additional complexity worthwhile? I. PurposeWhy is the additional complexity worthwhile?Rules of thumb are known to be of limited utility2N ≥ 200N/p ≥ 10N/q ≥ 5construct reliability and adequate NAdequate N for CFA depends on many factors that typically vary across any two studies that use real data and inexact theoretical modelsThese factors can be directly modeled using MC methods
4 Informal review of RQES (and others) I. PurposeInformal review of RQES (and others)Some observed trends in practice:single population modelsmany p and multiple hordinal dataa sig c2 value and e ≤ .05variable L and Y (parameters of interest, qi)few vc > 1 and/or off-diagonal Q ≠ 0A priori plan for N, for a desired level of p?An estimate of p, for a fixed N?....must have “knowns”Model the messiness of practice… N? p?3
5 II. A Conceptual Demonstration Coaching Efficacy: coach’s belief in his/her ability to influence the learning and performance of his/her athletes4Psych: Bandura (1997)5; Educ: Denham & Michael (1981)6Coaching Effectiveness: Horn (2002)7
6 II. A Conceptual Demonstration Coaching Efficacy Scale II-HST8
7 III. Methods/ResultsDesign Stage: Conceptualize experiment - “Intro” 9Research Question(s)Smallest N necessary to achieve at least .80 p for each qiGiven a particular N , what is the p estimate for each qi2. Derive theoretical model & population model3. Design ExperimentModel mis-specification (MODEL POPULATION versus MODEL)N = 800, 200, 300, 400, 500; NR = 10,000Ordinal data4. Choose values of population parametersrange of parameter estimates5. Choosing SoftwareMplus 5.2 (code is available by request)10,11
8 III. Methods/Results Generating Data Stage: Performing experiment “Data collection”6. Executing the simulationsSame code varied N7. File storageSaved where input file is located (stuff happens…)8. Trouble shootingData generation problem (e.g., no obs in a particular category)Data analytic problem (e.g., non-convergent, improper solution)
9 III. Methods/ResultsInterpreting Results Stage: Findings from experiment“Results/Discussion”9. Summarizing ResultsTheoretical Model fit to data generated from Population ModelQ1 (N?): A relatively small N (200) provides ample π (> 98%)Q2 (p?): common N (300, 400, 500) is similar to Q1 (π > 99%)Problematic bias values (> |10%|) and coverage values (< 91%), for a few parameters (~20%) are observed for all sample sizesPopulation Model fit to data generated from Population ModelTo what degree might the “relatively minor” model mis-specifications be responsible for these problems?Q1 (N?): and Q2 (p?) similar to findings from Theoretical Model except no problems with bias values and coverage values
10 IV. Importance Thank you. Ms is in press at RQES. MC methods have long been used to advance statistical theory. There have been several recent calls to use Monte Carlo methods as a tool to improve applications of quantitative methods in substantive research.Mplus code is available and can easily be alteredCES II – HST: N ≥ 200 for the theoretical modelN ≥ 300 for the population modela level of misfit that may be regarded as trivial in practice may have troubling effects on parameters of conceptual interestencourage sustained efforts toward generating closer approximations of population modelsThank you.Ms is in press at RQES.
11 Gentle, J. E. (2003). Random number generation and Monte Carlo methods (2nd ed.). New York: Springer.Marsh, H. W., Hau, K.-T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? Thenumber of indicators per factor in confirmatory factor analysis. Multivariate BehavioralResearch, 33,3. MacCallum, R. (2003). Working with imperfect models. Multivariate Behavioral Research, 38,Feltz, D.L., Chase, M.A., Moritz, S.E., & Sullivan, P.J. (1999). A conceptual model of coaching efficacy:Preliminary investigation and instrument development. Journal of Educational Psychology, 91,5. Bandura, A. (1997). Self-efficacy: The exercise of control. New York: Freeman.6. Denham, C.H., & Michael, J.J. (1981). Teacher sense of efficacy: A definition and a model for further research. Educational Research Quarterly, 5,7. Horn, T.S. (2002). Coaching effectiveness in the sport domain. In T.S. Horn (Ed.), Advances in sport psychology (2nd ed., pp ). Champaign, IL: Human Kinetics.8. Myers, N. D., Feltz, D. L., Chase, M. A., Reckase, M. D., & Hancock, G. R. (2008). TheCoaching Efficacy Scale II - High School Teams. Educational and PsychologicalMeasurement, 68,9. Paxton, P., Curran, P. J., Bollen, K. A., Kirby, J., & Chen, F. (2001). Monte Carlo experiments:Design and implementation. Structural Equation Modeling, 8,10. Muthén, L. K., & Muthén, B. O. (2002). How to use a Monte Carlo study to decide on samplesize and determine power. Structural Equation Modeling, 9,11. Muthén, L.K. and Muthén, B.O. ( ). Mplus User’s Guide (6th ed.). Los Angeles, CA:Muthén & Muthén.