Download presentation

Presentation is loading. Please wait.

Published byJocelyn Martinez Modified over 3 years ago

1
Common Factors versus Components: Principals and Principles, Errors and Misconceptions Keith F. Widaman University of California at Davis Presented at conference “Factor Analysis at 100” L. L. Thurstone Psychometric Lab, University of North Carolina at Chapel Hill, May 2004

2
**Goal of the Talk Flip rendition More serious**

(With apologies to Will) I come not to praise principal components, but to bury them Thus, we might inter the procedure beside its creator More serious To outline several key assumptions, usually implicit, of the “simpler” principal components approach Compare and contrast common factor analysis and principal component analysis

3
**Organization of the Talk**

Principals Major figures/events Important dimensions – factors/components Principles To organize our thinking Lead to methods to evaluate procedures Errors Structures of residuals Unclear presentations Misconceptions

4
**Principal Individuals & Contributions**

Spearman (1904) First conceptualization of the nature of a common factor – the element in common to two or more indicators (preferably three or more) Stressed presence of two classes of factors – general (with one member) and specific (with a potentially infinite number) Key: Based evaluation of empirical evidence on the tetrad difference criterion (i.e., on patterns in correlations among manifest variables) with no consideration of diagonal

5
**Principal Individuals & Contributions**

Thomson (1916) Early recognition of elusiveness of theory – data connection Single common factor implies hierarchical pattern of correlations, but so does an opposite conceptualization Key for this talk: Focus was still on the patterns displayed by off-diagonal correlation. Diagonal elements were of no interest or importance

6
**Principal Individuals & Contributions**

Thurstone (1931) First foray into factor analysis Devised a “center of gravity” method for estimation of loadings Led to centroid method Key: Again, diagonal values explicitly disregarded

7
**Principal Individuals & Contributions**

Hotelling (1933) Proposed method of principal components Method of estimation Least squares Decomposition of all of the variance of the manifest variables into dimensions that are: (a) orthogonal (b) conditionally variance maximized Key 1: Left unities on diagonal Key 2: Interpreted unrotated solution

8
**Principal Individuals & Contributions**

Thurstone (1935) – The Vectors of Mind “It is a fundamental criterion for a valid method of isolating primary abilities that the weights of the primary abilities for a test must remain invariant when it is moved from one test battery to another test battery.” “If this criterion is not fulfilled, the psychological description of a test will evidently be as variable as the arbitrarily chosen batteries into which the test may be placed. Under such conditions no stable identification of primary mental abilities can be expected.”

9
**Principal Individuals & Contributions**

Thurstone (1935) This implies invariant factorial description of a test (a) across batteries and (b) across populations Again, diagonal values explicitly disregarded Developed rationale for necessity for rotation Contra Hotelling: Unities on diagonal – imply manifest variables are perfectly reliably Need for # dimensions = # manifest variables No rotation! This appears, to me, to be the most important criticism of Hotelling by Thurstone.

10
**Principal Individuals & Contributions**

McCloy, Metheny, & Knott (1938) Published in Psychometrika Sought to compare Common FA (Thurstone’s method) vs. Principal Components Analysis (Hotelling’s method) Perhaps the first comparison of the two methods

11
**Principal Individuals & Contributions**

Thomson (1939) Clear statement of the differing aims of Common factor analysis – to explain the off-diagonal correlations among manifest variables Principal component analysis – to re-represent the manifest variables in a mathematically efficient manner

12
**Principal Individuals & Contributions**

Guttman (1955, 1958) Developed lower bounds for the number of factors Weakest lower bound was number of “factors” with eigenvalues greater than or equal to unity With unities on diagonal With population data Other bounds used other diagonal elements (e.g., strongest lower bound used SMCs), but these did not work as well

13
**Principal Individuals & Contributions**

Kaiser (1960, 1971) Described the origin of the Little Jiffy Principal components Retain components with eigenvalues >= 1.0 Rotate using varimax Later modifications – Little Jiffy Mark IV – offered important improvements, but were not followed

14
**Principles – Mislaid or Forgotten**

Principle 1: Common factor analysis and principal component analysis have different goals – à la Thomson (1939) Common factor analysis – to explain the off-diagonal correlations among manifest variables Principal component analysis – to re-represent the original variables in a mathematically efficient manner (a) in reduced dimensionality, or (b) using orthogonal, conditionally variance maximized way

15
**Principles – Mislaid or Forgotten**

Principle 2: Common factor analysis was as much a theory of manifest variables as a theory of latent variables Spearman – doctrine of the indifference of the indicator, so any manifest variable was a more-or-less good indicator of g Thurstone – test one’s theory by developing new variables as differing mixtures of factors and then attempt to verify presumptions Today, focus seems largely on the latent variables Forgetting about manifest variables can be problematic

16
**Principles – Mislaid or Forgotten**

Principle 3: Invariance of the psychological/ mathematical description of manifest variables is a fundamental issue “It is a fundamental criterion for a valid method of isolating primary abilities that the weights of the primary abilities for a test must remain invariant when it is moved from one test battery to another test battery” Much work on measurement & factorial invariance But, only similarities between common factors and principal components are stressed; differences are not emphasized

17
**Principles – Mislaid or Forgotten**

Principle 4: Know data and model Should know relation between data and model Should know all assumptions (even implicit) of model Frequently told: information in correlation matrix is difficult to discern so, don’t look at data run it through FA or PCA interpret the results This is not justifiable!

18
**Common FA & Principal CA Models**

Common Factor Analysis R = FF’ + U2 = PΦP’ + U2 where R is (p x p) correlation matrix among manifest vars F is a (p x k) unrotated factor matrix, with loadings of p manifest variables on k factors U2 is a (p x p) matrix (diagonal) of unique factor variances P is a (p x k) rotated factor matrix, with loadings of p manifest variables on k rotated factors Φ is a (k x k) matrix of covariances among factors (may be I, usually diag = I)

19
**Common FA & Principal CA Models**

Principal Component Analysis R = FcFc’ = PcΦcPc’ R = FcFc’ + GG’ = PcΦcPc’ + GG’ R = FcFc’ + Δ = PcΦcPc’ + Δ where Fc, Pc, & Φc have same order as like-named matrices for CFA, but with c subscript to denote PCA G is a (p x [p-k]) matrix of loadings of p manifest variables on the (p-k) discarded components Δ (= GG’) is a (p x p) matrix of covariances among residuals

20
**Present Day: Advice to Practicing Scientist**

Velicer & Jackson (1990): CFA vs. PCA Four “practical” issues Similarity between solutions Issues related to # of dimensions to retain Improper solutions in CFA Differences in computational efficiency Three “theoretical” issues Factorial indeterminacy in CFA, not PCA CFA can be used in exploratory and confirmatory modes, PCA only exploratory CFA is latent procedure, PCA is manifest

21
**Present Day: Advice to Practicing Scientist**

Goldberg & Digman (1994) and Goldberg & Velicer (in press): CFA vs. PCA Results from CFA and PCA are so similar that differences are unimportant If differences are large, “data are not well-structured enough for either type of analysis” Use “factor” to refer to factors and components Aim is to explain correlations among manifest vars

22
**Present Day: Quantitative Approaches**

Recent paper in Psychometrika (Ogasawara, 2003) Based work on oblique factors & components with: Equal number of indicators per dimension Independent cluster solution Sphericity (equal “error” variances), hence equal factor loadings Derived expression for SEs (standard errors) for factor and component loadings and intercorrelations SEs for PCA estimates were smaller than those for CFA estimates, implying greater stability of (i.e., lower variability around) population estimates

23
An Apocryphal Example Researcher wanted to develop a new inventory to assess three cognitive traits Knew to collect data in at least two initial, derivation samples Use exploratory procedures to verify initial, a priori hypotheses Then, move on to confirmatory techniques So, Sample 1, N = 1600, and 8 manifest variables 3 Components explain 51% of total variance

24
**Oblique Components, Sample 1**

Variable Fac 1 Fac 2 Fac h V – V – V – N N2 – N3 – – S1 – S2 – Fac 1 1.0 Fac Fac

25
**Orthogonal Components, Sample 1**

Variable Fac 1 Fac 2 Fac h2 . V V V N N N3 – – S S Fac 1 1.0 Fac Fac

26
An Apocryphal Example After confirming a priori hypotheses in Sample 1, the researcher collected data from Sample 2 Same manifest variables Sampled from the same general population Same mathematical approach – principal components followed by oblique and orthogonal rotation Got same results! Decided to “test” the theory in Sample 3 – using “replicate and extend” approach Major change: Switch to Confirmatory Factor Analysis

27
**Confirmatory Factor Analysis, Sample 3**

Variable Fac 1 Fac 2 Fac θ V (.18) V (.21) V (.25) N (.13) N (.14) N (.16) S (.44) S (.50) Fac 1 1.0 Fac (.04) 1.0 Fac (.10) (.10) 1.0

28
**Fully Standardized Solution, Sample 3**

Variable Fac 1 Fac 2 Fac h V V V N N N S S Fac 1 1.0 Fac Fac

29
**Oblique Component Solution, Sample 3**

Variable Fac 1 Fac 2 Fac h V – V – V – N N2 – N3 – – S1 – S2 – Fac 1 1.0 Fac Fac

30
**An Early Comparison McCloy, Metheny, & Knott (1938)**

Published in Psychometrika Sought to compare Common FA (Thurstone’s method) vs. Principal Components Analysis (Hotelling’s) Stated that Principal Components can be rotated So, both techniques are different means to same end Principal difference: Thurstone inserts largest correlation in row in the diagonal of each residual matrix Hotelling begins with unities and stays with residual values in each residual matrix

31
**Hypothetical Factor Matrix (McCloy et al.)**

Variable Fac 1 Fac 2 Fac h2 .

32
**Rotated Factor Matrix (McCloy et al.)**

Variable Fac 1 Fac 2 Fac h2 . – – – 6 – 7 – – –

33
**Rotated Component Matrix (McCloy et al.)**

Variable Fac 1 Fac 2 Fac h2 . – – – – 5 – 6 –.094 – 7 – –

34
**An Early Comparison McCloy, Metheny, & Knott (1938) Argued that:**

both CFA and PCA were means to same end both led to similar pattern of loadings, but Thurstone’s method was more accurate (Δh2 = .056) than Hotelling’s (Δh2 = .125) – [but these were average absolute differences] I averaged signed differences, and Thurstone’s method was much accurate (Δh2 = -.013) than Hotelling’s (Δh2 = .120)

35
**An Early Comparison McCloy, Metheny, & Knott (1938) Tukey (1969)**

Found similar pattern of high and low loadings from PCA and CFA But, they found (but did not stress) that PCA led to decidedly higher loadings Tukey (1969) Amount, as well as direction, is vital For any science to advance, we must pay attention to quantitative variation, not just qualitative

36
**Regularity Conditions or Phenomena**

Relations between population values of P and R Features of eigenvalues Covariances among residuals Need a “theory” of “errors” Recount my first exposure … Should have to acknowledge (predict? live with?) the patterns in residuals

37
**Practicing Scientists vs. “Statisticians”**

Interesting dimension along which researchers fall: Practicing “Statisticians” scientists (Dark side) use CFA prefer PCA use regression warn of probs analysis errors in vars

38
**Practicing Scientists vs. “Statisticians”**

At first seems odd Practicing scientist prefers CFA (which partials out errors of measurement and specific variance) Regression analysis – despite the implicit assumption of “perfect” measurement “Statistician” prefers To warn of ill-effects of errors in variables on results of regression analysis PCA (despite lack of attention to measurement error), perhaps due to elegant, reduced rank representation

39
**Practicing Scientists vs. “Statisticians”**

On second thought, is rational: Practicing scientist prefers Assumptions that residuals (in CFA or regression analysis) are independent, uncorrelated, normally distributed “Statistician” prefers To try to circumvent (or solve) problem of errors in variables in regression To relegate “errors in variables” problems in PCA to that part of solution (GG’) that is orthogonal to the retained part, thereby circumventing (or solving) this problem

40
**Regularity Conditions or Phenomena**

In Common Factor Analysis, Char. of correlations Char. of variables 1:1 Char. of correlations Char. of variables 1:1 In Principal Component Analysis, Char. of correlations Char. of variables 1:1 (??) Char. of correlations ~[] Char. of variables many:1

41
**Manifest Correlations**

Var V1 V2 V3 V4 V5 V6 V V V V V V

42
**Eigenvalues, Loadings, and Explained Variance**

Var EV P1 P2 h2 EVc Pc1 Pc2 hc2 V V V V4 V5 V6 P P2

43
**Residual Covariances: CFA**

Var V1 V2 V3 V4 V5 V6 V V V V4 V5 V6 Covs below diag., corrs above diag.

44
**Residual Covariances: PCA**

Var V1 V2 V3 V4 V5 V6 V V V V4 V5 V6 Covs below diag., corrs above diag.

45
**Eigenvalues, Loadings, and Explained Variance**

Var EV P1 P2 h2 EVc Pc1 Pc2 hc2 V V V V V V P P2

46
**Residual Covariances: CFA**

Var V1 V2 V3 V4 V5 V6 V V V V V V Covs below diag., corrs above diag.

47
**Residual Covariances: PCA**

Var V1 V2 V3 V4 V5 V6 V V V V V V Covs below diag., corrs above diag.

48
**Regularity Conditions or Phenomena**

In Common Factor Analysis, If (a) the model fits in the population, (b) there is one factor, and (c) communalities are estimated optimally, Single non-zero eigenvalue Factor loadings and residual variances for first three variables are unaffected by addition of 3 “identical” variables Residuals = specific + error variance Residual matrix is diagonal

49
**Regularity Conditions or Phenomena**

In Principal Component Analysis, If (a) the common factor model fits in the population, (b) there is one factor, and (c) unities are retained on the main diagonal, Single large eigenvalue, plus (p – 1) identical, smaller eigenvalues Residual component matrix G is independent of the space defined by Fc But, residual covariance matrix is clearly non-diagonal And, (a) “population” component loadings and (b) residual variances and covariances vary as a function of number of manifest variables!

50
**Manifest Correlations**

Var V1 V2 V3 V4 V5 V6 V V V V V V

51
**Eigenvalues, Loadings, and Explained Variance**

Var EV P1 P2 h2 EVc Pc1 Pc2 hc2 V V V V4 V5 V6 P P2

52
**Residual Covariances: CFA**

Var V1 V2 V3 V4 V5 V6 V V V V4 V5 V6 Covs below diag., corrs above diag.

53
**Residual Covariances: PCA**

Var V1 V2 V3 V4 V5 V6 V V V V4 V5 V6 Covs below diag., corrs above diag.

54
**Eigenvalues, Loadings, and Explained Variance**

Var EV P1 P2 h2 EVc Pc1 Pc2 hc2 V V V V V V P P2

55
**Residual Covariances: CFA**

Var V1 V2 V3 V4 V5 V6 V V V V V V Covs below diag., corrs above diag.

56
**Residual Covariances: PCA**

Var V1 V2 V3 V4 V5 V6 V V V V V V Covs below diag., corrs above diag.

57
**Regularity Conditions or Phenomena**

So, the difference between “population” parameters from CFA and PCA diverge more: (a) the fewer the number of indicators per dimension, and (b) the lower the true communality But, some regularities still seem to hold (although these vary with the number of indicators) “regular” estimates of loadings “regular” magnitude of residual covariance “regular” form of eigenvalue structure

58
**Regularity Conditions or Phenomena**

But, what if we have variation in loadings?

59
**Manifest Correlations**

Var V1 V2 V3 V4 V5 V6 V V V V V V

60
**Eigenvalues, Loadings, and Explained Variance**

Var EV P1 P2 h2 EVc Pc1 Pc2 hc2 V V V V V V P P2

61
**Residual Covariances: CFA**

Var V1 V2 V3 V4 V5 V6 V V V V V V Covs below diag., corrs above diag.

62
**Residual Covariances: PCA**

Var V1 V2 V3 V4 V5 V6 V V V V V V Covs below diag., corrs above diag.

63
**Regularity Conditions or Phenomena**

So, with variation in loadings One piece of approximate stability “regular” estimates of loadings But, sacrifice “regular” magnitude of residual covariance “regular” form of eigenvalue structure

64
**Regularity Conditions or Phenomena**

But, what if we have multiple factors? Let’s start with (a) equal loadings (b) orthogonal factors

65
**Eigenvalues, Loadings, and Explained Variance**

Var EV P1 P2 h2 EVc Pc1 Pc2 hc2 V V V V V V P P

66
**Residual Covariances: PCA**

Var V1 V2 V3 V4 V5 V6 V V V V V V Covs below diag., corrs above diag.

67
**Regularity Conditions or Phenomena**

So, “strange” result: Same factor inflation as with 1-factor, 3 indicators Same within-factor residual covariances as for 1-factor, 3 indicators But, between-factor residual covariances = 0! Let’s go to (a) equal loadings, but (b) oblique factors

68
**Eigenvalues, Loadings, and Explained Variance**

Var EV P1 P2 h2 EVc Pc1 Pc2 hc2 V V V V V V P P

69
**Residual Covariances: PCA**

Var V1 V2 V3 V4 V5 V6 V V V V V V Covs below diag., corrs above diag.

70
**Regularity Conditions or Phenomena**

So, “strange” result: Same factor inflation as with 1-factor, 3 indicators Reduced correlation between factors But, residual covariances matrix is identical! Let’s go to (a) unequal loadings, and (b) orthogonal factors

71
**Eigenvalues, Loadings, and Explained Variance**

Var EV P1 P2 h2 EVc Pc1 Pc2 hc2 V V V V V V P P

72
**Residual Covariances: PCA**

Var V1 V2 V3 V4 V5 V6 V V V V V V Covs below diag., corrs above diag.

73
**Regularity Conditions or Phenomena**

So, “strange” result: Different factor inflation than with 1-factor, 3 indicators Reduced correlation between factors But, residual covariances matrix has unequal covariances and correlations among residuals, but between-factor covariances = 0! Let’s go to (a) unequal loadings, and (b) oblique factors

74
**Eigenvalues, Loadings, and Explained Variance**

Var EV P1 P2 h2 EVc Pc1 Pc2 hc2 V V V V V V P P

75
**Residual Covariances: PCA**

Var V1 V2 V3 V4 V5 V6 V V V V V V Covs below diag., corrs above diag.

76
**Regularity Conditions or Phenomena**

So, “strange” result: Extremely different factor inflation than with 1-factor, 3 indicators Largest loading is now UNderrepresented Very different population factor loadings (.8, .6, & .4) have very similar component loadings Now, between-factor covariances are not zero, and some are positive!

77
**R from Component Parameters**

All the preceding from a CFA view: Develop parameters from a CF model Analyze using CFA and PCA CFA procedures recover parameters PCA procedures exhibit failings or anomalies So What? What else could you expect? Challenge (to me): Generate data from a PC model PCA should recover parameters, CFA should exhibit problems and/or anomalies

78
**R from Component Parameters**

Difficult to do Leads to Impractical, unacceptable outcomes, from the point of view of the practicing scientist Crucial indeterminacies with the PCA model

79
**R from Component Parameters**

Impractical, unacceptable outcomes, from the point of view of the practicing scientist

80
**Manifest Correlations**

Var V1 V2 V3 V4 V5 V6 V V V V4 V5 V6 First principal component has 3 loadings of .8 First principal factor has 3 loadings of (.46)1/2, or about .67

81
**Manifest Correlations**

Var V1 V2 V3 V4 V5 V6 V V V V V V First principal component has 6 loadings of .8 First principal factor has 6 loadings of (.568)1/2, or about .75 But, one would have to alter the first 3 tests, as their “population” correlations are altered

82
**R from Component Parameters**

Crucial indeterminacies with the PCA “model” Consider case of well-identified CFA model: 6 manifest variables loading on a single factor One could easily construct the population matrix as FF’ + uniquenesses to ensure diag(R) = I With 6 manifest variables, 6(7)/2 = 21 unique elements of covariance matrix 12 parameter estimates therefore 9 df

83
**R from Component Parameters**

Crucial indeterminacies with the PCA “model” Consider now 6 manifest variables with defined loadings on first PC To estimate the correlation matrix, must come up with the remaining 5 PCs A start: [Fc | G]’ [Fc | G] = diag, so orthogonality constraint yields 6(5)/2 = 15 equations Sum of squares across rows = 1, so 6 more equations In short, 15 equations, but 30 unknowns (loadings of 6 variables on the 5 components in G) Therefore, an infinite # of R matrices will lead to the stated first PC

84
**R from Component Parameters**

Crucial indeterminacies with the PCA “model” Related to the Ledermann number, but in reverse For example, with 10 manifest variables, one can minimally overdetermine no more than 6 factors (so use 6 or fewer factors) But, here, one must specify at least 6 components (to ensure more equations than unknowns) to ensure a unique R If fewer than 6 components are specified, an infinite number of solutions for R can be found

85
**Conclusions: CFA CFA factor models may not hold in the population**

But, if they do (in a theoretical population): The notion of a population factor loading is realistic The population factor loading is unaffected by presence of other variables, as long as the battery contains the same factors In one-factor case, loadings can vary from 0 to 1 (provided reflection of variables is possible) This generalizes to the case of multiple factors

86
**Conclusions: CFA CFA factor models may not hold in the population**

But, if they do: Residual (i.e., unique) variances are uncorrelated Magnitude of unique variance for a given variable is unaffected by other variables in the analysis

87
Conclusions: PCA PCA factor models cannot hold in the population (because all variables have measurement error) Moreover: The notion of “the population component loading” for a particular manifest variable is meaningless The “population” component loading is affected strongly by presence of other variables SEs for component loadings have no interpretation In the one-component case, component loadings can only vary from (1/m)1/2 to 1, where m is the number of indicators for the dimension Generalizes to multiple component case

88
Conclusions: PCA PCA factor models cannot hold in the population (because all variables have measurement error) Moreover: Residual variables are correlated, often in unpredictable and seemingly haphazard fashion Magnitude of unique variance and covariances for a given manifest variable are affected by other variables in the analysis

89
Conclusions: PCA PCA factor models cannot hold in the population (because all variables have measurement error) Moreover: Finally, generating data from a PC model leads either to Impractical, unacceptable outcomes Indeterminacies in the parameter – R relations

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google