An example of a CFA: Evaluating an existing scale

Normal operating procedures Most people who do research, search for existing measures and if they find one that measures one of their constructs, then they just use it. I would encourage you to be more critical. In particular, did the author of the scale do the proper groundwork? I will take you through an example which I hope will convince you that one benefits from being skeptical and careful.

Evaluation of the Spector Work Locus of Control Scale This scale measures attitudes about how much control a worker has in his/her work situation. There is a large literature on “internal and external loci of control”, and this measure is a specific example of this type of scale. Spector wrote the scale so that half of the items are “external” items: “promotion is usually a matter of good fortune”, and half are “internal” items: “promotions are given to employees who perform well on the job”. What happens if we do a Cronbach’s alpha on the entire scale (16 items)?

Results All 16 items:  =.597. One strives to obtain an alpha of at least.70, so this is subpar. However, does this make sense? Spector felt that there were two discriminable factors, so we should really do the internal reliability analyses on the two factors separately: –Internal (8 items):  =.828; and –External (8 items):  =.813. That’s what we would expect.

One factor, or more? Spector originally designed the scale so that the external items are reverse-scored and then added to the internal items. What does this mean? It suggests that he thinks that there is a single factor operating here. A simple test can determine whether these two clumps are really a single factor: a correlation. If the correlation is highly significant AND negative, then you would have evidence for this point of view. What did we find? I obtained: Pearson r(145) = -.360, p <.001. It is negative, but not “highly significant”. This should cast some doubt on his suggestion.

Psychometric work And let me point out that he didn’t perform even an exploratory factor analysis on the scale when he first published a research paper on it (Spector, 1988). Were psychometric techniques like factor analysis available in 1988? Yes, they were. Bear this in mind.

If you do an EFA, what do you find? We did a PCA with varimax rotation. Expected to find that two factors was optimal: what did we find?

1. Promotion is usually a matter of good fortune Making money is primarily a matter of good fortune The main difference between people who make a lot of money and people who make a little money is luck Getting the job you want is mostly a matter of luck In order to get a really good job you need to have family members or friends in really high places To make a lot of money you have to know the right people When it comes to landing a really good job, who you know is more important than what you know It takes a lot of luck to be an outstanding employee on most jobs Most people are capable of doing their jobs well if they make the effort A job is what you make of it On most jobs, people can pretty much accomplish whatever they set out to achieve If you know what you want out of a job, you can find a job that gives it to you If employees are unhappy with a decision made by their boss, they should do something about it People who perform their jobs well generally get rewarded for it Promotions are given to employees who perform well on the job Most employees have more influence on their supervisors than they think Table 2. Rotated Pattern matrix of Work Locus of Control Scale PCA. ItemFactor 1 (External) Factor 2 (Internal 1) Factor 3 (Internal 2)

Then we did a CFA in LISREL We examined three models: –One factor; –Two factors (Int and Ext); and –Three factors (from the previous page). A CFA gives you a number of “model fit indices” to tell you how well the data fit the proposed model. How did they turn out?

Model fit indices Table 5. Fit indices for the WLCS* factor models for Study 2 Fit Indices Model 22 df  2 / df GFINNFIRMRRMSEA One factor model Two factor model Three factor model *WLCS; Work Locus of Control Scale, GFI; Goodness-of-Fit Index, NNFI; Non-Normed Fit Index, RMR; Root Mean Square Residual, RMSEA; Root Mean Square Error of Approximation

Three factor model looks best The model fit indices are superior (and mostly acceptable) for the 3- factor solution. This is strong evidence to say that the scale is psychometrically flawed. Why would one factor (the Internal factor) fragment and fall apart? Heterogeneity of item meanings would be my guess. Joe Oliver and I are trying to get a paper published which reports these difficulties, but Spector is not too happy. I wonder why.

Implications for research? If you combine Int with reverse- coded Ext, then you will have a poor factor (remember, they are only moderately negatively correlated). If you use the 8 Int and the 8 Ext items, then you are using a deficient Int factor. Future directions? Someone needs to write a good I-E scale for the workplace.

How does one derive a good scale? 1.Begin with qualitative data—ask workers about ways in which they feel in control or not. 2.Write a whole bunch of questions (I would start with about 40). 3.Do a pretest with a large sample ( ?), and select items that are highly intercorrelated with each other in these two domains. 4.Do another data collection with about 24 items (12 each), and select the top 8 or 10 in each domain. Verify that they clump into two discriminable factors with EFA. 5.Obtain another large dataset, and perform a CFA. If you’ve done your work well, the CFA will confirm that you have a good factor structure. 6.Now publish. 7.Sit back and wait for the royalty checks to roll in.

Moral of the story Almost no one does what was described on the previous page (except people who publish their tests to make money). Most people write some questions, collect some data, do an EFA, and publish those results. Why are we surprised when we find that the measure doesn’t work psychometrically? If you create a scale, do the groundwork.