1. Administrator Beware: Checking the Validity of Published Subscales
Nancy D. Floyd
University of South Carolina
South Carolina Association for Institutional Research
Columbia, SC March 5, 2012

2. Holland’s theory of vocational personality: six types
Realistic
Investigative
Artistic
Social
Enterprising
Conventional

3. Academic Disciplines Smart, Feldman & Ethington (2000)
College freshmen completed CIRP Freshman Survey in 1986
Followed up with Senior Survey in 1990
Holland type estimated using subscales comprised of items on CIRP Freshman Survey
Consistency of typing with both freshman major and graduating major studied, as well as changes during four years

4. CIRP sections used in Holland type estimation (from 1996 Freshman Survey)
Rate yourself . . compared to the average person your age (1=Lowest 10%; 5=Highest 10%):
Academic ability
Artistic ability
Competitiveness
Cooperativeness
Creativity
Drive to achieve
Emotional health
Leadership ability
Mathematical ability
Physical health
Popularity
Public speaking ability
Self-confidence (intellectual)
Self-confidence (social)
Self-understanding
Spirituality
Understanding of others
Writing ability
Please indicate the importance to you personally of . . : (1=Not Important; 4=Essential)
Becoming accomplished in one of the performing arts (acting, dancing, etc.)
Becoming an authority in my field
Obtaining recognition from my colleagues for contributions to my special field
Influencing the political structure
Influencing social values
Raising a family
Having administrative responsibility for the work of others
Being very well off financially
Helping others who are in difficulty
Making a theoretical contribution to science
Writing original works (poems, novels, short stories, etc.)
Creating artistic work (painting, sculpture, decorating, etc.)
Becoming successful in a business of my own
Becoming involved in programs to clean up the environment
Developing a meaningful philosophy of life
Participating in a community action program
Helping to promote racial understanding
Keeping up to date with political affairs
Becoming a community leader

5. Smart & Associates’ (2000) Holland estimation
Item scores converted to T(50,10) scores
Items comprising a subscale are averaged
Student’s highest average is their primary Holland type (Smart, Feldman & Ethington 2000)
Subscales not constructed for Realistic and Conventional subscales; Smart et al had removed these from analysis

6. Sample Comparison USC 1996 incoming freshmen N=2309
Smart, Feldman & Ethington (2000)
USC 1996 incoming freshmen
N=2309
52.7 percent female
N=2438
57.9 percent female
26.4 percent nonwhite
40.7 percent undeclared majors

7. Principal Axis Factoring, Varimax Rotation, no # of factors specified
Principal Axis Factoring, Varimax Rotation, 4 factors specified
Maximum Likelihood, Oblique Rotation, no # of factors specified
Maximum Likelihood, Oblique Rotation, 4 factors specified
Percent Variance Explained
42.9 percent
36.8 percent
43.9 percent
36.9 percent
Number of Factors Extracted 6 4
Simple Factor Structure
Of 26 items, 12 have a cross loading of at least .15; another 7 have two cross loadings of at least .15
Of 26 items, 10 have a cross loading of at least .15; another 4 have two cross loadings of at least .15
Of 26 items, 7 have a cross loading of at least .15; none have two cross loadings of at least .15
Of 26 items, 8 have a cross loading of at least .15; another 3 have two cross loadings of at least .15
Models Tested

8. Proposed Investigative Subscale
Primary Loading
Others
Self-confidence (intellectual)
-.533 on Negative Perceived Self-Worth
.369 on General School Achievement
Academic ability
.803 on General School Achievement
Drive to achieve
.358 on General School Achievement
Mathematical ability
.541 on General School Achievement
Making a theoretical contribution to science
.316 on Prosocial Altruism
.222 on Responsible Success
Proposed Investigative Subscale

9. Proposed Artistic Subscale
Primary Loading
Others
Artistic ability
.578 on Artistic Orientation
Becoming accomplished in one of the performing arts (dancing, acting, etc.)
.550 on Artistic Orientation
Writing original works (poems, novels, short stories, etc.)
.616 on Artistic Orientation
Creating artistic work (painting, sculpture, decorating, etc.)
.742 on Artistic Orientation
Writing ability
.273 on Artistic Orientation, .259 on General School Achievement
Developing a meaningful philosophy of life
.496 on Prosocial Altruism
.163 on Artistic Orientation
Proposed Artistic Subscale

10. Proposed Social Subscale
Primary Loading
Others
Influencing the political structure
.517 on Community Influence Motivation
Influencing social values
.872 on Community Influence Motivation
Helping others who are in difficulty
.395 on Prosocial Altruism
.197 on Community Influence Motivation
Becoming involved in programs to clean up the environment
.630 on Prosocial Altruism
Participating in a community action program
.759 on Prosocial Altruism
Helping to promote racial understanding
.632 on Prosocial Altruism
Proposed Social Subscale

11. Proposed Enterprising Subscale
Primary Loading
Others
Leadership ability
-.443 on Negative Perceived Self-Worth
Popularity
-.628 on Negative Perceived Self-Worth
Self-confidence (social)
-.959 on Negative Perceived Self-Worth
Becoming an authority in my field
.683 on Responsible Success
.164 on Artistic Orientation
Obtaining recognition from my colleagues for contributions to my special field
.697 on Responsible Success
Having administrative responsibility for the work of others
.497 on Responsible Success
Being very well off financially
.551 on Responsible Success
-.204 on Artistic Orientation
Becoming successful in a business of my own
.353 on Responsible Success
.154 on Prosocial Altruism
Becoming an expert on finance and commerce
NA: not on 1996 CIRP Freshman Survey
Proposed Enterprising Subscale

12. Implications
If factor structure didn’t hold for one institution that should have been similar to development sample for Smart, Feldman & Ethington, have to question validity of original subscales.
If original subscales are invalid, one of two things: our sample is not representative of research sample, or the factors were not produced correctly or tested enough.
If the sample is not representative, why am I buying CIRP and NSSE like it is?
If the factors were not produced correctly or tested enough, can I have faith in other factor subscales?
If I can’t get original response data back from a survey developer, I can’t check factor structure to see if it holds.

13. Questions?