Administrator Beware: Checking the Validity of Published Subscales Nancy D. Floyd University of South Carolina South Carolina Association for Institutional Research Columbia, SC March 5, 2012

Holland’s theory of vocational personality: six types Realistic Investigative Artistic Social Enterprising Conventional

Academic Disciplines Smart, Feldman & Ethington (2000) College freshmen completed CIRP Freshman Survey in 1986 Followed up with Senior Survey in 1990 Holland type estimated using subscales comprised of items on CIRP Freshman Survey Consistency of typing with both freshman major and graduating major studied, as well as changes during four years

CIRP sections used in Holland type estimation (from 1996 Freshman Survey) Rate yourself . . compared to the average person your age (1=Lowest 10%; 5=Highest 10%): Academic ability Artistic ability Competitiveness Cooperativeness Creativity Drive to achieve Emotional health Leadership ability Mathematical ability Physical health Popularity Public speaking ability Self-confidence (intellectual) Self-confidence (social) Self-understanding Spirituality Understanding of others Writing ability Please indicate the importance to you personally of . . : (1=Not Important; 4=Essential) Becoming accomplished in one of the performing arts (acting, dancing, etc.) Becoming an authority in my field Obtaining recognition from my colleagues for contributions to my special field Influencing the political structure Influencing social values Raising a family Having administrative responsibility for the work of others Being very well off financially Helping others who are in difficulty Making a theoretical contribution to science Writing original works (poems, novels, short stories, etc.) Creating artistic work (painting, sculpture, decorating, etc.) Becoming successful in a business of my own Becoming involved in programs to clean up the environment Developing a meaningful philosophy of life Participating in a community action program Helping to promote racial understanding Keeping up to date with political affairs Becoming a community leader

Smart & Associates’ (2000) Holland estimation Item scores converted to T(50,10) scores Items comprising a subscale are averaged Student’s highest average is their primary Holland type (Smart, Feldman & Ethington 2000) Subscales not constructed for Realistic and Conventional subscales; Smart et al had removed these from analysis

Sample Comparison USC 1996 incoming freshmen N=2309 Smart, Feldman & Ethington (2000) USC 1996 incoming freshmen N=2309 52.7 percent female N=2438 57.9 percent female 26.4 percent nonwhite 40.7 percent undeclared majors

Principal Axis Factoring, Varimax Rotation, no # of factors specified Principal Axis Factoring, Varimax Rotation, 4 factors specified Maximum Likelihood, Oblique Rotation, no # of factors specified Maximum Likelihood, Oblique Rotation, 4 factors specified Percent Variance Explained 42.9 percent 36.8 percent 43.9 percent 36.9 percent Number of Factors Extracted 6 4 Simple Factor Structure Of 26 items, 12 have a cross loading of at least .15; another 7 have two cross loadings of at least .15 Of 26 items, 10 have a cross loading of at least .15; another 4 have two cross loadings of at least .15 Of 26 items, 7 have a cross loading of at least .15; none have two cross loadings of at least .15 Of 26 items, 8 have a cross loading of at least .15; another 3 have two cross loadings of at least .15 Models Tested

Proposed Investigative Subscale Primary Loading Others Self-confidence (intellectual) -.533 on Negative Perceived Self-Worth .369 on General School Achievement Academic ability .803 on General School Achievement Drive to achieve .358 on General School Achievement Mathematical ability .541 on General School Achievement Making a theoretical contribution to science .316 on Prosocial Altruism .222 on Responsible Success Proposed Investigative Subscale

Proposed Artistic Subscale Primary Loading Others Artistic ability .578 on Artistic Orientation Becoming accomplished in one of the performing arts (dancing, acting, etc.) .550 on Artistic Orientation Writing original works (poems, novels, short stories, etc.) .616 on Artistic Orientation Creating artistic work (painting, sculpture, decorating, etc.) .742 on Artistic Orientation Writing ability .273 on Artistic Orientation, .259 on General School Achievement Developing a meaningful philosophy of life .496 on Prosocial Altruism .163 on Artistic Orientation Proposed Artistic Subscale

Proposed Social Subscale Primary Loading Others Influencing the political structure .517 on Community Influence Motivation Influencing social values .872 on Community Influence Motivation Helping others who are in difficulty .395 on Prosocial Altruism .197 on Community Influence Motivation Becoming involved in programs to clean up the environment .630 on Prosocial Altruism Participating in a community action program .759 on Prosocial Altruism Helping to promote racial understanding .632 on Prosocial Altruism Proposed Social Subscale

Proposed Enterprising Subscale Primary Loading Others Leadership ability -.443 on Negative Perceived Self-Worth Popularity -.628 on Negative Perceived Self-Worth Self-confidence (social) -.959 on Negative Perceived Self-Worth Becoming an authority in my field .683 on Responsible Success .164 on Artistic Orientation Obtaining recognition from my colleagues for contributions to my special field .697 on Responsible Success Having administrative responsibility for the work of others .497 on Responsible Success Being very well off financially .551 on Responsible Success -.204 on Artistic Orientation Becoming successful in a business of my own .353 on Responsible Success .154 on Prosocial Altruism Becoming an expert on finance and commerce NA: not on 1996 CIRP Freshman Survey Proposed Enterprising Subscale

Implications If factor structure didn’t hold for one institution that should have been similar to development sample for Smart, Feldman & Ethington, have to question validity of original subscales. If original subscales are invalid, one of two things: our sample is not representative of research sample, or the factors were not produced correctly or tested enough. If the sample is not representative, why am I buying CIRP and NSSE like it is? If the factors were not produced correctly or tested enough, can I have faith in other factor subscales? If I can’t get original response data back from a survey developer, I can’t check factor structure to see if it holds.

Questions? nfloyd@mailbox.sc.edu