Presentation on theme: "Real and Imaginary Concerns in Assessing College Readiness Nathan R. Kuncel Department of Psychology."— Presentation transcript:
Real and Imaginary Concerns in Assessing College Readiness Nathan R. Kuncel Department of Psychology
Favored Career Choices: In Rank Order 1.Jedi Knight 2.Star Fleet Captain 3.Batman 4.Playboy Millionaire 5.Psychology Professor 6.Evolutionary Biologist 7.Economist 8.Brilliant but Misunderstood Novelist
Jedi Knight Innate heritable ability, unrelated to social class, focused through mentoring and intense training, high self control
Star Fleet Captain Born but seasoned leaders who succeed through managed team effectiveness, charismatic and bold
Batman Single formative childhood experience that motivates massive skill investment, augmented by tremendous family wealth, intense and disagreeable
Professor of Psychology Masochistic enthusiasm for studying nearly intractable research questions, fairly friendly
Stories We Tell About Achieving Success Predictable versus Unpredictable Book Smarts versus Street Smarts General versus Specific Aptitudes Social Class Dependent versus Not Talent versus Effort Linear versus Asymptotic
Stories about Tests –Tests don’t even predict grades well –Tests don’t add anything to prediction –Tests don’t predict anything other than grades –Cognitive tests only predict academic outcomes –Cognitive tests are only proxies for SES –Beyond a certain point, scores don’t matter Stories about Traditional Alternatives –Letters of Recommendation –Personal Statements –Interviews Frightening Cautionary Tales about the Future –Faking –Bias –Unreliability
Assertion: Tests Don’t Even Predict Grades Well What does “well” mean and how do we quantify it?
Zombie Plague Plague sweeps the world Plague transforms 60% of infected people into horrifying and mindless monsters.
A Partial Treatment Is Developed! ZombieLive No Treatment 60%40% Treatment 40%60% What is the correlation?
Tests and Grades Berry and Sackett (2009) examined SAT-grade r ’ s in a sample of 165,000 students from 41 schools Corrected for restriction of range using school- specific applicant pools Obtained individual course grades for each course, and computed validity for each course (148,072 validity coefficients) –This removes influence of student choice of courses –Estimate r between SAT and common course portfolio
Berry and Sackett (2009) results SAT-First Year College GPA r observed.36 corrected for range restriction.46 corrected for course difficulty.55
Story: Tests Don’t Add Anything to Prediction
Bridgeman, Burton, & Cline (2009) Percent Earning a 3.8 or Higher Graduate GPA
Story: Tests only predict 1 st year grades Paper submitted to American Psychologist: “ As is well known, the SAT predicts nothing but first year grades ” [with no citation]
Evidence Across All Admissions Tests: Kuncel & Hezlett (2007) Science 7 Admissions Tests –GRE-T –GRE-S –MCAT –LSAT –GMAT –MAT –PCAT 8 Outcomes –1 st Year GPA –Graduate GPA –Faculty Ratings –Degree Attainment –Citation Counts –Research Productivity –Quals./Comps. Exams –Licensing Exams Results are based on over 600,000 students across over 3,000 independent samples
Kuncel & Hezlett (2007). Science, 315,
If the critics were right….
Book Smarts vs. Street Smarts or Academic vs. Practical Intelligence or Book Learning vs. Common Sense
Book Smarts vs. Street Smarts Fundamental argument is that different types of intelligence are needed in academic versus real world contexts It is argued that academic contexts require an “academic intelligence” or book smarts It is my belief that these arguments are based (in part) on an overly narrow conceptualization of the tasks that comprise academic performance
Academic versus Practical Tasks Academic Formulated by others Well-defined Presents complete information Single method for obtaining answer Not embedded in ordinary experience Little or no intrinsic interest Practical Requires problem recognition and formulation Ill-defined Requires information seeking Multiple acceptable solutions Multiple paths to solution Embedded in ordinary experience Requires motivation and personal involvement
Things Students Do Before Taking a Test Determine study objectives and methods Manage goal conflicts Coordinate work with other classmates Seek additional information or materials Handle school related finances Negotiate with peers and faculty Avoid counterproductive school behaviors Structure effective communications
A Model of Undergraduate Student Performance Dimensions Traditional Classroom Success Written and Oral Communication Personal Discipline Resolving Goal Conflicts Studying and Learning Proficiency Sustained Goal Directed Effort Interactive Learning and Team Performance Administration Interpersonal Proficiency Non-Classroom Performance Development of Life Goals and Values Kuncel (2002); Kuncel, Campbell, Hezlett, & Ones (2001)
Cross-Situational Validity: A Direct Test However, we still lack a direct test: –Examine the validity of a single test developed for academic settings but used in both academic and work settings –Ideally we would also establish the relationship between this test and other cognitive ability measures Unfortunately, a single ability measure is rarely used for both personnel selection and educational admissions decisions With one notable exception….
MAT – Transitional Variables and Creativity Kuncel, Hezlett, & Ones (2004) JPSP
MAT – Work Criteria Kuncel, Hezlett, & Ones (2004) JPSP
Story: Predictive Power of Cognitive Ability Tests is an artifact of SES
Its Storytime… In the interest of truth in advertising, the SAT should simply be called a “wealth test” - Guiner (Undated) “The SAT merely measures the size of student’s houses” - Kohn (2001) “Only thing the SAT predicts well now is socioeconomic status” – Colvin (1997)
Recent Claims in the American Psychologist “…SAT I scores lose any ability to predict freshman year grades if the regression analyses control for socioeconomic status” (p. 100) – Crosby, Iyer, Clayton, and Downing (2003) “…SAT scores used for college admissions do not predict freshman year grades when socioeconomic status is controlled” (p. 1023) – Biernat (2003) Atkinson and Geiser (2009) claim that SAT coefficients are “decisively diminished” when SES and HSGPA are controlled.
Consistent SES Findings SourceSampler test-grade r test- grade.SES U California77, Sackett, Kuncel, et al. 2009a155, Sackett, Kuncel, et al. 2009b17, Moderate observed relationship Does not go to zero when controlling for SES
University of California System Data HSGPA.30 SAT I.22 Family Income.03 Parents Education.05
“Ok, but I know someone who had really high test scores and they didn’t do as well as another friend who had lower test scores. So scores only matters to a certain degree, right?”
More is also better at school Arneson and Sackett (2009)
Coward & Sackett (1990) 174 studies on the relationship between intelligence and job performance Studies used the 9 scale GATB (General Aptitude Test Battery) Total sample size across all studies was a substantial 36,614 Found overwhelming support for a linear relationship between ability and job performance The commonly held notion appears to be incorrect
Traditional Supplementary Predictors Letters of Recommendation Personal Statements Interviews
LettersGRE Result suggest no incremental validity for letters, save for degree completion. Note that all letters were structured and quantitative. Kuncel, Vanelli, & Ones (2009)
Cooper-Murphy, Klieger, Borneman, & Kuncel (2007) College and University Personal Statements
Admissions Interview: Results
Yet There is Good Stuff Out There
Study Habits, Skills, and Attitudes Crede & Kuncel (2007) Perspectives in Psychological Science Nkr Aggregate Measures 18, Study Skills 24, Study Habits 23, Study Attitudes 7,
Class Attendance: Being There Crede, Roch, & Kieszczynka (2010) Review of Educational Research Nkr Attendance 21,
Personality Predictors of Graduate Student Performance Rigdon & Kuncel, 2010 Nkr obs SD obs SD Graduate GPA Emotional Stability Achievement1, Task Performance Autonomy Ach. Via Indep Qualifying Exam Ach. Via Indep Obtain Ph.D. Emotional Stability1, Conscientiousness1,
Cautionary Tales Faking and Coaching –Most alternatives that have been proposed would be highly susceptible to deliberate faking or test preparation coaching Personality Assessments –Other ratings Study habits, attitude, and skill inventories –Placement and guidance Biodata –Verifiable content Situational judgments tests
Fairness and Bias Extensive research on standardized test scores for bias Social class has also been scrutinized. Data for letters of recommendation and personal statements in academic settings is thinner. A summary of current knowledge on the next slide…
New Alternatives Should be Subjugated to Extensive Study for Bias Before Implementation