Time on Test, Student Motivation, and Performance on the Collegiate Learning Assessment: Implications for Institutional Accountability Session 353 Braden J. Hosch, Ph.D. Director of Institutional Research & Assessment Central Connecticut State University AIR Annual Forum, Chicago, IL June 1, 2010

Overview Voluntary System of Accountability and the Collegiate Learning Assessment Institutional Profile Methodology and Limitations Findings Implications for Accountability

Voluntary System of Accountability Initiative among public colleges led by APLU and AASCU to provide public with comparable information in same format Pre-emptive response to findings issued by Spellings Commission (2006) Learning outcomes must be posted by Spring 2011

VSA Learning Outcomes Results from cross-sectional administration to first-year students and seniors of one of three tests: Collegiate Learning Assessment (CLA) Collegiate Assessment of Academic Proficiency (CAAP) Measure of Academic Proficiency and Progress (MAPP) Template reports scaled assessment scores, SAT/ACT scores of tested students, and an institutional relative-to-expected (RTE) score

Collegiate Learning Assessment Constructed response test that measures: Critical Thinking Problem Solving Analytical Reasoning Writing Two tasks: Performance task (90 minute time limit) Analytic writing task (60 minute time limit) Scored holistically and converted to scaled score,percentile score, and RTE score.

Central Connecticut State University Public – part of Connecticut State Univ. System Carnegie 2005 Masters-Larger Programs New Britain, CT (Hartford MSA: ~ 1.2 million pop.) Fall 2009 Enrollment: 12,461 headcount (9,989 undergraduate, 22% residential); 9,619 full-time equivalent enrollment 52% female; 17% minority Full-time, first-time students: 1,281 (56% residential) Mean SAT score: 1025 (F 2009) Six-year graduation rates of full-time, first-time students entering in Fall 2003: 49%

CCSU CLA Percentile Scores

Detailed Results by Semester ParticipantsFall 2007 First-Year Spring 2008 Senior Fall 2008 First-Year Spring 2009 Senior Fall 2009 First-Year Participants HS Rank57%ile64%ile61%ile63%ile62%ile SAT M+CR CLA Scaled Performance Task Analytic Writing Task CLA %ile Cum GPA Minutes

Test Administration Test administration procedures evolved over time because of difficulty in recruitment. First-year students recruited through FYE courses Incentives varied by instructor Seniors recruited primarily through Incentive = graduation regalia (~$25 - $40)

First-Year Students Fall 2007Fall 2008Fall 2009 ProctorPT Employee OIRA Director Grad Assistant Incentives[Various] FYE Sections779 Participants Pre-Test SurveyNYY

Seniors Spring 2008Spring 2009Spring 2010 ProctorPT Employee & OIRA Director Grad Assistant IncentivesInitially none $25 Regalia Discount $35 Regalia Voucher (Total Cost) $40 Regalia Voucher (Total Cost) Senior Capstones300 Total Participants Pre-test SurveyNYY

Methodological Issues & Limitations Methodological Issues Cross-sectional design Different proctors in than and Different incentives for FY students and seniors Within administrations Across administrations Hand-timing Time on test as an imperfect proxy for motivation

Minutes Spent on Test by Scaled CLA Score (All Available)

Minutes Spent on Test by Scaled CLA Score ( )

Minutes Spent on Test by Scaled CLA Score ( )

Minutes Spent on Test by Scaled CLA Score ( )

Minutes Spent on Test by Relative-to- Expected CLA Score (All Available)

Minutes Spent on Test by Relative-to- Expected CLA Score ( )

Minutes Spent on Test by Relative-to- Expected CLA Score ( )

Minutes Spent on Test by Relative-to- Expected CLA Score ( )

Correlations (First-Year Students) CLA Scaled Score Minutes Spent on CLA SAT CR SAT Writing SAT (Math + CR) High School Rank Minutes spent on CLA SAT Critical Reading SAT Writing SAT (Math + CR) High School Rank SAT Math

Correlations (Seniors) CLA Scaled Score SAT (Math + CR) SAT MathSAT CR End of term cum GPA High School Rank SAT (Math + CR) SAT Math SAT Critical Reading End of term cum GPA High School Rank Minutes spent on CLA

Regressions (First-Year Students) First-Year Student CLA Scaled Score (R 2 =0.261) βStd. Err. tSig. (Constant) *** Minutes spent on CLA *** Combined SAT Score *** First-Year Student CLA Percentile (R 2 =0.286) βStd. Err. tSig. (Constant) *** Minutes spent on CLA *** Combined SAT Score ***

Regressions (Seniors) Senior CLA Scaled Score (R 2 =0.261) βStd. Err. tSig. (Constant) *** Minutes spent on CLA *** Combined SAT Score *** Senior CLA Percentile (R 2 =0.286) βStd. Err. tSig. (Constant) *** Minutes spent on CLA *** Combined SAT Score ***

Survey Results Differences in test scores and time spent on test by self-reported motivation were suggestive but not statistically significant. Scores and time usage aside, the percentage of students who agreed or strongly agreed they were highly motivated to participate in CLA: 34% First-Year Students 70% Seniors

Self-Reported Motivation First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree104% %^^ Disagree198% %^^ Neutral12353% % Agree6428% % Strongly agree146% % I feel highly motivated to participate in this activity today Differences are suggestive but NOT statistically significant

Test Modality Preference First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree3515% % Disagree7332% % Neutral7432% % Agree3415% % Strongly agree156% % I perform better on essay tests than on multiple choice tests. Differences are suggestive but NOT statistically significant

Assessment Modality Preference First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree115% % Disagree3013% % Neutral5122% % Agree8136% % Strongly agree115% % I prefer to take a test rather than write a paper.

Test Anxiety First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree219% % Disagree5424% % Neutral7633% % Agree5323% % Strongly agree2411% %^^ I get so nervous when I take tests that I don't usually perform my best work.

Student Responsibility for Learning First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree00%^^0 ^^ Disagree10%^^22%^^ Neutral198%37*1022*118% Agree10647%45*1124*6147% Strongly agree10044%51*1121*5744% Students are responsible for learning material assigned by their professors * Sig. at p<0.05 ANOVA test.

Institution Responsibility for Learning First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree146% % Disagree6629% % Neutral9843% % Agree4018% % Strongly agree84% % Colleges and universities are responsible if students don't learn what they need to be successful after they graduate.

Mandatory College Exit Tests First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree4520% % Disagree8638% % Neutral6127% % Agree3114% % Strongly agree52% % All college students should be required to pass a standardized exit test in order to graduate Differences are suggestive but NOT statistically significant

Publications of College Rankings First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree73% % Disagree3716% % Neutral11852% % Agree6227% % Strongly agree31% % Students should use published college rankings (like US News and World Report) when deciding which school to attend Differences are suggestive but NOT statistically significant

Overall Findings Time spent on test MATTERS What students say about their motivation may not matter (much) The relationship between time and test scores is generally missing from discussions about accountability

Implications (1) 1. Acknowledge that test scores may be influenced by motivation/time spent on test; support further research into these effects. 2. Longitudinal testing may help control for some of the effects of motivation / time spent on test.

Implications (2) 3. Multi-year moving averages might improve meaningfulness of test-score information. 4. Statistical adjustments based on time spent on test should be explored but may not be technically or politically feasible.

Implications (3) 5. Explore portfolio or other contextual assessment strategies for accountability, esp. among consortia of institutions (not unlike athletics conferences) 6. Recognize that motivation and time effects are also likely present in elementary and secondary education; consider extent to which performance reflects cognitive vs. behavioral/motivational outcomes.

Questions Session 353 Dr. Braden J. Hosch Director of Institutional Research & Assessment Central Connecticut State University Paper, handout, and slides online at: (see Research and Presentations) Contact Information