Presentation is loading. Please wait.

Presentation is loading. Please wait.

Time on Test, Student Motivation, and Performance on the Collegiate Learning Assessment: Implications for Institutional Accountability Session 353 Braden.

Published byDeshaun Curtiss Modified over 10 years ago

Similar presentations

Presentation on theme: "Time on Test, Student Motivation, and Performance on the Collegiate Learning Assessment: Implications for Institutional Accountability Session 353 Braden."— Presentation transcript:

1 Time on Test, Student Motivation, and Performance on the Collegiate Learning Assessment: Implications for Institutional Accountability Session 353 Braden J. Hosch, Ph.D. Director of Institutional Research & Assessment Central Connecticut State University AIR Annual Forum, Chicago, IL June 1, 2010

2 Overview Voluntary System of Accountability and the Collegiate Learning Assessment Institutional Profile Methodology and Limitations Findings Implications for Accountability

3 Voluntary System of Accountability Initiative among public colleges led by APLU and AASCU to provide public with comparable information in same format Pre-emptive response to findings issued by Spellings Commission (2006) Learning outcomes must be posted by Spring 2011

4 VSA Learning Outcomes Results from cross-sectional administration to first-year students and seniors of one of three tests: Collegiate Learning Assessment (CLA) Collegiate Assessment of Academic Proficiency (CAAP) Measure of Academic Proficiency and Progress (MAPP) Template reports scaled assessment scores, SAT/ACT scores of tested students, and an institutional relative-to-expected (RTE) score

5 Collegiate Learning Assessment Constructed response test that measures: Critical Thinking Problem Solving Analytical Reasoning Writing Two tasks: Performance task (90 minute time limit) Analytic writing task (60 minute time limit) Scored holistically and converted to scaled score,percentile score, and RTE score.

6 Central Connecticut State University Public – part of Connecticut State Univ. System Carnegie 2005 Masters-Larger Programs New Britain, CT (Hartford MSA: ~ 1.2 million pop.) Fall 2009 Enrollment: 12,461 headcount (9,989 undergraduate, 22% residential); 9,619 full-time equivalent enrollment 52% female; 17% minority Full-time, first-time students: 1,281 (56% residential) Mean SAT score: 1025 (F 2009) Six-year graduation rates of full-time, first-time students entering in Fall 2003: 49%

7 CCSU CLA Percentile Scores

8 Detailed Results by Semester ParticipantsFall 2007 First-Year Spring 2008 Senior Fall 2008 First-Year Spring 2009 Senior Fall 2009 First-Year Participants10599110134130 HS Rank57%ile64%ile61%ile63%ile62%ile SAT M+CR1019994104510161019 CLA Scaled10571133112712481098 Performance Task 10661113111012141087 Analytic Writing Task 10481147114412821109 CLA %ile5137677053 Cum GPA2.733.132.873.242.88 Minutes--45496344

9 Test Administration Test administration procedures evolved over time because of difficulty in recruitment. First-year students recruited through FYE courses Incentives varied by instructor Seniors recruited primarily through email Incentive = graduation regalia (~$25 - $40)

10 First-Year Students Fall 2007Fall 2008Fall 2009 ProctorPT Employee OIRA Director Grad Assistant Incentives[Various] FYE Sections779 Participants105110130 Pre-Test SurveyNYY

11 Seniors Spring 2008Spring 2009Spring 2010 ProctorPT Employee & OIRA Director Grad Assistant IncentivesInitially none $25 Regalia Discount $35 Regalia Voucher (Total Cost) $40 Regalia Voucher (Total Cost) Senior Capstones300 Total Participants 99130105 Pre-test SurveyNYY

12 Methodological Issues & Limitations Methodological Issues Cross-sectional design Different proctors in 2007-08 than 2008-09 and 2009-10 Different incentives for FY students and seniors Within administrations Across administrations Hand-timing Time on test as an imperfect proxy for motivation

13 Minutes Spent on Test by Scaled CLA Score (All Available)

14 Minutes Spent on Test by Scaled CLA Score (2007-08)

15 Minutes Spent on Test by Scaled CLA Score (2008-09)

16 Minutes Spent on Test by Scaled CLA Score (2009-10)

17 Minutes Spent on Test by Relative-to- Expected CLA Score (All Available)

18 Minutes Spent on Test by Relative-to- Expected CLA Score (2007-08)

19 Minutes Spent on Test by Relative-to- Expected CLA Score (2008-09)

20 Minutes Spent on Test by Relative-to- Expected CLA Score (2009-10)

21 Correlations (First-Year Students) CLA Scaled Score Minutes Spent on CLA SAT CR SAT Writing SAT (Math + CR) High School Rank Minutes spent on CLA.468-- SAT Critical Reading.333.178-- SAT Writing.311.162.632-- SAT (Math + CR).326.186.807.610-- High School Rank.264.227.133.161.180-- SAT Math.201.127.331.370.824.145

22 Correlations (Seniors) CLA Scaled Score SAT (Math + CR) SAT MathSAT CR End of term cum GPA High School Rank SAT (Math + CR).505-- SAT Math.479.886-- SAT Critical Reading.409.890.576-- End of term cum GPA.400.271.285.263-- High School Rank.338.214.178.147.336-- Minutes spent on CLA.331.090.210.095.177.214

23 Regressions (First-Year Students) First-Year Student CLA Scaled Score (R 2 =0.261) βStd. Err. tSig. (Constant)65273.38.89*** Minutes spent on CLA3.67.4907.49*** Combined SAT Score0.281.0713.93*** First-Year Student CLA Percentile (R 2 =0.286) βStd. Err. tSig. (Constant)-28.912.6-2.28*** Minutes spent on CLA.685.0848.18*** Combined SAT Score.049.0123.94***

24 Regressions (Seniors) Senior CLA Scaled Score (R 2 =0.261) βStd. Err. tSig. (Constant)39593.94.20*** Minutes spent on CLA2.50.5354.67*** Combined SAT Score0.665.0917.34*** Senior CLA Percentile (R 2 =0.286) βStd. Err. tSig. (Constant)-69.014.2-4.85*** Minutes spent on CLA.372.0814.59*** Combined SAT Score.099.0147.19***

25 Survey Results Differences in test scores and time spent on test by self-reported motivation were suggestive but not statistically significant. Scores and time usage aside, the percentage of students who agreed or strongly agreed they were highly motivated to participate in CLA: 34% First-Year Students 70% Seniors

26 Self-Reported Motivation First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree104%40107111%^^ Disagree198%45110311%^^ Neutral12353%4611093728%591240 Agree6428%4811257053%631250 Strongly agree146%5111692217%671244 I feel highly motivated to participate in this activity today Differences are suggestive but NOT statistically significant

27 Test Modality Preference First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree3515%48108997%571220 Disagree7332%4511022418%611222 Neutral7432%4811275139%621292 Agree3415%4611293627%661207 Strongly agree156%451155118%651265 I perform better on essay tests than on multiple choice tests. Differences are suggestive but NOT statistically significant

28 Assessment Modality Preference First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree115%42108397%671205 Disagree3013%4611431612%621248 Neutral5122%4511223325%621290 Agree8136%4811034534%671233 Strongly agree115%42108397%671205 I prefer to take a test rather than write a paper.

29 Test Anxiety First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree219%4811571310%601286 Disagree5424%5011264635%671271 Neutral7633%4811143930%621209 Agree5323%4210923124%611261 Strongly agree2411%45111222%^^ I get so nervous when I take tests that I don't usually perform my best work.

30 Student Responsibility for Learning First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree00%^^0 ^^ Disagree10%^^22%^^ Neutral198%37*1022*118%541187 Agree10647%45*1124*6147%651271 Strongly agree10044%51*1121*5744%631233 Students are responsible for learning material assigned by their professors * Sig. at p<0.05 ANOVA test.

31 Institution Responsibility for Learning First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree146%48116854%741235 Disagree6629%4610973627%601233 Neutral9843%4811144031%631271 Agree4018%4711223627%641242 Strongly agree84%4311181411%621245 Colleges and universities are responsible if students don't learn what they need to be successful after they graduate.

32 Mandatory College Exit Tests First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree4520%4511091713%621254 Disagree8638%4811376348%631280 Neutral6127%4710933023%611226 Agree3114%4611191310%691176 Strongly agree52%42100986%611197 All college students should be required to pass a standardized exit test in order to graduate Differences are suggestive but NOT statistically significant

33 Publications of College Rankings First-Year (Fall 2008 and Fall 2009) Seniors (Spring 2009 only) NPct Avg. Minutes Avg. CLA ScoreNPct Avg. Minutes Avg. CLA Score Strongly disagree73%41119943%691239 Disagree3716%4711353325%611270 Neutral11852%4611144736%621231 Agree6227%4810964232%631260 Strongly agree31%4596554%701181 Students should use published college rankings (like US News and World Report) when deciding which school to attend Differences are suggestive but NOT statistically significant

34 Overall Findings Time spent on test MATTERS What students say about their motivation may not matter (much) The relationship between time and test scores is generally missing from discussions about accountability

35 Implications (1) 1. Acknowledge that test scores may be influenced by motivation/time spent on test; support further research into these effects. 2. Longitudinal testing may help control for some of the effects of motivation / time spent on test.

36 Implications (2) 3. Multi-year moving averages might improve meaningfulness of test-score information. 4. Statistical adjustments based on time spent on test should be explored but may not be technically or politically feasible.

37 Implications (3) 5. Explore portfolio or other contextual assessment strategies for accountability, esp. among consortia of institutions (not unlike athletics conferences) 6. Recognize that motivation and time effects are also likely present in elementary and secondary education; consider extent to which performance reflects cognitive vs. behavioral/motivational outcomes.

38 Questions Session 353 Dr. Braden J. Hosch Director of Institutional Research & Assessment Central Connecticut State University hoschbrj@ccsu.edu Paper, handout, and slides online at: http://www.ccsu.edu/page.cfm?p=1973 (see Research and Presentations) Contact Information

Download ppt "Time on Test, Student Motivation, and Performance on the Collegiate Learning Assessment: Implications for Institutional Accountability Session 353 Braden."

Similar presentations

1 Radio Maria World. 2 Postazioni Transmitter locations.

1 Radio Maria World. 2 Postazioni Transmitter locations.
Trend for Precision Soil Testing % Zone or Grid Samples Tested compared to Total Samples.

Trend for Precision Soil Testing % Zone or Grid Samples Tested compared to Total Samples.
AGVISE Laboratories %Zone or Grid Samples – Northwood laboratory

AGVISE Laboratories %Zone or Grid Samples – Northwood laboratory
In-Home Pantry Inventory Updated: November 2012. Background and Methodology Background In 1996 a National Eating Trends (NET) pantry survey found that.

In-Home Pantry Inventory Updated: November Background and Methodology Background In 1996 a National Eating Trends (NET) pantry survey found that.
NCAA Division I Progress Toward Degree

NCAA Division I Progress Toward Degree
AP STUDY SESSION 2.

AP STUDY SESSION 2.
Reflection nurulquran.com.

Reflection nurulquran.com.
Worksheets.

Worksheets.
Ph. D. Completion and Attrition: Baseline Program Data

Ph. D. Completion and Attrition: Baseline Program Data
Solving the Faculty Shortage in Allied Health 9 th Congress of Health Professions Educators 4 June 2002 Ronald H. Winters, Ph.D. Dean College of Health.

Solving the Faculty Shortage in Allied Health 9 th Congress of Health Professions Educators 4 June 2002 Ronald H. Winters, Ph.D. Dean College of Health.
Tammy Muhs-University of Central Florida

Tammy Muhs-University of Central Florida
HART RESEARCH P e t e r D ASSOTESCIA How Should Colleges Assess & Improve College Learning? Employers Views on the Accountability Challenge Key findings.

HART RESEARCH P e t e r D ASSOTESCIA How Should Colleges Assess & Improve College Learning? Employers Views on the Accountability Challenge Key findings.
HART RESEARCH ASSOTESCIA AAC&U Members On Trends In Learning Outcomes, General Education, and Assessment Key findings from online survey among 433 Chief.

HART RESEARCH ASSOTESCIA AAC&U Members On Trends In Learning Outcomes, General Education, and Assessment Key findings from online survey among 433 Chief.
Understanding Faculty Practice & Perspective for a Learning- Centered Campus Ashley Finley, PhD National Evaluator, Bringing Theory to Practice Director.

Understanding Faculty Practice & Perspective for a Learning- Centered Campus Ashley Finley, PhD National Evaluator, Bringing Theory to Practice Director.
Findings from a National Study Ashley Finley, Ph.D Director of Assessment & Research, AAC&U National Evaluator, Bringing Theory to Practice POD Conference,

Findings from a National Study Ashley Finley, Ph.D Director of Assessment & Research, AAC&U National Evaluator, Bringing Theory to Practice POD Conference,
Newark Kids Count 2011 A City Profile of Child Well-Being Advocates for Children of New Jersey 35 Halsey Street Newark, NJ 07102 973.643.3876 www.acnj.org.

Newark Kids Count 2011 A City Profile of Child Well-Being Advocates for Children of New Jersey 35 Halsey Street Newark, NJ
Sophomores: Ready or Not: College Readiness for All

Sophomores: Ready or Not: College Readiness for All
Teaching and Learning Science & Assessment Informational Webinars Presenter: Linda Cabe Smith, Science Assessment Specialist Ellen Ebert, Science Director,Teaching.

Teaching and Learning Science & Assessment Informational Webinars Presenter: Linda Cabe Smith, Science Assessment Specialist Ellen Ebert, Science Director,Teaching.
Create an Application Title 1Y - Youth Chapter 5.

Create an Application Title 1Y - Youth Chapter 5.
Overview for CTE Educators CTE Accountability, Budget and Grants Management: Data Reporting July 15-17, 2013 Murfreesboro, TN Susan Cowden: Director of.

Overview for CTE Educators CTE Accountability, Budget and Grants Management: Data Reporting July 15-17, 2013 Murfreesboro, TN Susan Cowden: Director of.

Similar presentations

Ads by Google