1. The Collegiate Learning Assessment (CLA) Roger Benjamin CAE April 15, 2008

2. Mission: Improve Teaching and Learning---Higher Order Skills Strategy –Initial institution-based comparison of student learning outcomes relative to the development of higher order skills –Encourage CLA blue print for continuous improvement –Launch Institute for Performance Assessment to advance role of performance assessment in education

3. Rationale We do not have strong theories of teaching and learning Comparative method useful to provide investigator with more cases to generalize from –Comparative based approach only makes sense if there is sufficient variation between colleges and universities on student learning outcomes

4. Collegiate Learning Assessment (CLA) The performance tasks are designed to measure higher order skills, critical thinking, analytical reasoning, problem solving and writing initially at institution level of analysis. CLA only measures certain aspects of undergraduate education.

5. Overview Principles driving the CLA CLA’s distinguishing features CLA’s measures Reporting results The CLA Blue Print for Continuous Improvement The Case for an Institute of Performance Assessment

6. Principles Driving the CLA 1.Cannot assess everything that is important 2.Can measure some important abilities 3.Need benchmarks 4.Use results to support improvement and curricular reforms 5.Use psychometrically sound measures

7. 1. One measure is not sufficient Colleges have different missions Students have different majors Multiple important outcomes require multiple measures

8. 2. Can measure some important abilities Competencies listed in college mission statements are applicable to all students – Writing – clear, organized, etc. – Critical thinking – Analytic reasoning – Problem solving Integrated rather than separate abilities Cut across academic majors

9. 3. Need benchmarks to assess progress How much did our students improve from the time they entered college? Is that degree of improvement comparable to that obtained by equally able students at other colleges?

10. 5. Use psychometrically sound tests Valid – assess important skills (and not something else) Reliable – consistent (not chance) Fair – standardized, calibrated, and unbiased measures Cost effective in terms of testing time and expense

11. Some of CLA’s Distinguishing Features Rely on open-ended holistic tests that are: –realistic “work samples” –engaging –applicable across academic majors College (not student) is the unit of analysis Sample students within schools No arbitrary standards, such as “proficient” Paperless administration, scoring, & reporting Control for differences in input between schools Report results in terms of “value-added”

12. CLA Is Not Like NCLB Constructed rather than selected response tests Use assessment for improvement (low stakes) rather than accountability (high stakes) Same tests used nationwide rather than a patchwork of measures and standards Matrix sampling of measures (like NAEP) rather than require that all students take all measures Test sample of students at each school Schools and students volunteer to participate Results released to the institution, not published

13. CLA’s Measures Analytic writing (essay) prompts –Make-an-argument (45 minutes) –Break-an-argument (30 minutes) Performance tasks (90 minutes) –Document based –Contextualized questions –Split screen/dialogue box format –Analytic and holistic scoring Background questionnaire

14. Make-An-Argument Prompt “In our time, specialists of all kinds are highly overrated. We need more generalists – people who can provide broad perspectives.” Directions: 45 Minutes, agree or disagree and explain the reasons for your position. Student selects one of two prompts to answer.

15. Break-An-Argument Prompt Students are asked to discuss how well reasoned they find an argument to be (rather than simply agreeing or disagreeing with it). A respected professional journal with a readership that includes school principals published the results of a two-year study on childhood obesity. This study sampled 50 children, ages 5-11, from Jefferson School. A fast food restaurant opened near the school just before the study began. After two years, students who remained in the sample were more likely to be overweight—relative to the national average. Based on this study, the principal of Jones Elementary School decided to address her school’s obesity problem by opposing the opening of any fast food restaurants near her school.

16. Performance Tasks 90-minute real life problems General directions and context Need to combine information from different types of sources A few open-ended questions, no one “right” answer—must explain rationale Split screen Right: Document Library Left: Question and answer block

17. Swiftaire 235 You advise the president of DynaTech DynaTech makes airplane navigation and communication equipment DynaTech’s sales manager suggests buying a Swiftaire 235 to visit customers and demo its products Recent accident – wing came off in flight Student’s tasks – Review document library – Write memo discussing pros and cons of DynaTech getting a Swiftaire 235 – Justify your recommendations

18. Document Library Newspaper article about recent accident Transcript of interview about accidents DynaTech email exchanges regarding reasons to buy a Swiftaire 235 Trade magazine article that compares Swiftaire 235’s performance and safety characteristics to similar planes Manufacturer specifications and required pilot training for Swiftaire 180 and 235 Charts about airplane accidents and sales

19. Scoring Rubric Writing skills – clear, organized, persuasive Analysis, problem solving, & reasoning skills –Integrates information from different sources –Recognize flaws and issues—not swayed by emotional arguments, faulty logic, irrelevant information, etc. –Anticipates consequences and implications of alternative solutions and strategies –See pros and cons of competing explanations, points of view, and arguments –Weighs evidence based on its credibility

20. Two Definitions of Value- Added Do seniors score higher than freshmen with the same SAT scores? Does a college’s seniors score higher than what would be expected given (1) their average SAT score and (2) the typical relationship between average SAT and CLA scores at other colleges?

21. Fig. 3: Relationship Between Mean ACT Scores and Mean Total CLA Scores for Freshmen and Seniors 31 27 23 19 15 19232731 ACT Score CLA Score

22. Status of the CLA, 07-08 238 four-year institutions testing (roughly mirrors the Carnegie classification) Community college and independent and suburban high school pilots underway Harvard Medical School-based consortium Feasibility study for OECD international version of CLA underway

23. The CLA Comparison Strategy: From the Institution to the Classroom There is up to 3.0 standard deviation of difference between similarly situated institutions along the regression plot. These large differences provide faculty and administrators a signal about where their institution stands This means there is a large canvas for studying best practices in the institutions that do better than predicted.

24. The CLA Blue Print for Continuous Improvement Correlate inputs, processes, and outputs with CLA results Drill down to departments and programs to discover which are strong or weak contributors to their CLA results Conduct audit of existing assessments Examine best practices, e. g., more analytic based writing appears to help

25. Get Published Performance Tasks Into The Classroom Assist faculty to develop performance tasks that are based on the scoring guide of the published tasks Choose case studies and problems for text material that is congruent with the documents in the performance tasks Adopt a student-centered approach to teaching that calls for more analytic-based writing

26. The Continuous Improvement System Do comparative value added institution- based testing with CLA Carry out variety of responses, particularly those focused on the classroom Re-test using CLA value added institution- based testing Repeat the cycle

27. The Evolution of the CLA (1) Attract a sufficient number of institutions using the CLA protocol to corroborate the hypothesis there is wide variation between similar institutions in producing CLA results Develop diagnostic logic necessary to “unpack” CLA results at institution level

28. The Evolution of the CLA (2) Develop resources useful for faculty and administrators to employ CLA blue print for improvement, –Analyze test results to provide more detailed information about strengths and weaknesses of students’ performance –Create CLAintheclassroom for classroom use –Launch CLA Academy workshops to teach faculty how to develop performance tasks –Design on line version of CLAintheclassroom

29. The Kalamazoo Story The differences between academic divisions---humanities, social sciences, natural sciences appears to be a function of writing –Students who write well and who have had more experience making judgments abut the value of information appear to perform better on the CLA

30. Sample Cabrini College Faculty Response If a faculty member is going to use this kind of assessment, they are going to rethink the way they teach because this type of assessment asks for students to pull together various forms of information and analyze it at a higher level.

31. The Case for an Institute of Performance Assessment Knowledge economy requires stronger higher order skills. This development portends significant changes in teaching and learning, –Shift from lecture format to student-centered approach that emphasizes more analytic writing –Change in the balance of curricular and textbook focus from emphasis on content to case and problem- based approach that requires students to apply what they know to new situations –Change from multiple choice tests to open-ended essay tests better aligned with the other two goals

32. Rationale for Institute Lesson of human capital school---its time to focus transforming the content of the stock of knowledge and skills of our citizens Multiple choice tests no longer sufficient The field of performance assessment must be developed Comparative advantage of CLA to launch the Institute

33. Frequently Asked Questions 1.Are not selective institutions penalized if value added growth is the metric? 2.What about absolute levels of performance compared to value added growth? 3.What is the methodological importance of establishing the base line characteristics of the students taking the CLA? 4.What about the motivation issue?