1. Incorporating Program Assessment into Your Annual Program Review June 29, 2006

2. 2 Changes to Program Review Program assessment is a new annual requirement – Easy 3-step process A program assessment plan is an integral part of the program review recommendations that must be submitted by September 15 th A program assessment report is due by the end of the academic year The results of annual program assessments will be evaluated during the comprehensive (5-year) program review process: – Explain how program assessment findings resulted in improved student learning

3. 3 Program Assessment Whereas classroom assessment focuses on gauging learning for individual students, program assessment gauges the learning of a group of students. The outcomes information in program assessment is used to improve courses, programs, and services. - D. Maldonado, Program Assessment Workshop, February 2006

4. 4 Annual Program Assessment Process Step 1: Start with program objectives, pick one Program Objectives are detailed aspects of the program that are accomplished by the successful completion of the course outcomes. Each program should have 5 or 6 objectives Faculty from each program/discipline select one objective to be assessed during the first discipline meeting of the year

5. 5 Annual Program Assessment Process Step 2: Pick two measures – One direct measure - Tangible evidence of what students have and have not learned – One indirect measure - Reveals students are learning but evidence of what students have learned is less clear

6. 6 Direct Measures Student work compared to a rubric: Classroom work from a capstone course Capstone project Portfolio Performance Scores on specific test questions related to the objective Licensure and certification results Field experience ratings by supervisors CATs* in Chapter 7 focus on student knowledge and skill *Angelo & Cross, Classroom Assessment Techniques

7. 7 Indirect Measures Student self-reflection or focus group interviews: How well did you achieve the learning goal? Analysis of grades (course, final exam, tests) Graduation rates Job placement rates Employer or alumni surveys Transcript or course content analysis CATs in Chapters 8 & 9 focus on student attitudes, values, and reaction to instruction

8. 8 Choose an indirect measure related to the direct measure Example Direct Measure: Scores on faculty developed rubric for an art project demonstrating applied competency at the end of an advanced art workshop Indirect Measure: Transcript analysis Sample analysis: Students lack proficiency in color theory. An analysis of student transcripts revealed less than half completed the course on color.

9. 9 The Basics of a Plan 2006 Program Assessment Plan for [Program Name] List objective to be assessed this year: List the two measures for assessing the objective: Direct: Indirect: Statement about method How will you assess achievement of this objective? Who? What? When?

10. 10 Examples Exceptional – Psychology – Math Acceptable – Foreign Language – Nursing – Communication in Any Program – Art Unacceptable – Developmental Math

11. 11 2006 Program Assessment Plan for Psychology List objective to be assessed this year: Define and explain the scientific method including the various research methods used by psychologists List the two measures for assessing the objective: Direct: Scores on embedded test questions Indirect: Analysis of test grades Statement about method All full-time faculty will evaluate the scores of embedded test questions on the first Psychology 101 test during the fall semester Exceptional Example

12. 12 2006 Program Assessment Plan for Math List objective to be assessed this year: Solve systems of linear equations in two and three variables List the two measures for assessing the objective: Direct: Scores on a faculty developed rubric Indirect: Analysis of final exam grades Statement about method Four instructors (one per campus) will evaluate pertinent questions on a sample of Fall 2006 Intermediate Algebra final exams Exceptional Example

13. 13 2006 Program Assessment Plan for Foreign Language List objective to be assessed this year: Students will exhibit fluency in a foreign language List the two measures for assessing the objective: Direct: Scores on a faculty designed rubric Indirect: Analysis of Productive Study-Time Logs (CAT 37) Statement about method Faculty will evaluate an oral presentation at the end of an intermediate-level course. Acceptable Example

14. 14 2006 Program Assessment Plan for Nursing List objective to be assessed this year: Demonstrate practical competency in the major List the two measures for assessing the objective: Direct: Rating forms completed by practicum supervisors Indirect: Analysis of self-rating forms completed by students Statement about method Faculty will analyze supervisors’ rating forms at the end of an externship or service learning experience Acceptable Example

15. 15 2006 Program Assessment Plan for Any Program List objective to be assessed this year: Communicate competently in the major List the two measures for assessing the objective: Direct: Scores on faculty developed rubric Indirect: Student reaction to the learning value of the assignment (suggest trying – CAT 49: Assignment Assessments) Statement about method Faculty will evaluate an oral report at the end of a capstone course Acceptable Example

16. 16 2006 Program Assessment Plan for Art List objective to be assessed this year: Demonstrate applied competency in the major List the two measures for assessing the objective: Direct: Scores on faculty developed rubric Indirect: Transcript analysis Statement about method Faculty will evaluate students’ final projects in an advanced art workshop Acceptable Example

17. 17 2006 Program Assessment Plan for Developmental Math List objective to be assessed this year: Understand the order of operations List the two measures for assessing the objective: Direct: final exam grades Indirect: Statement about method Analyze final exam grades Unacceptable Example

18. 18 Use a map to identify the appropriate place to assess your objective Obj 1Obj 2Obj 3Obj 4Obj 5 CFS 101 XXX CFS 103 XXX CFS 111 XX CFS 130 XXXXX CFS 135 XXXXX CFS 214 XXXX CFS 218 XXXX CFS 240 XXXXX -Abbreviated Example from CFS-

19. 19 Take it one step further Obj 1Obj 2Obj 3Obj 4Obj 5 Course 1 II Course 2 RRI Course 3 MRI Course 4 RRR Course 5 RMRI Course 6 RR Course 7 MRR Course 8 MM I – Introduce; R – Reinforce; M - Master

20. 20 Annual Program Assessment Process Step 3: Have an annual meeting to discuss data and identify action items Analysis of data: Rely on faculty expertise Identify student strengths and weaknesses during discussion of results Use the results of indirect measures to support direct evidence Example: The single most frequently mentioned weakness is…

21. 21 Writing the Report 2006 Assessment Results for [Program] List specific objective that was measured: Assessment criteria used to measure objective: Results of assessment study: Actions taken for improvement of learning: Problems encountered:

22. 22 Example of a Report Foreign language

23. 23 2006 Program Assessment Results for Foreign Language List specific objective that was measured: Students will exhibit fluency in a foreign language Assessment criteria used to measure objective: Scores on faculty developed rubric for oral presentation at the end of an intermediate-level course Results of assessment study: Students exhibit low levels of fluency in the language after taking several courses. Students have few opportunities for sustained practice in target language. Actions taken for improvement of learning: Require a study abroad or immersion experience for language majors Problems encountered: Time required to review video taped material was excessive. Next time select a smaller sample.

24. 24 Suggested schedule for 2006/07 Fall discipline meeting – identify program objective, measures and method Fall semester – conduct assessment Spring discipline meeting – discuss results Before the end of the academic year- prepare a report for coordinating deans September 2007 – monitoring report due on the use of assessment results

25. 25 Valuable Resources Middle States Commission on Higher Education. Student Learning Assessment: Options and Resources. 2003 Chapter 3 – Evaluating Student Learning Suskie, Linda. Assessing Student Learning: A Common Sense Guide. Anker Publishing Co. 2004. Part III – The Assessment Toolbox Angelo, Thomas, and Patricia Cross. Classroom Assessment Techniques: A Handbook for College Teachers. Jossey-Bass. 1993.

26. 26 Thank You Contact: Mary Kate Quinlan Learning Assessment Analyst 412-237-8191