Susan Agre-Kippenhan, Portland State University Professor, Art Department Evaluating the Effectiveness of Service Learning

Why evaluate?  Assessment as a continual Service Learning improvement tool  Clarification of critical learning objectives, ability to focus on Service Learning specifics  Yields useful information for programs, faculty, students  Evidence and support claims

SL Outcomes: Ù Service Learning Claims: Ù Understanding of course content Ù Theory to practice Ù Understanding of Community Ù Community related skills Ù Civic Engagement Ù Diversity Ù MORE?

Student Learning Focus  Value of authentic work samples  Moving from self-report  Moving from less integrated forms of assessment (i.e. observation, focus group)

Continual Improvement  Feedback to actual courses, programs  Applicable/actionable assessment  Can be course/assignment based  Can serve other programmatic goals

How do we know what works? A working model for assessment based on the premise that there are: Ù Key questions to ask Ù Logical steps to follow Ù Usable data Ù Transforms student learning and community experiences

1 Assessment Cycle Model 4 Develop Learning Objectives Identify Work Samples Locate Critical Junctures Determine Assessment Criteria/Quality Test Validity /Reliability Disseminate Revise/Revisit

Assessment Cycle Model 4 Develop Learning Objectives Identify Work Samples Locate Critical Junctures Determine Assessment Criteria/Quality Test Validity /Reliability Disseminate Revise/Revisit

Assessment Cycle Model What am I trying to assess? Action: Define Student Learning Outcome  Student Learning Outcomes in the context of:  Mission  Priorities  Program Goals 1

What are appropriate work samples? Action: Identify Student Work Samples  Qualitative vs. Quantitative  Examples: Portfolios, Papers (writing samples, essays, reflective papers, editorials, journals…) Performances, Artwork, Lab reports Assessment Cycle Model 2

When should I assess? Action: Identify critical junctures  Critical Junctures  Entering students  Passages  Culminating Assessment Cycle Model 3

What am I looking for in student work? Action: Define scoring criteria  Rubric Development: Scoring Criteria  What indicators will I use to demonstrate desired student learning?  Examples: Bloom's taxonomy 4 Assessment Cycle Model

How do I evaluate the range of student work? Action: Define performance levels  Rubric Development: Performance standards  How many levels will my rubric have and how will I define performance at each level?  Specific clear descriptors at each level  Qualitative differences  Common errors 4 Assessment Cycle Model

Sample Rubric Work Sample Scoring Criteria 1 Scoring Criteria 2 Scoring Criteria 3 Performance Level 1- Low Performance Level 2- Medium Performance Level 3- High

Once I have a rubric how do I know it's valid? How do I know if the rubric is reliable?  Evaluate the rubric against the learning objectives, learning opportunities etc… Is there clear alignment?  Pilot test with multiple evaluators  Discuss how well the rubric works  Compare scores for consistency  Determine procedures for inconsistent scores 5 Assessment Cycle Model

What's next? Action: Back to the cycle  Revise and revisit - learning outcomes, learning opportunities, assignments, critical junctures, rubrics, program, course offerings/order, curriculum, pedagogy… 6 Assessment Cycle Model

SL Case Study Ù University Studies - General Education ï 4 Year integrated program ï Interdisciplinary program Ù Critical Features ï Clearly defined goals Communication, Critical thinking, Diversity, Civic Responsibility ï Common Pedagogy ï Required Courses

SL Case Study Student Learning Outcomes Action: Define Student Learning Outcome Appreciation and Understanding of Civic Responsibility  PSU Mission - Let Knowledge Serve the City  University Studies Goal - Civic Responsibility 1

SL Case Study Work samples Action: Identify Student Work Samples Common assignment given in each class  Paper 2

Critical Junctures Action: Identify critical junctures Pilot in Senior Capstone Classes Ù Senior Capstone required course Ù Culminating group experience for major and general education Ù Application of expertise to a real project in the community SL Case Study 3

Common Assignment “Please write a 3-5-page reflective analysis summarizing the ways you have gained experience with the University Studies goal of Civic Responsibility. Use specific examples from your discussions and course readings, or in completing your project.” SL Case Study

Scoring Criteria and Performance Levels Action: Define scoring criteria/levels Civic Responsibility Stages  Based on Bloom's taxonomy SL Case Study 4

Valid Rubric How do I know if the rubric is reliable?  PSU Mission - Let Knowledge Serve the City  University Studies Goal - Civic Responsibility  Pilot test with multiple evaluators  Compare scores for consistency.  Discuss how well the rubric works. SL Case Study 5

Revisiting Process Suggestions for improvements  Revise assignment for clarity  Revise rubrics for specificity  Clarify pedagogy Confirmed  Program goals addressed in class  Basis of assignment is valid  Keep assessment in Capstone SL Case Study 6

 Focus on student learning objectives  What are appropriate work samples?  Are there existing critical junctures?  How do we evaluate quality?  How can this improve our practice? SL Assessment in practice

 Connect to goals and mission  Knowledge of the process that is transferable  Assessment as a tool for partnerships  Applicable/actionable assessment Lessons Learned

For More Information Susan Agre-Kippenhan