2. Effective Methods for Assessing the Impact of Service-Learning on Students, Institutions, and Communities ANDREW FURC0 University of Minnesota February 27, 2009

3. Problems with Service-Learning Evaluation 1. Program goals are unrealistic. 2. Expected outcomes cannot be easily measured. 3. Evaluation is an after thought. 4. Evaluation is not aligned to program goals 5. We claim impacts when we only measured outcomes. 6. The time frame is not commensurate for impacts to manifest. 7. We view all service-learning experiences to be the same. 8. We make bold claims after one study.

4. General Tips for Successful Evaluation  Set measurable, realistic goals  Focus the evaluation  Have your evaluation design match evaluation question(s)  Assess the quality of the service-learning practice  Secure an appropriate sample  Have a realistic timeline for data collection  Apply appropriate instruments & data collection techniques  Use a systematic approach for data collection  Have a plan for how the data will be analyzed systematically

5. Service-Learning Quality Matters  Principles (e.g., Wingspread Principles)  Balance Beam  Quadrant  Essential Elements

6. Service-Learning: A Balanced Approach to Experiential Education Learning FOCUS PRIMARY INTENDED BENEFICIARY Service Recipient Provider Volunteerism Community Service Service-Learning Field Education Internships

7. Assessing Service-Learning Quality I II III IV Integrated Learning Unrelated Learning High Service Low Service Adapted from the Service Learning Quadrant, Service Learning 2000 Center

8. ESSENTIAL ELEMENTS OF SERVICE-LEARNING CLUSTER 1: LEARNING 1) CLEAR EDUCATIONAL GOALS 2) CHALLENGING EDUCATIONAL TASKS: 3) ONGOING ACADEMIC ASSESSMENT CLUSTER 2: SERVICE 4) MEANINGFUL SERVICE TASKS 5) EVALUATION OF SERVICE OUTCOMES From Essential Elements of Service-Learning. National Youth Leadership Council. Minneapolis, MN.

9. Essential Elements of Service-Learning CLUSTER 3: CRITICAL COMPONENTS THAT SUPPORT SERVICE AND LEARNING 6) STUDENT VOICE 7) DIVERSITY OF PERSPECTIVES 8) OPEN AND EFFECTIVE COMMUNICATION 9) STUDENT PREPARATION 10) ONGOING AND CRITICAL REFLECTION 11) CELEBRATION.

10. Assessing Student Outcomes DOMAINOUTCOMES ACADEMICIncreased retention; content knowledge and skills; higher order thinking CIVICCivic responsibility; commitment to service CAREERCareer awareness and skills PERSONAL & SOCIAL Self-esteem, empowerment, motivation, engagement (academic, civic, social); prosocial behaviors ETHICAL/MORALValues development

11. MEASUREMENTS FOR STUDENT OUTCOMES 1) CART (http://cart.rmcdenver.com/)

12. Bringle, Phillips, and Hudson (2004): Contains a series of validated scales used in service-learning studies.

13. MEASUREMENTS FOR STUDENT OUTCOMES 1) CART (http://cart.rmcdenver.com/) 2) Bringle, Phillips, and Hudson (2004) 3) Higher Education S-L Survey (Civic Responsibility, Academic Attitude, Career Development, and Empowerment) (Furco, 2000) 4) CASQ (Civic Action, Social Justice, Appreciation of Diversity) (Moely et. al., 2002)

14. ISSUES IN S-L STUDENT ASSESSMENT Student service preferences matter

15. Individual Preferences for Service TYPE EXAMPLE Charity Serve food to the homeless on Saturdays Empowerment Service Educate the homeless about social services available to them Public Work Facilitate the opening of a homeless shelter Participatory Democracy Work to secure legislation and citizen support that will secure rights for persons. Social Action Students organize a camp out on campus to raise awareness about homeless Social Change Work to reduce the number of homeless persons; train homeless persons for jobs Social Justice Secure legal assistance for a homeless person who was denied health services

16. ISSUES IN S-L STUDENT ASSESSMENT Student service preferences matter Service-learning extends student learning

17. Aspects of Student Learning in Service-Learning Classroom-based Learning Experiences LEARNING THE COURSE CONTENT Connecting Learning and Service through Reflection LEARNING ABOUT THE SOCIAL ISSUE Community-based Learning Experiences LEARNING ABOUT SERVICE Civic Responsibility Academic Achievement …Learning…Service

18. Learning the Course Content Learning Aspects of Service-Learning Students learn the process of photosynthesis, the development of nutrients in plants, and the life cycles of various types of plants.. Plant Biology Edible Garden for the Homeless Students learn: why and how individuals become homeless; the ways in which the homeless can benefit from being served the food from the gardent; the nature of homeless services in the local area, etc. Learning about Service Learning about the Social Issue Students learn: to cultivate various edible plants; the nurturing that different types of plants require; assessment of plant quality; and garden care.

19. Assessing Institutional Outcomes 1) Institutionalization 2) Institutional climate (e.g. sense of community) 3) Learning communities 4) Faculty engagement in service- learning 5) Other

20. Instruments to Measure Institutionalization (Engagement and Service-Learning) 1) Kellogg Forum Checklist 2) Committee on Institutional Collaboration Benchmarks 3) Furco Rubric for Institutionalizing Service-Learning 4) Holland Matrix on Relevance to Mission 5) Campus Compact Indicators of Engagement 7) Bringle et al.’s Comprehensive Assessment for the Scholarship of Engagement (CASE) 8) Carnegie Elective Classification for Community Engagement

21. KELLOGG FORUM: DIMENSIONS OF ENGAGEMENT A CHECKLIST of ten principles: 1) Access to Learning 2) Enhanced Diversity 3) Civic Leadership 4) Public Scholarship 5) Social Well-Being 6) Trusted Voice 7) Public Spaces 8) Community Partnerships 9) Self Governance 10) Public Accountability

22. CIC BENCHMARKS OF ENGAGEMENT 7 BENCHMARKS that show evidence of: 1) Institutional commitment to engagement 2) Institutional resource commitments 3) Student Involvement in engagement activities 4) Faculty and staff partnerships with community 5) Institutional engagement with community 6) Assessing impact & outcomes 7) Resource/Revenue opportunities generated Committee on Institutional Cooperation

23. CIC BENCHMARKS OF ENGAGEMENT 1. Evidence of Institutional Commitment to Engagement 1.1. The institution’s commitment is reflected throughout its administrative structure. 1.2. The institution’s commitment is reflected in its reward structure for faculty and staff. 1.3. The institution’s commitment is reflected in its policies and procedures designed to facilitate outreach and engagement activities. 1.4. The institution’s commitment is reflected in its policies and procedures that are responsive to students. 2. Evidence of Institutional Resource Commitments to Engagement 2.1. The institution shows evidence of leadership for engagement and outreach activities. 2.2. The institution shows evidence of financial support for engagement through its budgetary process. 2.3. The institution shows evidence that faculty and staff time is devoted to outreach and engagement activities. 2.4. The institution includes engagement activities as part of its programs for faculty, student and staff development.

24. CAMPUS COMPACT INDICATORS OF ENGAGEMENT INDICATORS that suggest the presence of engagement at the institution:  Mission and Purpose  Academic and Administrative Leadership  Disciplines, Departments, and Interdisciplinary work  Teaching and Learning  Faculty Development  Faculty Roles and Rewards  Support Structures and Resources  Internal Budget & Resource Allocations  Community Voice External  Resource Allocation  Coordination of Community-Based Activities  Forums for Fostering Public Dialogue  Student Voice

25. OPERATIONALIZATION OF INDICATORS OF ENGAGEMENT A. Mission and Purpose The institution’s mission statement explicitly articulates its commitment to the public purposes of higher education and is deliberate about educating students for lifelong participation in their communities. This aspect of the mission is openly valued and is explicitly used to promote and to explain the civic engagement and community building activities on and off campus. The institution demonstrates a genuine willingness to review, discuss, and strengthen its commitment to civic engagement and community building. All members of the campus community demonstrate their familiarity with and ownership of the institution’s mission. From Burack and Saltmarsh (2007), Advancing Civic Engagement through Strategic Assessment. NERCHE

26. FURCO SELF-ASSESSMENT RUBRIC FOR SERVICE-LEARNING INSTITUTIONALIZATION A RUBRIC built on 22 components that are organized within five dimensions: 1) Philosophy and mission of service-learning 2) Faculty support and involvement in service- learning 3) Student support for and involvement in service- learning 4) Community participation and partnerships 5) Institutional support for service-learning Three stages: Critical mass building; Quality building; Sustained institutionalization

28. FINDINGS AND LESSONS LEARNED Based on data from 209 Colleges and Universities: 1) Overall Yr. 1 institutionalization level:  =1.6, s.d. =.21 Overall Yr. 3 institutionalization level:  = 2.1, s.d. =.31 Year 1-3 change:  =.50, p = 0.018 2) Estimated 3-5 years to move from one stage to next 3) Internal assessments are more positive than external assessments 3 out of 4 cases (n = 43) in Year 1. Internal assessments are more negative than external assessments 3 out of 4 cases (n = 42) in Year 3.

29. FINDINGS AND LESSONS LEARNED 4)The self-assessment process is more important than the rubric score 5)The purposeful, ambiguous wording of the rubric encouraged discussion and dialogue 6)The components of the rubric are interdependent 7) All components do not have to be at Stage Three to achieve institutionalization 8) As you progress you may regress

30. ASSESSING COMMUNITY OUTCOMES

31. STRATEGIES FOR EVALUATING COMMUNITY OUTCOMES  Involve community partners/members in the design of the evaluation  Narrow the impact area to isolate service- learning’s effect  Secure pre-post measures; benchmark change  Support personal stories with data  Use pictures to support the data