Presentation on theme: "Assessing K-12 Service-Learning Impacts and Quality Shelley H. Billig, Ph.D. RMC Research Corporation."— Presentation transcript:
Assessing K-12 Service-Learning Impacts and Quality Shelley H. Billig, Ph.D. RMC Research Corporation
Overview Evaluation Toolkit Logic Models and Theory of Change Measuring Impact –Academic impact –Civic impact –Personal/social impacts –Others Measuring Quality
Evaluation Toolkit How-To Guide For Traditional Evaluation Overview Steps for Planning and Implementation Worksheets Detailed Guidance
Session Experiences “Dip in” for various aspects of the evaluation and assessment process.
Logic Models Shows how and why the program will work; Connects program activities to program outcomes through outputs: –e.g., activity (input) Increased student engagement in schools (output) Increased student academic achievement (outcome).
TX K-12 Learn and Serve Logic Model
Try It! Start with your activities – what will you do? Specify your goals – what will you accomplish? Identify those factors that will influence results, both positively and negatively. Include demographics, context, implementation issues
Logic Model Use Once the logic model has been created with stakeholders, review it. Use the logic model to design or revamp the evaluation. Use the constructs identified in the logic model to guide the development of measures (instrumentation).
Logic Model Resources University of Wisconsin-Extension Program Development and Evaluation, W.K. Kellogg Foundation, –Evaluation Handbook, 1998 –Logic Model Development Guide, 2001
Assessing Academic Impact –Where is service-learning likely to have the greatest academic impact? Knowledge areas (general and specific) Types of skills (general and specific) Moderators –What measures are available?
Academic Skills Areas of likely impact –Higher order thinking skills such as problem solving, analysis, transfer of learning –Specific knowledge related to service activities, e.g., measurement, writing, synthesizing materials, research What are the available measures? –Rubrics –Test scores –Writing scores –Problem solving essays –Measures of academic engagement –Absenteeism –Surveys –More!
Assessing Civic Impacts Where is service-learning likely to have the greatest civic impact? –Knowledge? –Skills? –Dispositions? –Attachment to community? Moderators What measures are available?
Civic Impacts Areas of likely impact –Knowledge of community needs –Skills for planning and implementing projects –Taking responsibility –Ethic of service Available measures –Surveys –Focus groups/interviews –Hours of service –Actual changes
Assessing Social/Personal Impacts –Where is service-learning likely to have the greatest impact on social/personal areas? Knowledge areas (general and specific) Types of skills (general and specific) Moderators –What measures are available?
Social/Personal Outcomes Areas of Likely Impact –Development of character traits such as responsibility, trust, and caring –Respect for diversity –More bonds with adults –More protective factors such as resilience –Greater self-efficacy Measures –Surveys –Focus groups/interviews –Rubrics –Self-assessments
Sample Measures of Outcome Areas Examine the samples What is the best use of each? By whom and for what purpose? What do you like and dislike about each?
Measuring Moderators Moderators are characteristics that influence outcomes. Examples: Instructor experience; program design; program quality. Need to identify and measure those factors like to serve as moderators.
What moderators are likely to be in play in your settings? Consider demographics. Consider program design. Consider quality. Consider instructor characteristics like experience, levels of training, and expertise. Consider partner characteristics.
Characteristics of Good Assessments Valid Reliable Objective Useful Items measure only one construct at a time (not double barrelled…) Related to the logic model or assessment questions
Open Discussion Questions and Answers
Resources National Service-Learning Clearinghouse Billig, S. H., & Waterman, A. S., eds. (2003). Studying service-learning: Innovations in education research methodology. Mahwah, NJ: Lawrence Erlbaum Associates. Available at Bringle, R. G., Phillips, M. A., & Hudson, M. (2004). The measure of service learning: Research scales to assess student experiences. Washington, DC: American Psychological Association. Available at