1. Sustaining Community Projects: Logic Model Construction and Implementation CYFAR Evaluation Team CYFAR, 2005

2. Introduction  Across the United States there are a multitude of programs designed to improve the quality of life for individuals, families, and communities.  Evaluation is essential for successful programming to document what happens in the program, demonstrate which strategies work best, and assess the short-term and long-term outcomes of the program.

3.  Understanding how a program achieves results is critical to determine if program elements should be modified and provide stakeholders with evidence that the program is effective.  Evaluation theory provides the basic conceptual framework for thinking about problems and how change should occur.  Logic models provide valuable ways to organize that thinking around evaluation theory.

4. Logic Model  Program theory articulates program impact and program processes.  Logic models provide a visual picture of program theory in action (Mancini, Huebner, McCollum and Marek, 2005).  Of primary importance in developing logic models is establishing and maintaining a focus on results rather than on activities.

5.  Logic models help program professionals, evaluators and stakeholders reach consensus about which elements are essential to the program (Millar, Simeone, & Carnevale, 2001; Orthner & Bowen, 2004).  Consequently program professionals and researchers are more able to identify faulty or implausible links early on.

6. SCP Logic Model  The logic model you are asked to complete includes five elements: Identified needs and assets, Desired results (short-term and long- term), Indicators (short-term and long-term), Activities, and Resources.  These are found on the Logic Model Template Worksheet.

7.  “Desired Results” Worksheets are provided to help you detail information necessary to describe anticipated results.  An Analysis column is included on the Desired Results Worksheets that allows you to discuss how you will analyze the data that are collected.  To assist you in completing your Program Logic Model, please refer to the sample logic model of the After School Program.

8. Step 1: Identified Needs and Assets  The first step is to identify needs and assets in the community which allow program personnel to clearly identify and understand the issues.  Because this is the basis for the entire program, it is important to use accurate and valid information rather than uninformed hunches.  Be sure to look at needs and assets at many levels, from the community systems such as health, education, legal, etc. to families and to individuals.

9.  Needs and assets analysis is a systematic way of gathering information that helps set priorities for action while also recognizing the assets that already exist in a community.  In the logic model process the information that evaluators are gathering to understand severity of a problem also assists them in determining program priorities, and results that accrue from program activities.  Census data, observations, records and existing data bases, and literature reviews are sources of existing information to identify needs and assets (Witkin & Altshuld, 1995).

10. Step 2: Desired Results  The second step in the logic model process is to articulate the desired result or change you expect to see because of your program.  This is arguably the most important aspect of a logic model because it provides the focus that all successful programs need.  At the completion of the program, what will be different?

11.  Results should be defined as short-term and long- term.  Specifying results is useful because it enables program professionals to have a more realistic sense of how parts of the program theory fit, and what can reasonably be expected to change over a specific time period.  Demonstrated results are not expected to be documented by following the same participants for five years, but rather are focused on all participants, or each group, as they complete participation in programs.

12. Short-Term Results  Changes in knowledge, attitudes, and/or behaviors that are anticipated as a result of participating in programs can be considered short-term results.  These results should be reported at the end of each of the five years.

13. Long-Term Results  System change or changes in organizations, community capacity, program support, and/or program sustainability may be considered long-term results.  Progress toward long-term results can be reported for each or any of the five years but MUST be reported in year 5.

14. Step 3: Indicators  Indicators flow logically from the desired results. How will you know if the results have been achieved, with what evidence?  By determining program indicators, program activities and results can be measured and tracked. Consequently notions, hunches, hearsay, and anecdotes are not good indicator candidates.

15.  Indicators can be far-ranging, including results from surveys and focus groups, agency administrative data, population data, as well as other measures (DeVellis, 2003; McKillip, 1998).  Evaluation Work Sheets are included to help you specify program indicators for each of your program goals.

16. Step 4: Activities  Activities describe programs that are implemented to achieve the desired results.  Program activities become important only when they are linked to desired results (Bowen et al., 2001).  Attention to the choice of programs, research upon which it is based, population to be served and critical elements of successful programs must all be considered.  By referring to the Guiding Principles for the National Outcome that you have chosen for your project, you will be assisted in determining appropriate activities for your projects – based upon the desired results that you have determined.

17. Step 5: Resources  Resources include the people, stakeholders, curricula, spaces, approvals, funds and anything else needed to conduct the activities to achieve the desired results to meet the identified needs and capitalize on the available assets.

18. Monitoring and Evaluation  An important part of the logic model is how the intervention will be monitored and evaluated, including program implementation and its intended results.  Monitoring and evaluation permeate all parts of a logic model.

19.  There are multiple objectives for monitoring and evaluation: –to identify the initial extent of need; –to track how the need may change over time; –to document how well program implementation protocols are being followed; –to mark progress toward achieving results; –to know when results have been met; and –to discern how resource use has supported particular program efforts and subsequent results.

20. Logic Model Work Sheets  The Logic Model Work Sheets allow you to put your Program Logic Model to work for you. –These worksheets are provided to help you identify the anticipated results (short-term and long-term) for your program as well as indicators. –You will also need to detail the data collection process: sources of data, time of collection, and method of collection. –Finally, you will also note the analyses you anticipate conducting.

21.  Please refer to the Program Logic Model Example for more detail on how you can complete these sheets for your project. –For example, in the sample logic model, the Short-Term Desired Result Work Sheet shows the indicators or evidence for knowledge, attitude, and/or behavior change. –The work sheet clearly shows what data are needed, when it will be collected, by whom, and how it will be analyzed. –Completion of these evaluation work sheets will guide the entire data collection, analysis and reporting processes over the course of the project.

22. Identified Needs and Assets Indicators Short Term: Long-term: Resources Activities Logic Model Template Note:Monitoring and evaluation activities permeate all aspects of the logic model. Results Short Term: Long-term:

23. Example of Logic Model and Work Sheets  The following provides an illustration and application of the logic model using an after school program as the example.  This example is designed to stimulate your thinking about how to most effectively integrate your plan with this results-oriented logic model template.  We anticipate that your logic model will contain more detail and be more comprehensive.

24.  The Logic Model is based upon the School- Age (K-8) national outcome.  The example shows the relationships between the identified needs and assets, the short-term and long-term desired results, indicators, activities and resources.  The following work sheets for the Short- Term and Long-Term Desired Results provide further details regarding the indicators, including the sources of data, times and methods of data collection, followed by planned analyses.

25. Step 1: Identified Needs and Assets   Lack of parental monitoring after school   Many children in park and on streets in groups after school   Church building available for program use   Low homework completion rates among 3rd to 5th graders

26. Step 2: Desired Results  Short-Term: –Homework completion rates for children in grades 3-5 will increase –Parents will gain knowledge of after school resources –Parents will gain knowledge of the importance of parental monitoring

27. Step 2: Desired Results  Long-Term: –Community support for after school programs will be sustained –Parents will monitor their children

28. Step 3: Indicators  Short-Term: –Attendance rate at after school program –Increased rates of homework completion –Increased parental requests for after school programs  Long-Term: –Community collaboration and support for space rental, staff training, recruitment, parent education and funding –Fewer children on streets & in park after school

29. Step 4: Activities  Develop and implement a structured tutoring program  Develop and implement a multi-component after school program, including recreation  Develop community media campaign on importance of parental monitoring and after school care  Educate school, faith and business communities about importance of after school programs through information sessions

30. Step 5: Resources  School, faith & business community leaders  Parents  Children in K-5  Staff time of existing organizations  Local media cooperation  Grant funds

31. Work Sheet Completion: Short-Term Desired Results Homework completion rates for children in grades 3-5 will increase.  Short-Term Indicator: Homework completion rates in grades 3, 4, & 5.  Source of Data: Teachers’ classroom records.  Time of Collection: These records are currently kept and reported quarterly to parents.  Methods of Collection: –Individuals will be coded for confidentiality. –Enrollment in the after school program will be coded yes or no, and homework completion rate will be reported as percentage of assignments turned in. –Teachers will report to evaluator each quarter.  Data Analysis: –Completion rates for each participant will be compared to the rate for the previous year using paired t-tests. –Completion rates for children who are and are not in the after school program will be compared.

32. Work Sheet Completion: Long-Term Desired Results Community support for after school programs will be sustained.  Long-Term Indicators : –Increased community collaboration membership –Increased sources of fiscal responsibility for program  Source of Data: Collaboration meeting minutes, Annual budget and financial sources for program, Administrator interviews  Time of Collection : Prior to program implementation and annually thereafter. –Media & and other relevant materials to be collected or documented in a portfolio as they occur.  Methods of Collection: –Project evaluator and director will work together to document and collect minutes of collaboration meetings and other key events that influence system change and stakeholder involvement. –Portfolios will be used to document progress.  Data Analysis: –Formative evaluation during the first two years of the program will help refine and direct efforts at building community support. –Evaluator will report at least bi-annually to the collaboration on progress made and key areas that need to be strengthened.