1. Daniel Barutta and Sarah Yue, Program Officers
Problem/Need and Theory of Change/Logic Model AmeriCorps State and National External Review
Daniel Barutta and Sarah Yue, Program Officers
CNCS Template 2013

2. Learning Objectives
Understand how CNCS defines Problem/Need, Theory of change, and Logic Model
Know how to assess Problem/Need, Theory of Change, and Logic Model criteria when reviewing an AmeriCorps application
Practice assessing these criteria with sample application narratives and logic models
CNCS Template 2013

3. Theory of Change
Cause and effect relationship between a community problem or need, a program’s intervention, and the desired outcomes
Supported by data and evidence
Theory of Change elements:
Community Problem/Need
Specific Intervention
Intended Outcome
Evidence
Guides choice of intervention
Supports cause-effect relationship
Statistics documenting the need
CNCS Template 2013

4. Example: Riverton Literacy Corps
Community Problem/Need
Specific Intervention
Intended Outcome
Evidence: Research on building block skills leading to reading proficiency. Research on design, frequency, duration of tutoring sessions.
Children reading below grade level in 3rd grade
Individualized tutoring 3 times/week for 20 min on five “building block” literacy skills through reading, writing and verbal communication activities
Students are able to read at 3rd grade level (as measured by 3rd grade reading exam)
Statistics on the number of students at below grade level in program’s service area. Data explaining why this problem exists and why it is significant.

5. Logic Model A visual way of depicting the program’s Theory of Change
The community problem/need that exists
Resources available to operate the program
Planned activities (interventions)
Outcomes the program intends to achieve
Should have a logical “flow” from left to right
Should tell a coherent story about what the program will do and what it plans to accomplish
CNCS Template 2013

6. Reading a Logic Model
CNCS Template 2013

7. CNCS Logic Model Follows a prescribed template:
PROBLEM
INPUTS
ACTIVITIES
OUTPUTS
OUTCOMES
Short-Term
Medium-Term
Long-Term
The community problem that the program activities (interventions) are designed to address.
Resources that are necessary to deliver the program activities (interventions), including the number of locations/sites and number/type of AmeriCorps members.
The core activities that define the intervention or program model that members will implement or deliver, including duration, dosage and target population.
Direct products from program activities.
Changes in knowledge, skills, attitudes and opinions. These outcomes, if applicable to the program design, will almost always be measurable during the grant year.
Changes in behavior or action. Depending on program design, these outcomes may or may not be measurable during the grant year.
Changes in condition or status in life. Depending on program design, these outcomes may or may not be measurable during the grant year. Some programs, such as environmental or capacity-building programs, may measures changes in condition over a period as short as one year.
CNCS Template 2013

8. CNCS Logic Model (continued)
Programs are not required to have all three types of outcomes
Not all outcomes need to be measurable during the grant period or directly connected to performance measures
It’s OK if outcomes are in the “wrong” boxes as long as the elements are logically aligned
One type of outcome is not "better" than other types (i.e., long term outcomes are not necessarily more valuable than short-term ones, and changes in condition are not inherently better than changes in behavior)
CNCS Template 2013

9. Expectations for High-Quality Applications
Strong Theory of Change with logical alignment between problem/need, intervention, and outcome
Clearly articulated problem/need backed up by specific, relevant, current data
High-quality intervention supported by solid evidence showing that it is likely to achieve meaningful outcomes
AmeriCorps member service that provides a significant "value-add" in addressing the problem/need
CNCS Template 2013

10. Review Criteria for External Review
Problem/Need
The applicant clearly describes how the community problem/need will be addressed by the program.
The applicant clearly describes how the community problem/need is prevalent and severe in communities where members will serve and the need has been well documented with relevant data.
CNCS Template 2013

11. Review Criteria for External Review (cont.)
Theory of Change and Logic Model
The applicant clearly describes the proposed intervention including the roles of AmeriCorps members and (if applicable) the roles of leveraged volunteers.
The applicant clearly describes how the intervention is likely to lead to the outcomes identified in the applicant’s theory of change.
The applicant clearly describes how the AmeriCorps members will produce significant and unique contributions to existing efforts to address the stated problem.
All elements of the logic model are logically aligned.
CNCS Template 2013

12. Logistics for Review
Sections of the application that should be reviewed to assess the criteria:
Problem/Need narrative
Theory of Change and Logic Model narrative
Logic Model chart (located at end of application)
Evidence Base narrative
You do not need to read or consider the Performance Measures section of the application.
CNCS Template 2013

13. Logistics for Review (continued)
Level of evidence will be assessed outside of external review, but the Evidence Base section should still be read and considered in addressing the review criteria
3-page limit for logic model; do not review beyond this point
Information in logic model chart and narrative is complementary; details do not have to be repeated both places, but should not contradict each other
CNCS Template 2013

14. Scoring Rubric 4-point scale*: Exceeds the criterion
Meets the criterion
Partially meets the criterion
Does not meet the criterion
* One criterion (“All elements of the logic model are logically aligned”) will be assessed on a three-point scale without an “Exceeds the criterion” option
Refer to the Scoring Rubric document on the Reviewer Resource Webpage to read an overview of each rating.
CNCS Template 2013

15. Criterion Descriptions
The Reviewer Resource Webpage contains a chart that lists specific descriptions of the “Exceeds the criterion” and “Does not meet the criterion” ratings for each individual review criterion
The “Meets the criterion” and “Partially meets the criterion” ratings will fall between these two extremes
Please use both reference documents (the Scoring Rubric and the Criterion Descriptions) throughout your review process
CNCS Template 2013

16. Practice: Sample Application
Open the Sample Application (located on the Reviewer Resource Webpage)
Read the Problem/Need narrative, Theory of Change and Logic Model narrative, Evidence Base narrative, and Logic Model chart
Using the Scoring Rubric and Criterion Descriptions, score the Problem/Need and Theory of Change and Logic Model criteria for the Sample Application (you can record your scores on a blank IRF)
Write down your justification for each score
Pause the training until you have finished
CNCS Template 2013

17. Practice: Sample Application
The following slides provide CNCS’s assessment of what the score should be for each criterion, plus written justifications for each score.
The justifications are presented in the style of appropriate reviewer comments, but reviewers are NOT required to provide this many comments. Reviewers should limit their comments to significant strengths and weaknesses only.
It’s OK for your scores to differ slightly from CNCS’s scores. If your scores differ significantly, please re-read the Scoring Rubric and Criterion Descriptions and re-review the application.
CNCS Template 2013

18. Problem/Need – Criterion 1
Criterion: The applicant clearly describes how the community problem/need will be addressed by the program.
Score: Meets the criterion
Justification:
Strengths:
The applicant clearly defines the two needs in the target community that the program will address: environmental restoration and improved academic performance in science for economically disadvantaged youth.
The proposed member activities (planning and conducting environmental restoration projects, and developing and conducting environmental education service learning projects for sixth and seventh grade students) are logically connected to the need defined by the applicant.
CNCS Template 2013

19. Problem/Need – Criterion 1
Justification (continued):
Strengths (continued):
The proposed intervention is well suited to address the causes of declining environmental quality – reduced funding to support land maintenance and a lack of knowledge of how to increase and maintain volunteers.
Weaknesses:
It is unclear whether the environmental education service learning projects will address the cause(s) of low academic performance in science among economically disadvantaged youth.
CNCS Template 2013

20. Problem/Need – Criterion 2
Criterion: The applicant clearly describes how the community need/problem is prevalent and severe in communities where members will serve and the need has been well documented with relevant data.
Score: Meets the criterion
Justification:
Strengths:
The applicant cites specific, relevant, and reasonably current (within 6 years) data to substantiate the environmental challenges in the target region including invasive species, erosion of stream banks, and increased chemical load in the water system.
CNCS Template 2013

21. Problem/Need – Criterion 2
Justification (continued):
Strengths (continued):
The applicant provides subsidized lunch data indicating a significant level of economic disadvantage in the target schools. The applicant also cites convincing statistics indicating that economically disadvantaged students in the state have low achievement scores in science.
Weaknesses:
The academic achievement data cited by the applicant is statewide rather than specific to the target counties or schools.

22. Theory of Change and Logic Model – Criterion 1
Criterion: The applicant clearly describes the proposed intervention including the roles of AmeriCorps members and (if applicable) the roles of leveraged volunteers.
Score: Meets the criterion
Justification:
Strengths:
The application clearly describes the proposed activities of the AmeriCorps members and how they fit into the overall program design and goals.
The applicant explains how volunteers will be leveraged by the program to contribute to the desired outcomes.

23. Theory of Change and Logic Model – Criterion 1
Justification (continued):
Weaknesses:
It is not clear to what extent the proposed outcome targets will be accomplished by members vs. by leveraged volunteers. [Contextual note: either way is fine as long as it is clearly explained in the application.]

24. Theory of Change and Logic Model – Criterion 2
Criterion: The applicant clearly describes how the intervention is likely to lead to the outcomes identified in the applicant’s theory of change.
Score: Meets the criterion
Justification:
Strengths:
The proposed member activities are logically connected to the desired outcomes of improved environmental quality and increased student academic achievement in science.
Previous performance measure data indicates that the program has been effective in improving environmental quality in the target region.

25. Theory of Change and Logic Model – Criterion 2
Justification (continued):
Strengths (continued):
The applicant cites a recent, relevant study on a similar student demographic that demonstrates that high-quality service learning activities resulted in increased student academic achievement. The applicant has modeled their own program design after this study, with additional thoughtful modifications that are responsive to requests from the local schools.
Weaknesses:
Details are lacking regarding what the service-learning projects will be or how they will be implemented, making it hard to assess whether the program's effectiveness is likely to match that of the study results.
CNCS Template 2013

26. Theory of Change and Logic Model – Criterion 2
Justification (continued):
Weaknesses (continued):
The applicant does not cite sufficient evidence that the program activities will lead to the proposed mid- and long-term outcomes (volunteers increasing their understanding of the importance of conservation, and restoration and conservation remaining sustainable).
[Contextual note: the fact that the evidence cited by the program was non-experimental will be considered as part of the Level of Evidence determination, which is outside of the scope of external review. It should not be assessed here.]
CNCS Template 2013

27. Theory of Change and Logic Model – Criterion 3
Criterion: The applicant clearly describes how the AmeriCorps members will produce significant and unique contributions to existing efforts to address the stated problem.
Score: Meets the criterion
Justification:
Strengths:
AmeriCorps members will be providing environmental restoration services that state and local governments and nonprofits lack the resources to accomplish. Members will also be building the capacity of local organizations to recruit and support volunteers to help meet these needs.
The service-learning activities developed by AmeriCorps members will be new to the target schools and will enhance the existing curriculum.

28. Theory of Change and Logic Model – Criterion 3
Justification (continued):
Weaknesses:
The applicant does not describe in sufficient detail how the AmeriCorps members' roles will differ from those of existing staff at the public and private parks where they will serve.
CNCS Template 2013

29. Theory of Change and Logic Model – Criterion 4
Criterion: All elements of the logic model are logically aligned.
Score: Partially meets the criterion [Note: this criterion is scored on a three-point scale.]
Justification:
Strengths:
The needs, interventions, outputs, and outcomes in the logic model are directly connected to each other, and the "flow" of the logic model generally makes sense.
Weaknesses:
The logic model is lacking some important details, such as the inputs for two of the three rows, which negatively affect its alignment. Some of the descriptions provided in the boxes, such as the "Activities" description in the third row, are not sufficiently detailed, and the application narrative does sufficiently compensate for these missing details.
CNCS Template 2013

30. Theory of Change and Logic Model – Criterion 4
Justification (continued):
Weaknesses (continued):
The row in the logic model about preparing members for careers in environmental stewardship is not directly connected to the program's theory of change as described in the narrative. [Contextual note: some programs do include member development as a primary goal of the program, particular programs that recruit disconnected youth as members. However, this particular application does not appear to be a program of that type.]
CNCS Template 2013

31. Theory of Change and Logic Model – Criterion 4
Justification (continued):
[Contextual note: it is OK that the program listed changes in condition – such as improvement of trails and parkland - as short-term outcomes, because this is consistent with the program's theory of change. It is also OK that the program doesn't list all three types of outcomes for all of their proposed interventions; programs were not required to have all three kinds of outcomes.]
CNCS Template 2013

32. Recap: Learning Objectives
Understand how CNCS defines Problem/Need, Theory of Change, and Logic Model
Know how to assess Problem/Need, Theory of Change, and Logic Model criteria when reviewing an AmeriCorps application
Practice assessing these criteria with sample application narratives and logic models
After completing this training, you will take an assessment that will give you more practice in scoring sample application narratives.
CNCS Template 2013

33. Assessment Questions
To check for understanding and to verify that you have completed this orientation session, please complete the assessment at the following link:
(Please reference the PowerPoint on the Reviewer Resource Webpage to access the link)
CNCS Template 2013