2014 AmeriCorps External Reviewer Training Assessing Need, Theory of Change, and Logic Model

Topics  Purpose of Logic Model  Assessing Need, Theory of Change and Logic Model 2

Preview of Assessment Questions (True/False)  The Logic Model is a visual representation of the applicant’s Theory of Change.  External Reviewers should assess whether the outcomes in the Logic Model are measurable. (Multiple Choice)  Data about the number of economically disadvantaged beneficiaries in a community would most likely describe: (Practice Your Understanding)  Complete portion of Review Assessment Form 3

What is a Logic Model?  A visual summary or snapshot of your program that communicates how your program works, the resources you have to operate your program, the activities you plan, and the outcomes you hope to achieve  A well-specified conceptual framework that identifies key components of the proposed process, product, strategy, or practice and describes the relationships among the key components and outcomes, theoretically and operationally (Notice of Funding Opportunity (NOFO) p. 18)  The purpose of a Logic Model is to describe how a program will create change. Logic Model 4

How a Logic Model Works Logic Model 5 Resources/ Inputs ActivitiesOutputs Outcomes/ Impacts Long-Term Goal Certain resources are needed to operate your program If you have access to them, then you can use them to accomplish your planned activities If you accomplish your planned activities, then you will hopefully deliver the amount of product and/or service that you intended If you accomplish your planned activities to the extent you intended, then your participants will benefit in certain ways If these benefits to participants are achieved, then certain changes in organizations, communities, or systems might be expected to occur Figure 1: How to Read a Logic Model  Your Planned WorkYour Intended Results

Applicant Logic Models  Inputs – Resources, including AmeriCorps members  Activities (Interventions)  Outputs (Beneficiaries, other products of program services)  Outcomes  Short Term (Knowledge, Skills, Attitudes, Opinions)  Medium Term (Behavior)  Long-Term (Condition) Logic Model 6

Key Points  The Logic Model and narrative work together to depict the program’s Theory of Change  All the information in one doesn’t have to be in the other, as long as the two are aligned  The Logic Model is simply a visual representation of the Theory of Change and program design  Applicants do not measure all outcomes in Theory of Change; they continue to measure outcomes that make sense for their program; External reviewers do not assess applicants’ performance measures and so should not be concerned with whether outcomes are measurable  CNCS provided a template for applicants’ Logic Models Logic Model 7

Theory of Change Elements Assessment: Theory of Change 8 Community Problem/Need Specific Intervention Intended Outcome Evidence Guides choice of intervention Supports cause-effect relationship Evidence Guides choice of intervention Supports cause-effect relationship Statistics documenting the need

How Do The NOFO Pieces Fit? Applicant’s Overall Theory of Change  Need narrative  Theory of change narrative  Logic model Assessment 9

The Selection Criteria & Standards Four Key Themes  Completeness (Logic Model)  Alignment/Logic  Quality of Theory of Change  Level/Quality of Evidence Assessment 10

Review Rubric  Most items scored on a five-point scale  Some items scored on a two-point scale (Satisfactory/Unsatisfactory)  Refer to the Review Rubric in the handbook to read the descriptions for each Rating Assessment 11

Completeness (Logic Model)  Number of locations and sites where members are providing services  Number of members that will be delivering the intervention  Core activities that define the intervention or program model  Duration of the intervention  Dosage of the intervention  Target population for the intervention Assessment 12

How Assessed  Series of check boxes on the review form  Reviewers check the box if the item is present in the Logic Model Assessment 13

Alignment with Logic Model  The Theory of Change is described consistently in the Logic Model and application narrative  The inputs, activities and outputs are logically aligned  The Theory of Change and Logic Model cover comprehensively the applicant’s entire program Assessment: Theory of Change 14

Quality of Theory of Change Problem  The community problem/need is prevalent and severe in the communities where members serve  The applicant cites specific, relevant data to document the need Intervention  The intervention is an effective way to address the problem/need identified by the applicant Outcomes  The outcomes identified in the Logic Model are logically aligned with the problem/need and intervention Assessment: Theory of Change 15

Assessing Alignment and Quality Alignment and Quality are assessed using a five-point scale:  Greatly exceeds the standard  Exceeds the standard  Meets the standard  Partially meets the standard  Does not meet the standard at all Refer to the Review Rubric to read the descriptions for each Rating. Assessment: Theory of Change 16

Need vs. Evidence Applicants are asked to provide data to demonstrate:  Prevalence and severity of need versus the level and quality of evidence  Specific, relevant data to document the need  Evidence to support the choice of intervention Assessment: Need 17

Data documenting problem/need should demonstrate the prevalence and severity of the problem: Who and how many are directly affected? How severe is the problem? 18 Community Problem/Need Assessment: Need

Examples  Unemployment rate  Poverty rate  Percent of students eligible for free/reduced lunch  Population or size of area affected by natural disaster  Percent of adult population with diabetes Assessment: Need 19

Data Should Be  From a reputable source such as a government agency, foundation, university or research organization  Recent  Local Assessment: Need 20

Common Issues  State or national data is provided instead of local data describing the need in the specific communities where the service will occur  Statistics are unrelated to the problem/need  Statistics cited are not recent  Data sources are not identified or are not reputable Assessment: Need 21

Next Steps To check for understanding and verify that you have completed this orientation session, please reference the remaining slides for the two-part Assessment of this Orientation. 22

Practice Your Understanding  Answer the Assessment Questions at the following link:  Read and assess the norming/sample application and complete the Rating sections for the Problem/Need, Theory of Change, and Logic Model sections of the review form. Do not include the significant strengths, weaknesses, or other narrative sections.  You will receive a review form completed by a CNCS staff member. Compare your assessments to those in the key.  This will complete your verification requirement for this training module. 23 Assessment: Practice

Practice SELF-ASSESSMENT When you receive the key for the practice activity from CNCS, compare your Ratings to those of the CNCS standard example. If your Rating was more than one Rating higher or lower than the example provided by CNCS, your assessment may not be normed closely enough to CNCS’ expectations for applicants Assessment: Practice 24