Presentation on theme: "2014 AmeriCorps External Reviewer Training"— Presentation transcript:
1 2014 AmeriCorps External Reviewer Training Assessing Need, Theory of Change, and Logic Model of Tier 1 Education Applications2014 AmeriCorps External Reviewer Training
2 Topics Purpose of Logic Model Assessing Need, Theory of Change, and Logic Model
3 Preview of Assessment Questions (True/False)The Logic Model is a visual representation of the applicant’s Theory of ChangeExternal Reviewers should assess whether the outcomes in the Logic Model are measurable(Multiple Choice)Data about the number of economically disadvantaged beneficiaries in a community would most likely describe:(Practice Your Understanding)Complete portion of Review Assessment Form
4 Logic ModelWhat is a Logic Model?A visual summary or snapshot of your program that communicates how your program works, the resources you have to operate your program, the activities you plan, and the outcomes you hope to achieveA well-specified conceptual framework that identifies key components of the proposed process, product, strategy, or practice and describes the relationships among the key components and outcomes, theoretically and operationally (Notice of Funding Opportunity (NOFO) p. 18)The purpose of a Logic Model is to describe how a program will create change.
5 Certain resources are needed to operate your program Logic ModelHow a Logic Model WorksFigure 1: How to Read a Logic ModelIf these benefits to participants are achieved, then certain changes in organizations, communities, or systems might be expected to occurIf you accomplish your planned activities, then you will hopefully deliver the amount of product and/or service that you intendedIf you accomplish your planned activities to the extent you intended, then your participants will benefit in certain waysIf you have access to them, then you can use them to accomplish your planned activitiesCertain resources are needed to operate your programResources/ InputsActivitiesOutputsOutcomes/ ImpactsLong-Term GoaljklmnYour Planned WorkYour Intended Results
6 Applicant Logic Models Inputs – Resources, including AmeriCorps membersActivities (Interventions)Outputs (Beneficiaries, other products of program services)OutcomesShort Term (Knowledge, Skills, Attitudes, Opinions)Medium Term (Behavior)Long-Term (Condition)
7 Logic ModelKey PointsThe Logic Model and narrative work together to depict the program’s Theory of ChangeAll the information in one doesn’t have to be in the other, as long as the two are alignedThe Logic Model is simply a visual representation of the Theory of Change and program designApplicants do not measure all outcomes in Theory of Change; they continue to measure outcomes that make sense for their program; External reviewers do not assess applicants’ performance measures and so should not be concerned with whether outcomes are measurableCNCS provided a template for applicants’ Logic Models
8 Theory of Change Elements Assessment: Theory of ChangeTheory of Change ElementsCommunity Problem/NeedSpecific InterventionIntended OutcomeStatistics documenting the needEvidenceGuides choice of interventionSupports cause-effect relationship
9 How Do The NOFO Pieces Fit? AssessmentHow Do The NOFO Pieces Fit?Applicant’s Overall Theory of ChangeNeed narrativeTheory of Change narrativeLogic Model
10 The Selection Criteria & Standards AssessmentThe Selection Criteria & StandardsFour Key ThemesCompleteness (Logic Model)Alignment/LogicQuality of Theory of ChangeLevel/Quality of Evidence
11 Review Rubric Most items scored on a five-point scale AssessmentReview RubricMost items scored on a five-point scaleSome items scored on a two-point scale (Satisfactory/Unsatisfactory)Refer to the Review Rubric in the handbook to read the descriptions for each Rating
12 Completeness (Logic Model) AssessmentCompleteness (Logic Model)Number of locations and sites where members are providing servicesNumber of members that will be delivering the interventionCore activities that define the intervention or program modelDuration of the interventionDosage of the interventionTarget population for the intervention
13 How Assessed Series of check boxes on the review form AssessmentHow AssessedSeries of check boxes on the review formReviewers check the box if the item is present in the Logic Model
14 Eligible Intervention(s) in Logic Model or Narrative AssessmentEligible Intervention(s) in Logic Model or NarrativeReviewers will also indicate whether or not the applicant’s Logic Model and/or Narrative incorporates at least one of the following six required member interventions for Tier 1 Education applicants (listed on this slide and the next slide):1) Providing ongoing mechanisms for family and community engagement2) Establishing a school culture and environment that improves school safety, attendance, and discipline and addresses other non-academic factors that impact student achievement, such as students’ social, emotional, and health needs3) Accelerating students’ acquisition of reading and mathematics knowledge and skills
15 Eligible Intervention(s) in Logic Model or Narrative AssessmentEligible Intervention(s) in Logic Model or Narrative4) Increasing graduation rates through strategies such as early warning systems, credit-recovery programs, and re-engagement strategies. 5) Increasing college enrollment rates through college preparation counseling assistance to include completing the Free Application for Federal Student Aid (FAFSA) and college applications, and educating students and their families on financial literacy for college 6) Supporting school implementation of increased learning time
16 Alignment with Logic Model Assessment: Theory of ChangeAlignment with Logic ModelThe Theory of Change is described consistently in the Logic Model and application narrativeThe inputs, activities and outputs are logically alignedThe Theory of Change and Logic Model cover comprehensively the applicant’s entire program
17 Quality of Theory of Change Assessment: Theory of ChangeQuality of Theory of ChangeProblemThe applicant cites specific, relevant data to document the needs that the AmeriCorps members will be addressing and these needs were identified with the involvement of eligible partner school(s) and LEA leadershipThe need in community and in the school(s) where the proposed AmeriCorps members will serve is prevalent and severeInterventionThe potential contribution of AmeriCorps members is an effective way to address the needs identified by eligible school and LEA leadershipThe project effectively addresses multiple student needsOutcomesThe outcomes identified in the Logic Model are logically aligned with the problem/need and intervention
18 Assessing Alignment and Quality Assessment: Theory of ChangeAssessing Alignment and QualityAlignment and Quality are assessed using a five-point scale:Greatly exceeds the standardExceeds the standardMeets the standardPartially meets the standardDoes not meet the standard at allRefer to the Review Rubric to read the descriptions for each Rating.
19 Need vs. Evidence Applicants are asked to provide data to demonstrate: Assessment: NeedNeed vs. EvidenceApplicants are asked to provide data to demonstrate:Prevalence and severity of need versus the level and quality of evidenceSpecific, relevant data to document the needEvidence to support the choice of intervention
20 Community and School Problem/Need Assessment: NeedCommunity and School Problem/NeedData documenting problem/need should demonstrate the prevalence and severity of the problem:Who and how many are directly affected?How severe is the problem?
21 Examples Graduation rate Assessment: NeedExamplesGraduation ratePercent of students eligible for free/reduced lunchStudent academic achievement dataData about the school culture or environmentData about family engagementStudent attendance data
22 Assessment: NeedData Should BeFrom a reputable source such as a government agency, foundation, university or research organizationRecentLocal
23 Assessment: NeedCommon IssuesState or national data is provided instead of local data describing the need in the specific communities where the service will occurStatistics are unrelated to the problem/needStatistics cited are not recentData sources are not identified or are not reputable
24 Next StepsTo check for understanding and verify that you have completed this orientation session, please reference the remaining slides for the two-part Assessment of this Orientation.
25 Practice Your Understanding Assessment: PracticePractice Your UnderstandingAnswer the Assessment Questions at the following link:Read and assess the norming/sample application and complete the Rating sections for the Problem/Need, Theory of Change, and Logic Model sections of the review form. Do not include significant strengths, weaknesses, or other narrative sections.You will receive a review form completed by a CNCS staff member. Compare your assessments to those in the key.This will complete your verification requirement for this training module.
26 Assessment: PracticePracticeSELF-ASSESSMENT When you receive the key for the practice activity from CNCS, compare your Ratings to those of the CNCS standard example. If your Rating was more than one Rating higher or lower than the example provided by CNCS, your assessment may not be normed closely enough to CNCS’s expectations for applicants