Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Role of Program Evaluation in Federal Grant Applications Dixie L. King, Ph.D. 5500 Ming Ave, Suite 240 Bakersfield, CA 93309

Similar presentations


Presentation on theme: "The Role of Program Evaluation in Federal Grant Applications Dixie L. King, Ph.D. 5500 Ming Ave, Suite 240 Bakersfield, CA 93309"— Presentation transcript:

1

2 The Role of Program Evaluation in Federal Grant Applications Dixie L. King, Ph.D Ming Ave, Suite 240 Bakersfield, CA

3 Program Evaluation: What People Love to Hate by Any Name  Nearly every federal grant requires program evaluation; but this can be deceptive, because:  Not all grants discuss “program evaluation,” per se. However, if the grant application references “data tracking”; “data collection”; “performance outcomes”; “performance indicators”; “measureable objectives” or “GPRA (Government Performance & Results Act) measures”—it is requiring evaluation.  Ensuring that you adequately address program evaluation—by any other name—in your grant will substantially increase your chances of funding.

4 Local vs. Cross-site or National Evaluations  The Request for Proposal (RFP) or Request for Application (RFA) will usually specify whether you are required to participate in a cross-site or national evaluation, fund your own local evaluation, or both.  Participation in a national or cross-site evaluation means that you will be required to collect the same data as other grantees, and report them periodically to the granting agency. These data are then used to generate data that can be reported to Congress or used in research.  Local evaluation is concerned with YOUR objectives, without regard to what other grantees may be measuring.

5 When Conducting a Local Evaluation, Keep In Mind…  The local evaluator is on your side. He/she is there to help you determine whether you are on course to reach your desired outcomes—and whether and how to make mid- course adjustments if you are not.  The local evaluator is on your side; nevertheless, the evaluator is obligated to give you the facts and help you interpret them. This means you may not like what the evaluator has to say on occasion. Don’t kill the messenger.  Working closely with your local evaluator is one of the best steps you can take to ensure that your program reaches its potential, and changes lives for the positive.

6  Logic models come in all shapes and sizes, but they all have one thing in common: they structure the way you address your goals, your needs, your activities, and your performance indicators (that is, your anticipated outcomes).  Advice to consider: before you write your grant, use your first stakeholder meeting to develop your logic model. Invite an evaluator to assist you in this process—and if you want this person to be able to bid on the evaluation contract without a conflict of interest, pay them for their assistance. The Logic Model: A Tool to Address Evaluation and Structure Your Grant Narrative

7  There are five main components to a logic model:  Overarching Goal(s) of the Project  Needs/Gaps in Service to Be Addressed  Performance Indicators/Process Measures  Outcome Indicators/Measurable Objectives  Supporting Activities  Outcome Indicators or Objectives may be broken down into short, intermediate and long-term.  Additional components may include partners and timeline. Logic Model Components

8 Building a Logic Model: Examples based on a Project to Reduce Teen Pregnancy 5/3/2015

9 Breaking It Down: Your Goal(s)  A goal is the overarching purpose behind the project. Most projects have only one or two goals.  Goals tend to be broad and should encompass all of your activities and indicators/objectives.  Examples of goals:  Decrease the number of teen pregnancies in Kern County.  Reduce rates of criminal justice recidivism related to substance abuse.  Increase the percentage of low income families receiving preventive health care.

10 Breaking It Down Step 1: Identify Needs (start with the problem) 5/3/2015 9

11 Breaking It Down: Needs/Gaps in Services  Your statement of need will be part of your project narrative, but is also the foundation of your logic model.  Needs should provide baseline data against which your progress can be measured.  Indicate the source of your data if source is not implied.  Because your needs will be linked to specific outcome objectives, avoid combining needs in one statement.

12 Breaking It Down: Needs/Gaps in Services  Examples of a well-stated need:  During the school year, 47 students enrolled at HighRisk High School became pregnant out of a student population of 322 female students, for a pregnancy rate of 14.6%.  Examples of a poorly stated need:  Teen pregnancy is a problem at HighRisk High School. OR:  Abstinence curriculum needs to be adopted at HighRisk High. OR:  The teen pregnancy rate at HighRisk High in was 14.6% and 87% of these teens failed to return to school.

13 12 Goal: To reduce teen pregnancy in HighRisk School District. NeedsActivities Process Indicators Outcome Indicators 1.1 During the school year, 47 students enrolled at HighRisk High School became pregnant out of a student population of 322 female students, for a pregnancy rate of 14.6% % of 11 th grade students surveyed who indicated that they are sexually active reported that they do not use birth control. 1.3 The percentage of students screening positive for suicide ideation was 30% higher among and pregnant parenting students, as measured by the Columbia Teen Screen.

14 5/3/ Breaking It Down: Step 2: Build Outcome Indicators/Measurable Objectives around Needs

15 Breaking It Down: What an Outcome Indicator Is A credible means of showing that people’s lives have changed as a result of an activity, program or service.

16  An activity, a program, or a service.  A nose count of people served.  A description of program intensity.  An anecdote.  A feeling or intuition. Breaking It Down: What an Outcome Indicator Is NOT

17 Breaking It Down Outcome Indicators ARE: Measureable Specific Meaningful/Relevant

18 Goal: To reduce teen pregnancy in HighRisk School District. NeedsActivities Process Indicators Outcome Indicators 1.1 During the school year, 47 students enrolled at HighRisk High School became pregnant out of a student population of 322 female students, for a pregnancy rate of 14.6% % of 11 th grade students surveyed who indicated that they are sexually active reported that they do not use birth control. 1.3 The percentage of students screening positive for suicide ideation was 30% higher among pregnant and parenting students, as measured by the Columbia Teen Screen. 1.1 By June 2018, the pregnancy rate at HighRisk High School will decrease by 25% from baseline, from 14.6% to no more than 11%. 1.2 By June 2018, the percentage of 11 th grade students who indicate they are sexually active but not using birth control will decrease by 50% from baseline. 1.3 By June 2018, the percentage of pregnant and parenting students screening positive for suicide ideation will be within 5% of the general student population.

19 A few reminders: You want your outcomes to be REASONABLE and ACHIEVABLE, based on the actual services you are providing. It would NOT make sense to believe, for example, that by training teachers in an evidence-based teen pregnancy prevention curriculum will automatically lower the teen pregnancy rate at the school. You also need to ask: Are teachers actually teaching the curriculum? Are they teaching it with fidelity? How many students actually receive it? Breaking It Down: Outcome Indicators

20 Breaking It Down Step 3: Identify Activities/Services that Will Assist in Producing the Desired Outcome

21 Breaking It Down: Supporting Activities/Services  Activities/Programs should be clearly linked to the desired outcome, preferably through an evidence- based or best practices program.  Activities ideally should target universal populations (for prevention), select populations (early intervention), and indicated populations (intensive early intervention).  You don’t have to specify a large number of activities. It is better to select a few, well-targeted activities and programs that have a demonstrated track record in achieving the outcome you want.

22 Goal: To reduce teen pregnancy in HighRisk School District. NeedsActivities Process Indicators Outcome Indicators 1.1 During the school year, 47 students enrolled at HighRisk High School became pregnant out of a student population of 322 female students, for a pregnancy rate of 14.6% % of 11 th grade students surveyed who indicated that they are sexually active reported that they do not use birth control. 1.3 The percentage of students screening positive for suicide ideation was 30% higher among pregnant and parenting students, as measured by the Columbia Teen Screen. 1.1 In fall 2015, evidence- based prevention curriculum will be implemented at the beginning of the school year at each grade level. 1.2 Students will design and launch a teen pregnancy prevention campaign with local media partners. 1.3 Pregnant and parenting students will be linked to the evidence- based Nurse Family Partnership Program with an on-site nurse. 1.1 By June 2018, the pregnancy rate at HighRisk High School will decrease by 25%, from 14.6% to no more than 11%. 1.2 By June 2018, the percentage of 11 th grade students who indicate they are sexually active but not using birth control will decrease by 50% from baseline. 1.3 By June 2018, the percentage of pregnant and parenting students screening positive for suicide ideation will be within 5% of the general student population.

23 Step 4. Write Peformance Indicators that show how you will use your activities/services to achieve your Outcome Indicators

24 Breaking It Down: Performance Indicators  Performance indicators specify “dosage” (program intensity) and numbers served—in other words, the strategies you will use to achieve your outcome indicators.  Performance indicators can and should specify who is being served/targeted, and how services/interventions are being implemented.  When using evidence-based programs, it is important to specify how adherence (faithfulness) to the model will be measured—e.g., through fidelity assessments.

25 Breaking It Down: Performance Indicators  Some logic model formats specify short, intermediate and long-term performance indicators or outcomes, rather than distinguishing between performance and outcome indicators. In this case, performance indicators generally fall under the heading “short-term outcomes,” outcome indicators fall either into “intermediate” or “long- term outcomes.” Often changes in attitude and belief may fall into the category of “intermediate outcome,” and changes in behavior fall into “long- term outcomes.”

26 Breaking It Down: Performance Indicators  For the example used here, a short-term outcome might relate to teachers being trained in the pregnancy prevention curriculum.  An intermediate outcome might relate to the number of students receiving the curriculum and the “dosage” (“80% of students will receive at least six out of eight sessions annually”). Or it might relate to changes in students’ attitudes and beliefs as measured by a survey.  Long-term outcomes usually relate to changes in behavior (“the percentage of sexually active teens who report using birth control increased by x%).

27 Goal: To reduce teen pregnancy in HighRisk School District. NeedsActivitiesProcess Indicators Outcome Indicators 1.1 During the school year, 47 students enrolled at HighRisk High School became pregnant out of a student population of 322 female students, for a pregnancy rate of 14.6% % of 11 th grade students surveyed who indicated that they are sexually active reported that they do not use birth control. 1.3 The percentage of students screening positive for suicide ideation was 30% higher among and parenting students, as measured by the Columbia Teen Screen. 1.1 In fall 2015, evidence-based prevention curriculum will be implemented at the beginning of the school year at each grade level. 1.2 Students will design and launch a teen pregnancy prevention campaign with local media partners. 1.3 Pregnant and parenting students will be linked to the evidence-based Nurse Family Partnership Program with an on-site nurse % of continuing students will receive pregnancy prevention curriculum at each grade level. 1.2a Each year, a minimum of 12 students from each grade level will be trained in how to design and create public service announcements. 1.2b At least three PSAs will be aired on local television a minimum of 30 times each semester as part of a teen empowerment campaign % of pregnant and parenting teens will be linked to the Nurse Family Partnership Program, and 80% will receive a minimum of one year of intervention. 1.1 By June 2018, the pregnancy rate at HighRisk High School will decrease by 25%, from 14.6% to no more than 11%. 1.2 By June 2018, the percentage of students who indicate they are sexually active but not using birth control will decrease by 50%. 1.3 By June 2018, the percentage of pregnant and parenting students screening positive for suicide ideation will be within 5% of the general student population.

28 Breaking It Down Step 5: Specify the partners who will provide services and/or implement programs and the timeline for implementation.

29 Knowing when you need help…  When is it appropriate/important to work with a professional evaluator?  When you are grant writing and/or developing a logic model or evaluation plan.  When you are required by your funder to work with an outside evaluator.  When you need technical assistance in developing an evaluation design, surveys, or other instruments.  When you need assistance in analyzing your data.  When you need assistance in interpreting your data.

30 A local evaluator:  Should be a team player. He/she is not an auditor, and he/she is working for you, to help you improve your program and measure its effectiveness.  This doesn’t mean you’ll like everything you hear. Keep in mind, the evaluator is there to help you develop the best program you can! He/she can’t do that by protecting your feelings.  Becoming defensive because the data doesn’t look like you want it to is self-defeating. Use what you learn to make improvements.

31 Make evaluation FUN.  Evaluation gives you the key to creating an effective and exciting program.  Data will tell you whether and how to improve program content, and whether and how to improve program delivery.  Outcomes will tell you whether and how you changed the lives of the people you are serving.  These data will give you the evidence you need to make a case for further funding, or expansion of the program or service.  What’s not to like?!

32 Questions?  Call or write: Dixie L. King, Ph.D. Transforming Local Communities, Inc Ming Avenue, Suite 240 Bakersfield, CA , ext (fax)


Download ppt "The Role of Program Evaluation in Federal Grant Applications Dixie L. King, Ph.D. 5500 Ming Ave, Suite 240 Bakersfield, CA 93309"

Similar presentations


Ads by Google