Planning Data-driven, Evidence-based Programs Presented by: Jaime Costigan, M&E Technical Advisor, International Rescue Committee ECDC 23rd Annual National Conference Strengthening Communities for Refugees in a Changing Environment May 31, 2017 Arlington, Virginia   The IRC received $225,000 in competitive funding through the U.S. Department of Health and Human Services, Administration for Children and Families, Grant 90RB0051-02-00.   The META project is financed 100% through Federal funds.  The materials presented during this workshop are solely the responsibility of the authors and do not necessarily represent the official views of the U.S. Department of Health and Human Services, Administration for Children and Families.

Objectives By the end of this session, you will: Be able to explain what is meant by data-driven and evidence-based programming Be able to define what a theory of change is and how it can be used in evidence-based and data- driven programming Describe the components of a M&E plan Have some ideas for things you can do during program planning to become data-driven and evidence-based

What are data-driven programs? Programs where data shapes program design and informs decision making during the life of the program. Data-driven programming involves using data to make decisions and take action.

For data-driven programming: Data must be good quality, purposeful, timely Processes must be in place to analyze, reflect and act on data

What are evidence-based programs? Programs where existing evidence has been used to inform the theory of change and implementation. There are different levels of evidence—from anecdotal evidence to evidence from experimental research designs and meta-analyses that look across studies to determine average impact. Typically in the context of evidence-based programs, we are talking about information that proves with a high level of certainty that an intervention leads to a particular outcome or result. This type of quantitative causal evidence is typically generated by high-quality research studies. In many cases, there isn’t evidence to support our intervention, not because it doesn’t work, but because the research hasn’t been done. In such instances, we have an opportunity to plan evidence-generating programs.

What can you do to plan data-driven and evidence-based programs?   What can you do to plan data-driven and evidence-based programs?

Develop or map your theory of change “Theory of change” may be a new term. The diagram, however, may be familiar, known as a logic model.

A theory of change is a type of logic model that defines all building blocks required to bring about a given long-term objective and provides a graphic representation of the change process, explicitly illustrating the causal pathways between activities, outputs, outcomes, and objectives. ToC diagrams may be basic or complex, with the most basic one shown here often being referred to as a results chain. Theory of change diagrams explicitly illustrate the causal pathways between activities, outputs, outcomes, and objectives. The process of developing a theory of change can help you make sure that the causal relationship between your activities, outputs, outcomes and objectives is logical. The process of developing a theory of change, if done correctly, contributes to evidence-based and data-driven programming in part by providing an opportunity to think through WHY you are implementing your program, for this population, in this way.

Based on the data and evidence we have, what issue should we address and why? Goal How do you develop an evidence-based, data-driven theory of change? During design of new programs, use existing data and evidence to help you define your program’s goal and objective. As you go through the process of backwards mapping consider questions like, “based on the data and evidence we have, what issue should we address and why?”

Based on the data and evidence we have, what part of the goal should we address, for whom and where? Why should this be a priority? Goal In evidence-based and data-driven programs, data and evidence should also be considered when you are defining your objective. Objective

Is there any evidence that shows us what pre-conditions, or outcomes need to be met in order to achieve this objective? Goal Outcome Data and evidence can help you define your outcomes as well. Objective Outcome

Is there any evidence that shows us what immediate results or sub-outcomes will lead to the outcomes we want to see? Goal Output Outcome Looking at evidence when you think about your outputs can help you define good quality outputs and output indicators that specify the quality standards that need to be met by an output in order to achieve an outcome. For example, evidence might show that providing information is important, but that crucially, for the outcome to be achieved, people need to receive information that is linguistically accessible on a particular topic, delivered in a particular way—for example, in person by peers as opposed to pamphlets. Objective Outcome Output

Goal Activity Output Outcome Objective Outcome Activity Output Is there any evidence that shows us what activities will enable us to achieve these outcomes? Goal Activity Output Outcome If you are designing evidence-based and data-driven programs, when you get to defining your activities, you should be asking: is there any evidence that shows us what activities will enable us to achieve these outcomes? For ongoing programs, or for programs you are looking to expand or improve, you might use program data to evaluate whether changes are needed to your program model. For example, if you are seeing from your data that women aren’t retaining jobs at the same rate as men, you may want to adjust your program model and adapt your theory of change to address the causes of this. Objective Outcome Activity Output

Discuss: Where might you find the data or evidence needed to develop a data-driven, evidence- based the theory of change?

Use your theory of change to guide your M&E plans What else can you do during program planning processes to become more data-driven and evidence based? Use your theory of change to guide your M&E Plans. An M&E plan is a narrative document that serves as a planning tool and reference for project staff for how to go about monitoring and evaluation throughout the life of their project. It outlines what will be monitored and evaluated and why, how data will be collected and protected, how data will be analyzed and shared, the staff responsible, and other resources needed.

Components of an M&E Plan Introduction & Project Summary Data Flow Chart Data Management Processes (Storage, Analysis, Privacy) Logical Framework (Logframe) Indicator Matrix Monitoring Budget Roles & Responsibilities Appendices (questionnaires, interview guides, procedures) The complexity of your M&E plan will vary based on the complexity of your project.

When developing your M&E plan: First, determine what you want to learn about your program and why Look at your theory of change to guide the process of determining what you want to learn and why. Questions might include: Did we deliver the program as planned? Did we achieve the outcomes we expected to achieve? What was the impact of our program? Was our program accessible to all clients? What “dosage” of this particular activity is needed to achieve this outcome? Is the theoretical foundation of our program sound? When determining what questions are most important to you, consider not just what is interesting to know, but what you need to learn in order to demonstrate results, improve your program and achieve better outcomes for clients. Consider how you will use and act on what you learn.

Discuss: You are developing an M&E plan for a new employment program for clients who have been in the U.S for more than 180 days. What might you want to learn about your program and why? How would you use the information?

Determine what data you need to collect in order to learn what you want to learn Description Indicator Means of Verification Assumptions Objective   Outcome Output Activities Developing a logframe is good standard practice even if the donor doesn’t require it. Use your theory of change to develop your logframe. When filling in your logframe, consider not just “how will I measure this” but also what data you need to collect in order to learn what you want to learn. Framing your data collection plans around questions you want to answer helps shift the focus from reporting and tracking for donors to being data-driven, focused on learning and program improvement.

Question / What we want to learn Do we already collect this data? Consider mapping out what data is required to learn what you want to learn Question / What we want to learn Data needed Do we already collect this data?  Are male and female clients regularly attending the job training program? Is the program accessible to both male and female clients? Participation rates of male and female clients Yes While not part of a standard M&E plan, before filling in your logframe, it might be helpful to map your data to what you want to learn. This can help make sure your are tracking the information you need. It can also help with organizing data for presentations, reports, & external publications.

Consider how you will use data It is important to have clear plans for how the data you collect will be used and, where needed, acted on. This can help you avoid collecting too much data—particularly important when you have limited data collection and management capacity. An indicator matrix is a tool that can help you map out not only the specific data points you need to collect and the tools you will use, but also the purpose of the data you collect and how it will be used. It, like a logframe, can be used as project management tools and help inform your workplan.

Discuss: What is an example of data that your program/organization collects that you use for something other than just reporting to the donor?

Consider how and when you will analyze data An M&E plan typically includes plans for how data will be analyzed, using excel or SPSS, for example. But for data-driven programming, think also about what questions you will use to guide your analysis—think back to the questions you want to answer/ what you want to learn. Also consider the process for conducting analysis, reflecting on the data and planning action based on it. Will you set aside time on a regular basis during staff meetings, for example, to reflect on data? What questions will guide your analysis? Also consider who should be involved in data analysis process.

Plan how you will ensure the quality and timeliness of the data you collect Data quality and timeliness is key to data driven programming. If you want to be able to use data to make programmatic decisions, when does aggregate data need to be available and by whom? What processes/ systems need to be put in place to enable this? SoPs, data flow charts, roles and responsibilities mapping are tools that can help.

Budget appropriately Data-driven programs require adequate budget for M&E activities. Ensure your budget is adequate to support your plans! A general rule is that M&E activities should make up 10 % of your project budget. Make sure you have or budget for an adequate number of staff to collect, enter and analyze data on a regular basis, staff to provide oversight of data quality and M&E activities, and any staff training needed. You will also want to budget for any consultant fees and expenses required for external evaluations. Other things you will want to consider budgeting for include: equipment and software required for M&E activities translation and piloting of M&E tools, such as surveys communications costs including mobile data, and copy editing, formatting, and printing of final evaluations.

What is one thing you are doing now to try Discuss: What is one thing you are doing now to try to ensure that programs are data-driven and evidence-based? What is one step your program, office, or organization could take to become more data-driven and evidence-based?

Questions? 

Find resources online at www.METASupport.org Follow us: @USMETASupport Contact us via email at: META@Rescue.org