NASCSP March 1, 2012

2 Guidance About Evidence Success should be judged by results, and data is a powerful tool to determine results. We cant ignore facts. We cant ignore data. President Barack Obama July 24, 2009 The test of a performance management system is whether its actually used….Federal managers and employees at all levels must use performance goals and measures to set priorities, monitor progress, and diagnose problems. Chief Performance Officer Jeffrey Zients October 29, 2009

Two key points: Results are supported by data Data from performance measurement systems must be put to use. In recent OCS guidance, several models and references to evidence have been identified. Here is a quick review of these terms. 3

Approaches to prevention or treatment that are validated by some form of documented scientific evidence. This could be findings established through scientific research, such as controlled clinical studies, or other comparable and rigorous methods. Identifies proven effectiveness, supported by objective and comprehensive research and evaluation. 4

Approaches that use the best available research and practice knowledge to guide program design and implementation within context. This informed practice allows for innovation and incorporates the lessons learned from the existing research literature. 5

An innovative and consistently applied policy, process, practice, or procedure that takes a comprehensive approach to developing and implementing activities using strategies that are related to the intended service recipients and community. This practice model is culturally competent, data-driven, measurable, and replicable and incorporates a method for documenting programmatic results. 6

A practice with at least preliminary evidence of effectiveness in small-scale interventions, or for which there is potential for generating data that will be useful for making decisions about taking the intervention to scale the results diverse populations and settings. 7

A program, activity or strategy that has been shown to work effectively and produce successful outcomes This is supported to some degree by subjective and objective data sources 8

What do we know about our measurement tools? Are they Accessible Reliable Valid (appropriate) Affordable Scalable (able to measure change) 9

Bank accounts Copy of Diploma or Certificate Employment records Escrow accounts Financial reports Health or nutrition records Inspection results Lease agreements Legal documents Loan monitoring reports Mortgage documents Observation log Pre-post tests Progress reports Questionnaire Rent receipts Scales and Matrices Survey Testing results 10

Agency database Case notes Centralized database Computer spreadsheets File cabinets Individual case records Manual tallies Public database School records Specialized database Tax Assessor database Training center Work plan reports 11

If no one is assigned the task, no one will do it. Consider various agency staff responsibilities Include partner reports 12

Home visits Office appointments Phone follow up Review of case notes/progress reports (scales?) 13

Daily Weekly Monthly Quarterly Biannually Annually Upon incident 14

15

16 Data about community needs and resources This can identify the scope of community issues Used to prioritize community needs for agency intervention Data used during strategic planning Identify resources ($, facilities, staff, etc) the agency has/needs What results does the agency expect to achieve? Data collected during implementation of services How many individuals, families, communities are projected to be served? How many are projected to achieve results? How many individuals, families, communities were actually served? Who were they? Data collected to identify achievement of results How many made movement toward their goals? How many achieved the end results previously identified?

17 Evaluation Data Used in comparing CAA performance with a control group to determine the quality of the results Done by an independent reviewer Used to validate that results actually impact on identified community needs

18 Source: Child Trends

The collection of NPI data for a report to OCS is a comprehensive effort by the CSBG Network to identify the results of the efforts of local CAAs. This is sometimes called the ROMA report, but we have just seen that ROMA data includes more than these indicators. 19

Measure and report performance Identify patterns and relationships Consider new actions based on analysis Create new outcome-focused goals Develop new resources 20

When you have your data, remember that you have to turn them into information. They must be interpreted to become evidence Consider this piece of data and the questions that arise: 35 individuals got a job. Is that good? What is the unemployment rate? Opportunity for employment? Characteristics of those who got jobs. Etc. 21

What are the top 10 services provided by your agency? What are the top 10 outcomes achieved by your customers? What is the connection between these two lists? 22

Build on ROMA principles to identify ways to collect credible evidence in systematic ways (making the connections among mission, community need, agency strategies and well documented results) Find ways to compare CAA performance with established standards and performance of other similar programs. 23

We must recognize the value of evidence-based or evidence-informed interventions…. but our definition of what counts as credible evidence should be expanded to allow for continuing improvement and innovation. Evidence-based does not have to mean experimental-based. We draw on evidence from many kinds of research, including program evaluations, and practice. From Lisbeth Schorr 24

Lisbeth Schorr tells us to identify the clear, measurable results sought by a complex intervention as the essential first step toward both successful implementation and to a successful evaluation. We have a start on this with the NPIs. 25

It is essential to have some way of comparing results to establish that the observed change has a high probability of resulting from the practices. How you will compare results from your programs to those who have not been involved in your programs. The community-specific nature of place-based interventions makes it very hard to find a comparison group that would allow for a clinical control group. 26

It is possible to compare outcomes among the populations those served by a specific initiative to (a) similar populations in the geographic area before the intervention began for whom baseline data are available, (b) current populations that did not receive similar services and supports, but for whom data are already available, i.e. does not have to be collected as part of the evaluation, and (c) national, state or local norms. 27

The elements of ROMA principles and practices can help agencies establish evaluation frameworks that include the evidence that will be used to support their results Establish CSBG industry standards Remember what baseball did that we havent yet done: 28

Clearly identified the indicator to be measured Collect data consistently Analyze the data against records of wins Publish the data and the analysis, so the public recognizes success 29

Support knowledge collection, analyses, and evidence syntheses that yield a more complete body of evidence. Develop network tools and capacities to gather knowledge at greater scale. Expand the menu of available evaluative techniques that can be matched to different types of interventions. Combine findings from research, theory, and practice, for informed decision-making Promote use of a results framework to strengthen measurement for accountability and learning 30

Real-time learning can be achieved by using a well developed results framework to track progress toward those results, and using the data for real-time learning. 31

As Peter Drucker has pointed out, The greatest danger in times of turbulence is not the turbulence; it is to act with yesterdays logic. We do not want to rely on fads, hunches, anecdotes, or good intentions. Nor are we reluctant to identify and end support for the efforts that are ineffective. 32

Barbara Mooney National ROMA Training Project 243 E. High St. Waynesburg, PA