1 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Welcome! We will begin our webinar at 9:00 am PT As you log on, do not be surprised if you don’t hear anyone else; participants are placed on mute in order to limit background noise

National CASA Performance Measurement Lori Delale-O’Connor Sarah Catherine Williams Child Trends  Twitter/childtrends  April/May 2014

3 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Presenter Information Lori Delale-O’Connor Scientist Child Trends Having difficulty with the sound? Please send us a message in the chat box.

4 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Roadmap Part I: Introductions and Overview of the National CASA project  In-depth interviews  Survey development, testing, and fielding Part 2: Survey Results  Survey completion demographics  Survey data  Potential “priority areas” identified Part 3: Process of Becoming Evidenced-Based  What it means to be evidence-based  “Arrow” and process  National CASA’s work to becoming evidence-based

5 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams What you will walk away with……. Knowledge about the current data collection activity of CASA/GAL programs across the country Understanding of what it means to be “Evidence Based” – and why it matters for your CASA/GAL program Insight into how National CASA is moving forward with performance measurement and working to build an evidence base

6 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Part I: Introductions and Overview

Child Trends’ Mission To improve outcomes for children by providing research and analysis to the people and institutions whose decisions and actions affect children. 7

8 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Research Areas 8 FATHERHOOD AND PARENTING EARLY CHILDHOOD CHILD WELFARE INDICATORS EDUCATION (K-12) DATA DEVELOPMENT YOUTH DEVELOPMENTFERTILITY AND FAMILY STRUCTURE

9 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams  Develop and track data on children and youth  Help to design, implement, and evaluate child and youth-serving programs  Analyze and synthesize data and evidence  Provide training and technical assistance A key focus of our work is identifying evidence- based programs, practices, and policies Child Trends’ Work 9

10 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams National CASA/Child Trends Performance Measurement Goal: Develop key measures to help National CASA better understand performance and move toward developing an evidence base. Phase 1: Scan of current data collection and use (Sept-March)  In-depth interviews  Survey development & testing  Survey fielding Phase 2: Presentation of findings (April-May) Phase 3: Recommendation of performance measures (May-June)

11 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Part II: Survey Findings

12 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Survey Completion Overview Category % % LevelUrbanicity local 89 rural 50 state 9 suburban/mixed 36 state – direct advocacy 2 urban 14 Age (in years)Program Model 10 or fewer 20 friend of the Court – guardian ad litem or more 44 friend of the Court/Attorney team 10 Size (# of children served)Type of Agency 50 or less 24 independent non-profit – part of a government agency or more 18 part of a non-profit umbrella agency 16

13 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Topics Addressed in the Survey Data collection responsibility and burden Data collection systems Systems used Meeting needs Data use Priorities for future data collection

14 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Data collection and level of burden Type of staff%Hours per month CASA supervisors/coordinators7020 CASA administrators5621 Volunteer164 Assistant or support staff931 Other519

15 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Data collection systems and satisfaction Current Data System% % CASA Manager39Other4 Microsoft Excel36Optima4 COMET16CASA Tracker4 State system12Own system2 Efforts to Outcomes (ETO)6Microsoft Access2

16 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Data use Activity % Reporting requirements90 Demonstrate effectiveness73 Make program improvements62 Fundraising58 Service improvements43

17 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Data support needs Area of desired support% Training on how to analyze data53 Guidance on collecting data46 Funding for data collection/analysis software 44 Guidance in identifying outcomes44 Funding for personnel40

18 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Data priorities Domain% Permanency60 Well-being44 Placements44 Education39 Mental health31

19 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Participant Feedback

20 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Participation What do you think about the priority areas identified in terms of their alignment with CASA? I think they are an accurate reflection of CASA’s mission and the reach of a volunteer. I think one or more of them are outside of CASA’s mission and the reach of a volunteer. I think one or more of them are outside of CASA’s mission, but a volunteer may impact these areas anyway.

21 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Participation Do you/local programs you work with already collect data on measures in the priority areas? Yes, I/they collect data on ALL of these measures. Yes, I/they collect data on SOME of these measures. No, I/they do not collect data on ANY of these measures.

22 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Participation What potential challenges do you see to getting data on measures in the priority areas? My program/the programs I work with does not have access to the data needed for these measures. My program/the programs I work with would need to administer additional assessments (i.e., mental health assessments/screenings) in order to capture data on these measures. I am not clear on what type of data are needed for these measures. Programs may interpret the meaning of these measures differently, and therefore data quality would be inconsistent.

23 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Participation What potential challenges do you see to reporting on measures in the identified priority areas? It is a challenge to get the data for these measures, and therefore my program/the programs I work with would most likely be unable to report on these measures. I/the programs I work with can get the data on these measures, but I am not confident I/they would be able to report on them. There is a danger that programs could interpret the measures differently, and therefore the data would not be as useful.

24 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Part III: Becoming Evidence Based

25 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams  Evidence-based programs: – Interventions that have been found to have positive effects or impacts on targeted outcomes  Evidence-based practices: – Intervention strategies or program practices that have been found to improve outcomes for children and youth  Evidence-based policies: – Regulations and laws that have been found to have positive effects on child and youth outcomes What We Mean by “Evidence-Based”? 25

26 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams High Low Quasi- experimental Experimental Replicated Experimental Evidence-informed, non-experimental Levels of Evidence 26

27 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Collect Data on Performance & Outcome Measures Conduct Needs Assessment Identify Your Population Targeting Conduct a Quasi-Experimental Outcomes Evaluation Conduct an Implementation Evaluation Select Intervention, Develop Logic Model & Identify Indicators Implement Program/Approach & Conduct Ongoing Performance Management Conduct a Randomized-Controlled Impact Evaluation 27

28 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Participant Feedback

29 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Participation Where is your CASA/GAL program/the programs that you work with on the arrow? We/They are not on the arrow at all. We/They are in the preliminary stages of performing a needs assessment and identifying target population. We/They are in the more secondary stage of developing a logic model and identifying indicators. We/They are in the conducting ongoing performance management phase. We/They are in the more advanced stage of conducting an implementation evaluation, quasi-experimental evaluation, or a randomized control trial (RCT).

30 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Food for thought—post-webinar survey question Where do you see your CASA/GAL program/the programs you work with being able to move on the arrow? Not any further than we/they currently are. Performing a needs assessment and identifying target population. Developing a logic model and identifying indicators. Conducting ongoing performance management. Conducting an implementation evaluation, quasi- experimental evaluation, or a randomized control trial (RCT).

31 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Building the Evidence Base

32 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Set up a clear path to success or a roadmap for achieving program goals Provide a visual test of whether inputs, and activities align with outputs and outcomes Help to get everyone “on the same page” Facilitate performance management and evaluation Benefits of Logic Models 32

Outcomes Outputs Activities InputsOutputs Outcomes Program Component InputsOutputs Outcomes Program Mission Program Component Logic Model Example Inputs Activities 33

Logic Model Example Outcomes Outputs Inputs Activities Inputs Activities Outputs 34

35 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Outcomes and Outputs Outcomes: The observed changes in attitudes, knowledge, or behavior among program participants that result from a program / service / initiative. Output: Measures of evidence of program implementation or quality.

36 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Outputs are sometimes confused with outcomes OutputsChild Outcomes Numbers of volunteers Number of children served Changes in health Changes in behavior Changes in knowledge Changes in placement Changes in permanency

37 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams What exactly is impact? Researcher definition of impact: Linking (through study/research) the observed outcomes to a program / service / initiative.  Does the program lead to outcomes?  Can we control for as many other explanations as possible? Impact could also mean the very long term changes attributable to a program / service / initiative.  Child welfare advocates – do children who return home to their parents STAY with their parents? Or do they re-enter foster care at a later date or repeatedly? Impact could also mean the collective outcomes across multiple programs / services / initiatives.  Does the same program offered in multiple locations have outcomes that can be aggregated across the locations?

38 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams How does measuring impact differ? Court Appointed Special Advocates ProgramOutcome(s) Decrease in length of stay in foster care Decrease in trauma symptoms Change in social worker at public child welfare agency Initiative at local child welfare agency Alternative programming Are other conditions or programs causing these changes?

39 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams National CASA’s next steps for consideration… Developing a logic model Identifying a small set of “key measures” for all CASA/GAL programs Addressing the evaluation challenges of networks structured like National CASA Conveying potential benefits and collaborating across the network to build an evidence base

40 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Contact Information Lori Delale-O’Connor Sarah Catherine Williams

Research in the Service of Children 56