1. No Place Like Home Cross-Site Evaluation Training

2. General Staff Survey

3. NPLH Summary Evaluation of three child welfare agencies’ use of family meetings with families receiving in-home services –Process What does practice look like? Tools: focus groups, observations, surveys –Outcomes Placements (number and type), re-reports, recurrence of maltreatment, etc. Satisfaction and service provision Tools: administrative data, surveys

4. Your role… Is critical! To provide data and feedback about your experience working in child welfare and with family meetings To recruit families to participate, so that they can provide data and feedback about their experience To participate in the feedback loop – ongoing communication

5. WhoWhat is the value? …the fieldResearch is mixed and evidence-base of FGDM is moderate—more research is needed Additional research can support the national spread of family meetings or suggest the need for alternative practices …familiesFamilies receive interventions with known effectiveness Opportunity for families to provide input about their experiences with CPS What’s in it for…?

6. WhoWhat is the value? …CPS Staff and Agencies (You!) Accountability. Opportunity to get input from staff who know the most about the families (You!) Knowledge of effectiveness of family meetings can increase investment in the practice Visibility and respect for advancing the field and systematically examining internal practices Increased knowledge about the use and potential of evaluation Results can impact system and practice change What’s in it for…?

7. Human Subjects What is the CITI program? History and Ethical Principles The Belmont Report

8. Human Subjects Defining Research with Human Subjects Federal Regulations Assessing Risks Basic Institutional Review Board (IRB) Regulations and Review Process

9. Human Subjects Informed Consent –Risks to you personally are minimal –There are potential benefits to children, families, and to staff in the agency –Your participation is voluntary Privacy and Confidentiality –All survey information is confidential and used only for the study –Survey data is maintained by Kempe Center NPLH evaluation team –Individually identifiable information will NOT be shared outside of the Kempe Center evaluation team

10. Human Subjects Records-Based Research Research with Protected Populations – Vulnerable Subjects Unanticipated Problems and Reporting Requirements

11. Evaluation Basics Evidence-Based Practice Confidentiality vs. Anonymity Bias Design Types –Experimental –Quasi-Experimental Process vs. Outcome evaluation –How vs. What Qualitative vs. Quantitative methods –Depth vs. Breadth

12. Evaluation Design - TX Staff Consent; General Staff Survey Families meet eligibility criteria and are consented into the study; Caregiver Survey Random Group Assignment Meeting participants consented into the study; Fidelity Survey Follow Up Survey Case-Specific Questionnaire Right now FBSS worker makes FGC referral FGC 1-2 months post-FGC FBSS Case closure/transfer to CVS

13. Evaluation Design - SD Staff Consent; General Staff Survey Families meet eligibility criteria and are consented into the study; Caregiver Survey Random Group Assignment Meeting participants consented into the study; Fidelity Survey Follow Up Survey Case-Specific Questionnaire Right now IFA worker makes FGC or FGC/TDM referral FGC/TDM; IFA and Ongoing worker attend 1-2 months post-FGC Ongoing case closure/transfer to out-of-home services

14. Evaluation Design - CO Staff Consent; General Staff Survey Families meet eligibility criteria and are consented into the study; Caregiver Survey Group Assignment – Propensity Score Match Meeting participants consented into the study; Fidelity Survey Follow Up Survey Needs-Services Assessment; Case- Specific Questionnaire Right now Intake or FAR worker makes 1 st FSRT or FUM referral 2 nd FUM/FGC 1-2 months post- FUM/FGC Ongoing case closure/transfer to out-of-home services

15. NPLH Surveys 1.General Staff Survey 2.Caregiver Survey 3.Meeting Fidelity Survey a)Participant b)Facilitator/Coordinator 4.Case-Specific Questionnaire 5.Follow-Up Survey a)Caregiver – Intervention b)Caregiver – Control c)Meeting Participant

16. Surveys that are distributed TO you General Staff Survey –All staff who have a role in the evaluation (case workers, coordinators/facilitators, supervisors) Coordinator-Facilitator Survey –Coordinators/Facilitators only Case-Specific Questionnaire –Case-workers only

17. General Staff Survey Purpose: –To understand more about: Who utilizes family meetings in CPS How much experience staff have with family meetings How staff perceive family meetings The context in which meetings are happening. –We can use this information to inform our process and outcome evaluations

18. General Staff Survey

19. Coordinator-Facilitator Fidelity Survey Purpose: –To understand more about: How, why, and when family meetings are used Who is invited to and who attends family meetings Other characteristics of family meetings –Fidelity –Logistics –Meeting fidelity is part of our process evaluation

20. Coordinator-Facilitator Fidelity Survey

21. Case-Specific Questionnaire Purpose: –To understand more about: What services in-home families receive Are services provided impacted by family meetings? –Services provided is one of our outcome evaluation variables

22. Case-Specific Questionnaire

23. Surveys that are distributed BY you Caregiver Survey –By case workers –At face-to-face meetings with family –At point of referral for a family meeting Participant Fidelity Survey –By coordinators/facilitators –At family meetings

24. What you ARE responsible for 1.Informed Consent Participation is voluntary Will families agree to be part of the pilot? If so, they have rights which are outlined in the Information Sheet, including the right to change their mind at any time. If not, no penalty will come to them. 2.Distribute Paper and pencil surveys with Caregiver Survey – self-addressed and stamped envelope with a flyer with a phone number to call for assistance Meeting Participant – large self-addressed, stamped envelope for all surveys in the center of the room/table

25. What you are NOT responsible for Explaining the content of surveys to families Helping families/meeting participants fill out surveys People with questions can call the number provided on the Information Sheet (have extra copies available for them to keep) Other household members or family supports, etc. can help. In order to maintain confidentiality, it is important that you merely handout surveys and provide no more information than what is included on the information sheets.

26. Caregiver Survey Purpose: –To understand more about: The characteristics of families receiving in-home child welfare services (e.g. demographics) The protective capacities of parents The parent-child relationship and child behavior –This information can be linked to outcomes

27. Caregiver Survey

28. Participant Fidelity Survey Purpose: –To understand more about: How, why, and when family meetings are used Who is invited to and who attends family meetings Other characteristics of family meetings –Fidelity –Logistics –Meeting fidelity is part of our process evaluation –We want to hear from families and other participants as well as staff.

29. Participant Fidelity Survey

30. Surveys administered by evaluation team How? Paper and pencil; via mail 3 follow-up survey versions with the following components: –Meeting Follow-Up Who? All meeting participants who filled out the initial Meeting Participant Fidelity Survey –Caregiver Follow-Up Who? All (intervention and control group) parents –Family Satisfaction Who? All (intervention and control group) parents AND meeting participants

31. NPLH Survey Summary Case workers: –Complete: General Staff Survey (1 time) Case Specific-Questionnaire (for every study case) –Distribute: Caregiver Survey (to every study case) Coordinators/Facilitators: –Complete: General Staff Survey (1 time) Coordinator-Facilitator Fidelity Survey (for every study family meeting) –Distribute: Meeting Participant Fidelity Survey (for every study family meeting)

32. Coordinators/Facilitators Meeting Log [placeholder]

33. THANK YOU! If, throughout the life of this project, you have any questions please contact: Heather Allan, NPLH Project Coordinator Email: heather.allan@childrenscolorado.orgheather.allan@childrenscolorado.org Phone: 720.336.8283