Effectively Utilizing Data Collection: A Case Study of a Functioning System Robert Bartelt & Kristen Gay Silver Springs – Martin Luther School

Where to Start Agency Values Agency Values Frame Questions Frame Questions Work with Others Work with Others Leaders Leaders Direct Care Staff Direct Care Staff Families and Children Families and Children Pilot Pilot Improve Assess Measure Design

Data Collection What you get out of your data is only as good as what you put in….

Data Collection Process Choosing What Data to Collect Choosing What Data to Collect Clinically Meaningful Clinically Meaningful Does the data add to your understanding of how clients are doing? Does the data add to your understanding of how clients are doing? Will your findings help you effect change in your program? Will your findings help you effect change in your program?

Data Collection Process Choosing What Data to Collect Choosing What Data to Collect User Friendly for Staff User Friendly for Staff Who is going to be collecting the data? Who is going to be collecting the data? Are measures easy to understand? Are measures easy to understand? How much training is required? How much training is required? How many measures do you need? How many measures do you need? How much time is available for data collection? How much time is available for data collection?

Choosing What Data to Collect Choosing What Data to Collect “Crunchable” “Crunchable” Does the measure provide information that is meaningful for individual analysis? Does the measure provide information that is meaningful for individual analysis? Does the measure provide information that is meaningful for aggregate analysis? Does the measure provide information that is meaningful for aggregate analysis? Do you have/how will you get the tools to manage the data? Do you have/how will you get the tools to manage the data? Data Collection Process

Choosing What Data to Collect Choosing What Data to Collect Information Required from Third Parties Information Required from Third Parties What are the requirements from third parties? What are the requirements from third parties? Can your data collection system be set up so as to satisfy these requirements? Can your data collection system be set up so as to satisfy these requirements? How will the data represent your agency and services? How will the data represent your agency and services? Data Collection Process

Choosing What Data to Collect Choosing What Data to Collect Is the data collection system flexible? Is the data collection system flexible? When a managed care company or a county calls to request data on only “their” children, can you do that easily? When a managed care company or a county calls to request data on only “their” children, can you do that easily? Can you respond to new requirements without expending a lot of additional resources? Can you respond to new requirements without expending a lot of additional resources? Data Collection Process

Choosing What Data to Collect Choosing What Data to Collect Do you want to engage in benchmarking? Do you want to engage in benchmarking? Other similar organizations Other similar organizations MCO expectations MCO expectations County expectations County expectations National norms National norms

Examples of Data to Collect Behavioral Rating Scales Behavioral Rating Scales Frequencies/Counts (i.e., Lengths of Stay) Frequencies/Counts (i.e., Lengths of Stay) Patterns of Diagnostic Groups/Presenting Problems Patterns of Diagnostic Groups/Presenting Problems Demographics of Population Demographics of Population Contacts with Families Contacts with Families Critical Incidents Critical Incidents Satisfaction Surveys Satisfaction Surveys Aftercare Data Aftercare Data County, MCO, etc. Requests County, MCO, etc. Requests

Evolution of Data Collection As the agency and/or service provided becomes more sophisticated, you want to know more information. As the agency and/or service provided becomes more sophisticated, you want to know more information. Start collecting data Start collecting data Ask new questions Ask new questions What other data will What other data will you want to collect? Do you have the flexibility to ask more questions and gather more data? Do you have the flexibility to ask more questions and gather more data? Improve Assess Measure Design

A Look at a Specific Outcome Restraints Restraints A high-risk activity A high-risk activity Upfront goal to reduce the use of restraints Upfront goal to reduce the use of restraints What data is needed by internal committees: What data is needed by internal committees: How do you provide the most meaningful data to internal committees? How do you provide the most meaningful data to internal committees? How frequently will the data be required? How frequently will the data be required? What variables are important? What variables are important?

Restraint Reduction Collection of Data Collection of Data What do you want to know? What do you want to know? Frequency of Restraints Frequency of Restraints Child Child Staff Who Initiates Staff Who Initiates Time of Day Time of Day Length of Restraint Length of Restraint Result (injury of child, injury of staff) Result (injury of child, injury of staff) Children with multiple restraints Children with multiple restraints Type of restraint used Type of restraint used

Restraint Reduction Flowchart: Developing a data collection system Flowchart: Developing a data collection system

Data Input

List Manager

Data Input

Data Output

Restraint Data

Restraint Reduction Different Types of Reports Different Types of Reports Trends Over Time Trends Over Time

Restraint Reduction Different Types of Reports Different Types of Reports By Unit By Unit Unit AUnit BUnit CUnit D January21021 February9882 March April28772 May June July49111 August14970 September15991 October9861 November6371

Different Types of Reports Different Types of Reports Restraints by Type of Restraint Restraints by Type of Restraint Restraint Reduction

Different Types of Reports Different Types of Reports Restraints by Child Restraints by Child

Restraint Reduction Different Types of Reports Different Types of Reports Restraints and Injury Restraints and Injury

Restraint Reduction Different Types of Reports Different Types of Reports Children with Frequent Restraints Children with Frequent Restraints

Reporting to Third Parties HCSIS reporting HCSIS reporting

Reporting to Third Parties Reports to MCOsReports to MCOs

Reporting to Third Parties Reports to MCOs Reports to MCOs

Reporting to Third Parties Reports to MCOs Reports to MCOs

Reporting to Third Parties Reports to MCOs Reports to MCOs

Responding to Requests by Funders Dilemma: Dilemma: Multiple expectations for reporting Multiple expectations for reporting Various requirements re data structure Various requirements re data structure Example: Example: Specific request from DHS to produce structured case notes Specific request from DHS to produce structured case notes

Structured Case Note: A Request by DHS

Measuring Symptom Reduction: An Example of the Use of Individual vs. Aggregate Data Youth Outcomes Questionnaire (YOQ) 64-item standardized measure 64-item standardized measure Total Score and Six Subscales Total Score and Six Subscales Intrapersonal Distress Intrapersonal Distress Somatic Somatic Interpersonal Relations Interpersonal Relations Social Problems Social Problems Behavioral Dysfunction Behavioral Dysfunction Critical Items Critical Items

Measuring Symptom Reduction Youth Outcomes Questionnaire (YOQ) Individual Results

Measuring Symptom Reduction Individual Results

Measuring Symptom Reduction: A Look at One Child - Two Measures

Measuring Symptom Reduction YOQ – Aggregate Results YOQ – Aggregate Results Of the children discharged in , 42% showed a decrease in YOQ scores from the second administration of the YOQ until the time they were discharged. Of the children discharged in , 42% showed a decrease in YOQ scores from the second administration of the YOQ until the time they were discharged. Children with higher YOQ Discharge Scores (suggesting more clinical symptoms) have a higher numbers of restraints. Children with higher YOQ Discharge Scores (suggesting more clinical symptoms) have a higher numbers of restraints.

Development of a New Measure: Family Work What do we want to know? What do we want to know? How often do families have therapy sessions? How often do families have therapy sessions? Do families see therapy sessions as helpful? Do families see therapy sessions as helpful? Do children participate in the sessions? Do children participate in the sessions? Do meetings address issues that are important to the family? Do meetings address issues that are important to the family? How can we measure it? How can we measure it?

Development of a New Measure: Family Work Design Design Measure Measure Assess Assess Improve Improve Improve Assess Measure Design

Development of a New Measure: Family Work

Discharge and Aftercare Data At Discharge At Discharge Restrictiveness of Living Scale (ROLES) Restrictiveness of Living Scale (ROLES) Aftercare Survey Aftercare Survey Interviews occur at four points in time: Interviews occur at four points in time: 3-6 months post discharge 3-6 months post discharge 1 year post discharge 1 year post discharge 2 years post discharge 2 years post discharge 3 years post discharge 3 years post discharge

At Discharge The Restrictiveness of Living Scale (ROLES) The Restrictiveness of Living Scale (ROLES) 85% were discharged to a home setting

Aftercare The intent of the Aftercare Study is to look at whether children are: The intent of the Aftercare Study is to look at whether children are: At Home At Home Still in less restrictive setting Still in less restrictive setting Disruptions in placement Disruptions in placement In School In School Suspensions Suspensions Expulsions Expulsions Out of Trouble Out of Trouble Reported Drug/Alcohol Use Reported Drug/Alcohol Use Involvement with Legal System Involvement with Legal System Healthy Healthy Have regular medical and dental appointments Have regular medical and dental appointments

Aftercare Data Collection At Home

Aftercare Data Collection In School  At the first interview, 84% of families report being satisfied with their child’s school placement.  Two years after discharge, 74% of families report no disruptions in their child’s education.

Aftercare Data Collection Out of Trouble A large percentage of the children were reported not to be using either drugs or alcohol. 96% of the children at the 24-month interval reportedly were not using either drugs or alcohol.

Aftercare Data Collection Out of Trouble

Aftercare Data: Representation of Needs Leads to a referral to Family Resource Services

More Sophisticated Data Analysis Use of Excel or SPSS to explore trends in data: Use of Excel or SPSS to explore trends in data: Run Charts Run Charts Pareto Charts Pareto Charts Correlations Correlations Regressions Regressions T-tests T-tests

Effectively Utilizing Data Collection Developing an effective data collection system is a critical part of providing quality services today. Developing an effective data collection system is a critical part of providing quality services today. Performance-based service delivery is essential to our practice. Performance-based service delivery is essential to our practice.

Questions?