1. Using Data to Inform Practice February, 15, 2011 David Colton, Ph.D., M.P.A.

2. ARE YOU JUST WHINING OR CAN YOU VERIFY THAT WITH DATA???? 2

3. How Data Fits Into The Process: Because organizational factors create the context for treatment delivery, they appear to play a greater part in facilitating reduction of seclusion and restraint than does knowledge solely about the individual being secluded or restrained. 3

4. Data Use and the Six Core Strategies GOAL TWO: To reduce the use of S/R by using data in an empirical, non-punitive, manner. Includes using data to analyze characteristics of facility usage by unit, shift day, and staff member; identifying facility baseline; setting improvement goals and comparatively monitoring use over time in all care areas, units and/or state system’s like facilities. 4

5. What is the evaluation question? What do we want to know and why: To understand what we are doing To improve organizational performance To report for external oversight (e.g. licensing agencies) 5

6. The evaluation question helps to determine the data you need and how to collect it (1) Example: Is treatment effective? What clinical outcome is anticipated? How often was this particular intervention offered? Is this treatment approach more effective than an alternative? What is the unit cost for providing this treatment? What factors enhanced or limited the client’s access to this treatment intervention? In what ways could we have improved treatment for this client? How can we enhance delivery of this intervention? 6

7. The evaluation question helps to determine the data you need and how to collect it (2) Will the information you need produce quantitative or qualitative data? How will I collect this data? Who will collect this data? How often will I collect this data? How will I process this data? 7

8. The evaluation question helps to determine the data you need and how to collect it (3) Examples of quantitative data: What is the current level of activity (frequency and duration)? What is the current level of activity taking into account variables such as age, gender, time of day (shift), day of week, location (residential unit vs. cafeteria, school, gymnasium, etc.)? How does our data compare to the data of other facilities serving similar populations? 8

9. The evaluation question helps to determine the data you need and how to collect it (4) Examples of qualitative data: Interview transcripts Focus group results Observational notes Attitude and opinion questionnaires (Reducing Seclusion and Restraint: Organizational Questionnaire (Colton and Xiong); The Management of Aggression and Violence Attitude Scale (Duxbury) 9

10. Addressing Data Collection (1) Is the data collection process organized, systematic, and consistent? Does the process help to ensure the trustworthiness of the data - is it valid and reliable? Are the individuals who collect the data adequately trained? Does the data get to the people who need it and does it get to them in a timely manner? Are the definitions well understood? 10

11. Addressing Data Collection (2) Definitions – is there a standard definition of physical restraint and does everyone know what it is? Does the definition take into consideration duration(?); type of hold(?); and/or situation(?) For example, is a hold used to sooth a child counted as a restraint? 11

12. Addressing Data Collection (3) How is the data collected and does the process ensure for data trustworthiness? Valid – does it measure what it intends to measure? Reliable – does the process produce consistent results over time? Useable – Is the data in a form where it can be used immediately or does the data have to be transformed (for example, converted to a ratio)? 12

13. Addressing Data Collection (4) How is the data collected and by whom? How are data collectors prepared to collect data? How often are they retrained? Is the collection instrument standardized? For example, does each RTF have it’s own form or is there a standardized form with standardized definitions in use across the state? 13

14. Data Analysis – Answering Our Evaluation Questions (1) Frequency: how many how often how long when (time of day, week, month) where (location), who (staff and child – gender, age, weight, height, diagnosis, history) 14

15. Data Analysis – Answering Our Evaluation Questions (2) For trending purposes, unless outliers are removed from the data, it will always be skewed somewhat by a few individuals requiring multiple interventions and/or of extended duration. Reliability issue: Is there a standard definition for an outlier? Reliance on quantitative data and trend analysis has limitations. For example, trend analysis will not explain underlying causation. 15

16. Data Analysis – Answering Our Evaluation Questions (3) Comparison to the performance of other facilities is helpful in understanding how your organization performs in relation to other facilities. However, performance data is not necessarily benchmarking data. Benchmarking can be used to identify facilities that have implemented “best practice” approaches. Qualitative analysis, such as content analysis of data collection forms and observation notes, when coupled with quantitative analysis can broaden our understanding of causative factors. 16

17. Data Analysis – Answering Our Evaluation Questions (4) Trend Analysis using SPC: Common cause variation: variation is systematic, for example repetitive pattern and fluctuations. This typically indicates the need to “fix” the process; i.e. it is a systems issue. Special cause variation: variation is unique and typically indicates the influence of outliers. This requires addressing the unique needs of the client. 17

18. Data Analysis (5) Feedback mechanisms should be in place to provide aggregate data to treatment teams in a timely manner. Data should be presented in various forms to illustrate patterns and trends, including tables and graphs. 18

19. Data Analysis (6) Restraint use indicates a downward trend. Hurray! We’re winning the battle. 19

20. Data Analysis (7) In reality, restraint use shows a downward trend after an increase. Also, the most recent low was not much lower than it was sixteen months ago. Oh dear! Things are quite what they seem. 20

21. Data Analysis (8) The pattern actually indicates that seclusion use is cyclic. The current downward trend is most likely a low point in a cycle. The data is actually indicating that restraint use is quite variable. Variability suggests that the organization is not having much influence on the process. In turn, this suggests that a goal might be to reduce the use of seclusion and to minimize variation from month-to-month. 21

22. Using Data (1) Data is a tool to measure the efforts of attaining goals and improving services. Data in and of itself cannot solve problems or make improvements: ◦ Self-assessment based on “factors that contribute to the reduction of seclusion and restraint” can be used to identify areas in need of development/improvement. ◦ Based on an internal needs assessment, a plan of action should be devised. This should include specific goals and objectives, and a strategic plan. ◦ Data can be used to measure the effectiveness of the plan to achieve desired goals. 22

23. Systems Evaluation and Quality Improvement (1) The organization has established policies, procedures, and systems for continuous evaluation of the need for and appropriate use of seclusion and restraint. There is a systematic data management process (data collection and analysis) in place relevant to seclusion and restraint use. The data management process ensures for the accuracy of seclusion and restraint data. Data is made available to treatment teams so that they can measure the effects of their efforts to reduce/eliminate the use of seclusion and restraint. Data about the frequency and duration of restrictive interventions is made available for review and analysis on a daily basis. Data provides information about both long-term (months/years) and short-term (days/weeks/months) utilization of seclusion and restraint. Data is used to examine the relationship between the use of seclusion and restraint and other factors, such as patient injuries; staff injuries; use of medications; patient and staff demographics; and the like. 23

24. Systems Evaluation and Quality Improvement (2) There is an internal audit system to investigate incidents and provide information that can be used to correct problems and improve the quality of care. The organization makes use of quality improvement tools, such as cause and effect analysis, Pareto analysis, scattergrams, statistical process control, and the ‘repetitive why’ approach to analyze the data. Qualitative data is reviewed, including incident reports and seclusion and restraint documentation, to assess opportunities for improvement. Client satisfaction questionnaires include items that collect data about the client’s experience in the use of seclusion and/or restraint. Data is used to measure the extent that seclusion and restraint reduction goals and plans are being achieved. There is written evidence of action taken to reduce the use of seclusion and restraint in response to data analysis, such as meeting minutes and/or quality improvement documents. 24

25. Finis 25