Presentation on theme: "Presented by Jennifer Holmes Drug & Alcohol Service Langton Centre May 2012 The Australian Treatment Outcome Profile (ATOP): Clinical Monitoring for the."— Presentation transcript:
Presented by Jennifer Holmes Drug & Alcohol Service Langton Centre May 2012 The Australian Treatment Outcome Profile (ATOP): Clinical Monitoring for the Drug & Alcohol Sector Anni Ryan, Adrian Dunlop, Jennifer Holmes, Vi Hunt, Kristie Mammen, Raka Tierney, Nicholas Lintzeris.
Acknowledgements Patients and staff of Jacaranda House, the Langton Centre and Newcastle OPT Clinics especially: Yvonne Sutton, Rohan Holland, Betty Jago, Skye O’Donnell, Gonzalo Rivas, Terry Schofield and Doungkamol Sindhusake. MHDAO, Ministry of Health, D&A Research Program Grant 2009/2010 MHDAO, Ministry of Health, Research Grant 2011/2012 UK National Treatment Agency, NHS
Overview Background to the ATOP Stage 1: Validating the ATOP in Australian populations Stage 2: Expanding across services in NSW Stage 3a: Australian dissemination / uptake ?Stage 3b: Developing Composite Measure of Treatment Outcome (ATOP Treatment Episode Global Outcome)
A clinician administered 1-page validated ‘instrument’ Self-report measures across 2 key domains in the preceding 4 weeks –Substance use –Health & well being –Global ratings (0-10) of physical & mental health, quality of life –Housing, employment & study, violence, child protection What is the ATOP?
Benefits of routine clinical outcome monitoring in health services are well documented –clinician administered e.g. HoNOS –repeat blood tests (CD4 counts in HIV) Few examples of successful implementation of in D&A services Background: Clinical Outcome Monitoring
length of the instruments (e.g. >6−10 pages long, taking >20 minutes to complete) + inadequate attention to training + excessive data entry requirements + poor feedback for patients, clinicians and administrators = limited utility and resistance Why is implementation so difficult?
Treatment Outcome Profile (TOP) –a validated tool introduced by the National Treatment Agency in the UK in 2007 –introduced across all NHS funded D&A services in England Modified for Australian conditions = ATOP –Reflecting substances commonly used in Australia –Made more ‘intuitive’ to complete –Validated in Australian populations & against common Australian instruments Developing the ATOP
to assist in systematic, documented and ongoing client assessment to provide feedback to clients about ‘progress’ over time, and assist in treatment care planning and motivation to assist in clinical handover when transferring/referring clients to other services to assist in service evaluation – helping us to answer the broader questions of “do our clients get better?” as a quality and potential research tool with services collecting standardised data Why use the ATOP? Different purposes for different people
Stage One: Project Aims Validate the Treatment Outcome Profile (TOP) under Australian conditions Examine implementation and feasibility issues in 3 NSW public OPT clinics
What do we mean by ‘validated scale’ Face validity: “does it look right” Does it measure things as well as other instruments –K-10, WHO-QOL, SF-12, PHQ-15 –OTI (substance use, injecting practices) Inter-rater reliability: –do 2 different raters administering the instrument on the same client get the same score?
Methods Concurrent validity and inter-rater reliability: ATOP administered as part of routine care by clinic staff at 3 month intervals Research interview within 72 hrs of last ATOP: gold standard instruments + repeat ATOP Implementation and feasibility issues (incl. data management): clinician, service manager and patient perspectives satisfaction surveys and focus groups
Results – concurrent validity Concurrent validityKappaSpearman ’ s Section 2: Injecting Risk Behaviour Injected with a needle/syringe used by someone else 0.81 (p<0.001)0.83 9p<0.001) Section 3: Crime Drug Selling1 (p<0.001) Committing Assault or violence1 (p<0.001)1 (p<.0001)
Results - concurrent validity Section 4: Health and Social FunctioningSpearman ’ s Psychological Health vs WHOQuOL Q (p<0.001) vs K10 total (p<0.001) Physical Health vs PHQ15 Total (p<0.001) SF12 Physical Component Score total0.65 (p<0.001) Quality of Life vs WHOQuOL Q10.69 (p<0.001) vs WHOQuOL Social Relationship Domain total0.55 (p<0.001) vs WHOQuOL Environment Domain Total0.66 (p<0.001) vs SF12 Physical Component Score total0.35 (p<0.001) vs SF12 Mental Component Score total0.62 (p<0.001)
Results - inter-rater reliability (n=103) No significant difference between researcher mean and clinician mean on all continuous items All dichotomous items correlated (p<0.001)
Clinician feedback (n=20) easy to administer: 80% agreed appropriate for my client population: 85% agreed format & style easy to understand: 85% agreed length appropriate for routine practice: 70% agreed appropriate for my setting: 65% agreed, 30% unsure useful in developing a case plan: 65% agreed, 25% unsure useful for identifying important problems 45% agreed, 35% ambivalent, 20% disagree happy to use as part of regular client reviews 45% agreed, 30% ambivalent, 25% disagree
Dislike… crime questions “too many, too difficult, all clients report nil…” “Not sure about the crime questions, although they are important.” At first ATOP clients say “no” to crime questions but by 3 to 6 months clients start to answer “yes” Clients express concerns over privacy/confidentiality/how this information may be used esp. where there is legal/child protection issues
Client feedback (n=123) ATOP questions were easy to understand 93% agreed Helpful way of looking at how well treatment is working for me 85% agreed Helpful to have this same review every few months 85% agreed Length of the ATOP was “about right” 90% agreed
Other comments… “ I don’t think everyone will be honest about the crime questions…” “asks the right questions: interesting to do it again…” “some questions feel like they are a stereotype of drug users…” “I think there should be more questions about how much the client feels the program is helping them…”
Conclusions Stage One ATOP is a valid instrument for measuring treatment outcomes in an Australian opioid maintenance treatment population ATOP is compatible with routine clinical practice ATOP can feasibly be implemented as part of routine clinical practice in public OPT Clinics Investigate feasibility of ATOP in other treatment settings
Stage 2 Pilot across NSW Services Beyond OTP Training Protocol & Business Rules Data Systems
Stage 2 Currently trialled across sites –Counselling –OTP –Withdrawal Management Feasilbility and clinician feedback Establishing protocol and business rules for services Training Package –Train the trainer workshops –Presentations and training manual
Putting the ATOP into Practice: a crash course for clinicians and service managers
How to complete the ATOP Enter –Client MRN, date of birth and sex –Your name –Date of ATOP –The stage at which the ATOP is being done Enter client responses –Use the Conversion Table to calculate standard drinks –Nil drug/alcohol use – enter “00” in the total box –Timeline – invite the client to recall the number of days in each of the past four weeks on which they did something –Before asking Section 2: Items (e) to (h) remind the client about confidentiality (see box) –Yes and no – a simple tick for yes or no –Rating scale – a 10-point scale from poor to good. Together with the client, CIRCLE a number. –Refused/can’t recall – write “NA” (short for Not Answered) next to the total box, tick box or rating scale.
ATOP across Australia ATOP being explored/used in –South Australia (DASSA) –Tasmania (Tas govt sector) –WA –Victoria (Turning Point) –NADA data set (NSW NGO sector)
Designing a Composite Measure of clinical outcomes in D&A treatment The ATOP Research Team March 2012
Composite performance measures The combination of 2 or more indicators into a single number to summarize multiple dimensions of performance and to facilitate comparisons. –integrate a large amount of information in a format that is easily understood. –increasing use by governments to assess performance. –examples: Dow Jones Index, IQ ratings, NAPLAN, SF-36
In substance abuse treatment Challenge: To establish a measure that summarises multiple clinical outcomes commonly associated with D&A treatment –Substance use (primary & secondary drugs) –General health & well-being (physical health, mental health, overall QOL) –High-risk behaviours: injecting, homelessness, child protection, violence, crime
Why a composite outcome index To be able to broadly describe whether or not a treatment episode is associated with changes in patient well-being To be able to state for a particular program: –56% patients significantly improved –27% had no significant change –17% deteriorated Allows identification of benchmarks by: –Drug type (alcohol, opiates etc…) –Treatment type (e.g. counselling, OTP, withdrawal etc…) –Patient factors (e.g. ATSI, age, rurality)
Examples to date: opioid treatment Dutch Heroin Trial (van den Brink et al BMJ 327) Used a pre-specified dichotomous, multidomain outcome index as the primary outcome parameter. Patients considered ‘responders’ if they showed –at least 40% improvement in at least one of the three domains (physical, mental, social) at end of treatment compared with baseline; –if this improvement was not at the expense of a serious ( ≥ 40%) deterioration in functioning in any of the other outcome domains; and –if the improvement was not accompanied by a substantial ( ≥ 20%) increase in use of cocaine or amphetamines.
Examples to date: cocaine treatment Paul Crits-Christoph et al. Psychosocial Treatments for Cocaine Dependence: National Institute on Drug Abuse Collaborative Cocaine Treatment Study. Arch Gen Psychiatry. 1999;56: Composite Cocaine Use Measure –pool information from multiple measures (urine drug tests, ASIs, and weekly cocaine use inventory) to code each month of treatment as abstinent vs not abstinent. Any indication of cocaine use from the 3 measures would lead to a “not abstinent” month. Composite Psychiatric Measure –To test hypothesis: degree of psychiatric symptoms interacting with the treatment condition, a composite measure of 4 scales—the Hamilton Rating Scale for Depression, the Beck Anxiety Inventory, the Brief Symptom Inventory, and the ASI–Psychiatric Severity Composite score— was created by converting each scale to a standard score and then averaging the scores.
Examples to date: Alcohol treatment Zweben & Cisler Clinical and Methodological Utility of a Composite Outcome Measure for Alcohol Treatment Research. Alcoholism: Clinical and Experimental Research 27 (10), 1680–1685 Project MATCH data: composite outcome measure to capture multiple treatment outcomes among diverse client populations. –self-reported alcohol consumption, alcohol problems, biological markers, other areas of functioning (psychiatric dysfunction, QOL). Findings on composite measure: –30% clients sustained a ‘remitted’ status (i.e., abstinent or moderate drinking without problems) over 1-year follow-up; –70% of the clients had reached a nonremitted status (i.e., heavy drinking and/or problems). “The composite outcome index could be used usefully along with singular measures of consumption to obtain a more complete picture of what has occurred among clients”.
Methods for combining items into a Composite Measure Linear combinations –Composite = [indicator1×weight1] + [indicator2×weight2] [indicatorN×weightN] –Expert Panel determine which indicators & which weightings Regression-Based Composite Measures –If a certain outcome is a ‘gold standard’, the weighting of individual items may be determined empirically by optimizing the predictability of the gold standard end point. Latent Trait Composite Measures –Identify clusters of correlated items & latent trait modeling may be used to combine items within clusters but not across clusters. –e.g. substance use outcomes; general health/well-being; high risk / harm behaviours Any-or-none Scoring of outcome measures –In this method, a patient is counted as failing if he or she experiences at least 1 adverse outcome from a list of 2 or more adverse outcomes. Opportunity Scoring –Opportunity scoring counts the number of times a given care process was actually performed (numerator), divided by the number of chances a provider had to give this care correctly (denominator). –e.g. individual treatment care plan completed; MH-COPES completed All-or-None Scoring of Process Measures –Only those patients who receive all indicated processes of care are counted as successes.
Recommendations re: Developing Composite Measures (Peterson et al 2010) 1.The intended audience & purpose of a composite measure should be explicitly stated. 2.Decisions about which measures should be based on clinical importance of patient outcomes and the reliability of individual performance measures. 3.Each individual component should be precisely defined to ensure consistent application in different settings. 4.The description of the methods used for weighting and combining individual measures into a composite performance measure should be transparent. 5.Developers should explore a variety of alternative methods for combining measures and document whether conclusions about provider performance differ with use of alternative methods. 6.Empirical testing needed to assess the properties of a composite measure score. 7.Reporting of composite performance measures should be accompanied by detailed reporting of individual domains and components. 8.Reporting of composite performance measures should include a measure of the degree of uncertainty surrounding composite estimates for providers. 9.Composite performance measures must be reevaluated as that evidence changes.
Could the ATOP be used to develop a composite measure of D&A treatment outcome / success? Instrument needs to be used by the sector Instrument shown to be robust across diverse populations, drug types, treatment types Measures the main domains that we are most interested –in over time (e.g. beginning & end of treatment) –substance use, general health, high-risk behaviours Linked to other aspects of NMDS –identifying primary/secondary drug use, demographics, treatment types delivered )
ATOP Treatment Episode Global Outcome Aim To develop & validate the ‘ATOP Treatment Episode Global Outcome’, as a means of assigning a global outcome for each D&A treatment episode that broadly reflects whether each treatment episode was associated with a significant improvement, no change, or significant deterioration in the main clinical domains.
Proposed methodology ATOP Treatment Episode Global Outcome National, multisite, project, 2-3 year duration –Reference group of expert clinicians, consumer reps, researchers, data experts Recruit large number of clients (100’s) entering a variety of treatment types (counselling, OTP, withdrawal, rehab) and using different primary drug types (alcohol, opioids, cannabis, other) –ATOP at beginning & end of treatment episode (or intervals for OTP, rehab), & –“Gold standard”: client & clinician global outcome ratings for each treatment episode (confirmed by independent clinician & client panel, with broad criteria identified by Reference group for different treatment types). Identify algorithms for attributing ATOP Treatment Episode Global Outcome for different treatment modalities & different primary drug types. Statistical methods (e.g. RUC) to identify algorithms against ‘gold standard’ measure of client & clinician global ratings. –How much change in (a) substance use, (b) general health & (c) high-risk practices is required for the ATOP scores to match client & clinician global ratings.
The holy grail? Data systems that enable us to triangulate data re: –client characteristics –services provided (treatment episode data such as type of service, number of contacts, by whom) –client outcome Would enable benchmarking of services NMDS possibly only mechanism to ensure this occurs.
For Further Information Contact ATOP Project Coordinator –Kristie Mammen –Langton Centre Composite Measure Project –Nicholas Lintzeris Validation of ATOP –Anni Ryan Data Collection Systems –Jennifer Holmes –