Presentation on theme: "Acknowledgements Patients and staff of Jacaranda House, the Langton Centre and Newcastle OPT Clinics especially: Yvonne Sutton, Rohan Holland, Betty."— Presentation transcript:
0The Australian Treatment Outcome Profile (ATOP): Clinical Monitoring for the Drug & Alcohol Sector Anni Ryan, Adrian Dunlop, Jennifer Holmes, Vi Hunt, Kristie Mammen, Raka Tierney, Nicholas Lintzeris.Presented by Jennifer Holmes Drug & Alcohol ServiceLangton CentreMay 2012
1AcknowledgementsPatients and staff of Jacaranda House, the Langton Centre and Newcastle OPT Clinics especially: Yvonne Sutton, Rohan Holland, Betty Jago, Skye O’Donnell, Gonzalo Rivas, Terry Schofield and Doungkamol Sindhusake.MHDAO, Ministry of Health, D&A Research Program Grant /2010MHDAO, Ministry of Health, Research Grant 2011/2012UK National Treatment Agency, NHS1
2Overview Background to the ATOP Stage 1: Validating the ATOP in Australian populationsStage 2: Expanding across services in NSWStage 3a: Australian dissemination / uptake?Stage 3b: Developing Composite Measure of Treatment Outcome (ATOP Treatment Episode Global Outcome)
3What is the ATOP?A clinician administered 1-page validated ‘instrument’Self-report measures across 2 key domains in the preceding 4 weeksSubstance useHealth & well beingGlobal ratings (0-10) of physical & mental health, quality of lifeHousing, employment & study, violence, child protection
4Background: Clinical Outcome Monitoring Benefits of routine clinical outcome monitoring in health services are well documentedclinician administered e.g. HoNOSrepeat blood tests (CD4 counts in HIV)Few examples of successful implementation of in D&A services
5Why is implementation so difficult? length of the instruments (e.g. >6−10 pages long, taking >20 minutes to complete)+inadequate attention to trainingexcessive data entry requirementspoor feedback for patients, clinicians and administrators=limited utility and resistance
6Developing the ATOPTreatment Outcome Profile (TOP)a validated tool introduced by the National Treatment Agency in the UK in 2007introduced across all NHS funded D&A services in EnglandModified for Australian conditions = ATOPReflecting substances commonly used in AustraliaMade more ‘intuitive’ to completeValidated in Australian populations & against common Australian instruments
7Why use the ATOP? Different purposes for different people to assist in systematic, documented and ongoing client assessmentto provide feedback to clients about ‘progress’ over time, and assist in treatment care planning and motivationto assist in clinical handover when transferring/referring clients to other servicesto assist in service evaluation – helping us to answer the broader questions of “do our clients get better?”as a quality and potential research tool with services collecting standardised data
8Stage One: Project Aims Validate the Treatment Outcome Profile (TOP) under Australian conditionsExamine implementation and feasibility issues in 3 NSW public OPT clinics
9What do we mean by ‘validated scale’ Face validity: “does it look right”Does it measure things as well as other instrumentsK-10, WHO-QOL, SF-12, PHQ-15OTI (substance use, injecting practices)Inter-rater reliability:do 2 different raters administering the instrument on the same client get the same score?9
11Methods Concurrent validity and inter-rater reliability: ATOP administered as part of routine care by clinic staff at 3 month intervalsResearch interview within 72 hrs of last ATOP: gold standard instruments + repeat ATOPImplementation and feasibility issues (incl. data management):clinician, service manager and patient perspectivessatisfaction surveys and focus groups11
15Results - inter-rater reliability (n=103) No significant difference between researcher mean and clinician mean on all continuous itemsAll dichotomous items correlated (p<0.001)15
16Clinician feedback (n=20) easy to administer: 80% agreedappropriate for my client population: 85% agreedformat & style easy to understand: 85% agreedlength appropriate for routine practice: 70% agreedappropriate for my setting: 65% agreed, 30% unsureuseful in developing a case plan: 65% agreed, 25% unsureuseful for identifying important problems 45% agreed, 35% ambivalent, 20% disagreehappy to use as part of regular client reviews 45% agreed, 30% ambivalent, 25% disagree16
17Dislike… crime questions “too many, too difficult, all clients report nil…”“Not sure about the crime questions, although they are important.”At first ATOP clients say “no” to crime questions but by 3 to 6 months clients start to answer “yes”Clients express concerns over privacy/confidentiality/how this information may be used esp. where there is legal/child protection issues17
18Client feedback (n=123)ATOP questions were easy to understand 93% agreedHelpful way of looking at how well treatment is working for me 85% agreedHelpful to have this same review every few months 85% agreedLength of the ATOP was “about right” 90% agreed18
19Other comments…“ I don’t think everyone will be honest about the crime questions…”“asks the right questions: interesting to do it again…”“some questions feel like they are a stereotype of drug users…”“I think there should be more questions about how much the client feels the program is helping them…”19
20Conclusions Stage OneATOP is a valid instrument for measuring treatment outcomes in an Australian opioid maintenance treatment populationATOP is compatible with routine clinical practiceATOP can feasibly be implemented as part of routine clinical practice in public OPT ClinicsInvestigate feasibility of ATOP in other treatment settingsATOP demonstrated acceptable concurrent validity and inter-rater reliabilityPatient/Clinician feedback:Easy to useApplicable/compatible with routine careLiked brevityTraining and support materialsTime management issuesSuggested improvements to enhance clinical utility of questionnaire:visual longitudinal feedback for patient/clinicianelectronic reminder systemsintegration with existing electronic clinical information systems20
22Stage 2 Pilot across NSW Services Beyond OTPTrainingProtocol & Business RulesData Systems
23Stage 2 Currently trialled across 10-15 sites Counselling OTP Withdrawal ManagementFeasilbility and clinician feedbackEstablishing protocol and business rules for servicesTraining PackageTrain the trainer workshopsPresentations and training manual
24Putting the ATOP into Practice: a crash course for clinicians and service managers 24
25How to complete the ATOP EnterClient MRN, date of birth and sexYour nameDate of ATOPThe stage at which the ATOP is being doneEnter client responsesUse the Conversion Table to calculate standard drinksNil drug/alcohol use – enter “00” in the total boxTimeline – invite the client to recall the number of days in each of the past four weeks on which they did somethingBefore asking Section 2: Items (e) to (h) remind the client about confidentiality (see box)Yes and no – a simple tick for yes or noRating scale – a 10-point scale from poor to good. Together with the client, CIRCLE a number.Refused/can’t recall – write “NA” (short for Not Answered) next to the total box, tick box or rating scale.25
26ATOP across Australia ATOP being explored/used in South Australia (DASSA)Tasmania (Tas govt sector)WAVictoria (Turning Point)NADA data set (NSW NGO sector)
27Designing a Composite Measure of clinical outcomes in D&A treatment The ATOP Research TeamMarch 2012
28Composite performance measures The combination of 2 or more indicators into a single number to summarize multiple dimensions of performance and to facilitate comparisons.integrate a large amount of information in a format that is easily understood.increasing use by governments to assess performance.examples: Dow Jones Index, IQ ratings, NAPLAN, SF-36
29In substance abuse treatment Challenge: To establish a measure that summarises multiple clinical outcomes commonly associated with D&A treatmentSubstance use (primary & secondary drugs)General health & well-being (physical health, mental health, overall QOL)High-risk behaviours: injecting, homelessness, child protection, violence, crime
30Why a composite outcome index To be able to broadly describe whether or not a treatment episode is associated with changes in patient well-beingTo be able to state for a particular program:56% patients significantly improved27% had no significant change17% deterioratedAllows identification of benchmarks by:Drug type (alcohol, opiates etc…)Treatment type (e.g. counselling, OTP, withdrawal etc…)Patient factors (e.g. ATSI, age, rurality)
31Examples to date: opioid treatment Dutch Heroin Trial (van den Brink et al BMJ 327)Used a pre-specified dichotomous, multidomain outcome index as the primary outcome parameter. Patients considered ‘responders’ if they showedat least 40% improvement in at least one of the three domains (physical, mental, social) at end of treatment compared with baseline;if this improvement was not at the expense of a serious ( ≥ 40%) deterioration in functioning in any of the other outcome domains; andif the improvement was not accompanied by a substantial ( ≥ 20%) increase in use of cocaine or amphetamines.
33Examples to date: cocaine treatment Paul Crits-Christoph et al. Psychosocial Treatments for Cocaine Dependence: National Institute on Drug Abuse Collaborative Cocaine Treatment Study. Arch Gen Psychiatry. 1999;56:Composite Cocaine Use Measurepool information from multiple measures (urine drug tests, ASIs, and weekly cocaine use inventory) to code each month of treatment as abstinent vs not abstinent. Any indication of cocaine use from the 3 measures would lead to a “not abstinent” month.Composite Psychiatric MeasureTo test hypothesis: degree of psychiatric symptoms interacting with the treatment condition, a composite measure of 4 scales—the Hamilton Rating Scale for Depression, the Beck Anxiety Inventory, the Brief Symptom Inventory, and the ASI–Psychiatric Severity Composite score—was created by converting each scale to a standard score and then averaging the scores.
34Examples to date: Alcohol treatment Zweben & Cisler. 2003 Examples to date: Alcohol treatment Zweben & Cisler Clinical and Methodological Utility of a Composite Outcome Measure for Alcohol Treatment Research. Alcoholism: Clinical and Experimental Research 27 (10), 1680–1685Project MATCH data: composite outcome measure to capture multiple treatment outcomes among diverse client populations.self-reported alcohol consumption, alcohol problems, biological markers, other areas of functioning (psychiatric dysfunction, QOL).Findings on composite measure:30% clients sustained a ‘remitted’ status (i.e., abstinent or moderate drinking without problems) over 1-year follow-up;70% of the clients had reached a nonremitted status (i.e., heavy drinking and/or problems).“The composite outcome index could be used usefully along with singular measures of consumption to obtain a more complete picture of what has occurred among clients”.
35Methods for combining items into a Composite Measure Linear combinationsComposite = [indicator1×weight1] + [indicator2×weight2] [indicatorN×weightN]Expert Panel determine which indicators & which weightingsRegression-Based Composite MeasuresIf a certain outcome is a ‘gold standard’, the weighting of individual items may be determined empirically by optimizing the predictability of the gold standard end point.Latent Trait Composite MeasuresIdentify clusters of correlated items & latent trait modeling may be used to combine items within clusters but not across clusters.e.g. substance use outcomes; general health/well-being; high risk / harm behavioursAny-or-none Scoring of outcome measuresIn this method, a patient is counted as failing if he or she experiences at least 1 adverse outcome from a list of 2 or more adverse outcomes.Opportunity ScoringOpportunity scoring counts the number of times a given care process was actually performed (numerator), divided by the number of chances a provider had to give this care correctly (denominator).e.g. individual treatment care plan completed; MH-COPES completedAll-or-None Scoring of Process MeasuresOnly those patients who receive all indicated processes of care are counted as successes.
36Recommendations re: Developing Composite Measures (Peterson et al 2010) The intended audience & purpose of a composite measure should be explicitly stated.Decisions about which measures should be based on clinical importance of patient outcomes and the reliability of individual performance measures.Each individual component should be precisely defined to ensure consistent application in different settings.The description of the methods used for weighting and combining individual measures into a composite performance measure should be transparent.Developers should explore a variety of alternative methods for combining measures and document whether conclusions about provider performance differ with use of alternative methods.Empirical testing needed to assess the properties of a composite measure score.Reporting of composite performance measures should be accompanied by detailed reporting of individual domains and components.Reporting of composite performance measures should include a measure of the degree of uncertainty surrounding composite estimates for providers.Composite performance measures must be reevaluated as that evidence changes.
37Could the ATOP be used to develop a composite measure of D&A treatment outcome / success? Instrument needs to be used by the sectorInstrument shown to be robust across diverse populations, drug types, treatment typesMeasures the main domains that we are most interestedin over time (e.g. beginning & end of treatment)substance use, general health, high-risk behavioursLinked to other aspects of NMDSidentifying primary/secondary drug use, demographics, treatment types delivered )
38ATOP Treatment Episode Global Outcome Aim To develop & validate the ‘ATOP Treatment Episode Global Outcome’, as a means of assigning a global outcome for each D&A treatment episode that broadly reflects whether each treatment episode was associated with a significant improvement, no change, or significant deterioration in the main clinical domains.
39Proposed methodology ATOP Treatment Episode Global Outcome National, multisite, project, 2-3 year durationReference group of expert clinicians, consumer reps, researchers, data expertsRecruit large number of clients (100’s) entering a variety of treatment types (counselling, OTP, withdrawal, rehab) and using different primary drug types (alcohol, opioids, cannabis, other)ATOP at beginning & end of treatment episode (or intervals for OTP, rehab), &“Gold standard”: client & clinician global outcome ratings for each treatment episode (confirmed by independent clinician & client panel, with broad criteria identified by Reference group for different treatment types).Identify algorithms for attributing ATOP Treatment Episode Global Outcome for different treatment modalities & different primary drug types. Statistical methods (e.g. RUC) to identify algorithms against ‘gold standard’ measure of client & clinician global ratings.How much change in (a) substance use, (b) general health & (c) high-risk practices is required for the ATOP scores to match client & clinician global ratings.
40The holy grail? Data systems that enable us to triangulate data re: client characteristicsservices provided (treatment episode data such as type of service, number of contacts, by whom)client outcomeWould enable benchmarking of servicesNMDS possibly only mechanism to ensure this occurs.
41For Further Information Contact ATOP Project CoordinatorKristie MammenLangton CentreComposite Measure ProjectNicholas LintzerisValidation of ATOPAnni RyanData Collection SystemsJennifer Holmes