Presentation is loading. Please wait.

Presentation is loading. Please wait.

Acknowledgements Patients and staff of Jacaranda House, the Langton Centre and Newcastle OPT Clinics especially: Yvonne Sutton, Rohan Holland, Betty.

Similar presentations

Presentation on theme: "Acknowledgements Patients and staff of Jacaranda House, the Langton Centre and Newcastle OPT Clinics especially: Yvonne Sutton, Rohan Holland, Betty."— Presentation transcript:

0 The Australian Treatment Outcome Profile (ATOP): Clinical Monitoring for the Drug & Alcohol Sector
Anni Ryan, Adrian Dunlop, Jennifer Holmes, Vi Hunt, Kristie Mammen, Raka Tierney, Nicholas Lintzeris. Presented by Jennifer Holmes Drug & Alcohol Service Langton Centre May 2012

1 Acknowledgements Patients and staff of Jacaranda House, the Langton Centre and Newcastle OPT Clinics especially: Yvonne Sutton, Rohan Holland, Betty Jago, Skye O’Donnell, Gonzalo Rivas, Terry Schofield and Doungkamol Sindhusake. MHDAO, Ministry of Health, D&A Research Program Grant /2010 MHDAO, Ministry of Health, Research Grant 2011/2012 UK National Treatment Agency, NHS 1

2 Overview Background to the ATOP
Stage 1: Validating the ATOP in Australian populations Stage 2: Expanding across services in NSW Stage 3a: Australian dissemination / uptake ?Stage 3b: Developing Composite Measure of Treatment Outcome (ATOP Treatment Episode Global Outcome)

3 What is the ATOP? A clinician administered 1-page validated ‘instrument’ Self-report measures across 2 key domains in the preceding 4 weeks Substance use Health & well being Global ratings (0-10) of physical & mental health, quality of life Housing, employment & study, violence, child protection

4 Background: Clinical Outcome Monitoring
Benefits of routine clinical outcome monitoring in health services are well documented clinician administered e.g. HoNOS repeat blood tests (CD4 counts in HIV) Few examples of successful implementation of in D&A services

5 Why is implementation so difficult?
length of the instruments (e.g. >6−10 pages long, taking >20 minutes to complete) + inadequate attention to training excessive data entry requirements poor feedback for patients, clinicians and administrators = limited utility and resistance

6 Developing the ATOP Treatment Outcome Profile (TOP) a validated tool introduced by the National Treatment Agency in the UK in 2007 introduced across all NHS funded D&A services in England Modified for Australian conditions = ATOP Reflecting substances commonly used in Australia Made more ‘intuitive’ to complete Validated in Australian populations & against common Australian instruments

7 Why use the ATOP? Different purposes for different people
to assist in systematic, documented and ongoing client assessment to provide feedback to clients about ‘progress’ over time, and assist in treatment care planning and motivation to assist in clinical handover when transferring/referring clients to other services to assist in service evaluation – helping us to answer the broader questions of “do our clients get better?” as a quality and potential research tool with services collecting standardised data

8 Stage One: Project Aims
Validate the Treatment Outcome Profile (TOP) under Australian conditions Examine implementation and feasibility issues in 3 NSW public OPT clinics

9 What do we mean by ‘validated scale’
Face validity: “does it look right” Does it measure things as well as other instruments K-10, WHO-QOL, SF-12, PHQ-15 OTI (substance use, injecting practices) Inter-rater reliability: do 2 different raters administering the instrument on the same client get the same score? 9


11 Methods Concurrent validity and inter-rater reliability:
ATOP administered as part of routine care by clinic staff at 3 month intervals Research interview within 72 hrs of last ATOP: gold standard instruments + repeat ATOP Implementation and feasibility issues (incl. data management): clinician, service manager and patient perspectives satisfaction surveys and focus groups 11

12 Results - client demographics (n=104)
Gender Male Female 62 (59.6%) 42 (40.4%) Age 0-30 31-40 41-50 51+ 32 (30.8%) 37 (35.6%) 22 (21.2%) 13 (12.5%) 12

13 Results – concurrent validity
Kappa Spearman’s Section 2: Injecting Risk Behaviour Injected with a needle/syringe used by someone else 0.81 (p<0.001) 0.83 9p<0.001) Section 3: Crime Drug Selling 1 (p<0.001) Committing Assault or violence 1 (p<.0001) 13

14 Results - concurrent validity
Section 4: Health and Social Functioning Spearman’s Psychological Health vs WHOQuOL Q26 0.62 (p<0.001) vs K10 total -0.60 (p<0.001) Physical Health vs PHQ15 Total -0.58 (p<0.001) SF12 Physical Component Score total 0.65 (p<0.001) Quality of Life vs WHOQuOL Q1 0.69 (p<0.001) vs WHOQuOL Social Relationship Domain total 0.55 (p<0.001) vs WHOQuOL Environment Domain Total 0.66 (p<0.001) vs SF12 Physical Component Score total 0.35 (p<0.001) vs SF12 Mental Component Score total 14

15 Results - inter-rater reliability (n=103)
No significant difference between researcher mean and clinician mean on all continuous items All dichotomous items correlated (p<0.001) 15

16 Clinician feedback (n=20)
easy to administer: 80% agreed appropriate for my client population: 85% agreed format & style easy to understand: 85% agreed length appropriate for routine practice: 70% agreed appropriate for my setting: 65% agreed, 30% unsure useful in developing a case plan: 65% agreed, 25% unsure useful for identifying important problems 45% agreed, 35% ambivalent, 20% disagree happy to use as part of regular client reviews 45% agreed, 30% ambivalent, 25% disagree 16

17 Dislike… crime questions
“too many, too difficult, all clients report nil…” “Not sure about the crime questions, although they are important.” At first ATOP clients say “no” to crime questions but by 3 to 6 months clients start to answer “yes” Clients express concerns over privacy/confidentiality/how this information may be used esp. where there is legal/child protection issues 17

18 Client feedback (n=123) ATOP questions were easy to understand 93% agreed Helpful way of looking at how well treatment is working for me 85% agreed Helpful to have this same review every few months 85% agreed Length of the ATOP was “about right” 90% agreed 18

19 Other comments… “ I don’t think everyone will be honest about the crime questions…” “asks the right questions: interesting to do it again…” “some questions feel like they are a stereotype of drug users…” “I think there should be more questions about how much the client feels the program is helping them…” 19

20 Conclusions Stage One ATOP is a valid instrument for measuring treatment outcomes in an Australian opioid maintenance treatment population ATOP is compatible with routine clinical practice ATOP can feasibly be implemented as part of routine clinical practice in public OPT Clinics Investigate feasibility of ATOP in other treatment settings ATOP demonstrated acceptable concurrent validity and inter-rater reliability Patient/Clinician feedback: Easy to use Applicable/compatible with routine care Liked brevity Training and support materials Time management issues Suggested improvements to enhance clinical utility of questionnaire: visual longitudinal feedback for patient/clinician electronic reminder systems integration with existing electronic clinical information systems 20


22 Stage 2 Pilot across NSW Services
Beyond OTP Training Protocol & Business Rules Data Systems

23 Stage 2 Currently trialled across 10-15 sites Counselling OTP
Withdrawal Management Feasilbility and clinician feedback Establishing protocol and business rules for services Training Package Train the trainer workshops Presentations and training manual

24 Putting the ATOP into Practice: a crash course for clinicians and service managers

25 How to complete the ATOP
Enter Client MRN, date of birth and sex Your name Date of ATOP The stage at which the ATOP is being done Enter client responses Use the Conversion Table to calculate standard drinks Nil drug/alcohol use – enter “00” in the total box Timeline – invite the client to recall the number of days in each of the past four weeks on which they did something Before asking Section 2: Items (e) to (h) remind the client about confidentiality (see box) Yes and no – a simple tick for yes or no Rating scale – a 10-point scale from poor to good. Together with the client, CIRCLE a number. Refused/can’t recall – write “NA” (short for Not Answered) next to the total box, tick box or rating scale. 25

26 ATOP across Australia ATOP being explored/used in
South Australia (DASSA) Tasmania (Tas govt sector) WA Victoria (Turning Point) NADA data set (NSW NGO sector)

27 Designing a Composite Measure of clinical outcomes in D&A treatment
The ATOP Research Team March 2012

28 Composite performance measures
The combination of 2 or more indicators into a single number to summarize multiple dimensions of performance and to facilitate comparisons. integrate a large amount of information in a format that is easily understood. increasing use by governments to assess performance. examples: Dow Jones Index, IQ ratings, NAPLAN, SF-36

29 In substance abuse treatment
Challenge: To establish a measure that summarises multiple clinical outcomes commonly associated with D&A treatment Substance use (primary & secondary drugs) General health & well-being (physical health, mental health, overall QOL) High-risk behaviours: injecting, homelessness, child protection, violence, crime

30 Why a composite outcome index
To be able to broadly describe whether or not a treatment episode is associated with changes in patient well-being To be able to state for a particular program: 56% patients significantly improved 27% had no significant change 17% deteriorated Allows identification of benchmarks by: Drug type (alcohol, opiates etc…) Treatment type (e.g. counselling, OTP, withdrawal etc…) Patient factors (e.g. ATSI, age, rurality)

31 Examples to date: opioid treatment
Dutch Heroin Trial (van den Brink et al BMJ 327) Used a pre-specified dichotomous, multidomain outcome index as the primary outcome parameter. Patients considered ‘responders’ if they showed at least 40% improvement in at least one of the three domains (physical, mental, social) at end of treatment compared with baseline; if this improvement was not at the expense of a serious ( ≥ 40%) deterioration in functioning in any of the other outcome domains; and if the improvement was not accompanied by a substantial ( ≥ 20%) increase in use of cocaine or amphetamines.


33 Examples to date: cocaine treatment
Paul Crits-Christoph et al. Psychosocial Treatments for Cocaine Dependence: National Institute on Drug Abuse Collaborative Cocaine Treatment Study. Arch Gen Psychiatry. 1999;56: Composite Cocaine Use Measure pool information from multiple measures (urine drug tests, ASIs, and weekly cocaine use inventory) to code each month of treatment as abstinent vs not abstinent. Any indication of cocaine use from the 3 measures would lead to a “not abstinent” month. Composite Psychiatric Measure To test hypothesis: degree of psychiatric symptoms interacting with the treatment condition, a composite measure of 4 scales—the Hamilton Rating Scale for Depression, the Beck Anxiety Inventory, the Brief Symptom Inventory, and the ASI–Psychiatric Severity Composite score—was created by converting each scale to a standard score and then averaging the scores.

34 Examples to date: Alcohol treatment Zweben & Cisler. 2003
Examples to date: Alcohol treatment Zweben & Cisler Clinical and Methodological Utility of a Composite Outcome Measure for Alcohol Treatment Research. Alcoholism: Clinical and Experimental Research 27 (10), 1680–1685 Project MATCH data: composite outcome measure to capture multiple treatment outcomes among diverse client populations. self-reported alcohol consumption, alcohol problems, biological markers, other areas of functioning (psychiatric dysfunction, QOL). Findings on composite measure: 30% clients sustained a ‘remitted’ status (i.e., abstinent or moderate drinking without problems) over 1-year follow-up; 70% of the clients had reached a nonremitted status (i.e., heavy drinking and/or problems). “The composite outcome index could be used usefully along with singular measures of consumption to obtain a more complete picture of what has occurred among clients”.

35 Methods for combining items into a Composite Measure
Linear combinations Composite = [indicator1×weight1] + [indicator2×weight2] [indicatorN×weightN] Expert Panel determine which indicators & which weightings Regression-Based Composite Measures If a certain outcome is a ‘gold standard’, the weighting of individual items may be determined empirically by optimizing the predictability of the gold standard end point. Latent Trait Composite Measures Identify clusters of correlated items & latent trait modeling may be used to combine items within clusters but not across clusters. e.g. substance use outcomes; general health/well-being; high risk / harm behaviours Any-or-none Scoring of outcome measures In this method, a patient is counted as failing if he or she experiences at least 1 adverse outcome from a list of 2 or more adverse outcomes. Opportunity Scoring Opportunity scoring counts the number of times a given care process was actually performed (numerator), divided by the number of chances a provider had to give this care correctly (denominator). e.g. individual treatment care plan completed; MH-COPES completed All-or-None Scoring of Process Measures Only those patients who receive all indicated processes of care are counted as successes.

36 Recommendations re: Developing Composite Measures (Peterson et al 2010)
The intended audience & purpose of a composite measure should be explicitly stated. Decisions about which measures should be based on clinical importance of patient outcomes and the reliability of individual performance measures. Each individual component should be precisely defined to ensure consistent application in different settings. The description of the methods used for weighting and combining individual measures into a composite performance measure should be transparent. Developers should explore a variety of alternative methods for combining measures and document whether conclusions about provider performance differ with use of alternative methods. Empirical testing needed to assess the properties of a composite measure score. Reporting of composite performance measures should be accompanied by detailed reporting of individual domains and components. Reporting of composite performance measures should include a measure of the degree of uncertainty surrounding composite estimates for providers. Composite performance measures must be reevaluated as that evidence changes.

37 Could the ATOP be used to develop a composite measure of D&A treatment outcome / success?
Instrument needs to be used by the sector Instrument shown to be robust across diverse populations, drug types, treatment types Measures the main domains that we are most interested in over time (e.g. beginning & end of treatment) substance use, general health, high-risk behaviours Linked to other aspects of NMDS identifying primary/secondary drug use, demographics, treatment types delivered )

38 ATOP Treatment Episode Global Outcome
Aim To develop & validate the ‘ATOP Treatment Episode Global Outcome’, as a means of assigning a global outcome for each D&A treatment episode that broadly reflects whether each treatment episode was associated with a significant improvement, no change, or significant deterioration in the main clinical domains.

39 Proposed methodology ATOP Treatment Episode Global Outcome
National, multisite, project, 2-3 year duration Reference group of expert clinicians, consumer reps, researchers, data experts Recruit large number of clients (100’s) entering a variety of treatment types (counselling, OTP, withdrawal, rehab) and using different primary drug types (alcohol, opioids, cannabis, other) ATOP at beginning & end of treatment episode (or intervals for OTP, rehab), & “Gold standard”: client & clinician global outcome ratings for each treatment episode (confirmed by independent clinician & client panel, with broad criteria identified by Reference group for different treatment types). Identify algorithms for attributing ATOP Treatment Episode Global Outcome for different treatment modalities & different primary drug types. Statistical methods (e.g. RUC) to identify algorithms against ‘gold standard’ measure of client & clinician global ratings. How much change in (a) substance use, (b) general health & (c) high-risk practices is required for the ATOP scores to match client & clinician global ratings.

40 The holy grail? Data systems that enable us to triangulate data re:
client characteristics services provided (treatment episode data such as type of service, number of contacts, by whom) client outcome Would enable benchmarking of services NMDS possibly only mechanism to ensure this occurs.

41 For Further Information Contact
ATOP Project Coordinator Kristie Mammen Langton Centre Composite Measure Project Nicholas Lintzeris Validation of ATOP Anni Ryan Data Collection Systems Jennifer Holmes

Download ppt "Acknowledgements Patients and staff of Jacaranda House, the Langton Centre and Newcastle OPT Clinics especially: Yvonne Sutton, Rohan Holland, Betty."

Similar presentations

Ads by Google