A FOCUS ON QUALITY OUTCOMES DAO and WANADA Session 1 st May 2012 ANITA GHOSE B. App Sci; MBA; MAICD DIRECTOR Anita Ghose and Associates

AGENDA 1. An overview of outcomes 2. Developing outcomes 3. Measuring outcomes

1. An Overview of Outcomes

INPUTSOUTPUTSOUTCOMES Resources Used Activities Products Participation ShortMedium What we invest What we do Who we reach What are the results? What changed? Long- term Eg: Money Staff time Facilities Equipment Eg: Provide counselling Provide diversion services Provide residential rehab services What we produce Eg: Number of hours of service delivered Number of counselling sessions provided Number of people in residential rehab services Eg: People who secured stable housing Improved attitudes towards cannabis use Increased knowledge of the effects of alcohol and drug use Reduction in harmful use of alcohol and drugs REFRESHER – KEY CONCEPTS

INPUTSOUTPUTSOUTCOMES Resources Used Activities Products Participation ShortMedium What we invest What we do Who we reach What are the results? What changed? Long- term What we produce Based on Productivity Commission Report 2010; United Way of America (1996) PROCESSES – eg waiting lists, response times, client/staff ratios

INPUTSOUTPUTSOUTCOMES Resources Used Activities Products Participation ShortMedium What we invest What we do Who we reach What are the results? What changed? Long- term What we produce Based on Productivity Commission Report 2010; United Way of America (1996) Population

Inputs—the resources used by the organisation to achieve the objectives of an activity – “what we invest” eg money, staff time, facilities, equipment Processes – the related tasks and systems of the organisation to fulfil its mission (eg. response times, staff trained, worker safety, client/staff ratio, length of wait lists) Outputs— the direct products of the organisation’s activities or intervention (“what we produce”) and the participation (“who did we reach”). Eg number of service hours delivered, number of counselling sessions provided, number of clients in residential rehabilitation services Outcomes—the effects or change on an individual or group as a result of the activities and outputs (“what was the result?) Eg people who secured stable housing, reduction in harmful use of alcohol and drugs, increased knowledge of the effects of cannabis Impacts—the broader effects of an activity reflected in community wellbeing domains (what changed for the population?) Eg. reduced unemployment, increased mental health, stronger families

INPUTSOUTPUTSOUTCOMES Resources Used Activities Products Participation ShortMedium What we invest What we do Who we reach What are the results? What changed? Long- term What we produce The effects or change as a result of the activities or outputs

INPUTSOUTPUTSOUTCOMES Resources Used Activities Products Participation ShortMedium What we invest What we do Who we reach What are the results? What changed? Long- term Eg: Money Staff time Facilities Equipment Eg: Provide counselling Provide diversion services Provide residential rehab services What we produce Eg: Number of hours of service delivered Number of counselling sessions provided Number of people in residential rehab services Eg: People who secured stable housing Improved attitudes towards cannabis use Increased knowledge of the effects of alcohol and drug use Reduction in harmful use of alcohol and drugs Government Agencies are moving here in OBP

INPUTSOUTPUTSOUTCOMES Resources Used Activities Products Participation ShortMedium What we invest What we do Who we reach What are the results? What changed? Long- term Eg: Money Staff time Facilities Equipment Eg: Provide counselling Provide diversion services Provide residential rehab services What we produce Eg: Number of hours of service delivered Number of counselling sessions provided Number of people in residential rehab services Eg: People who secured stable housing Improved attitudes towards cannabis use Increased knowledge of the effects of alcohol and drug use Reduction in harmful use of alcohol and drugs This isn’t being prescribed by Govt – BUT organisations do need to understand their business here

OUTCOMES There will generally be effects, results, changes in one or more of these core areas: KNOWLEDGE OR SKILLS ATTITUDE OR OPINION VALUES BEHAVIOUR CIRCUMSTANCE CONDITION OR STATUS

OUTCOMES KNOWLEDGE OR SKILLS - increased knowledge of the effects of alcohol and drug use on the body ATTITUDE OR OPINION – improved attitudes or opinion about the dangers of cannabis use VALUES - enhanced values on the impact of drug use on others around the person BEHAVIOUR – reduction in the harmful use of drugs and alcohol CIRCUMSTANCE – increased ability to secure and maintain stable housing CONDITION OR STATUS – improved general health and wellbeing

OUTCOMES – KEY CONSIDERATIONS Proximal/Distal Attribution/Contribution Pathways approach Link to outputs Reasonable number Perverse Measurement versus achievement Positive or negative Intended or unintended

EXAMPLE Community Drug Service Team Counselling model Proximal outcome: Amphetamine users have reduced their drug and alcohol use Amphetamine users have an improved level of mental health and wellbeing Distal outcome: Amphetamine users have reduced levels of offending

EXAMPLE Mother and Child Program Specialist support Proximal outcome: Mothers reduce their alcohol and drug use Mothers demonstrate an increased awareness of the harmful effects of alcohol and drug use on their child Mothers remain the main caregiver of their child Distal outcome: Mothers participate more in their children’s schooling program

How efficient is the organisation? How effective is the organisation?

Efficient and effective INPUTSOUTPUTSOUTCOMES Resources Used Activities Products Participation ShortMedium Long- term Efficiency = Inputs Outputs Example: the average cost per session of counselling delivered Effectiveness = Inputs Outcome Example: the average cost per completed as planned episode

Outcomes Frameworks

A Logic Model Source: University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation Is a framework for describing the relationships between investments, activities, and results. Depicts a program showing what the program will do and what it is to accomplish (reasoning) Provides a series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes Provides a common approach for integrating planning, implementation, evaluation and reporting. Applied at a program, process, multi component program, organisational level

Fully detailed logic model Source: University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation

Logic model and reporting Source: University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation

RESULTS BASED ACCOUNTABILITY

Strong and stable families, positive family relationships Rate of family breakdown due to alcohol and drug use Rate of domestic violence 1. How much did we do? 2. How well did we do it? 3. Is anyone better off? RESULT or OUTCOME INDICATOR or BENCHMARK PERFORMANCE MEASURE A condition of well-being for children, adults, families or communities. A measure which helps quantify the achievement of a result. A measure of how well a program, agency or service system is working. Three types: = Customer Results Population Performance Population Performance = Customer Results Source: Mark Friedman 2005

How much did we do? How well did we do it? Is anyone better off? # Clients/customers served # Activities (by type of activity) % Common measures e.g. client/staff ratio, unit cost, clients from a target group, staff turnover, staff morale, staff fully trained, worker safety, client satisfaction (were they treated well), wait time, wait lists/turnaways, meeting specific industry standards % Skills / Knowledge (e.g. parenting skills) # % Attitude / Opinion (e.g. toward drugs) # % Behaviour (eg drug and alcohol use) # % Circumstance (e.g. working, in stable housing) # % Activity-specific measures e.g. timely, clients completing activity, correct and complete processes, meeting standards Source: Mark Friedman 2005; Anglicare 2011 EFFECT EFFORT QUANTITYQUALITY

How Population & Performance Accountability FIT TOGETHER

Contribution relationship Alignment of measures Appropriate responsibility THE LINKAGE Between POPULATION and PERFORMANCE POPULATION ACCOUNTABILITY General Physical Health Prevalence of liver disease Stable Families and Relations Rate of alcohol and drug use Mental Health Rate of co-occurrence with mental health issues # of bed days % receiving assessment within agreed timeframe # increased awareness of the impact of alcohol and drug use on physical health PERFORMANCE ACCOUNTABILITY POPULATION RESULTS Residential Rehab Services Source: Mark Friedman 2005 % increased awareness of the impact of alcohol and drug use on physical health

2. Developing Outcomes

The Request Key Assumptions Partnership and collaboration between agency, organisation and individuals, families to identify community outcomes Appropriate consultation framework and engagement strategies with key stakeholders Appropriate mechanisms in place (formal and informal) to capture the information required to get the outcomes and time frames right This is the stage of developing and identifying outcomes which is outside of scope for this workshop but is the important precursor to defining and writing outcomes

Individual outcomes for people with alcohol and drug issues Family outcomes Program or Service level outcomes Organisational outcomes Corporate – Financial; Human resource; Risk Community outcomes Population outcomes procurement Outcomes - different levels and purposes

Types of Levels Individual OutcomesService Level OutcomesOrganisational OutcomesCommunity OutcomesPopulation Outcomes

Outcomes – Responding to a Request Key considerations: Statement of the desired change or result (what does it look like?) Interpretation – get clarity and agreed understanding across stakeholders Be clear about target group (eg child, mother, parent) Outcomes – at what level (eg individual, service level, population) What are you trying to assess? Measurable and meaningful; reasonable number

Outcomes – Responding to a Request Key considerations: Attribution – circle of control and influence; proximal and distal System – how does this fit within the whole? Communities of practice Think about time frame – short term to long term; longitudinal change Consider context eg chronic relapsing condition Outcomes in the AOD sector are generally about results/effects or change in knowledge or skills; attitude or opinion; values; behaviour; circumstance; condition or status

A useful approach to writing an outcome statement SMART objectives – specific, measurable, achievable, results oriented, time frame Who/whatChange/desired effect In whatBy when Parents of people with alcohol and drug issues participating in the parenting program increasetheir knowledge of alcohol and drug use at the completion of the program Young people in a specialist alcohol and drug treatment centre reducetheir harmful alcohol and drug use after the first year of the program

3. Measuring Outcomes

Indicator - is a measure for which data is available which provides a specific and reliable means to quantify the achievement of the outcome. These results indicators are what we will see if change occurs. Ask the questions: What will we see if change occurs? What will be the evidence? Each outcome can have a set of indicators, which, if taken together gives a reasonable approximation of the achievement of the outcome or result

Indicator – can be expressed as a # or % (quantifies the achievement of the outcome) Eg: The percentage of participants reporting increased awareness about alcohol and drugs The number of people who have secured stable housing since the programme’s commencement The percentage of mothers who have retained primary care of their children since the program’s completion

Sometimes, one indicator may be sufficient to evidence the outcome’s achievement. Often, several indicators are required. Select carefully to be meaningful and not over burdensome. Outcome: SOBER UP CENTRE – Clients are kept safe overnight Reduction in complaints about public drunkeness Reduction in incidence of domestic violence Reduction in street arrests of clients Reduction in clients attending A&E depts in hospital

About measurement tools There is no magical measurement tool! There are a range of tools (eg. clinical profiling tools, instruments, surveys and questionnaires etc) that can be used but they all have limitations Develop an “evidencing the outcome” approach that is context specific, in collaboration with the sector organisations, individuals and families which can give a reasonable approximation of the achievement of the outcome There is always a story, assumptions and a context which needs to be explicitly stated

Reliability refers to the extent to which a measure produces accurate information. Validity refers to the extent to which the measure actually measures what it is intended to measure. Suitability refers to the criteria for selecting measures that meet the target group or intent – and are appropriate for those administering it Relevancy refers to measures clearly related to the outcome being evaluated, providing useful information and allowing meaningful conclusions to be drawn THIS IS A JOURNEY – CONSIDER THESE AND WORK TOWARDS

We need a baseline (pre-post)

Outcomes, Indicators, Data OutcomesIndicatorsData collection SourcesMethodsSampleTiming Linkages and relationship, alignment, what is important, reducing administrative burden Evidence based practice

SUMMARY OF THE DAY 1. An overview of outcomes 2. Developing outcomes 3. Measuring outcomes

QUESTIONS

Consultants to Government and Community