Alcohol Improvement Programme Evaluation Michelle Cornes & Michael Clark

Aims Outline plans for the evaluation of the AIP – What support will be available to Early Implementers? Explain the underlying rationale - What is a ‘programme evaluation’ and what do we want to achieve? In terms of the ‘how to’ – introduce an outcomes based approach.

A Co-research Model Each site will be responsible for designing their own evaluation strategy and incorporating this as an ‘integral’ part of their Local Improvement Plan. An evaluation co-ordinator will offer support and guidance (site visits, telephone, ) A National Collaborative will act as a forum for the 20 Early Implementer Sites (‘peer support’ and ‘peer review’) At the end of the Programme, an overview report will pull together the learning from all 20 Early Implementer Sites.

Evaluation Timetable (to be confirmed) Preliminary work on evaluation strategies to be completed by March 2009 National collaborative to meet every quarter Each site to bring to the collaborative a short progress report of their emerging findings Final evaluation report to be produced by each site by month 36 Overview report completed 2 months later

Evaluation Requirement ‘[The AIP] will establish a group of early implementers who will go further a little bit faster in defining and implementing solutions for tackling alcohol-related harm. These PCTs will have priority access to [resources] and their experiences will contribute to the bank of good practice which will be disseminated via [a] learning centre to the rest of the NHS’ ( [Accessed ]

Evaluation Focus High Impact Changes Based on the best available evidence the Department of Health has identified key actions that Primary Care Trusts can take which will make the highest impact on reducing alcohol related harm and admissions…

“Spreading information about best practice is one thing; the real challenge is to secure successful exploitation and adaptation of that information” Stephenson (2002)

Key questions for the evaluation: How effective is your ‘Local Improvement Plan’ as a strategic change mechanism in reducing alcohol related harm/admissions to hospital ?

Is it helpful/useful to work together collaboratively in this way? Has the support offered to you as part of the AIP been helpful?

Expectations Undertake a series of planned actives Collect evidence in a systematic way Use more than one method (quantitative/qualitative) Include multiple perspectives Make a judgement about the effectiveness of your local improvement programme Make a judgement about the effectiveness of working together in this way

Being realistic A DIY Evaluation will inevitably be a small- scale piece of work, employing simple techniques but this is not to down play its value… It can provide a wealth of interesting and useful information, and be a positive learning experience for all concerned.

Why design programmes on the basis of outcomes? It is what we do (implicitly) already - focus on needs To more clearly demonstrate impact To more clearly show added value An improvement on inputs-outputs approaches Means of clearly agreeing what we are to do Making clearer what we can control or only influence A means to more clearly make and show links - e.g. across programmes, between national & regional levels It shows what our ‘products’ & ‘services’ are for It leads to a clearer understanding of a programme logic Leads to a better position for monitoring/evaluation

What do we mean by outcomes? the changes, benefits, learning or other effects that result from the work that we do.

Inputs Processes Outputs Immediate Outcomes Intermediate Outcomes Ultimate Outcomes Impacts Control / Influence ? Plausibility

Measuring outcomes Hard and soft outcomes Direct indicators vs indirect indicators Single indicator vs a basket of indicators Plausibility Can we challenge ourselves to measure more?

Outcomes and the programme logic Inputs Processes Immediate Outcomes Intermediate Outcomes Ultimate Outcomes Evidence Programme Logic/ Theory/Rationale Change management Education theory Psychological theory Social Marketing This is sensible/plausible

Potential Danger Limiting ourselves to the more easily controlled outcomes Do not limit your ambitions, just be honest about what is in your immediate control, what you can influence and what you can lay down some ground work for but which is ultimately within the control of others.

Examples of outcomes from DRE Delivering Race Equality in Mental Health Immediate - Links to and activity in all relevant statutory agencies; work with local 3rd sector and communities;better local information; more awareness of personalisation and recovery; safer in- patient care; Intermediate - improved access to psychological therapies; stronger community capacity to lead; more confidence in communities; increased satisfaction Ultimate - equal access to services; equal outcomes; better recovery and social inclusion.

Examples of outcomes from DRE Ultimate Outcomes - priorities and rationale AreaDefinition of indicator(s) What the indicator means What the expected direction of travel would be Data source(s) and levels of coverage (national/local) Early Interventi on in Psychosis services 1. Numbers of service users by ethnicity 2. DUP 1. We would expect rates of using services to reflect the ethnicity of local populations and their incidence of psychosis. 2. We would expect DUP to be no longer for ethnic minority groups As services provide equal access we would expect this to influence the proportions of BME users to fully reflect their psychosis incidence. Overall an upward trend in the proportion of BME users is likely. 1. LDPR (Q ) 2. Possibly datasets from services and related research (LEO, Worcester, EDEN, MiData, FERN) 3. MHMDS