1. Research methods in clinical psychology: An introduction for students and practitioners Chris Barker, Nancy Pistrang, and Robert Elliott CHAPTER 11 Evaluation research

2. Outline Background concepts Preparing to evaluate a service Monitoring service delivery Outcome evaluation

3. What is evaluation? Evaluation is applied research that aims to assess the worth of a service, often judging it against specified goals Two central questions: –“What is the service trying to do?” –“How will you know if it has done it?”

4. Characteristics of evaluation For local consumption only Aid to decision making –“utilisation-focused” Takes place in an “action setting” Participants are service users Tight timetable

5. Some terminology Audit: “the audit cycle” Quality assurance Clinical governance Formative versus summative evaluation Stakeholders Structure, process, outcome (Donabedian)

6. Observe practice Compare Make changes The audit cycle Set standards

7. Socio-political issues Evaluation is threatening –Being scrutinised –Time consuming –May be used against you –May not validly assess your work External versus in-house evaluations Different stakeholders may have different agendas

8. Preparing to evaluate a service Aims and objectives Needs assessment Service design (Rossi, Freeman & Lipsey, 1999)

9. Aims and objectives Aims: global statements of desired outcomes Objectives: specific goals –ideally occur within a specific time period –detail what the service will do to achieve its aims –indicate whether or not the aims will have been met –are clear, simple and, if possible, measurable

10. Needs assessment specify the target problem and the target population estimate the extent of the problem assess the need for the service –need (professionally determined) –demand (client determined) –supply (availability)

11. Service design Impact model –links problem cause, service action, expected outcomes “Delivery system design” –operational plan

12. Process evaluation: monitoring service delivery “Who does what to whom?” Coverage –“Who is getting the service?” –(consider issues of bias) Implementation –“What service is being given?”

13. Outcome evaluation: “What was the impact of the service?” Efficacy: –evaluation under controlled experimental conditions Effectiveness: –evaluation of the service as delivered in practice (see Seligman, 1995) –Client satisfaction surveys Cost-benefit analysis