Evaluating Complex Health Programmes Fraser Battye

“Programmes chart out a perceived course whereby wrongs might be put to rights, deficiencies of behaviour corrected, inequalities of condition alleviated. Programmes are thus shaped by a vision of change and they succeed or fail according to the veracity of that vision. Evaluation…has the task of testing out the underlying programme theories… is that basic plan sound, plausible, durable, practical and, above all, valid?” A word from our gurus(!)

Points & structure  Main Points: 1.Of course it’s complex! 2.It’s our job to respond 3.Mix of approaches needed: but always underpinned by programme theory  Structure: o What is GHK? o Why are these evaluations difficult? o How have we addressed this? o Conclusions

What is GHK?  Multi-disciplinary  Independent and employee-owned  Various policy areas  Specialism in evaluation  (Working with ETHNOS)

Why might programmes like COFSS be hard to evaluate?  Complexity: neighbourhoods not labs  Interactions with context: o Other interventions o The real world (e.g. residential ‘churn’ / global catastrophe & war!)  Timescales, effects and attribution: o Intervention output outcome impact  Determinants of health o Lack of / debated evidence o Standards of evidence

How have we addressed this?  Approach based on programme theory 1.Define theory behind the programme: o What is COFSS? o What does it do? o Desired effects? o How does it expect to work? 2.Design research to test it: o Quantitatively (neighbourhood; individual) o Qualitatively (lit reviews; interviews; case studies; tracking beneficiaries) o Mixed methods

Defining the theory 1.Aims to reduce health inequalities 2.Significant resources at its disposal 3.Multi-agency and multi-disciplinary 4.Significant investment in management and ‘True’ partnership working 5.Community based and uses an assertive outreach approach 6.Aims to change mainstream service provision 7.Resident input is central

Testing the theory (example) Feature:Hypotheses:Tested by… 4) Significant investment in management & ‘True’ partnership working Outcomes achieved by beneficiaries are better because services are coordinated Service providers spend less time managing cases, doing admin. etc & more time delivering services Interviews with providers & management Interviews with beneficiaries Lit review: costs & benefits of partnership VFM assessment

Concluding points  It’s always going to be complex (GOOD!)  Evaluation must respond  Theory-driven approaches promising o Best mixed with ‘traditional’ methods  Challenge to accepted public health understandings of ‘evidence’