What evaluation attributes, stakeholder characteristics & contextual factors are important for evaluation influence? Sarah Appleton-Dyer Dr Janet Clinton, Dr Peter Carswell & Dr Rob Mc Neill AES Conference; Sydney, August 2011

Overview Identify the aims of the presentation. Describe the research used to inform the presentation. Present a theory of evaluation influence within a partnership context. Use survey and interview data to highlight the evaluation attributes, stakeholder characteristics and contextual factors that are important for evaluation influence. Get some feedback.

Background

Evaluation influence The multiple pathways and mechanisms through which evaluation can hope to influence attitudes and action, due to exposure to evaluation findings or to participation in evaluation (Mark & Henry, 2004). Mark, M. M., & Henry, G. T. (2004). The Mechanisms and Outcomes of Evaluation Influence. Evaluation, 10: 35-48

Aims To identify the types of influences experienced by population health partnership members. To identify the factors that facilitate evaluation influence.

Date The conceptual model Phase 2: Survey with partnership members Phase 3: Site visits with partnerships Phase 4: Data integration and analysis Similarities and differences are explored Phase 1: Literature review Overview of evaluation influenceIn-depth understanding of evaluation influence New insights and understandings

Conceptual model

Study 1 Online survey – 187 PHP members. April to May 2010 (6 weeks). Female (71%). Aged 35 to 49 (33%). NZ Euro (66%), Maori (18%) or Pacific Islanders (3%). 2 to 5 organisations (50%). Involved high levels of collaboration (55%). Provided programs or services (65%) and shared information between partners (57%). Range of roles and initiatives.

Evaluation 63% were involved in evaluation or had been over last 6 months (n=187). 30% had received feedback. 62% were underpinned by an evaluation theory.

Study 2 Site visits with 4 partnerships. –Document analysis & interviews. Identified in Study 1. Nov 2010 to March to 8 organisations. 5 to 23 members. Range of roles. Range of PH initiatives.

Evaluation Current/6 months evaluation. Range of reasons: –‘to see if it works, and how it was working’ –‘to improve our program and support learning’ –‘It was mainly driven by the [funders]’ External evaluators. Theory driven and participatory approaches based on Centres for Disease Control model.

Influences on individuals & partnerships ‘It made me understand more about the information we collect. Um, and how we could collect information to show that we are making progress’. Partnership B ‘As a result of the evaluation findings, they agreed to reduce the price for retailers to re- new their licence, so they could re-grade’. Partnership C

Data Integration Survey –Kruskal-Wallis and Mann Whitney U. –PCA and Linear Regression. Interviews Thematic analysis Integration –* statistically significant –Themes identified

Evaluation Attributes Approach –Theory* –Learning* –Sophistication & quality ‘I don’t think there was a very robust methodological approach. You know the way the scoping was carried out, there wasn’t much planning before it actually happened, so I’m not sure that was very helpful for us.’

Eval Attributes cont.. Feedback –Credibility –Timeliness Evaluation capacity building** Evaluator –External* –Skills & expertise –‘I rely on the credibility and independence of [organisation] when I share the findings, because it is what people base their opinion on’. Partnership C

Stakeholder Characteristics Existing knowledge & attitudes Evaluation readiness* –‘Everyone was on the same page, and we were all involved in the evaluation. It was something that we recognised the value of, and something that we still go back to’. Partnership D Participation*

Partnership Evaluation Behaviour Partnership evaluation readiness –‘Some members are more involved than others. I guess for some it was an extra expense and they would rather just get on with it’. Partnership D Partnership support* Leadership participation*

Partnership Functioning Clarity of purpose and functioning –‘I think if you ask everyone what we do you’ll get a different answer, so we all say yeah that’s good and agree, it just doesn’t seem to happen. We all have a different focus’. Partnership B Adaptation or change processes* –‘I lead that, I love that, I get so excited and I am reading this stuff and writing notes you know. I use the evaluators to help me, so I don’t misinterpret and then we take it to the Steering Group’. Partnership A Shared commitment to partnership Funding & competitive environments

Contextual Characteristics Policy context Funding environment Resources, time & support for evaluation –‘They’ve been very supportive. We’ve been given the time to get involved in this, and it (evaluation) is listened to. So that’s been hugely beneficial for us’. Partnership A Existing attitudes of key decision-makers –‘I rather worry about collecting quantitative data because funders, well, stories on their own for funders are not influential’. Partnership C

Summary Provides support for the model. Highlights the complexity of evaluation influence. Is evaluation an intervention? Evaluation influence should be judged by its inputs and within its context.

Facilitating Influence Evaluation influence is facilitated by: Evaluation theory & focus on learning. Skilled & credible evaluator. Stakeholder evaluation readiness. Stakeholders’ existing functioning and systems. Stakeholders’ capacity to respond to evaluation. Wider support for evaluation.

Acknowledgements Supervision team: Dr Janet Clinton, Dr Peter Carswell and Dr Rob McNeill Population health partnership members and health sector staff from across New Zealand. The University of Auckland Full Doctoral Scholarship.

Sarah Appleton-Dyer

Principal Components Analysis Reduced data set into 6 factors explaining 52.36% of variance in the data: –Partnership functioning. –Evaluation attributes. –Evaluation capacity building. –Partnership evaluation behaviour. –Evaluation readiness. –Evaluation influence.

Linear Regression Model accounts for 42.3% of variance in evaluation influence. Factors predict evaluation influence significantly well. Following factors make a significant contribution to evaluation influence: –Evaluation capacity building, (Beta =.576, p<0.05). –Evaluation readiness, (Beta =.248, p<0.05).

Potential effects and influences Changes organizations (Williams; Davidson). Develops learning environment (Preskill). Enhances program success – when embedded (Clinton). Influences attitude, motivation & behaviour – individual, interpersonal & collective level (Mark & Henry). Empowers (Fetterman; Patton).