Evaluating Engagement Judging the outcome above the noise of squeaky wheels Heather Shaw, Department of Sustainability & Environment Jessica Dart, Clear Horizon

Engagement Services in DSE Internal consultants within DSE Engagement planning & implementation advice and support Engaging people in decision making

Research Question How could the E&C team add value concerning engagement evaluation?

What is special about engagement evaluation? A lack of published materials and guidelines on criteria for engagement evaluation The nature of engagement – it is context specific and qualitative And some hurdles to overcome Engagement covers a wide range of activities - needs a wide range of evaluation approaches Attribution difficulties Avoiding overemphasis on “squeaky wheels” Avoiding engagement fatigue

IAP2 public participation spectrum INCREASING LEVEL OF PUBLIC IMPACT INFORM CONSULTINVOLVECOLLABORATEEMPOWER Public Participation Goal: Public Participation Goal: Public Participation Goal: Public Participation Goal: Public Participation Goal: To provide the public with balanced and objective information to assist them in understanding the problems, alternatives and/or solutions. To obtain public feedback on analysis, alternatives and/or decisions. To work directly with the public throughout the process to ensure that public issues and concerns are consistently understood and considered. To place final decision- making in the hands of the public. To partner with the public in each aspect of the decision, including the development of alternatives and the identification of the preferred solution. Promise to the public: Promise to the public: Promise to the public: We will keep you informed. We will work with you to ensure that your concerns and issues are directly reflected in the alternatives developed and provide feedback on how public input influenced the decision. We will keep you informed, listen to and acknowledge concerns and provide feedback on how public input influenced the decision. We will look to you for direct advice and innovation in formulating solutions and incorporate your advice and recommendations into the decisions to the maximum extent possible. We will implement what you decide.

What is special about engagement evaluation? A lack of published materials and guidelines on criteria for engagement evaluation The nature of engagement – it is context specific and qualitative And some hurdles to overcome Engagement covers a wide range of activities - needs a wide range of evaluation approaches Attribution difficulties Avoiding overemphasis on “squeaky wheels” Avoiding engagement fatigue

Proposed guidelines for engagement evaluation include: Start with an evaluation plan Consider the criteria for successful engagement processes Acknowledge the context and the purpose of the engagement task Make sure that the level of evaluation effort is commensurate with the engagement expenditure Employ mixed methods Consider relationship capital – beyond project timeframes

Engagement criteria

Engagement specific evaluation criteria Engagement was adequately scoped and planned People who are affected in a decision were engaged sufficiently early in the process People engaged were adequately representative of those affected by the decision (both those affected directly, indirectly, short term and long term) Materials and methods used sufficient and appropriate for people to meaningfully engage Those engaged received feedback about how the engagement affected the ultimate decision The engagement was conducted in an independent and unbiased manner The promises that were made by the engagement were upheld Where appropriate: influence the engagement had on decision makers The cost effectiveness of the engagement process.

Proposed guidelines for engagement evaluation include: Start with an evaluation plan Consider the criteria for successful engagement processes Acknowledge the context and the purpose of the engagement task Make sure that the level of evaluation effort is commensurate with the engagement expenditure Employ mixed methods Consider relationship capital – beyond project timeframes

Bronze level evaluation When? Low risk project, reasonable financial investment in engagement Why? Mainly for learning Which Criteria?  Extent to which the people engaged were adequately representative of those affected by the decision  Extent to which the promises made (IAP2 spectrum) were upheld

Silver level evaluation When? Medium risk project, significant financial investment in engagement Why? Summative evaluation and possibly formative Which Criteria?  As per bronze level evaluation plus others such as:  The influence the engagement had on decision makers  Degree to which people who are affected in a decision were engaged sufficiently early in the process  Extent to which materials provided were sufficient and appropriate for people to meaningfully engage  Extent to which those engaged received feedback about how the engagement affected the ultimate decision

Gold level evaluation When? High risk, high profile project, significant financial investment in engagement Why? Summative evaluation for accountability. Highly robust defensible design needed. Which Criteria?  Most, if not all of them.

Relevant criteria Suggested evaluation questions Gold How efficient was the implementation of the engagement process? (scope, planning, and management and cost)? How appropriate was the implementation of the engagement process in terms of enabling people to meaningfully engage (timing, resources provided, unbiased conduct, feedback received)? To what extent were the people engaged/(or communicated with) adequately representative of those affected by the decision? To what extent were the promises that were made by the engagement upheld? What was the expected & unexpected impact of the engagement process? How efficient was the implementation of the engagement process? (scope, planning,) Silver How appropriate was the implementation of the engagement process in terms of enabling people to meaningfully engage? To what extent were the people engaged/(or communicated with) adequately representative of those affected by the decision? To what extent were the promises that were made by the engagement upheld? What was the expected and unexpected impact of the engagement process? Bronze What were the strengths and weaknesses of the engagement process? To what extent were the people engaged/(or communicated with) adequately representative of those affected by the decision? To what extent were the promises that were made by the engagement upheld?

Bronze level case study Developing a new coastal risk policy Engaging internal DSE stakeholders Engagement promise – Collaborate External organisation delivered a collaborative framework Evaluation of the engagement process Purpose - to assess whether this collaboration framework might be useful for developing future DSE policies.

Bronze level case study Evaluation questions What were the strengths and weaknesses of the E&C implementation process? To what extent were the people engaged with adequately representative of those affected by the decision? To what extent were the engagement promises upheld? How appropriate was the E&C approach for this policy development? How valuable was the consensus on key issues and next steps achieved from the process?

Bronze level case study Data collection methods An online survey of 60 stakeholders. Participant observation from the DSE Project Team and Engagement and Partnerships Team member. Interviews of Project Team members. PCB perception of value of the policy produced.

Bronze level case study Conclusions Engagement promise upheld/buy-in achieved. Representative & inclusive participation. Sufficient resources provided for people to collaborate. Flaws in the delivery of the process. High resource requirement from DSE. Framework suitable for future policy development with changes in the way it is delivered. Identified where the engagement framework is most appropriate.

Bronze level case study – reflections “I thought the evaluation process was great. I couldn’t believe how easy it was” “I would be comfortable to use the process again though I may need some assistance with some steps.”

Integrating engagement evaluation into program/policy evaluation 1:Scope the rationale and boundaries of the program or policy 2:Design the program/policy and map the logic 3:Design the engagement and communications plan 4:Scope and develop a monitoring and evaluation plan to covers steps 2&3 5:Develop a reporting and improvement strategy to cater for step 3 and step 4

StepsHow steps fit integrated evaluation plan Step 1. Scope the evaluation the scope, audience and purpose of the evaluation identify depth of evaluation required Here engagement evaluation can be scoped with the program evaluation question Attention needs to be paid to depth of evaluation required for engagement process Step 2. Clarify the engagement task Consider the outcomes of program/policy and logic Indentify targeted stakeholders in engagement Consider promises that were made by the engagement  The logic will be already scoped during the program evaluation questions. The targeted stakeholders and clarifying the promises may be an extra step. Step 3. Develop questions Develop evaluation questions based on purpose and depth of evaluation required Here the evaluation questions relating to the engagement t need to be added Step 4. Determine how questions will be addressed  This will be covered by the program evaluation plan Step 5. Effective reporting and improvement strategy  This will be covered by program evaluation plan