1. EEN [Canada] Forum Shelley Borys Director, Evaluation September 30, 2010 Developing Evaluation Capacity

2. Page 2 Context The importance of building evaluation capacity is linked to the following factors: –Increased requirements for evaluation ▪The requirements under the new Policy on Evaluation to move towards coverage of 100% of both transfer program and direct program spending on a five-year cycle. –The increased interest in the use of evaluation evidence to provide useful feedback regarding program improvement, to demonstrate progress toward outcomes, and to support decision-making.

3. Page 3 Overview This presentation will discuss ways in which Environment Canada is working to increase its own evaluation capacity. This topic will be explored from three perspectives: –How we build evaluation capacity within the Evaluation Division; –Activities to increase the capacity for programs to be evaluated; and –How we build capacity to use evaluation.

4. Page 4 Evaluation Competencies Two frameworks exist which elaborate a series of evaluator competencies which can serve as a guide to capacity development: –Canadian Evaluation Society Credentialing Initiative ▪Five competency domains: reflective (7), technical (16), situational (9), management (7), interpersonal (10) –Treasury Board Secretariat Centre for Excellence in Evaluation ▪Four competency clusters: intellectual (3), management (2), relationship (3), personal (2)

5. Page 5 Building Evaluator Capacity: Environment Canada Evaluation Model Intent is to conduct as much evaluation work as is possible in-house –Use contractors for evaluations that are technically beyond our capacity, interdepartmental evaluations, and where we have contract dollars but not staff time This helps build evaluation capacity by: –Having evaluators gain hands-on evaluation experience –Helping evaluators become subject-matter experts –Facilitating evaluators to develop a network of contacts within the Department

6. Page 6 Building Evaluator Capacity: Project Teams Evaluation Division teams are formed for each project, with: –A project lead, who coordinates communications, directs the work of team members, and is the lead pen on deliverables; –Team members, who support the efforts of the project lead in performing evaluation activities and preparing reports. –A peer reviewer, who is another evaluation manager whose role is to review in-progress deliverables, before they are submitted for management review; and Teams are not necessarily hierarchical. This helps build evaluation capacity by ▪Providing exposure to different styles of evaluation and project management ▪Providing an opportunity to work on different tasks and a range of projects ▪Enabling evaluators to take on additional responsibilities while managing risk.

7. Page 7 Building Evaluator Capacity: Templates Evaluation projects use a common file folder structure and basic document formats. These templates build evaluation capacity by: –Ensuring that all required issues are addressed; –Laying out a somewhat standard process for evaluation; –Reducing the risk of uneven evaluation conduct.

8. Page 8 Building Evaluator Capacity: Generic Evaluation Framework One templated item of particular importance is the generic evaluation framework. This is a matrix table contained within the evaluation plan template that sets out: –Key questions to be asked; –Indicators for each question; –Sources and methods that would support the analysis; and –The specific Treasury Board Secretariat Policy on Evaluation issue each question relates to. This evaluation framework helps build evaluator capacity by: –Ensuring that all evaluations begin from the same foundation –Serving as guidance around the identification of indicators and data sources.

9. Page 9 Building Evaluator Capacity: Training Staff within the Evaluation Division are encouraged to seek out opportunities for training and professional development –Conferences, networking events, lunch and learn sessions –Resource library This helps to build evaluation capacity by: –Helping evaluation staff remain abreast of developments within the theory and practice of evaluation

10. Page 10 Building Capacity to be Evaluated: Shadow Plan Two years in advance of a planned evaluation, we conduct a mini evaluability assessment: –Performance measurement? –Logic Model? –Key issues of interest? –Timing consideration? Intent is that improvements can be introduced before the evaluation starts, thereby strengthening the evaluation –Performance measurement –Relevance to managers

11. Page 11 Building Capacity to be Evaluated: Performance Measurement Improvements The Evaluation Division regularly assesses the quality of performance measurement in its evaluations, as shown in the following table: In recommending improvements to performance measurement, subsequent evaluations will have better data to draw upon.

12. Page 12 The Department has established several mechanisms for the Division to provide feedback on potential programs at an early stage. The Division is consulted on: –Memoranda to Cabinet submissions; –Treasury Board Secretariat submissions; and –The design of performance measurement strategies. This helps build the capacity to be evaluated and to use evaluation by: –Establishing a dialogue between program staff and evaluators in advance of evaluation; –Ensuring evaluation results are considered in the design of new initiatives; and –Ensuring that program design supports evaluation e.g. defined outcomes, performance measurement. Building Capacity to be Evaluated and to Use Evaluation: Integration with Program Design

13. Page 13 Building Capacity to Use Evaluation: Management Response Guidelines The Evaluation Division has prepared guidelines for management to draw upon when preparing their response to evaluation recommendations. These guidelines describe the essential components of a management response: –In particular, the development of specific actions with deadlines and clear responsibilities. This helps build the capacity to use evaluation by: –Ensuring the clarity of management responses; and –Facilitating the measurement of progress against action plans.

14. Page 14 Building Capacity to Use Evaluation: Follow-up Process After evaluation reports and the management responses to their recommendations have been approved, the Department works to ensure that these recommendations are followed through on. This is done formally at the Departmental Evaluation Committee meetings, where the status of actions stemming from past evaluations is a standard agenda item. This helps build the capacity to use evaluation by: –Formalizing the implementation process; –Ensuring that senior management remain aware of outstanding action items; and –Providing a forum for any barriers to implementation to be discussed and responded to.

15. Page 15 Building Capacity to Use Evaluation: Management Support A successful evaluation unit requires a suitably enabling environment. At Environment Canada, senior management understanding and support of evaluation is evident. Departmental Evaluation Committee is active and engaged This helps build evaluation capacity by: –Signalling to program personnel that evaluation is seen as a furtherance of departmental work – not an interruption; and –Strengthening the connection between evaluation evidence and senior decision-making.

16. Page 16 Summary of Practices Evaluator Capacity In-house Conduct Team Structure Templates Generic Evaluation Framework Training Capacity to be Evaluated / Use Evaluation Performance Measurement Improvements Shadow Plan Integration with Program Design Management Response Guidelines Follow-up Process Management Support