Presentation on theme: "Evaluation at NRCan: Information for Program Managers Strategic Evaluation Division Science & Policy Integration July 2012."— Presentation transcript:
1 Evaluation at NRCan: Information for Program Managers Strategic Evaluation Division Science & Policy Integration July 2012
2 PurposeThe purpose of this document is to provide program managers with an overview of the evaluation function at NRCan.The TBS Policy on Evaluation (April 2009) requires that all direct spending, including all G&C Programs, be evaluated every five yearsmost program managers will find themselves participating in an evaluation at some point.
3 What is Evaluation? Evaluations Audits Evaluations are the systematic collection and analysis of evidence on the outcomes of programs to make judgments abouttheir relevance;performance; andalternative ways to deliver them or to achieve the same results.”Evaluations must be neutral, and evidence-based.An evaluation is not the same as an audit.EvaluationsAuditsFocus on whether we are doing the right things, and the extent to which a program is achieving its expected outcomes, in a cost-effective manner.Internal audit is looking at financial management, processes, controls and risk.Make assessments on the relevance & performance of programsIdentify strengths and weaknesses in the management control framework
4 Why do Evaluations?The objective of evaluation is to create a comprehensive and reliable base of evidence to support:policy and program improvement;expenditure management;Cabinet decision-making; andpublic accountability.Evaluations are often required to support TB submissions and Memoranda to Cabinet.They are also a critical source of evidence for Strategic Reviews, to support resource reallocation (next review will occur in 2014).
5 Background: Evaluation Stakeholders Evaluation ReportsPolicy & Program ImprovementExpenditure Management/Cabinet Decision-makingAccountability/Public ReportingCabinetParliamentNRCan DMStrategic ReviewCanadian PublicSector ADMsProgram ManagersTBSThe challenge is to meet the diverse information needs of many stakeholders for decision-making AND public accountability in a timely fashion.
6 Some phases may overlap. The Evaluation CycleEvaluationPlanningApprovals/Posting(2-3 months)Assessment(1-3 months)Report &Recommendations(1-4 months)ImplementingChangeContracting(1-2 months)Field Work/Analysis(6-8 months)Management Responses(1 month)Including planning and approvals/posting, large evaluations typically take months to complete.Some phases may overlap.
7 Evaluation PlanningNRCan must evaluate all direct program spending, including all ongoing grant and contribution programs, every five years.NRCan has developed a five-year Evaluation Plan based on PAA units that is updated annually and approved by the Evaluation Committee.In most cases, the evaluation of an individual program will be conducted within the scope of a broader evaluation of a PAA unit.The current plan summary appears on the Strategic Evaluation Internet site at:
8 Phases of an Evaluation Evaluation Assessment (1-3 months)research and planning to understand the programsdevelop the Terms of Referenceobtain approval from the Evaluation CommitteeContracting (1-2 months)Consultants are often used to supplement in-house staff. Their roles will vary by project.Fieldwork or Data Collection/Analysis (6-8 months)develop a detailed methodology reportmethodologies: key informant interviews; focus groups; file/document/literature reviews; surveys; case studies; and data and economic analysisanalyse information collected from these multiple lines of evidence to develop conclusions
9 Phases of An Evaluation, cont’d Reporting & Development of Recommendations (2-4 months)prepare preliminary findings and discuss with programsdraft reportaddress comments and revisionsdevelop recommendationsManagement Responses (1 month)obtain ADM-approved management responses and actionplans to the recommendationsApprovals/Posting of report (2-3 months)recommendation by the Evaluation Committeeapproval by the DMtranslation, ATIP review, media lines, release on Internet
10 Evaluation Questions and Issues Evaluations address relevance and performance.Relevance issues focus on:continued need for program;alignment with government priorities; andalignment with federal roles and responsibilities.Performance issues focus on effectiveness:achievement of expected outcomes; anddemonstration of efficiency and economy.Evaluators work with program managers to develop more detailed evaluation questions relevant to their program.
11 Roles and Responsibilities Under the TBS Evaluation Policy, Deputy Ministers are responsible for the evaluation function.NRCan’s Departmental Evaluation Committee – an ADM-level Committee – is chaired by the DMNRCan’s Head of Evaluation – who is also the DG of Planning and Performance Management Reporting – reports to the Evaluation Committee.
12 The Role of the Evaluation Division The Strategic Evaluation Division (SED) is responsible for:Proposing a five year departmental evaluation plan to the Evaluation Committee, and updating it annually;Managing and conducting evaluation studies, including managing contracts and deliverables when consultants are used and issuing reports in a timely manner.Additionally, SED will help program managers develop their performance measurement strategies, with the goal of ensuring that good data is collected to support future evaluations.Evaluation will work with your team to develop objectives, a logic model, a performance measurement framework and evaluation requirements.NRCan's Strategic Evaluation Division is also responsible for reviewing and providing advice on the accountability and performance provisions in Cabinet documents (Memoranda to Cabinet (MCs) and TB Submissions).
13 The Role of Program Managers Program managers are key to conducting evaluations.They are responsible for developing, implementing and monitoring ongoing performance measurement – the foundation of evaluation.Additionally, during an evaluation, they must be actively involved in:explaining how their programs work;contributing to evaluation planning, including identifying more detailed evaluation questions;providing performance measurement information on resources used, activities undertaken and results achieved;providing detailed documentation (see next slide) and suggestions on potential interviewees, case studies etc.;participating in working groups to review questionnaires, preliminary findings, draft evaluation reports, etc.;developing management responses and action plans for their ADMs and implementing them after the evaluation.
14 Key Documents for An Evaluation In preparation for an evaluation, program managers will be asked to provide key documents as early as possible, including:Legislation, Regulations, MCs, TB SubmissionsRMAFs, RBAFs or Performance Measurement Frameworksreferences in budgets, SFTs, DPRs, RPPsbriefing notes, reports --including annual and project reports, studies, databaseswebsites and communications productsfive-years of financial expenditures for the PAA unit including G&C expenditures, O&M, and salaries
15 Questions and Assistance If you have any questions on evaluation, or wish evaluation assistance in developing performance measurement information or Cabinet documents, please contact:The Director of Strategic Evaluation(613)Electronic copies of this document, completed evaluation reports and the Terms of Reference for the Evaluation Committee are available at: