Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada.

Similar presentations


Presentation on theme: "Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada."— Presentation transcript:

1 Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada

2 REL Solutions Inc.Brasilia, November 23, Background on the Canadian System 2. Historical Perspective on how Evaluation has evolved 3. Government Accountability Structure 4. Global View on how Evaluation Supports Governments Vision Outline PART ONE: An Overview of the Evaluation System in Canada

3 REL Solutions Inc.Brasilia, November 23, Population 31 million Sovereign parliamentary democracy Federation of 10 provinces and 3 territories A federal Parliament and 13 provincial / territorial legislatures Areas of federal responsibility include: national defence, criminal law, postal service, census, copyrights, trade, external relations, finance, transportation, citizenship, Indian affairs, fisheries and agriculture Provincial / territorial governments: authority for education, hospitals, property and local government Federal Government Over 100 departments and agencies Each has a Minister who reports to Parliament Budgets approved annually Formal Evaluation Policy in Canada for more than 30 years Background on Canadian System of Government

4 REL Solutions Inc.Brasilia, November 23, Growing demand for information to support increasingly complex and costly programs New management systems created for financial administration and planning, programming and budgeting Evaluation as a practice not yet formalized Historical Perspective: The 1960s

5 REL Solutions Inc.Brasilia, November 23, Building of key infrastructure elements in departments Treasury Board Evaluation Policy (1977) was the first formalized evaluation policy in Canada Evaluation Policy centre created within the new Office of the Comptroller General (1978) Considerable focus on building evaluation capacity in federal departments Historical Perspective: The 1970s

6 REL Solutions Inc.Brasilia, November 23, High expectations for evaluations not always met Government-wide review led by Parliament Historical Perspective: The 1980s

7 REL Solutions Inc.Brasilia, November 23, Evaluation capacity affected by general government downsizing AG report (1993) indicated renewal of evaluation capacity needed New Review Policy (1994) linked evaluation closer to internal audit New emphasis on performance measurement / monitoring: Departmental Performance Reports to Parliament Historical Perspective: The 1990s

8 REL Solutions Inc.Brasilia, November 23, Capacity gap for Evaluation (and Internal Audit) identified – need to reinvest in evaluation capacity within departments – both $ and skilled evaluators New Evaluation Policy (2001): Commitment to link Evaluation to broader accountability and reporting requirements (strategic) as well as management decision-making (focused) Emphasis on Results based Management Measuring Results Results based Management and Accountability Framework (RMAF) Re-establishment of Policy Centre: Oversight of Evaluation practice in departments capacity building role; quality control role; a champion Move in direction of participatory evaluation Objectivity, not independence, emphasized Greater link to management and performance measurement / monitoring Historical Perspective: The 2000s

9 REL Solutions Inc.Brasilia, November 23, Need for formal evaluation increases as resources become scarcer and the identification of priorities becomes more important Necessary, but not sufficient, conditions for an effective evaluation system are: government investment and support; formalized policies and standards; and, leadership for capacity building Evaluation quality depends on an approach that balances: timeliness, usefulness, methodological purity, client requirements and cost Critical mass in capacity is required to ensure evaluation remains credible, relevant and strategic Performance measurement / monitoring will typically not address ongoing issues of cause and effect; for that, ad-hoc evaluation is generally required The priority given Evaluation is greatly enhanced if there is a perspective on modern management shared by the political level and the Public Service and evaluation is seen as integral to that vision Historical Perspective: Lessons Learned

10 REL Solutions Inc.Brasilia, November 23, Canadian Government Accountability Structure Parliament of Canada House of Commons Prime Minister Cabinet (Privy Council) Parliamentary Committees TB Minister Treasury Board Secretariat (TBS) (Secretary appointed) Governments management board (financial, management and human resources) Comptrollership function Centre of Excellence for Evaluation Policies and standards Capacity building Links evaluation and performance measurement Public Servants All Other Ministers Federal Depts. /Agencies (DMs appointed) Deputy Ministers (DMs) accountable for the application of Evaluation Policy within their departments Heads of Evaluation implement policy as per TBS standards and guidelines Internal accountability and reporting to DMs External accountability and reporting to TBS and Parliament Public Servants Auditor General (AG) (appointed) Conducts independent audits of government operations Produces periodic oversight reports on the conduct of evaluation Promotes accountability and best practices Reports directly to Parliament

11 REL Solutions Inc.Brasilia, November 23, Global Views on Public Sector Management in Canada Vision: Results for Canadians – Improve government services and respond to demands for better value and transparency Tools and Initiatives: Applied through a wide-reaching series of initiatives Commitment to link evaluation and performance measurement / monitoring to broader accountability and reporting requirements Incentives and Use: Perspective on modern management shared by the political level and the Public Service

12 REL Solutions Inc.Brasilia, November 23, Generate and use reliable information on results for the purposes of: Sound management, resource allocation and decision making Accountability to Parliament and Canadians Continuous process of cultural change with no static end state Global Views: Goals to Achieve

13 REL Solutions Inc.Brasilia, November 23, Government Commitments Roles & Responsibilities Agenda for Change Results Responsible Spending Values Citizen Focus The Results for Canadians Management Framework

14 REL Solutions Inc.Brasilia, November 23, Learn & Adjust Strategic Analysis Plan for Results Implement Monitor, Measure, Evaluate Report on Results Analysis of the current environment, past performance, emerging priorities and significant risks to achievement of desired results Developing Strategic and Operational Plans – identify and plan to achieve key results and mitigate risk Action taken to achieve results Ongoing performance measurement and periodic evaluation to determine progress and allow for corrective action Providing integrated Financial and non-financial information on: results and accountability for internal and external use Evaluation: A Critical Link to Results-Based Management

15 REL Solutions Inc.Brasilia, November 23, Gain broad support for the Vision Create the necessary enablers (policies, guidelines, standards) Invest in capacity building across the system Human capital Tools necessary to support implementation Provide the necessary incentives for implementation Monitor and evaluate implementation Learn and adjust, as necessary Approach to Implementing the RFC Vision

16 REL Solutions Inc.Brasilia, November 23, Tools and Initiatives to Support the Vision Modern Comptrollership Broad direction for management reforms Strengthened Evaluation and Internal Audit functions Investment in capacity building Functional specialists in departments (Evaluators; Auditors) Learning for managers Results-based Management and Accountability Frameworks (RMAFs) Improved Performance Reporting to Parliament Departmental Performance Reports Canadas Performance Monitoring implementation Modern Comptrollership Capacity Check Managing for Results Self-Assessment Tool Evaluation and Active Monitoring of Policy implementation New government-wide initiatives Expenditure and Management Review Management Accountability Framework

17 Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART TWO: A Detailed Examination of the Process and Conduct of Evaluation in Federal Departments

18 REL Solutions Inc.Brasilia, November 23, Module 1: Purpose of Evaluation 2. Module 2: Developing the Capacity to Do Evaluation 3. Module 3:Developing Capacity to Use Evaluation 4. Module 4:Evaluation vs. Performance Measurement/Monitoring 5. Module 5: Results based Management and Accountability Frameworks Outline PART TWO: A Detailed Examination of the Process and Conduct of Evaluation in Federal Departments

19 REL Solutions Inc.Brasilia, November 23, Outline of Module Definitions and Terminology Key Uses and Users of Evaluation A Logic Model for the Use of Evaluation in a Federal Department Performance Measures for the Evaluation Function MODULE ONE: Purpose of Evaluation.. Why is it important? And, to whom?

20 REL Solutions Inc.Brasilia, November 23, Evaluation Summative Evaluation Formative Evaluation Performance Measurement Performance Monitoring Results Results Based management DEFINITIONS AND TERMINOLOGY

21 REL Solutions Inc.Brasilia, November 23, How and where do results of Evaluation get used? Within federal departmentsseveral users Deputy Ministers Assistant Deputy Ministers (ADMs) Program Managers Input to Corporate functions (policy, planning) Treasury Bard of Canada Secretariat (TBS) Funding renewal for individual programs Expenditure and Management Reviews Management Accountability Frameworks (MAF) Elected Officials Departmental Performance Reports Canadas Performance Parliamentary Committees General Public Access to Evaluation Reports (ATIP) Web-based increases transparency KEY USES AND USERS OF EVALUATION

22 REL Solutions Inc.Brasilia, November 23, Defined four broad ways Evaluation supports senior management Four activity areas where skill set of Evaluators and Evaluation information is used Traditional role (conducting program and policy evaluation) represents major use of Evaluation resources (some 75%) A LOGIC MODEL FOR THE USE OF EVALUATION IN A FEDERAL DEPARTMENT

23 REL Solutions Inc.Brasilia, November 23, ACTIVITIES OUTPUTS IMMEDIATE OUTCOMES INTERMEDIATE OUTCOMES ULTIMATE OUTCOMES A LOGIC MODEL FOR THE USE OF EVALUATION IN A FEDERAL DEPARTMENT

24 REL Solutions Inc.Brasilia, November 23, How well are we doing in terms of expectations? Derive measures from Logic Model Caution how information is used, particularly during capacity building period PERFORMANCE MEASURES FOR THE EVALATION FUNCTION

25 REL Solutions Inc.Brasilia, November 23, Outline of Module Policies and Standards Champion for the Function Investment in Evaluation Capacity Recruitment, training and development of Evaluators Vehicles to Train and Develop Educating the Users of Evaluation Organizing an Evaluation Unit in a Federal Department MODULE TWO: Developing the Capacity To Do Evaluation. A Key Role for the Central Agency

26 REL Solutions Inc.Brasilia, November 23, Identifies expectations, roles and responsibilities Federal Departments given flexibility in their application Deputy Minister (DM) accountable Treasury Board of Canada Secretariat (TBS) oversight of individual departments…through the Centre of Excellence for Evaluation (CEE) Periodic review by the Auditor General of Canada (AG) of how/how well Evaluation Policy is being implemented …provides system-wide oversight POLICIES AND STANDARDS

27 REL Solutions Inc.Brasilia, November 23, Important to have a champion/advocate for the function Several roles: Setting and adjusting policy and standards for the function Developing and executing human resource (HR) strategies to train and develop Evaluators Advisory role (practical advice) to individual departments Oversight and monitoring quality of evaluation practices and products Corporate resource on Evaluation and accountability for government officials Establish a network for Evaluators (continuous learning) CHAMPION FOR THE FUNCTION: THE CENTRE OF EXCELLENCE FOR EVALUATION (CEE)

28 REL Solutions Inc.Brasilia, November 23, Need for both $ and trained personnel Policy not prescriptive on size of departmental Evaluation units… a decision of the DM Accountabilities of the DM (several) serve to leverage resources for the function In recent years, focus of TBS/CEE on building a sustainable Evaluation function Infrastructure Product Use With flexibility in resourcing, important that DMs understand importance and use of Evaluation Canadian experience....periods of general downsizing (fewer resources for the function) and capacity building (new investment) Danger of fluctuating resource levels….losing skilled/experienced Evaluators. A long-term investment to replace. INVESTMENT IN BUILDING EVALUATION CAPACITY

29 REL Solutions Inc.Brasilia, November 23, Recruitment University graduates generally not Evaluation-specific Entry into the function from many disciplines…social sciences (Economics, Psychology, Sociology); Education; Mathematics/Statistics; Public Administration Both a strength and a weakness Training Need to distinguish between training for entry level, mid level and senior Evaluators Link to the competencies required for Evaluators (Competency Profile) Various vehicles for formal training Development Importance of on the job training Continuous learning…link to changing/evolving role of Evaluation Communication skills particularly important for senior level Evaluators (Departmental Heads of Evaluation) Several vehicles and occasions for development No formal certification to work as an Evaluator in federal government RECRUITMENT, TRAINING AND DEVELOPMENT OF EVALUATORS

30 REL Solutions Inc.Brasilia, November 23, Formal curriculum developed by CEE for entry level Evaluators (Evaluator Internship Program)…not mandatory Training needs identified for mid-level Evaluator CEE partnering with others for training delivery…e.g. National Statistical agency (Statistics Canada); Canadian Centre for Management Development; Professional Associations (Canadian Evaluation Society (CES); department-led training; private sector trainers/course deliverers Network of Evaluators encourages informal communities of practice and sharing of lessons learned across like evaluations Link to other professional groups (e.g. Internal Auditors; Comprehensive Audit) VEHICLES TO TRAIN AND DEVELOP EVALUATORS

31 REL Solutions Inc.Brasilia, November 23, Users of evaluation (in departments, central agencies, politicians) Need to be educated on how, where, and why evaluation can/should be used/resourced Importance of aligning user expectations with reality of what can/will be delivered by Evaluators Useful to use a variety of vehicles: briefing notes; presentation decks; training modules/workshops aimed at senior managers Avoid technical jargon and deliver at appropriate level of understanding Education an ongoing activity; not a one-time event Basic distinctions need to be made…e.g. difference between Internal Audit and Evaluation, an ongoing area of confusion for managers: Internal Audit: Are we doing things right? (efficiency issues) Evaluation: Are we doing the right things? (effectiveness and performance issues) EDUCATING THE USERS OF EVALUATION

32 REL Solutions Inc.Brasilia, November 23, In Canada, federal departments determine their own arrangements Much variation in terms of number of full-time staff hired vs use of consultants Generally, most departments use some consultants… e.g. for surveys, specialized tasks, etc. Important for Unit to have minimum core of full-time Evaluators Wide range in size of Units dedicated to Evaluation… e.g. 1 person (small agency) to f/t staff (largest department) Typical Evaluation Unit: 10 or fewer f/t staff, plus resources to hire consultants for Evaluation ORGANIZING AN EVALUATION UNIT

33 REL Solutions Inc.Brasilia, November 23, Outline of Module Central Agency-driven demand for Evaluation Steering the Evaluation function in departments Credibility building for Evaluators Developing a Comprehensive Evaluation Plan Advisory Committees Educating Users of Evaluation Participatory Evaluation and Accountability for Follow-up Evaluations in full public view MODULE THREE: Developing Capacity To Use Evaluation: Overcoming Resistance and Making Effective Use of Evaluation

34 REL Solutions Inc.Brasilia, November 23, CENTRAL AGENCY-DRIVEN DEMAND FOR EVALUATION Deputy Minister in department faces variety of accountabilities requiring performance/results information Required for good management practices and internal and external reporting Program-specific funding renewal decisions Horizontal and vertical expenditure reviews Accountability report card of the DM (Management Accountability Framework (MAF) Departmental Performance Report to Parliament Ability to respond to queries of Parliamentary Committees; issues raised by the Auditor General

35 REL Solutions Inc.Brasilia, November 23, Steering the Evaluation Function in Departments Evaluation and Audit Committee, chaired by Deputy Minister Requirement of TBS for every government department Senior level committee; formal Terms of Reference; meet minimum four times a year Review and approve annual and multi-year plan for Evaluation Senior managers identify new/changing priorities Table completed Evaluation reports and management response Forum to alert senior managers of areas of risk to department Vehicle to inform/educate senior managers (senior users of evaluation

36 REL Solutions Inc.Brasilia, November 23, Credibility Building for Evaluators How Evaluators see themselves. How others see Evaluation. Two critical factors for credibility: Understanding the needs/requirements of senior decision makers Delivering quality product on time Answering the right questions at the right time Critical factors for success: Head of Evaluation well linked to senior management Appropriate balance between methodological rigour and delivery in a timely fashion Responsive to changing priorities

37 REL Solutions Inc.Brasilia, November 23, Developing a Comprehensive Evaluation Plan Three key questions: i. What are the guiding principles for evaluation planning? ii. What is the relevant Evaluation Universe? iii. How best to determine priorities and schedule evaluations over a multi-year period?

38 REL Solutions Inc.Brasilia, November 23, Advisory Committees: Building trust and avoiding surprises Useful vehicle to assist conduct of an Evaluation study Representation from broad set of stakeholders Forum to clarify study objectives; table early findings; review conclusions and recommendations Creates more open evaluation process Avoid surprises at the end Helps ensure follow-through on study recommendations

39 REL Solutions Inc.Brasilia, November 23, Users of evaluation (in departments, central agencies, politicians) Need to be educated on how, where, and why evaluation can/should be used/resourced Importance of aligning user expectations with reality of what can/will be delivered by Evaluators Avoid technical jargon and deliver at appropriate level of understanding Education an ongoing activity; not a one-time event EDUCATING THE USERS OF EVALUATION

40 REL Solutions Inc.Brasilia, November 23, Conduct of evaluation not independent of program managers New Policy (2001) removed independence; stressed objectivity and transparency Attempt to bring evaluation closer to managers…not a threat, but a tool to assist good management practices Puts onus on Managers for ongoing performance monitoring as well as accountability for implementing evaluation recommendations Participatory Evaluation and Accountability for Follow-up

41 REL Solutions Inc.Brasilia, November 23, Broad demand for greater accountability in government Evaluation reports accessible to wide audience (via ATIP) Web posting increases accessibility to general public Evaluation Policy stresses transparency Requires a culture allowing for: full disclosure of information unbiased analysis and reporting openness in government Evaluation In Full Public View

42 REL Solutions Inc.Brasilia, November 23, Outline of Module Historical Context Definitions and Distinctions Departmental Performance Framework Using Logic Models to Derive Performance Measures Results Measurement Continuum Model Roles and Responsibilities for Delivery General Strategy for Implementing M&E Systems MODULE FOUR: Evaluation versus Performance Measurement/Monitoring: A Critical Distinction

43 REL Solutions Inc.Brasilia, November 23, Mid-to-late 90s onward: Increasing government emphasis on performance measurement, monitoring and results Drivers: Results based management; Greater accountability to Parliament and to citizens Put onus on managers to measure and report on performance/results Required clarification of terminology, roles and responsibilities in the process Historical Context

44 REL Solutions Inc.Brasilia, November 23, How is Evaluation different from Performance Monitoring? How does one link to the other? Are they both necessary tools to support good management practices and accountability? Definitions and Distinctions

45 REL Solutions Inc.Brasilia, November 23, Planning Reporting and Accountability Structure (PRAS) provides broad departmental framework Policy being updated (Fall 2004) to Management Resources and Results Structure (MRRS) Business line structure: architecture and basis for cascading set of performance measures through organization Brings greater focus to managers on measuring results Evaluation specialist has the skill set to facilitate this process Departmental Performance Framework: Basis for Ongoing Performance Monitoring

46 REL Solutions Inc.Brasilia, November 23, Performance indicators derive from logic model What needs to be measured and reported? What are the key questions that need to be answered? How will it be used? By whom? How often? Monitoring versus ad hoc Evaluation Performance measurement development/implementation Technical cost considerations An iterative process Using Logic Models to Derive Meaningful Performance Measurement

47 REL Solutions Inc.Brasilia, November 23, Results Measurement Continuum Model

48 REL Solutions Inc.Brasilia, November 23, Managers ultimately accountable for performance measurement and reporting Evaluation specialists support to the development and implementation of the performance measurement system through: creating PM understanding---learning events, seminars, information packages facilitating PM development applying needed rigour and discipline---challenge guidelines Evaluation study as one option for measuring performance, where: ad hoc study more cost-effective cause and effect (attribution) issue important IM/IT specialist---advice re systems and data capture issues Roles and Responsibilities for Delivery

49 REL Solutions Inc.Brasilia, November 23, Challenges: political; organizational; technical; resourcing Useful to do a readiness assessment: Is there an enabling environment: political will; commitment; a champion Technical capacity, including HR skills Necessary infrastructure to produce, collect, analyze and report relevant information Necessary infrastructure/institutions to use evaluation and monitoring information Priority needs to be established from the top Resourcing: a corporate responsibility. Needs to be linked to accountability and modern management practices General Strategy for Implementing Evaluation and Performance Monitoring Systems

50 REL Solutions Inc.Brasilia, November 23, Outline of Module Why have we introduced the RMAF? RMAF Components Using the RMAF to Greatest Gain Helpful Hints for RMAF Development/Implementation MODULE FIVE: Results Based Management and Accountability Framework (RMAF): An Important Vehicle

51 REL Solutions Inc.Brasilia, November 23, Tool designed to assist the move to results measurement/reporting Broader government commitment on results reporting (Results for Canadians) Formal requirement for certain programs Facilitates development to performance measurement and establishes priority and timeframe for evaluation Why have we introduced the RMAF?

52 REL Solutions Inc.Brasilia, November 23, All RMAFs have five major sections: A: Profile B: Logic Model C: Ongoing Performance Measurement Strategy D: Evaluation Strategy E: Reporting Strategy Together, describes the rationale, resources and performance logic of program or initiative Presents the measurement, evaluation and reporting strategies that will be implemented to track progress RMAF Components

53 REL Solutions Inc.Brasilia, November 23, A collaborative effort of key program stakeholders to clearly articulate: Program objectives in terms that allow for measuring/monitoring performance/success Linkages of program activities to program goals Brings technical (Evaluator) and non-technical (Manager) to common agreement on meaningful measures of performance and a measurement strategy Up-front investment in RMAF pays off in Clarifying program role and resourcing needs Establishment of meaningful measurement systems More systematic and disciplined monitoring of program performance Improved management information, supporting RBM Improved accountability, both to internal and external stakeholders (senior managers; elected officials; citizens) Using the RMAF to Greatest Gain

54 REL Solutions Inc.Brasilia, November 23, Accountability with RMAF development; performance monitoring; reporting…lies with Manager Carry out development and implementation as collaborative effort (Manger; Evaluation specialist; IM/IT specialist) Flexibility, particularly in early stages: acceptable RMAF versus perfection performance measurement as an iterative process Select realistic performance indicators based on what will provide the best information, not the easiest to implement Helpful Hints for Implementing RMAFs

55 REL Solutions Inc.Brasilia, November 23, Evaluation and performance measurement: Important link to effective pubic sector governance Broad application across: individual programs sectors or business lines whole organization Not an end in itself, a means to an end Evaluation capacity building to a self-sustaining state requires: Commitment/political will Resources (both $ and HR) Time Energy Final Thoughts


Download ppt "Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada."

Similar presentations


Ads by Google