1. Evaluation in the Government of Canada
PART ONE: An Overview of the Evaluation System in Canada
Robert Lahey
23 November 2004

2. Outline Background on the Canadian System
PART ONE: An Overview of the Evaluation System in Canada
Background on the Canadian System
Historical Perspective on how Evaluation has evolved
Government Accountability Structure
Global View on how Evaluation Supports Government’s Vision
Bob’s notes: Add – standards, HR strategy, and competency profile
REL Solutions Inc. Brasilia, November 23, 2004

3. Background on Canadian System of Government
Population 31 million
Sovereign parliamentary democracy
Federation of 10 provinces and 3 territories
A federal Parliament and 13 provincial / territorial legislatures
Areas of federal responsibility include: national defence, criminal law, postal service, census, copyrights, trade, external relations, finance, transportation, citizenship, Indian affairs, fisheries and agriculture
Provincial / territorial governments: authority for education, hospitals, property and local government
Federal Government
Over 100 departments and agencies
Each has a Minister who reports to Parliament
Budgets approved annually
Formal Evaluation Policy in Canada for more than 30 years
REL Solutions Inc. Brasilia, November 23, 2004

4. Historical Perspective: The 1960s
Growing demand for information to support increasingly complex and costly programs
New management systems created for financial administration and planning, programming and budgeting
Evaluation as a practice not yet formalized
Increasingly complex and costly programs
Growing demand for information to support program changes
Established a new planning, programming and budgeting system
Introduced new financial administration and auditing systems
Politicians, senior managers and Canadians demanded information to support program improvement, modification and management
Efforts made to report on accountability and to justify expenditures through a new planning, programming and budgeting system
REL Solutions Inc. Brasilia, November 23, 2004

5. Historical Perspective: The 1970s
Building of key infrastructure elements in departments
Treasury Board Evaluation Policy (1977) was the first formalized evaluation policy in Canada
Evaluation Policy centre created within the new Office of the Comptroller General (1978)
Considerable focus on building evaluation capacity in federal departments
Period of increased funding and decentralization
Building of evaluation infrastructure (policies, standards and central agencies) and setting the stage for more formalized evaluation
Treasury Board Evaluation Policy (1977) was the first formalized evaluation policy in Canada
Program Evaluation Branch of the new Office of the Comptroller General created (1978)
Branch issued Policy, guides and standards to assist departments in establishing and maintaining an evaluation function
REL Solutions Inc. Brasilia, November 23, 2004

6. Historical Perspective: The 1980s
High expectations for evaluations not always met
Government-wide review led by Parliament
High expectations for evaluations in spite of known theoretical and methodological challenges
Government-wide review led by Parliament changed organizations and reduced spending
Review process and recommendations changed government, reduced spending and highlighted the lack of adequate and timely evaluations and performance reports
Highlighted the lack of adequate and timely evaluations and performance reports for departments, central agencies and parliamentary committees
REL Solutions Inc. Brasilia, November 23, 2004

7. Historical Perspective: The 1990s
Evaluation capacity affected by general government downsizing
AG report (1993) indicated renewal of evaluation capacity needed
New Review Policy (1994) linked evaluation closer to internal audit
New emphasis on performance measurement / monitoring: Departmental Performance Reports to Parliament
Evaluation capacity and critical mass needed for evaluation lacking due to downsizing
Auditor General reported (1993) on need to improve the evaluation system and renew evaluation capacity
New approach linked evaluation closer to internal audit with the new Review Policy (1994)
Evaluation resources often diverted to other areas related to program design and management
REL Solutions Inc. Brasilia, November 23, 2004

8. Historical Perspective: The 2000s
‘Capacity gap’ for Evaluation (and Internal Audit) identified – need to reinvest in evaluation capacity within departments – both $ and skilled evaluators
New Evaluation Policy (2001): Commitment to link Evaluation to broader accountability and reporting requirements (strategic) as well as management decision-making (focused)
Emphasis on Results based Management
Measuring Results
Results based Management and Accountability Framework (RMAF)
Re-establishment of Policy Centre: Oversight of Evaluation practice in departments capacity building role; quality control role; a champion
Move in direction of ‘participatory’ evaluation
Objectivity, not independence, emphasized
Greater link to management and performance measurement / monitoring
Evaluation capacity and critical mass needed for evaluation lacking due to downsizing
Auditor General reported (1993) on need to improve the evaluation system and renew evaluation capacity
New approach linked evaluation closer to internal audit with the new Review Policy (1994)
Evaluation resources often diverted to other areas related to program design and management
REL Solutions Inc. Brasilia, November 23, 2004

9. Historical Perspective: Lessons Learned
Need for formal evaluation increases as resources become scarcer and the identification of priorities becomes more important
Necessary, but not sufficient, conditions for an effective evaluation system are: government investment and support; formalized policies and standards; and, leadership for capacity building
Evaluation quality depends on an approach that balances: timeliness, usefulness, methodological purity, client requirements and cost
Critical mass in capacity is required to ensure evaluation remains credible, relevant and strategic
Performance measurement / monitoring will typically not address ongoing issues of ‘cause and effect’; for that, ad-hoc evaluation is generally required
The priority given Evaluation is greatly enhanced if there is a perspective on modern management shared by the political level and the Public Service and evaluation is seen as integral to that vision
Historical Perspective: Lessons Learned
REL Solutions Inc. Brasilia, November 23, 2004

10. Canadian Government Accountability Structure
Parliament of Canada House of Commons
Prime Minister
Cabinet (Privy Council)
Parliamentary Committees
Auditor General (AG) (appointed)
Conducts independent audits of government operations
Produces periodic oversight reports on the conduct of evaluation
Promotes accountability and best practices
Reports directly to Parliament
TB Minister
Treasury Board Secretariat (TBS) (Secretary appointed)
Government’s management board (financial, management and human resources)
Comptrollership function
Centre of Excellence for Evaluation
Policies and standards
Capacity building
Links evaluation and performance measurement
Public Servants
All Other Ministers
Federal Depts. /Agencies (DMs appointed)
Deputy Ministers (DMs) accountable for the application of Evaluation Policy within their departments
Heads of Evaluation implement policy as per TBS standards and guidelines
Internal accountability and reporting to DMs
External accountability and reporting to TBS and Parliament
Public Servants
REL Solutions Inc. Brasilia, November 23, 2004

11. Global Views on Public Sector Management in Canada
Vision:
“Results for Canadians” – Improve government services and respond to demands for better value and transparency
Tools and Initiatives:
Applied through a wide-reaching series of initiatives
Commitment to link evaluation and performance measurement / monitoring to broader accountability and reporting requirements
Incentives and Use:
Perspective on modern management shared by the political level and the Public Service
Evaluation capacity and critical mass needed for evaluation lacking due to downsizing
Auditor General reported (1993) on need to improve the evaluation system and renew evaluation capacity
New approach linked evaluation closer to internal audit with the new Review Policy (1994)
Evaluation resources often diverted to other areas related to program design and management
REL Solutions Inc. Brasilia, November 23, 2004

12. Global Views: Goals to Achieve
Generate and use reliable information on results
for the purposes of:
Sound management, resource allocation and decision making
Accountability to Parliament and Canadians
Continuous process of cultural change with no static end state
REL Solutions Inc. Brasilia, November 23, 2004

13. The Results for Canadians Management Framework
Citizen Focus
Responsible Spending
Values
Results
4 Government Commitments
Roles & Responsibilities
Agenda for Change
REL Solutions Inc. Brasilia, November 23, 2004

14. Monitor, Measure, Evaluate
Evaluation: A Critical Link to Results-Based Management
Learn & Adjust
Monitor, Measure, Evaluate
Strategic Analysis
Plan for Results
Report on Results
Implement
Analysis of the current environment, past performance, emerging priorities and significant risks to achievement of desired results
Developing Strategic and Operational Plans – identify and plan to achieve key results and mitigate risk
Action taken to achieve results
Ongoing performance measurement and periodic evaluation to determine progress and allow for corrective action
Providing integrated Financial and non-financial information on: results and accountability for internal and external use
REL Solutions Inc. Brasilia, November 23, 2004

15. Approach to Implementing the RFC Vision
Gain broad support for the “Vision”
Create the necessary ‘enablers’ (policies, guidelines, standards)
Invest in capacity building across the system
Human capital
Tools necessary to support implementation
Provide the necessary incentives for implementation
Monitor and evaluate implementation
Learn and adjust, as necessary
REL Solutions Inc. Brasilia, November 23, 2004

16. Tools and Initiatives to Support the Vision
Modern Comptrollership
Broad direction for management reforms
Strengthened Evaluation and Internal Audit functions
Investment in capacity building
Functional specialists in departments (Evaluators; Auditors)
Learning for managers
Results-based Management and Accountability Frameworks (RMAFs)
Improved Performance Reporting to Parliament
Departmental Performance Reports
Canada’s Performance
Monitoring implementation
Modern Comptrollership “Capacity Check”
Managing for Results Self-Assessment Tool
Evaluation and Active Monitoring of Policy implementation
New government-wide initiatives
Expenditure and Management Review
Management Accountability Framework
TBS: The items listed are roles. Responsibilities: policy, standards, evaluation of policy and capacity building initiatives
Departments: Responsibilities: Evaluation reports on programs and departmental policies; performance reports on departments
REL Solutions Inc. Brasilia, November 23, 2004

17. Evaluation in the Government of Canada
PART TWO: A Detailed Examination of the Process and Conduct of Evaluation in Federal Departments
Robert Lahey
23 November 2004

18. Outline Module 1: Purpose of Evaluation
PART TWO: A Detailed Examination of the Process and Conduct of Evaluation in Federal Departments
Module 1: Purpose of Evaluation
Module 2: Developing the Capacity to ‘Do’ Evaluation
Module 3: Developing Capacity to ‘Use’ Evaluation
Module 4: Evaluation vs. Performance Measurement/Monitoring
Module 5: Results based Management and Accountability Frameworks
Bob’s notes: Add – standards, HR strategy, and competency profile
REL Solutions Inc. Brasilia, November 23, 2004

19. MODULE ONE: Purpose of Evaluation.. Why is it important? And, to whom?
Outline of Module
Definitions and Terminology
Key Uses and Users of Evaluation
A Logic Model for the Use of Evaluation in a Federal Department
Performance Measures for the Evaluation Function
REL Solutions Inc. Brasilia, November 23, 2004

20. DEFINITIONS AND TERMINOLOGY
Evaluation  Summative Evaluation  Formative Evaluation
Performance Measurement
Performance Monitoring
Results
Results Based management
REL Solutions Inc. Brasilia, November 23, 2004

21. KEY USES AND USERS OF EVALUATION
How and where do results of Evaluation get used?
Within federal departments—several users
 Deputy Ministers  Assistant Deputy Ministers (ADMs)  Program Managers  Input to Corporate functions (policy, planning)
Treasury Bard of Canada Secretariat (TBS)
 Funding renewal for individual programs  Expenditure and Management Reviews  Management Accountability Frameworks (MAF)
Elected Officials
 Departmental Performance Reports  Canada’s Performance  Parliamentary Committees
General Public
 Access to Evaluation Reports (ATIP)  Web-based increases transparency
REL Solutions Inc. Brasilia, November 23, 2004

22. Defined four broad ways Evaluation supports senior management
A LOGIC MODEL FOR THE USE OF EVALUATION IN A FEDERAL DEPARTMENT
Defined four broad ways Evaluation supports senior management
Four activity areas where skill set of Evaluators and Evaluation information is used
Traditional role (conducting program and policy evaluation) represents major use of Evaluation resources (some 75%)
REL Solutions Inc. Brasilia, November 23, 2004

23. INTERMEDIATE OUTCOMES
A LOGIC MODEL FOR THE USE OF EVALUATION IN A FEDERAL DEPARTMENT
ACTIVITIES
OUTPUTS
IMMEDIATE OUTCOMES
INTERMEDIATE OUTCOMES
ULTIMATE OUTCOMES
REL Solutions Inc. Brasilia, November 23, 2004

24. PERFORMANCE MEASURES FOR THE EVALATION FUNCTION
How well are we doing in terms of expectations?
Derive measures from Logic Model
Caution how information is used, particularly during capacity building period
REL Solutions Inc. Brasilia, November 23, 2004

25. MODULE TWO: Developing the Capacity ‘To Do’ Evaluation
MODULE TWO: Developing the Capacity ‘To Do’ Evaluation. A Key Role for the Central Agency
Outline of Module
Policies and Standards
Champion for the Function
Investment in Evaluation Capacity
Recruitment, training and development of Evaluators
Vehicles to Train and Develop
Educating the ‘Users’ of Evaluation
Organizing an Evaluation Unit in a Federal Department
REL Solutions Inc. Brasilia, November 23, 2004

26. POLICIES AND STANDARDS
Identifies expectations, roles and responsibilities
Federal Departments given flexibility in their application
Deputy Minister (DM) accountable
Treasury Board of Canada Secretariat (TBS) oversight of individual departments…through the Centre of Excellence for Evaluation (CEE)
Periodic review by the Auditor General of Canada (AG) of how/how well Evaluation Policy is being implemented …provides system-wide oversight
REL Solutions Inc. Brasilia, November 23, 2004

27. Important to have a champion/advocate for the function Several roles:
CHAMPION FOR THE FUNCTION: THE CENTRE OF EXCELLENCE FOR EVALUATION (CEE)
Important to have a champion/advocate for the function
Several roles:
 Setting and adjusting policy and standards for the function  Developing and executing human resource (HR) strategies to train and develop Evaluators  Advisory role (practical advice) to individual departments  Oversight and monitoring quality of evaluation practices and products  Corporate resource on Evaluation and accountability for government officials
Establish a network for Evaluators (continuous learning)
REL Solutions Inc. Brasilia, November 23, 2004

28. INVESTMENT IN BUILDING EVALUATION CAPACITY
Need for both $ and trained personnel
Policy not prescriptive on size of departmental Evaluation units… a decision of the DM
Accountabilities of the DM (several) serve to leverage resources for the function
In recent years, focus of TBS/CEE on building a ‘sustainable’ Evaluation function  Infrastructure  Product  Use
With flexibility in resourcing, important that DMs understand importance and use of Evaluation
Canadian experience....periods of general downsizing (fewer resources for the function) and capacity building (new investment)
Danger of fluctuating resource levels….losing skilled/experienced Evaluators. A long-term investment to replace.
REL Solutions Inc. Brasilia, November 23, 2004

29. RECRUITMENT, TRAINING AND DEVELOPMENT OF EVALUATORS
Recruitment  University graduates generally not ‘Evaluation-specific’  Entry into the function from many disciplines…social sciences (Economics, Psychology, Sociology); Education; Mathematics/Statistics; Public Administration  Both a strength and a weakness 
Training  Need to distinguish between training for ‘entry level’, ‘mid level’ and ‘senior’ Evaluators  Link to the competencies required for Evaluators (Competency Profile)  Various vehicles for formal training
Development  Importance of ‘on the job’ training  Continuous learning…link to changing/evolving role of Evaluation  Communication skills particularly important for senior level Evaluators (Departmental Heads of Evaluation)  Several vehicles and occasions for development  No formal ‘certification’ to work as an ‘Evaluator’ in federal government
REL Solutions Inc. Brasilia, November 23, 2004

30. VEHICLES TO TRAIN AND DEVELOP EVALUATORS
Formal curriculum developed by CEE for ‘entry level’ Evaluators (Evaluator Internship Program)…not mandatory
Training needs identified for mid-level Evaluator
CEE partnering with others for training delivery…e.g. National Statistical agency (Statistics Canada); Canadian Centre for Management Development; Professional Associations (Canadian Evaluation Society (CES); department-led training; private sector trainers/course deliverers
Network of Evaluators encourages informal ‘communities of practice’ and sharing of ‘lessons learned’ across like evaluations
Link to other professional groups (e.g. Internal Auditors; Comprehensive Audit)
REL Solutions Inc. Brasilia, November 23, 2004

31. EDUCATING THE ‘USERS’ OF EVALUATION
‘Users’ of evaluation (in departments, central agencies, politicians)
Need to be educated on how, where, and why evaluation can/should be used/resourced
Importance of aligning ‘user’ expectations with reality of what can/will be delivered by Evaluators
Useful to use a variety of vehicles: briefing notes; presentation decks; training modules/workshops aimed at senior managers
Avoid technical jargon and deliver at appropriate level of understanding
Education an ongoing activity; not a ‘one-time’ event
Basic distinctions need to be made…e.g. difference between Internal Audit and Evaluation, an ongoing area of confusion for managers:  Internal Audit: Are we doing things right? (efficiency issues)  Evaluation: Are we doing the right things? (effectiveness and performance issues)
REL Solutions Inc. Brasilia, November 23, 2004

32. ORGANIZING AN EVALUATION UNIT
In Canada, federal departments determine their own arrangements
Much variation in terms of number of full-time staff hired vs use of consultants
Generally, most departments use some consultants… e.g. for surveys, specialized tasks, etc.
Important for Unit to have minimum core of full-time Evaluators
Wide range in size of Units dedicated to Evaluation… e.g. 1 person (small agency) to f/t staff (largest department)
Typical Evaluation Unit: 10 or fewer f/t staff, plus resources to hire consultants for Evaluation
REL Solutions Inc. Brasilia, November 23, 2004

33. MODULE THREE: Developing Capacity ‘To Use’ Evaluation: Overcoming Resistance and Making Effective Use of Evaluation
Outline of Module
Central Agency-driven demand for Evaluation
Steering the Evaluation function in departments
Credibility building for Evaluators
Developing a Comprehensive Evaluation Plan
Advisory Committees
Educating ‘Users’ of Evaluation
Participatory Evaluation and Accountability for Follow-up
Evaluations in ‘full public view’
REL Solutions Inc. Brasilia, November 23, 2004

34. CENTRAL AGENCY-DRIVEN DEMAND FOR EVALUATION
Deputy Minister in department faces variety of accountabilities requiring performance/results information
Required for good management practices and internal and external reporting
 Program-specific funding renewal decisions
 Horizontal and vertical expenditure reviews
 Accountability ‘report card’ of the DM (Management Accountability Framework (MAF)
 Departmental Performance Report to Parliament
 Ability to respond to queries of Parliamentary Committees; issues raised by the Auditor General
REL Solutions Inc. Brasilia, November 23, 2004

35. Steering the Evaluation Function in Departments
Evaluation and Audit Committee, chaired by Deputy Minister
Requirement of TBS for every government department
Senior level committee; formal Terms of Reference; meet minimum four times a year
Review and approve annual and multi-year plan for Evaluation
Senior managers identify new/changing priorities
Table completed Evaluation reports and management response
Forum to alert senior managers of areas of risk to department
Vehicle to inform/educate senior managers (senior ‘users’ of evaluation
REL Solutions Inc. Brasilia, November 23, 2004

36. Credibility Building for Evaluators
How Evaluators see themselves. How others see Evaluation.
Two critical factors for credibility:
 Understanding the needs/requirements of senior decision makers
 Delivering quality product on time
‘Answering the right questions at the right time’
Critical factors for success:
 Head of Evaluation well linked to senior management
 Appropriate balance between methodological rigour and delivery in a timely fashion
 Responsive to changing priorities
REL Solutions Inc. Brasilia, November 23, 2004

37. Developing a Comprehensive Evaluation Plan
Three key questions:
What are the guiding principles for evaluation planning?
What is the relevant ‘Evaluation Universe’?
How best to determine priorities and schedule evaluations over a multi-year period?
REL Solutions Inc. Brasilia, November 23, 2004

38. Useful vehicle to assist conduct of an Evaluation study
Advisory Committees: Building trust and avoiding surprises
Useful vehicle to assist conduct of an Evaluation study
Representation from broad set of stakeholders
Forum to clarify study objectives; table early findings; review conclusions and recommendations
Creates more ‘open’ evaluation process
Avoid surprises at the end
Helps ensure follow-through on study recommendations
REL Solutions Inc. Brasilia, November 23, 2004

39. EDUCATING THE ‘USERS’ OF EVALUATION
‘Users’ of evaluation (in departments, central agencies, politicians)
Need to be educated on how, where, and why evaluation can/should be used/resourced
Importance of aligning ‘user’ expectations with reality of what can/will be delivered by Evaluators
Avoid technical jargon and deliver at appropriate level of understanding
Education an ongoing activity; not a ‘one-time’ event
REL Solutions Inc. Brasilia, November 23, 2004

40. Participatory Evaluation and Accountability for Follow-up
Conduct of evaluation not ‘independent’ of program managers
New Policy (2001) removed ‘independence’; stressed ‘objectivity’ and ‘transparency’
Attempt to bring evaluation closer to managers…not a threat, but a tool to assist good management practices
Puts onus on Managers for ongoing performance monitoring as well as accountability for implementing evaluation recommendations
REL Solutions Inc. Brasilia, November 23, 2004

41. Evaluation ‘In Full Public View’
Broad demand for greater accountability in government
Evaluation reports accessible to wide audience (via ATIP)
Web posting increases accessibility to general public
Evaluation Policy stresses ‘transparency’
Requires a culture allowing for:
 full disclosure of information
 unbiased analysis and reporting
 openness in government
REL Solutions Inc. Brasilia, November 23, 2004

42. MODULE FOUR: Evaluation versus Performance Measurement/Monitoring: A Critical Distinction
Outline of Module
Historical Context
Definitions and Distinctions
Departmental Performance Framework
Using Logic Models to Derive Performance Measures
Results Measurement Continuum Model
Roles and Responsibilities for Delivery
General Strategy for Implementing M&E Systems
REL Solutions Inc. Brasilia, November 23, 2004

43. Historical Context
Mid-to-late 90’s onward: Increasing government emphasis on ‘performance measurement’, ‘monitoring’ and ‘results’
Drivers: Results based management; Greater accountability to Parliament and to citizens
Put onus on managers to measure and report on performance/results
Required clarification of terminology, roles and responsibilities in the process
REL Solutions Inc. Brasilia, November 23, 2004

44. Definitions and Distinctions
How is Evaluation different from Performance Monitoring?
How does one link to the other?
Are they both necessary tools to support good management practices and accountability?
REL Solutions Inc. Brasilia, November 23, 2004

45. Departmental Performance Framework:
Departmental Performance Framework: Basis for Ongoing Performance Monitoring
Planning Reporting and Accountability Structure (PRAS) provides broad departmental framework
Policy being updated (Fall 2004) to Management Resources and Results Structure (MRRS)
Business line structure: architecture and basis for cascading set of performance measures through organization
Brings greater focus to managers on measuring ‘results’
Evaluation specialist has the skill set to facilitate this process
REL Solutions Inc. Brasilia, November 23, 2004

46. Using Logic Models to Derive Meaningful Performance Measurement
Performance indicators derive from logic model
What needs to be measured and reported?
 What are the key questions that need to be answered?
 How will it be used? By whom? How often?
‘Monitoring’ versus ad hoc ‘Evaluation’
Performance measurement development/implementation
 Technical cost considerations
 An iterative process
REL Solutions Inc. Brasilia, November 23, 2004

47. Results Measurement Continuum Model
REL Solutions Inc. Brasilia, November 23, 2004

48. Roles and Responsibilities for Delivery
Managers ultimately accountable for performance measurement and reporting
Evaluation specialists support to the development and implementation of the performance measurement system through:
 creating PM understanding---learning events, seminars, information packages
 facilitating PM development
 applying needed rigour and discipline---‘challenge’ guidelines
Evaluation study as one option for measuring performance, where:
 ad hoc study more cost-effective
 ‘cause and effect’ (attribution) issue important
IM/IT specialist---advice re systems and data capture issues
REL Solutions Inc. Brasilia, November 23, 2004

49. Challenges: political; organizational; technical; resourcing
General Strategy for Implementing Evaluation and Performance Monitoring Systems
Challenges: political; organizational; technical; resourcing
Useful to do a ‘readiness assessment’:
 Is there an enabling environment: political will; commitment; a champion
 Technical capacity, including HR skills
 Necessary infrastructure to produce, collect, analyze and report relevant information
 Necessary infrastructure/institutions to ‘use’ evaluation and monitoring information
Priority needs to be established from the top
Resourcing: a corporate responsibility. Needs to be linked to accountability and modern management practices
REL Solutions Inc. Brasilia, November 23, 2004

50. MODULE FIVE: Results Based Management and Accountability Framework (RMAF): An Important Vehicle
Outline of Module
Why have we introduced the RMAF?
RMAF Components
Using the RMAF to Greatest Gain
Helpful Hints for RMAF Development/Implementation
REL Solutions Inc. Brasilia, November 23, 2004

51. Why have we introduced the RMAF?
Tool designed to assist the move to results measurement/reporting
Broader government commitment on results reporting (Results for Canadians)
Formal requirement for certain programs
Facilitates development to performance measurement and establishes priority and timeframe for evaluation
REL Solutions Inc. Brasilia, November 23, 2004

52. RMAF Components All RMAFs have five major sections: A: Profile
B: Logic Model
C: Ongoing Performance Measurement Strategy
D: Evaluation Strategy
E: Reporting Strategy
Together, describes the rationale, resources and performance logic of program or initiative
Presents the measurement, evaluation and reporting strategies that will be implemented to track progress
REL Solutions Inc. Brasilia, November 23, 2004

53. Using the RMAF to Greatest Gain
A collaborative effort of key program stakeholders to clearly articulate:
 Program objectives in terms that allow for measuring/monitoring performance/success
 Linkages of program activities to program goals
Brings technical (Evaluator) and non-technical (Manager) to common agreement on meaningful measures of performance and a measurement strategy
Up-front investment in RMAF pays off in
 Clarifying program role and resourcing needs
 Establishment of meaningful measurement systems
 More systematic and disciplined monitoring of program performance
 Improved management information, supporting RBM
 Improved accountability, both to internal and external stakeholders
(senior managers; elected officials; citizens)
REL Solutions Inc. Brasilia, November 23, 2004

54. Helpful Hints for Implementing RMAFs
Accountability with RMAF development; performance monitoring; reporting…lies with Manager
Carry out development and implementation as collaborative effort (Manger; Evaluation specialist; IM/IT specialist)
Flexibility, particularly in early stages:
 ‘acceptable’ RMAF versus perfection
 performance measurement as an ‘iterative’ process
Select realistic performance indicators based on what will provide the best information, not the easiest to implement
REL Solutions Inc. Brasilia, November 23, 2004

55. Final Thoughts
Evaluation and performance measurement: Important link to effective pubic sector governance
Broad application across:
 individual programs
 sectors or business lines
 whole organization
Not an end in itself, a means to an end
Evaluation capacity building to a self-sustaining state requires:
 Commitment/political will
 Resources (both $ and HR)
 Time
 Energy
REL Solutions Inc. Brasilia, November 23, 2004