1. Ian Goldman Head: Evaluation and Research AEA, 17 October 2012
The Presidency
Department of Performance Monitoring and Evaluation
Using evaluation to influence policy and implementation and accountability:
The case of South Africa
Ian Goldman
Head: Evaluation and Research
AEA, 17 October 2012

2. Outline
Background – emergence of evaluation system
Key findings and side effects
Where next

3. Political context for M&E in SA
1 Semi-federal system
2 New administration in 2009 focuses on M&E
SA is a semifederal state with 9 provinces and around 280 local governments, all autonomous spheres. This means that implementation of policy requires cooperation across spheres, including for M&E
As part of the public service reform process, some attention was paid to developing M&E between 1994 and However, there was a major increase in emphasis on M&E after 2009, as the new administration saw M&E as one way of improving the performance of the public service
As a result of the increased emphasis on performance and M&E from 2009, the Department of Performance Monitoring and Evaluation was created in the Presidency in It has an annual budget of approximately 20 million dollars and a staff complement of approximately 200 people. It is part of the executive
3 DPME established in 2010

4. Evaluations not being used
Evaluation not informing planning, policy-making and budgeting sufficiently, so we are missing the opportunity to improve Government’s effectiveness, efficiency, impact and sustainability.

5. National Evaluation Plan
July 2011 – study tour to Mexico, Colombia, US November National Evaluation Policy Framework approved by Cabinet June 2012 First evaluation completes, First National Evaluation Plan with 8 evaluations approved November Second National Evaluation Plan approved with 15 evaluations Currently - 3 evaluations completed, 4 close to completion, 16 others underway, next 15 for 2014/15 selected

6. System design Evaluation and Research Unit in DPME/Presidency
National Evaluation Plan
Evaluation and Research Unit in DPME/Presidency
Policy Framework setting out common approach/language
Focus on utilisation
Maximise ownership so evaluations proposed by depts
Cost sharing by dept/DPME
Improvement plans, evaluations public
Focus on strategic priorities through 3 year National and Provincial Evaluation plans
National Evaluation System
Support by cross-Government Evaluation Technical Working Group
Use of Steering Committee as semi-independent bodies to make decisions, with DPME
Guidelines, standards, training, peer reviewers, quality assurance, evaluation repository etc - minimum standards
Widen use through Provincial/Departmental Evaluation Plans

7. Early days but key emerging results
First evaluations are being influential
Early Childhood Development (ECD) evaluation changing policy and resources for services
Grade R leading to rethinking of strategies
Business Process Services leading to tweaking implementation
Change
President, Parliament and party structures anxious to get results
Pressure
Need to improve planning system (Programmes, Diagnostics prior)
Logframe
Expenditure reviews
And better understand budgets

8. Key findings - side effects
Some fear leading to gaming strategies:
Fear that evaluations public
Running parallel internal evaluation
Seeking to influence sample
Seeking to delay findings being made public
Senior managers keeping away
Challenge of adequate supply of good quality evaluators
Concern at use of consultants

9. Way forward This year is critical – monitor and document:
How to deal with gaming – link to management process
How government clusters/Cabinet deal with difficult results (and elections are next year)
Response from Portfolio Committees
Some elements planned
Demand - training in analysis and use of evidence for Directors-General/Deputies as well as Parliament
Demand – support provinces/depts
Supply of evaluators – interact with universities
Attitude - encourage evaluators to express constructively
Planning - training around programme planning
Budget - project on expenditure reviews
Learning - bring stakeholders together to reflect on learnings

10. Contact Dr Ian Goldman Head: Evaluation and Research
Department of Performance M&E, The Presidency

11. Additional slides

12. Implementation evaluation
Different types of evaluations related to questions around the outcome model
Impact evaluation
Has the intervention had impact at outcome and impact level, and why
Economic Evaluation
What are the
cost-benefits?
Implementation evaluation
- what is happening and why
If presenting the following slides on each type of evaluation, then can be brief here. Mention evaluation synthesis too.
DESIGN
Diagnostic
what is the underlying situation and root causes of the problem, what are options?
Design evaluation
Does the theory of change seem strong?