1. Department of Performance Monitoring and Evaluation
The Presidency
Department of Performance Monitoring and Evaluation
Developments in Government: National Evaluation System and broader context
Dr Sean Phillips
Director General
Presentation to SAMEA
19 September2013

2. Focus of DPME to date M&E of national priorities
Management performance M&E
M&E of front-line service delivery
Government-Wide M&E System
Plans for the 12 priority outcomes (delivery agreements)
Monitoring (ie tracking) progress against the plans
Evaluating to see how to improve programmes, policies, plans ( evaluations, then 15 per year)
Monitoring of experience of citizens when obtaining services (joint with provinces), including citizen-based monitoring
Presidential Hotline – analysing responses and follow-up
Assessing quality of management practices in individual departments (MPAT) at all levels of government
Moderated self assessment and continuous improvement
M&E platforms across gov – nationally, provincially
Data quality/Structures of M&E units/Capacity development
Emerging focus on (implementation) programmes
National Evaluation System
National M&E Policy Framework
To date my Department has been focused on three levels of performance monitoring and evaluation. At the highest level, our focus since 2010 has been on facilitating the development of plans for priorities such as basic education, health, reducing crime and creating employment.
The main aims of this initiative have been to increase the strategic focus of government, to introduce results-based planning on a sectoral basis, to increase coordination between departments and across spheres of government, and to use M&E of progress against the plans to foster a culture of evidence-based continuous improvement.
The 10 and 15 year reviews of the post-apartheid government’s performance which were carried out by the Presidency came to the conclusion that a key challenge in SA is implementation of policies, which in turn is related to management weaknesses. At the next level we have therefore introduced a management performance assessment mechanism, informed by the Canadian Management Assessment Framework.
The methodology is based on self assessment against standards in key management areas, coupled with verification against secondary data. It is being implemented in partnership with the provinces and the results of the assessments will be presented to Cabinet and the Provincial Executive Councils on an ongoing basis, together with monitoring reports on the implementation of improvement plans.
A key political imperative in South Africa is to improve the quality of services provided directly to citizens. We are therefore also carrying out M&E at the level of frontline service delivery by visiting service delivery sites together with the Offices of the Premier in the provinces.
The focus is on whether service delivery standards are in place and are being adhered to. The results and monitoring reports on the implementation of improvement plans are similarly presented to Cabinet and the Provincial Executive Councils. We have also introduced a Presidential Hotline through which the public can lodge service delivery complaints. In future we would like to introduce more citizen-based monitoring, in partnership with civil society organisations.

3. Timelines
2010 DPME established. 12 outcomes agreed, Minister’s performance agreements, delivery agreements for each outcome, quarterly reports start, using traffic light system 2011 Systems for Management Performance Assessment created, monitoring of front-line services, and Evaluation Policy Framework. MPAT assessment of 103/156 national and provincial departments 2012 Reports against all of these, first evaluation completed. First National Evaluation Plan (NEP) approved for 2012/13 with 8 evaluations. Second NEP approved for 2013/14 to 2015/16 with 15 evaluations in 2013/14. 8 evaluations start for 2012/13 MPAT assessment of all 156 national and provincial departments evaluations underway or being scoped for 2013/14 Work with Treasury to develop system for performance dialogue Systems for monitoring local government and citizen-based monitoring being piloted Government-wide M&E Policy Framework drafted

4. Why evaluate?
Improving policy or programme performance (evaluation for continuous improvement):
this aims to provide feedback to programme managers.
Evaluation for improving accountability:
where is public spending going? Is this spending making a difference?
Improving decision-making:
Should the intervention be continued? Should how it is implemented be changed? Should increased budget be allocated?
Note Minister clear he wants all 4
Evaluation for generating knowledge (for learning):
increasing knowledge about what works and what does not with regards to a public policy, programme, function or organization.

5. Key aspects of the SA national evaluation system
Departments submit proposals for interventions to evaluate (policies, programmes, projects) – as they have to own the evaluation and implement the findings. 3rd parties, eg Treasury, Parliament can propose evaluations, but departments should normally submit
Selection by cross-government Evaluation Technical Working Group – based on importance (either by scale or because strategic or innovative)
Evaluations must be made public unless security concerns
All evaluation reports in the national system go to Cabinet (which approves the Plan)
To ensure independence:
Evaluations implemented as partnership between department(s) and DPME
Steering Committee makes decisions on evaluation not department
External service providers undertake the evaluation reporting to the Steering Committee
To ensure quality:
Peer reviewers (normally 2) per evaluation
Evaluation panel, standards, guidelines, training etc
Quality assessment once completed
Joint funded – department and DPME, in some cases donors
There must be an improvement plan which is monitored

6. Implementation evaluation
Different types of evaluations related to questions around the outcome model
Economic Evaluation
What are the
cost-benefits?
Impact evaluation
Has the intervention had impact at outcome and impact level, and why
Implementation evaluation
- what is happening and why
If presenting the following slides on each type of evaluation, then can be brief here. Mention evaluation synthesis too.
DESIGN
Diagnostic
what is the underlying situation and root causes of the problem
Design evaluation
Does the theory of change seem strong?

7. Progress with Evaluations
National Evaluation Plan :
2012/13: 7 evaluations
2013/14: 15 evaluations
2014/15: 16 evaluations selected but not yet approved by Cabinet
Evaluations completed - ECD completed June 2012 (on DPME website), Grade R and Business Process Outsourcing reports completed and on way to Cabinet
17 other evaluations underway or being scoped from 2012/13 and 2013/14
Audit of evaluations from 2006 identified 83 evaluations , quality assessment tool developed and applied, 71 passed and are up on the DPME website, and assessment contributed to the awards for best evaluation yesterday. Evaluation repository launched yesterday

8. Progress with National Evaluation System
>12 Guidelines and templates - ranging from Guideline on TORs to Guideline on Improvement Plans
Standards for evaluations and competences drafted, and standards have guided a quality assessment tool
2 courses developed, over 250 government staff trained
Evaluation panel developed with 42 organisations which simplifies procurement – however evaluation capacity is still limited – and a particular problem that few universities are bidding for evaluations
Gauteng, W Cape provinces have developed provincial evaluation plans. DPME is working with other provinces who wish to develop PEPs, starting with Free State
3 departments have developed a departmental evaluation plan (dti, DST, DRDLR)

9. Evaluation challenges emerging
Overall the system is working well but some challenges. These include:
Poor programme plans (for the government programmes which are being implemented) and so difficult to evaluate - need for minimum standards for programme plans – DPME has issued guideline on this
Poor communication channels - programme managers often not aware of the possibility of conducting evaluations on their programmes
Some senior managers wary of evaluation and don’t see it as an opportunity to improve their performance
Making sure evaluations proposed are strategic ones and that key sectors covered
Sometimes departments not budgeting for evaluations and expecting DPME to provide all the money
Departments not planning ahead – very important for impact evaluations in particular where need to plan 3+ years ahead, also affects how rollout happens
Reluctance to rollout in carefully planned way which facilitates impact evaluation. To be clear on impact must compare with/without the intervention

10. Conclusions
Interest is growing – more departments getting involved, more provinces, and more types of evaluation
Some 35 evaluations are underway or planned
Development of guideline for planning implementation programmes and for depts to do design evaluation will potentially have very big impact – will build capacity in departments to undertake these
Challenges emerging as the evaluation reports start being finalised and raise challenges which some departments are sensitive about results
Parliament is playing a key oversight role in this regard – committees are starting to request departments to present the evaluation results to them