Mexico 8th Meeting of the Steering Committee of INTOSAI Committee on Knowledge Sharing and Knowledge Services Mexico 7-9 September 2016 Cour des Comptes - Evaluation Working Group - 15/16 october 2015

Cour des Comptes - Evaluation Working Group – 7-9 September 2016 OUR AMBITION Long term : «help SAIs which practice evaluation or wish to do so by providing them with methodological tools and practical recommendations » (creation of EWG in 1992) Immediate : approval by KSC Steering Committe of the endorsement version of the guidelines Cour des Comptes - Evaluation Working Group – 7-9 September 2016

Cour des Comptes - Evaluation Working Group – 7-9 September 2016 I – Where we are Achievements Major issues II – Next steps Cour des Comptes - Evaluation Working Group – 7-9 September 2016

Cour des Comptes - Evaluation Working Group - 15/16 october 2015 I. Where we are Cour des Comptes - Evaluation Working Group - 15/16 october 2015

Cour des Comptes - Evaluation Working Group – 7-9 September 2016 Achievements (1/3) Sept 2012– 4th KSC Decision of the 4th KSC in Sept. 2012 To aim at a definition of what is evaluation and what could be common principles of Public policies evaluation, based on different experiences To build and to strengthen links with the Performance Audit WG, because the process of elaborating standards of audit performance is far more ahead To draft guidelines for the SAI’s willing to improve their capabilities in this field Cour des Comptes - Evaluation Working Group – 7-9 September 2016

Cour des Comptes - Evaluation Working Group – 7-9 September 2016 June 2013 – EWG Achievements (2/3) Elaboration of a preparatory document Discussion of its content Agreement on some key points : It is neither desirable not possible to achieve an ISSAI, at least in a short-term It is necessary to go further into the definition and the methodology of evaluation : Some aspects of evaluation clearly don’t belong to performance audit, for example the impact of a policy : how can we define and determine what is an impact ? The process of evaluation involves a wide range of stakeholders : how shall we manage this complexity ? Cour des Comptes - Evaluation Working Group – 7-9 September 2016

Cour des Comptes - Evaluation Working Group – 7-9 September 2016 2014/2016 – meetings in Paris Achievements (3/3) Three meetings in Paris respectively in November 2014, June 2015 and June 2016 Objective of the meetings : reach a consensus on the distinctive characteristics and methodology of evaluation; modify the preparatory document and summarize it (Nov 2014) Ultimate Goal : establish guidelines (not a « standard ») to be endorsed by KSC, GB 2015 and INCOSAI 2016 (June 2015 for the project proposal and June 2016 for the endorsement version) Several amendments both in form and substance were adopted resulting in the current endorsement version The document, primarily intended to be a methodological guide of the different types of evaluations conducted by member countries now focuses mainly on the evaluation of public policies. Cour des Comptes - Evaluation Working Group – 7-9 September 2016

Crucial Importance of knowledge sharing Major issues (1/3) A questionnaire to collect case studies and practical information on Evaluation methodology and experiences (elaborated by a sub-group composed of 5 SAIs) was and sent to 191 SAIs. The main lessons learnt (44 SAIs have answered) were presented to KSC in 2012. One SAI out of two deals somehow with evaluation Cour des Comptes - Evaluation Working Group – 7-9 September 2016

Crucial Importance of knowledge sharing Major issues (2/3) A field where definitions are complex, practices are different, comparisons are specifically useful. Some pitfalls to avoid : focus only on definition problems (even if they are important) ; set wrong ambitions ; lose ourselves in never-ending discussions between standard and guidelines ; complexify instead of clarify. Cour des Comptes - Evaluation Working Group – 7-9 September 2016

Crucial Importance of knowledge sharing Major issues (3/3) Guidelines are even more necessary since evaluation of public policies could involve specific risks: close to politics, even to political philosophy far from SAIs usual methods has to deal with new and not always robust techniques Cour des Comptes - Evaluation Working Group – 7-9 September 2016

Cour des Comptes - Evaluation Working Group - 15/16 october 2015 II. NEXT STEPS Cour des Comptes - Evaluation Working Group - 15/16 october 2015

AND NOW THE NEXT STEPS – Four stages in developing INTOSAI GOV Project proposal Drafting of project proposal and preliminary draft of INTOSAI GOV by relevant working group (i.e. EWG) Approval of project proposal and preliminary draft of INTOSAI GOV by relevant steering committee (i.e. KSC) Exposure draft Drafting of exposure draft by EWG Approval of exposure draft by KSC Steering Committee Exposure period of exposure draft Endorsment version Assessment of comments received on exposure draft Drafting of endorsement version and executive summaries by EWG Approval of endorsement version by KSC Steering Committee Approval of endorsement version by Governing Board Final INTOSAI GOV Endorsement of final INTOSAI GOV by INCOSAI Cour des Comptes - Evaluation Working Group – 7-9 September 2016

STAGE 1 –PROJECT PROPOSAL Drafting of the project proposal by EWG 15 June 2015: EWG meeting in Paris (discussions on the last drafts of the project proposal and the guidelines on the evaluation of public policies). 16 June to July 14 2015 : drafting of the project proposal and the guidelines according to the received comments during the last EWG meeting, approval of the resulting document by the members of the group. Approval of the Project Proposal by KSC 15 July 2015: project proposal and guidelines sent to KSC for approval. 15 October 2015: KSC Steering Committee (approval of the project proposal). Cour des Comptes - Evaluation Working Group – 7-9 September 2016

Cour des Comptes - Evaluation Working Group – 7-9 September 2016 STAGE 2 : exposure draft Drafting of the exposure draft by EWG 16 October to 31 October 2015 : drafting of the exposure draft by EWG Approval of the exposure draft by KSC 1 November 2015 : exposure draft sent to KSC for approval 17 February 2016: exposure draft approved by the KSC, document posted on the issai.org website 17 February - 14 May 2016: exposure period : 149 comments were received, from SAIs and international organizations. Out of the 6 international organizations that were contacted, 3 of them sent us comments: the OECD (67 comments), the UN (23 comments) and the World Bank (6 comments). Cour des Comptes - Evaluation Working Group – 7-9 September 2016

STAGE 3 : ENDORSEMENT VERSION (1/2) Overview of comments received by EWG 14 May to 31 May 2016 : analysis of the comments received during the exposure period 1 June 2016 : new draft of the guidelines to be sent to EWG members for comment with table including comments and responses to these comments. 20 June 2016 : EWG meeting (discussions on the new draft of the guidelines): 10 members of the group present, minor changes on the document, technical rather than substantial. Drafting of the endorsement version of the guidelines by EWG 21 June to 31 June 2016 : drafting of the endorsement version according to comments received during the last EWG meeting 1 July 2016 : Draft endorsement version sent to EWG members for comment. 11 July 2016 : drafting of the endorsement version according to comments received from EWG members (done by email exchange) 13 July to 31 August 2016: Translation of the final endorsement version in the 5 official languages of INTOSAI. Cour des Comptes - Evaluation Working Group – 7-9 September 2016

Stage 3 : endorsement version (2/2) Approval of the endorsement version by KSC and by the Governing Board 15 August 2016: Endorsement version sent to KSC for approval December 2016 : Endorsement version to be sent to Governing Board for approval Stage 4 : endorsement of final intosai gov XXI INCOSAI, Abu Dhabi, 2016 Cour des Comptes - Evaluation Working Group – 7-9 September 2016

Cour des Comptes - Evaluation Working Group – 7-9 September 2016 Thank you for your attention. INTERNATIONAL@ccomptes.fr, Stéphane gaillard, auditor, deputy director, international relations departement Cour des Comptes - Evaluation Working Group – 7-9 September 2016