1. Evaluating the Effectiveness of OECD Economic Surveillance John C. Whitehead School of Diplomacy and International Relations Seton Hall University

2. PURPOSE Our objective is to understand the overall impact of OECD economic surveillance on member countries. We analyzed economic surveys for randomly selected OECD Country Members across the three most recent review cycles. We determined the extent to which OECD recommendations were implemented, and we also evaluated the content of these reports.

3. IMPORTANCE The OECD is understudied relative to other international economic organizations. Given the global economic crisis, surveillance is important. What are its strengths and weaknesses, and how can it be made better? OECD surveillance is conducted every 18 months, and functions by peer review. How well does peer review work?

4. OUR APPROACH Two project team members were randomly assigned to each of 24 OECD countries. Each team member performed two tasks: Assess the extent to which past recommendations by the OECD were adopted or pending in the two most recent Economic Surveys. Assess the content of surveillance by completing a survey of the three most recent Economic Surveys. Each country team initially worked independently, then collaborated to develop a composite assessment of surveillance effectiveness and surveillance content. Project data is available on our website http://oecdproject.wordpress.com/

5. COUNTRIES IN STUDY Twenty-Four randomly selected OECD Member Countries (out of 34 total) AustraliaFranceItalySlovakia AustriaGermanyJapanSpain CanadaGreeceKoreaSweden ChileHungaryMexicoTurkey Czech RepublicIcelandNorwayUnited Kingdom DenmarkIrelandPortugalUnited States

6. MEASURING EFFECTIVENESS Assessment of “Progress in Structural Reform” tables in two most recent Economic Surveys. Coded actions taken by member countries as either “adopted” (any official policy change in the spirit of OECD recommendations) and/or “pending” (recommendations pending, proposed, or under review) Total of 1680 unique recommendations by OECD (average of 70 per country)

7. OVERALL FINDINGS FOR EFFECTIVENESS 52% ADOPTED (Median: 53%) Range: Portugal 75%, Slovakia 25% 28% PENDING (Median: 27%) Range: UK 61%, Slovakia 15%

8. SPECIFIC FINDINGS FOR EFFECTIVENESS We disaggregated these results in three ways: Changes in effectiveness over time comparing the two most recent reviews Comparing effectiveness in crisis and non-crisis countries Comparing effectiveness across type of economic issue

9. CHANGES IN EFFECTIVENESS OVER TIME Recommendation Adoption 11 Improved, 12 Worsened, 1 Stayed the Same Overall worsening with net average of -2% Range: Turkey +32%, France -29% Recommendations Pending 10 Improved, 14 Worsened Overall worsening with net average of -1% Range: Italy +32%, Canada -25%

10. CRISIS VS. NON-CRISIS COUNTRIES We classified countries in crisis based on whether they received IMF adjustment loans: Greece (2011) Hungary (2010) Iceland (2009, 2011) Ireland (2009, 2011) Turkey (2008, 2010) Mexico (2009-2011)

11. EFFECTIVENESS IN CRISIS AND NON-CRISIS COUNTRIES % RECOMMENDATIONS ADOPTED % RECOMMENDATIONS PENDING Greece 201162%36% Hungary 201073%8% Iceland43%23% Ireland51%30% Turkey66%20% Mexico66%16% TOTAL:59%24% Non-Crisis Mean51%29%

12. CRISIS VS. NON-CRISIS COUNTRIES We compare within-case variations for Greece and Hungary These countries had Economic Surveys in years preceding crisis as well as crisis years. Greece: Adopted 16% more recommendations in crisis years Hungary: Adopted 14% more recommendations in crisis years Implication: Surveillance can make a difference for countries in crisis.

13. EFFECTIVENESS ACROSS ISSUE AREAS % RECOMMENDATIONS ADOPTED % RECOMMENDATIONS PENDING Social Policy*5333 Fiscal Policy5324 Financial Sector5621 Market Competition4827 *Social Policy includes education, healthcare, housing, labor, environment, immigration, human capital, pensions, family, aging, and infrastructure.

14. SURVEILLANCE CONTENT Assessed three most recent OECD Economic Surveys Coded each survey based on four questions: 1). Is the political and social context of the country explained? 2) Does the economic survey include a self-contained comprehensive statement assessing the effectiveness of past surveillance in the country? 3) What reasons (if any) were provided where past OECD advice was not followed? 4) Are dissenting views by the authorities clearly identified and explained?

15. OVERALL FINDINGS FOR SURVEILLANCE CONTENT Question 1 91% of Surveys had political or social context. Question 2 86% of Surveys had a comprehensive effectiveness statement. Question 3 61% of Surveys contained reasons for past non-compliance. Question 4 23% of Surveys contained dissenting views by state authorities.

16. SPECIFIC FINDINGS FOR SURVEILLANCE CONTENT We disaggregated these results in two ways: Changes in content over time comparing the two most recent reviews Comparing content in crisis and non-crisis countries

17. CHANGES IN SURVEILLANCE CONTENT OVER TIME Economic Survey Year 1 Economic Survey Year 2 Economic Survey Year 3 Question One (Q1)91%87%96% Question Two (Q2)58%100% Question Three (Q3)58%62% Question Four (Q4)16%25%29%  Increased transparency over the span of the most three recent reports  Can improve surveillance by including more dissenting views which would provide a detailed explanation why past advice was not adopted

18. SURVEILLANCE CONTENT IN CRISIS AND NON-CRISIS COUNTRIES Q1Q2Q3Q4 Greece (2010)100% Hungary (2011)100% Iceland (2009, 2011) 100% 0% Ireland (2009, 2011)100% 50%0% Mexico (2009, 2011)100% 0%50% Turkey (2008, 2010)100% 50%0% TOTAL100% 66%41% Non-Crisis Mean91%83%59%24%

19. CRISIS VS. NON-CRISIS COUNTRIES More transparency about non-adoption of reforms in crisis countries. Greece and Hungary had detailed discussions of dissents in pre- crisis years as well Finding on Q4 for greater transparency about dissents should be tempered.

20. FINDINGS ON SURVEILLANCE CONTENT Surveillance reports include contextual information. Surveillance reports provide a benchmark for evaluating prior recommendations. Explanations for why past recommendations were not adopted appear in only 3 out of 5 reports. Discussions of non-adoption are more anecdotal than substantive.

21. FINDINGS ON SURVEILLANCE CONTENT Surveillance reports include contextual information. Surveillance reports provide a benchmark for evaluating prior recommendations. Explanations for why past recommendations were not adopted appear in only 3 out of 5 reports. Discussions of non-adoption are more anecdotal than substantive.

22. CAN BETTER CONTENT PRODUCE MORE EFFECTIVE SURVEILLANCE? Adoption and pending percentages in the most recent Economic Survey by whether the prior survey had positive answers to Questions 3 and 4. % Recommendations Adopted % Recommendations Pending Questions 3 & 4 = NO57%23% Questions 3 & 4 = YES60%45%

23. POLICY RECOMMENDATIONS Economic Surveys should take a multi-year perspective to their recommendations. The issue of non-adoption of reforms should receive greater attention. More consideration should be given to the domestic political context when developing recommendations.

24. RECOMMENDATION 1: MULTI-YEAR PERSPECTIVE Economic Surveys should take a multi-year perspective to their recommendations. Helps to track whether pending recommendations become adopted in subsequent years Helps call attention to potential for reform backsliding Could be implemented thematically initially

25. RECOMMENDATION 2: ADDRESSING NON-ADOPTION The issue of non-adoption of reforms should receive greater attention. Economic Surveys should include a section on why prior recommendations have not been adopted. More detail on whether countries dissent with recommendations would make dialogue more substantive Consider developing a publicly available dataset (i.e. the IMF’s MONA data) Use greater media outreach

26. RECOMMENDATION 3: TAKE POLITICS SERIOUSLY More consideration should be given to the domestic political context when developing recommendations. Consider prioritized recommendations Consider tradeoffs between optimal reforms and politically feasible reforms

27. PROJECT TEAM MEMBERS

28. FOR MORE INFORMATION Project Website: http://oecdproject.wordpress.com/ Email address for Project Team: whiteheadOECDproject@gmail.com