Presentation on theme: "Building Evaluation Capacity in Africa: Strategies and Challenges"— Presentation transcript:
1Building Evaluation Capacity in Africa: Strategies and Challenges By Dr. Frannie A. LéautierExecutive SecretaryThe African Capacity Building Foundation (ACBF)AFDB Evaluation Week on the theme: Evaluation for DevelopmentDecember 3 – 6, 2012, Tunis, Tunisia
2How should evaluation be structured “ catching the short dash but sustaining the long march”
3Order of PresentationGeneral overview of Evaluation Capacity issues in AfricaIntroduction (The challenge)Status of Evaluation in AfricaWhy Evaluate?Levels and Dimensions of Evaluation CapacityOpportunitiesStrategiesChallengesThe ACBF CaseConclusion and Way forward
5Introduction: The Challenge Terrain, Demography,Infrastructure, Admin UnitsProductionEnvironment &ConstraintsProductionSystems & PerformanceLinkage toMacroModelsAggregate to FPUsInterventions/ResponsesDistribution of Welfare BenefitsAgroecological ZonesElevationYield Responses to Inputs, Management, CCFarming SystemsProfitability of small scale irrigationCrop Distribution & YieldsCropland extent & intensityCrop Suitability: Rainfed WheatSlope, aspect, drainageValue of Production per Rural PersonQuantity of Nutrients RemovedPests & Diseases (Maize Stem Borer)Fertilizer ProfitabilitySettlements, ports, marketsDrought Incidence & SeverityRoad, rail, river, ICT networksRunoffMarket travel times & costsPort travel times & costsAdministrative UnitsSource: HarvestChoice/IFPRI 2010
6Strategic Choices and Evaluation There are generally two approaches to strategy making—the root method and the branch method--evaluation practice needs to be aligned to the strategy approach in order to capture impactsThe root method relies on the ability to define objectives very well, outline a range of options in a comprehensive manner, evaluate the options and select from the one that maximizes the attainment of the objective.The branch method involves building out step by step and in small degrees from the current situation. It is the state of practice used by leaders in political and complex environments. It is many times referred to as “muddling through”.Lindblom (1959) presents us with the virtues of muddling through in his example of dealing with inflation and Kay (2009) illustrates how companies have used these strategies with very different effect, offering learning strategies for individuals to be better at managing portfolio risks by knowing why some companies succeed and some fail.
7Evaluation and the Science of Muddling Through There would also be a premium on piloting and learning from trials that can be monitored and scaled upSuch an approach would suit very well post conflict and fragile states, investments with long gestation periods like infrastructure and education, as well as investments in building capacityGrowing out a results chain gradually and systemically over time would be valued as it would show a resilient and robust approach to resultsIf we were to use the science of muddling through then what would matter would be trends in results achievements and the sustainability of results
8The Challenge Cont’dUnderstanding the root causes, impediments, and enablers to Africa's development require good data analysisGoing beyond physical achievement to also gauge long term achievement and implications for the future requires good quantitative and qualitative informationNeed to deal with Africa’s problems of yesterday, today and tomorrow simultaneously required evidence- based information
9Status of Evaluation in Africa Evaluation capacity is varied across the continent- the statement of lack of evaluation capacity can not be applied across boardThere are increasing expectations for countries to develop national evaluation systemsPlanning and monitoring have received more resources and have been strengthened more that evaluationTechnical soundness is a cornerstone for crediblility but does nt gurantee useEvaluation of public policies and programs must beembedded in the political process. Both technical and political dimensions of evaluation needs to be considered.
10Why Evaluate?Speaking the Truth to the KingEvaluation is a powerful tool for public accountability and learningAfrican Governments have improved their knowledge and appreciation of the value of evaluations as enhancing the efficiency and effectiveness of public policies, programs and projects.Evaluation provides a means of assessing which public initiatives work well, which do not work well and most importantly, why.provide credible way of demonstrating the outcome of government effort in a transparent and consistent manner including how public resources were used and what informed the prioritization in the allocation of public resources.
11Levels of Evaluation Capacity Enabling environment, organizations and individualsSystemic factors, i.e., relationships between theIndividual level(experience, knowledge & technical skills)Organizational level(systems, procedures & rules)Influences by means of incentives it createsEnabling environment(institutional framework, power structure & influence)Successful capacity development requires not only skills &organizational procedures, but also incentives & good governance11(OECD paper )
12Seeing Through LensSupply LensDemand LensMichael Q. Patton, 2010
13Dimensions of Evaluation Capacity Evaluation capacity can also be categorized into three dimensions:Capacity for Conducting Evaluations - Conducting an evaluation involves both producing the study and communicating and disseminating it, which requires specialized technical capacity;Capacity for Managing Evaluations: Managing evaluations requires a broad understanding of evaluation but can be done without the specialized skills to conduct evaluation; andCapacity for Using Evaluations -The capacity to use evaluations is completely different; users of evaluations are mainly decision-makers and in some cases policymakers.
14Building Evaluation Capacity- Key Considerations Evaluation capacity must be ‘unbundled’: Different evaluation capacities should be taken into account. One size fit all approach should be avoided. It is important to distinguish between the capacity to manage evaluations and the capacity to conduct them, as well as capacity to use evaluations. These are all different capacities; it is not practical to lump them all together under the single term ‘capacity’Individual training on how to conduct evaluations is not sufficient for development of national evaluation capacity: For quite some time evaluation capacity was reduced to ‘the capacity to carry out evaluations’ and to a certain extent this continues today. Experience shows that enhancing individual capacities without strengthening the organization and the enabling environment can be counterproductive as the individual experts may be frustrated by the institutional arrangement and processesIndividual training on how to conduct evaluations is not sufficient for development of national evaluation capacity: Also needed is the capability to use evaluations for learning and adapting methods to objectives
15OpportunitiesGrowing interest among universities and national and regional research institutions to provide services in evaluation. This provides opportunities to work with these institutions in further developing capacities and promoting specialized training in evaluation.Increasing appreciation and demand from African Governments to reinforce institutional capacities to develop evaluation policy and evaluation coordination at national levelIncreasing pressure on Governments to be transparent and accountable in the use of national resources as well as demonstrating results of their policies, programs and projects. The complexity of governance in the modern world requires officials to have more knowledge for optimal decision-making.Strong and growing demand by donors in civil society organizations (a requirement for accountability of public action);
16ChallengesThe current trend in evaluation capacity across the continent is broadly rooted on two main challenges:First is the low demand for credible evidence about performance and the scant use of the information generated through evaluation efforts to inform public decision-making.Of particular concern are, on the one hand, the poor quality of the evidence generated by M&E systems, and on the other, the lack of interest from legislative bodies and citizens, key players in democracies with the authority to demand accountability for results vis-à-vis public investments such as the media. (National Evaluation Capacities, Proceedings from the International Conference, UNDP-2011) The second problem is the poor integration of institutions and actors associated with the effective evaluation of public policies, programmes and institutions, as well as the lack of convergence among cycles of various public administration processes relevant to broad M&E efforts, such as planning, budgeting and personnel.
17Demand & Supply of Evaluation Excess supply, or surplus, is the condition that exists when quantity supplied exceeds quantity demanded at the current price (EUROPE & N. AMERICA).Asia at equilibrium , where quantity (E Capacity) supplied equals quantity (E Capacity) demanded.Excess demand, is the condition that Africa has where quantity of E Capacity demanded exceeds quantity supplied at the current price
18Challenges Cont’dThe third problem is the lack harmony among donors and the national evaluation system. Donors tend to use their own evaluation systems rather than country systems to ensure visibility of their efforts. These broad challenges are manifested through:Weak of demand for MfDR;Weak human resources (Inadequate evaluation professionals);Weak statistical capacity;Absence of National Evaluation Policy and an incentivizing regulatory framework;Weak management capacity of the government;Low participation of non-government stakeholders in the evaluations; andLack of specialized training programs in evaluation
19StrategiesSupport to national systems: support countries to develop/use in-built quality assurance mechanisms, comply with evaluation norms and standards, and set up codes of conduct and ethical principles for evaluation. It is also important to balance the use of self-assessments (which may compromise independence and result in conflict of interest) and independent evaluation—such as ZANSTAT and ZIMSTATDevelop/strengthen networks of evaluation practitioners and national evaluation capacities to ensure continuous capacity building through knowledge and experience sharing and peer learning. Support the development and implementation of national Evaluation Associations—such as AfCOPDevelop and implement comprehensive capacity building programs on evaluation with selected higher level institutions in Africa—in progress for Agriculture under EWA
20StrategiesStimulating demand for evaluation with a focus on utility whilst addressing supply issues—skills, procedures, methodology, data systems, manuals—have to be addressed. Organise regional and higher level for decision makers. Work with the Media.Facilitate the design and implementation of specialized evaluation training programs at selected higher educational institution across the continent.Promote involvement of policy institutes/thinks tanks and non-state actors in the national evaluation process including providing short term practical training and conducting evaluations.
22Results & Efficiency: Long-term Secure resources to deliverTA, training, institutional support, KMIdentify an appropriate institutionReform process/procedureSelect individuals/ champions to supportSustained funding over timeBulk of spending on personnel costs, degrees, institutional infrastructure and some spending on idea generation and spreadingChoose the areas to strengthen or buildNetworking around CB issuesTrain and develop skills
23Value for money in the long term Strengthened platforms, networks and dialogueGreater accountabilityImproved discourseStrengthened economic governanceStrengthened reputationIncreased engagementStrengthened policy debate supportSustainable TTs and CSOsInstitutional supportSustainable systemIncreased demandReputation improvedPolicy influenceImproved quality and credibilityResearch, analysis and dissemination
25Evaluation of Complex Networks: Attribution Challenge
26Results: what do independent evaluations show about the process of change? Economic policy analysis and managementFinancial management and accountabilityPublic administrationGovernance and accountabilityKnowledge and learning
31Results in Knowledge and Learning Knowledge products that have transformed practice (RECs Study, Zimbabwe Currency Reform, ACIR)
32Features of the ACBF results-efficiency frontier: Think Tanks Results have been achieved with relatively little support from ACBFIndividual interventions to policy units and think tanks are relatively small compared to other projectsThe interventions focus on short routes to results but also longer term change and impact, including in systems and processesACBF support has been in many ways significantly cost effective and efficientSupport has resulted in both planned and unplanned positive outcome level results
33Results & Efficiency: Special Role of ACBF "the tendency by most donors is to target their support to areas which are relatively easier to generate results as well as easier to justify supporting. Consequently, the focus of most donor support has tended to be biased towards areas deemed relatively cost-effective and efficient.""ACBF on the other hand understands the African context...invests in elements that "make or break an institution" and yet they are elements whose specific benefits are difficult to trace, measure and quantify."Going forward, ACBF will select which instruments work best in which contexts, monitor and track the trends in use of expatriate expertise for the services produced, and shift to broader country support.
35ConclusionLack of sufficient Evaluation capacity in Africa has led to unexplainable issues like:High average annual economic growth but lower real per capita income today than in 1970 and more than million people still living below poverty lineDependency on external and food aid co-existing with growth in domestic revenues and food surpluses in many of the African countries
36Way Forward There is need for: Greater investment at the individual level where evaluation capacity is weakest, more investment in enhancing Africa's evaluation knowledge base, and finding better ways to use skills and resourcesGreater utilization and rationalization of existing evaluation capacity and improved mobilization of resources to enhance overall evaluation capacity in AfricaEnhancing governance and bring about leadership transformation at individual and institutional levels
37Way Forward There need for: Culture of responsibility, mutual accountability, commitment to performance, Monitoring, excellence and results evaluationContinuous assessment of evaluation capacity to respond to gaps and adapt to new and emerging M&E challenges.Continental (AFRICA) evaluation capacity building initiative that is practical, responsive to needs and motivated and designed by the continent itself (ACTUALLY this is long overdue)