Using results frameworks to shift the focus of evaluation to a strategic level Emerging research on the principles underpinning results frameworks Kate Averill AES Conference Sydney 2011 Building futures Using results frameworks to shift the focus of evaluation to a strategic level Emerging research on the principles underpinning results frameworks Kate Averill AES Conference Sydney 2011 Building futures

Presentation  Why frameworks?  Shifting evaluation focus – to strategic and sector levels  Architecture and use of results frameworks  What are results frameworks?  Principles underpinning results frameworks  Implications for practitioners

Why frameworks?  Frameworks are promoted as providing the structural, strategy and performance links between the different actors in development cooperation (Aid Effectiveness Review, AusAID 2011)  Since 2002, move towards more systematic approach to planning, monitoring and evaluating for results - both outputs and outcomes, and impacts (Aucoin & Jarvis, 2004)  Monitoring and evaluation systems/frameworks include: (a) identify intended results (outputs, outcomes, impacts) (b) measure results (monitoring and evaluation) (c) use evidence-based information to improve programs (Leonard & Bayley, 2008; Kusek & Rist 2004)  To enhance development and aid effectiveness  How ‘evaluation’ fits within this agenda is under debate  Personal experience and literature

Research problem/need - starting point  Paris Declaration on Aid Effectiveness includes five partnership commitments that need to be interpreted in light of the specific situation of each partner country of ownership, alignment, harmonisation, managing for results, and mutual accountability (OECD-DAC, 2005)  12 indicators of progress measured nationally and monitored internationally (OECD-DAC, 2006)  Indicator 11: Sound Frameworks to Monitor Results - number of countries with transparent and monitorable performance assessment frameworks to assess progress against: (a) national development strategies (b) sector programmes  2008 survey monitoring Paris Declaration on use of frameworks showed increase from 7% in 2005 to 9% in OECD target for reduce gap by one-third. Further survey in 2011

Context – development and evaluation

Shifting evaluation focus to a strategic level  New paradigms in development cooperation are emerging :  shift of focus to developing countries becoming driving force of their own development and country systems  country system - national arrangements and procedures for public financial management, procurement, audit, monitoring and evaluation and social and environmental procedures (OECD, 2009)  Increasingly, emphasis is now being placed on strengthening national-level monitoring and evaluation systems led by countries (Ba Tall, K. 2009)  Focus is now at country and sector levels but program level activities are important and their alignment to sector goals  agencies and donors align activities and programs to sector/national goals  capacity and design considerations

What are results frameworks? Key terms and components of frameworks are not commonly defined in the literature and in practice:  Results framework (and diagram) - links between country strategic goals, higher level sector outcomes, country organisational structures, key stakeholder relationships and development partners (adapted from Binnendijk, 2001)  Outcomes frameworks - shows hierarchy of key outcomes for sector or overarching multi-program (adapted from Duignan, 2004). May include multiple outcome layers—sector, region, agency, individual  Program logic - links at program level between inputs-outputs- outcomes/impacts, theory of change, context and assumptions. (Results-chain term used interchangeably with logic model or program logic)  Literature: draws from evaluation, development, management and governance knowledge areas

Results Framework (Averill 2011)  Three components: 1.Stakeholder / process map 2.Results/outcomes models – strategic and program levels 3.Associated measurement frameworks  Purpose of frameworks – planning, on-going monitoring, evaluation or ‘evaluative’ monitoring  Principles underpinning architecture and use of frameworks are important to enhance development and aid effectiveness (research)  Reposition evaluation as management strategy – linking strategy, learning and performance  Three phases: cyclical and iterative (1/3 principle) 1.Plan 2.Monitor, research and assess 3.Report and change

Example: Uganda - PEAP Key Strategic Results within Greater Accountability Framework (Tumusiime-Mutebile, 2002)

Results/outcomes model Outcome Indicator Quantitative indicatorMechanism / example Aligning results at different levels: Using strategic and program models to align results at different levels Context and need Inputs Outputs Outcomes / impacts Outcome Quantitative indicator Mechanism / example Quantitative indicator Programme one Programme two Outcome Quantitative indicator Mechanism / example Quantitative indicator

Measurement Framework

Learning and implications  Values - collaborative, ownership, participatory, cultural competency, credible, ‘fit for purpose’, ethical  Reposition evaluation as management strategy linking strategy and ‘evaluation’  Results Framework - foundation tool - 3 components  Purpose and design of results frameworks (planning, monitoring and ‘evaluation’ or ‘evaluative monitoring’)  Approach - on-going, point in time knowledge  Internal/external – ownership and building capacity for sustainability

Learning and implications  Three phases - cyclical and systematic (1/3 principle) 1.Plan - needs, context, scope, theory of change – framework 2.Monitor, research & assess – measurement, data collection, recording, analysis and assessment 3.Report and change – timely, credible, cyclical, iterative.  Building capacity for iterative learning with support  Challenges – ‘tunnelling’, indicators, scope, use of information  Value – reporting on progress and enabling timely response to emerging evidence  Enhancing development and aid effectiveness

Next steps research and practice  Research process:  key informant interviews  case study fieldwork:  New Zealand  Papua New Guinea  Samoa  Laos  Examining:  approaches and the use of results frameworks in public sector  different perspectives of stakeholders  Aim - identify principles underpinning the architecture and use of results frameworks to enhance development and aid effectiveness and the implications for practitioners  Continue working with and supporting people and organisations in their planning and evaluation to build capacity and enhance decision making  Share learning from research and practice contributing to knowledge base

Further details Kate Averill Director Evaluation Consult Building futures