Quality & Impact in DE defined: a selection An overview of selected informants Harm-Jan Fricke October 2011 hjfricke@btinternet.com

DARE Forum so far (May 2011) Q&I of DE depends particularly on Quality of organisation and management of work Quality of ways of working with audiences, partners, beneficiaries Specific to DE are issues of: Coherence: between values and actions, between actions and relevance to the own organisation and the audience/beneficiaries/partners Partnerships: explicitly defined and operated Participation: In methods used and in participants defining work done Perspectives: Plurality of viewpoints

Johannes Krause: Global Education quality standards Project/ programme design based on quality standards Monitoring& evaluation (a) M&E instruments, (b) M&E practice Official policies: quality standards translated into grant guidelines, funding criteria Organisational Quality Management Sharing good practices/ lessons learnt within the org. & with networks; peer reviews (e.g. Norway), etc. Work on quality frameworks (a) in the organisation, (b) in networks, (c) for policy level

Irish DEA: Quality and Measuring the Impact of DE Attention to the context in which work takes place: the relationship between the ‘development agenda’ and the ‘learning agenda’ Distinguish between process and impact evaluation: show progress against defined milestones or progress markers Suggested criteria and indicators at output level: # of schools/groups/individuals participating in a programme/course/event # of sessions/meetings held % of participants who complete a programme # of publications/resources/webpages produced # of visits to a website # of posts to a blog, facebook page etc. # of people who join an organisation Criteria and indicators at outcome level: % of participants who take an action/ join relevant activist groups or campaigns % of participants who go on to further study/training in the area % of participants who change their consumer behaviour and/or lifestyle choices Progress towards ‘markers’ of desired behavioural changes

Irish DEA: example of Q&I at national levels Output criteria Indicator Cooperation with DES is enhanced Number of meetings with the DES Number of times DE is mentioned at the cross-departmental working group Educational policies contain DE messages Number of educational policies containing dev ed messages Quality Research Number of research articles on DE Outcome criteria Indicator Cooperation with DES is enhanced (Enhancement is defined as increased awareness and buy-in from people at senior management level and increased commitment to integrate messages) Senior members of DES attend DE events and are open to conversations and dialogue DE becomes visible in DES communications and policies Educational policies contain DE messages Quality and depth of dev ed messages contained in policies Quality Research Indicators for research should follow academic standards (peer- reviews, quality of publications, etc.) but also bear in mind that relevance for practice is an important indicator of success for quality DE research.

ACODEV – COPROGRAM: Quality in Development Education Looking for quality in DE means the following: awareness of the relationship DE action - context, culture, reality, power relationships (society, north-south relationships, institution, funding,…) To act in coherence with our values. Quality on 2 main levels, Level 1 : the 3 main groups of actors involved in a DE action : participants, educators (and the institution to which they belong) and partners Level 2 : the 3 main phases in the setting up and development of any DE action: action design, action process or processes, results and outcomes

Teles and Schmitt: Elusive craft of evaluating advocacy “Successful advocacy efforts are characterized not by their ability to proceed along a predefined track, but by their ability to adapt to changing circumstances” Attribution of effect and of effectiveness is difficult Therefore: Assess the (policy-social) context within which interventions take place Use a "spread betting" approach to advocacy, and thus to evaluation since success or pay-off is rarely due to any one project/intervention alone Adopt a Long Time Horizon Pay Attention to Policies’ Impact on Politics, Not Just on Target Populations Evaluate organisations as a whole, not the single project

DESS 2011: Q&I in advocacy, campaigns and events Clarify concepts and definitions: Advocacy v. Campaigns Advocacy v. Awareness Raising Advocacy/Campaigning v. Education Q&I shown through: Strategic approach to do with for instance: Choice of topic/theme/issue; Problem analysis Stakeholder analysis Definition of objectives Critical path/Intended change process Participation including to do with: Vision Values Reflection Theory of change made explicit

DESS 2011: Q&I in adult education Criteria: Participation of audience/target group and relevance of issues to learners’ needs Methodology to promote engagement, motivation, critical reflection Engagement as outcome: with skills useful in various situations

DESS 2011: Q&I in youth work Issues to address include: Plan for impact Address relevance Show coherence with vision Be based on explicit values Use participatory approach Show different perspectives Stimulate reflection (on what is being gained) Be sensitive to gender and other power relations

DESS 2011: Q&I in resources Does it engage the user in questioning, discussion and action? Is it relevant and educational Does it show different perspectives on the issue – including minority and southern Does the resource enforce/reinforce stereotypes? Is it relevant to the target group Is it accessible to the user Is the information accurate and up-to-date

OECD-DAC & EC: evaluation framework for projects Relevance Effectiveness Efficiency Sustainability Impact DG DevCo: the above plus Coherence Added value

Next steps for the Q&I working group? Given the foregoing and given the DEEEP objective (a pan-European quality framework on Development Education): What can the Q&I group do next? What is needed? What is useful?