EVALUATING, REVIEWING REFLECTING & REVISING Ros Tennyson

Some definitions: EvaluatingMeasuring ReviewingTaking stock ReflectingUnderstanding RevisingChanging Essentially we seek to understand / describe the process of acquiring, analyzing, interpreting, adapting, disseminating and (ultimately) transforming experience and knowledge

Some further definitions: INPUTSACTIVITIESOUTPUTSOUTCOMESIMPACT What you need / bring What you doWhat you produce Benefits of your work to those involved Benefits of your work to wider society Eg: Human resources Facilities / equipment Contacts / networks Specialist knowledge Funding Eg: Specific development project(s) Comms Creating mechanisms Capacity bldg. Eg: Physical products Events Services delivered Number of people trained Eg: Changes in levels of skill / confidence / behaviour Improved systems or organisational efficiency / capacity Eg: More stable communities Increased cross-sector respect / collaboration Influence on policy and strategy

In Phase 1: Set up parameters, baselines and performance indicators Creating review / evaluation systems Ensuring commitment (‘buy-in’) to evaluation and review from partners

In Phase 2: Collecting project data (outputs and outcomes) Tracking performance / Keeping records (decisions, events, changes) Maximising partner resource contributions / relationships and engagement

In Phase 3: Evaluating project outcomes and impacts Assessing the the partnership as an effective mechanism for sustainable development activity Reviewing partner relationships to assess and maximise value

In Phase 4: Undertaking and disseminating a final evaluation Or: Agreeing new parameters and performance indicators Revising systems Creating new review and evaluation procedures

Evaluating Partnerships: What do we most want to know? 1.That the partnership has been effective in achieving its aims 2.That the partners have all benefited from their involvement 3.That the partnership approach was / is the best way to do it We need to understand the bits that are not obvious and that we can’t see Painting by Magritte

AN EMERGING EVALUATION APPROACH FOR PARTNERSHIPS: 1 IMPACT ASSESSMENT 2. PARTNER RELATIONSHIP REVIEW 3 EVALUATING PARTNERING AS A MECHANISM

To evaluate:  Tangible impacts: –Technical / Physical –Human –Financial  Intended / unintended outcomes  Value (of investment) 1. Impact assessment

To understand:  Transaction costs / value added  Sustainability of outcomes  Strategic influence  Systemic impact / reform Whether a partnership approach was ultimately better than the next best alternative 3. Evaluating partnering as a mechanism

Formula for assessing ‘added value’ of a partnership AV =  (OP + SC) – (RC + NA + EC + OC + FC) Key: AV = A dded V alue of a Partnership OP = O utcomes of P artnership SC = S ocial C apital RC = R esources C ontributed NA = Net Benefit of the Next Most Likely Alternative EC = E nvironmental C ontributions OC = O pportunity C osts (eg time spent) FC = F acilitation C osts

To reveal:  Value for partner organisations (& other stakeholders) –expected –unexpected –potential  Degree of effectiveness / efficiency / impact  Level of influence (sectoral / strategic) 2. Partner relationship review

This form of ‘evaluation’ links to: A review of partnering principles in practice (ie is the partnership equitable, transparent and mutually beneficial?) Whether the partnership is achieving individual goals / underlying interests of partner organisations or not Exploring whether the partners have made maximum use of the range of resources available Whether the partnership could work better – if so, how? Whether the partnership could do something quite different – if so, what?

Who are the audiences for a partner review? Brainstorm…

This is a new ‘science’ and it raises questions about:  Validity?  Reliability?  Legitimate measurement?  Ethical considerations?  Added value?

A working hypothesis… Any truly valid and effective review of a partnership always need to: Involve all partners & key stakeholders in design and data collection Include a genuine feedback loop so that the process truly informs the development of the partnership Find a good balance between external ‘objectivity’ and internal knowledge / experience of the partnership’s history

Sources of knowledge: Your own and others experiences and observations Formal records (eg minutes, proposals, action plans,agreements) Physical evidence Other relevant materials giving contextual information (eg newspapers, legislation, local events) Theories and hypotheses (to challenge your thinking and understanding) Guidelines / manuals Case studies

Ways of Knowing… Sensory (sight, smell, touch, hearing, taste) Mental (memory, patterns, shapes) Intellectual (concepts, logic, assumptions) Intuitive (instinct, feelings) Imaginative (vision, empathy)

Capturing knowledge involves… Looking for tangible evidence Empowering and inviting people to reveal what they know Active listening / Astute observation Asking ‘open’ questions Record keeping (endless note-taking; keeping a ‘log book’) Good filing systems (or a good memory)

Being aware of the ‘filter’ of the reviewer… Preconceptions, assumptions and prejudices Cultural / sectoral / political ‘lens’ Personal values / belief system Professional discipline / training Experiences (good or bad) of partnering to date Capacity to make sense of complex and multi-source data

Making sense of data involves… Active engagement and interest Attention to detail Willingness to explore contradictions Sifting and selecting material Interpretation and clarification ‘Triangulation’ (confirmation) of findings and above all…

Time for quiet reflection! Painting by Salvador Dali

Tools and technologies Tools:  Questionnaires  SWOT analysis  Open questioning  Observed activity  Tangible evidence  Document analysis  Other? Processes:  Written responses  1:1 interviews  Partner groups  Partnership groups  Third-party inputs  Contextual evidence  Other?

“As a reviewer, make no assumptions and always remember to expect the unexpected” THE UNEXPECTED ANSWER by Rene Magritte