1. Engaging with external evaluators
Kim Grey
15 June 2017
Disclaimer – the views expressed are those of the author

2. How do we get - Right skills Right time Right problem?
What’s right?
In general – it means:
The ‘right’ technical needs
The relevant ‘decision points’, and
A significant or strategic focus
But - what’s ‘right’ might depend on circumstances

3. When might this vary? When doing evaluation in:
Chaotic situations
Complex
Complicated
Simple

4. Complicated situations Complex situations
Technical solutions are available
Stakeholders in relative agreement
Problems are knowable and stable
Complex situations
Techniques under development
Stakeholder views differ
Problems not clearly understood, multiple options & emergence

5. Complicated situations Complex situations
Diagnosis relatively straightforward
Use checklists
Complex situations
Greater uncertainty
Adaptation needed

6. What to do in areas of complex social policy?
Suggestions from my experience:
1. Technical challenges: test new approaches together
For example
Seek partners willing to implement new methods
Diversify your partnerships
Try mixed method combinations
And build uncommon skills
Be prepared to share risk and develop bespoke approaches.

7. Recognise the need to develop a tailored response
What's the strategy?
Recognise the need to develop a tailored response
Implement bespoke evaluation design
Recommended reference:
Forss, Marra & Schwartz (2011)
Evaluating the Complex. Attribution, Contribution and Beyond.
Comparative Policy Evaluation, Volume 18
Transaction Publishers

8. 2. Seek common ground among stakeholders
Ask:
What needs do we share?
What facts do we all need to see?
and
Strive to see the world from others’ points of view

9. 3. Explore the problem together
External evaluators aren’t the font of all knowledge
They can’t read your mind.
Share program insights, build on your stock of knowledge
Seek methods that reveal the unexpected
… and the unintended effects
Be prepared to question your assumptions

10. Options for working with external evaluators in areas of complex social policy
Summary:
1. Technical challenges: test new approaches together
2. Find common ground among stakeholders
3. Explore the problem together

11. Tools for starting off well… 1st 2nd
Clear requirements
Careful selection…
Tender document
State assumptions about the evaluand
Share your preferences
Ask for ideas and suggestions
Tender assessment
Consider panel membership and debate strengths and weaknesses
Get references for projects involving the staff listed

12. How to continue?
Talking
Debating
Negotiating

13. Take-home messages: Three things we need to do - Build a relationship
Be prepared to discuss your epistemological assumptions 
Have the difficult conversations

14. Thank you