Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluation linked to strategy A presentation for the Trust Company MBS Symposium, October 24, 2013 Kate McKegg, Director of The Knowledge Institute & member.

Similar presentations


Presentation on theme: "Evaluation linked to strategy A presentation for the Trust Company MBS Symposium, October 24, 2013 Kate McKegg, Director of The Knowledge Institute & member."— Presentation transcript:

1 Evaluation linked to strategy A presentation for the Trust Company MBS Symposium, October 24, 2013 Kate McKegg, Director of The Knowledge Institute & member of the Kinnect Group

2 However beautiful the strategy, you should occasionally look at the results. Winston Churchill Trust Company MBS Symposium, October 2013

3 Some common thinking about evaluatio n Its time consuming Its expensive Its produces long reports that are hard, and often boring, to read! It doesnt necessarily provide us with answers to the really important questions we have Its something we feel we have to do – but would rather not! Trust Company MBS Symposium, October 2013 3 Its just not that useful!

4 What is evaluation? It is the systematic determination of the quality, value or importance of something (investment, project, initiative, program, organisation etc) in order to take action It is underpinned by the collection of information / evidence about the inputs, activities and outcomes of something i.e., a project, program, initiative or activity Its key purposes are to determine how well something is doing or was done, what its value is or was, and if it is worth doing more of it Trust Company MBS Symposium, October 2013

5 Evaluation measurement May 2003 Harvard Business Review "The High Cost of Accuracy." Kathleen Sutcliffe and Klaus Weber. They concluded that "the way senior executives interpret their business environment is more important for performance than how accurately they know their environment." Trust Company MBS Symposium, October 2013

6 Results and Performance Mainstay of private sector and increasingly important in social/public / philanthropic sectors In the public / social / philanthropic sectors - measurement is often considered to be one of biggest challenges… However…. Establishing what results are to be achieved – and what performance actually means is probably one of the most difficult aspects of managing for results or outcomes Trust Company MBS Symposium, October 2013

7 Measurement & results Private sector results – the bottom line reasonably straightforward Social / public sector / philanthropic results or outcomes are different – not as straightforward Linkages between various levels of results are not always known and measurement is often very challenging We often do not control outcomes …but we do seek to influence them Trust Company MBS Symposium, October 2013

8 Some challenges Unpredictability of initiative direction and likelihood of emergent unknowable outcomes Many changes that are difficult to measure e.g., resiliency, equity, etc. Lags between interventions, activities and outcomes Difficulty in sorting out the extent to which changes have been influenced by our initiatives Trust Company MBS Symposium, October 2013

9 Common responses Tail wags the dog: Only do what can be easily measured Miss the parade: Evaluate only those easy-to- evaluate things and miss the rest Sink the ship: Try to evaluate everything extensively and end up capsizing along the way under the volume of effort, for very little return Trust Company MBS Symposium, October 2013

10 Defining performance, quality and value For good evaluation, we need an explicit way of defining performance, quality and value for ourselves and for others Trust Company MBS Symposium, October 2013

11 Performance & goodness = values E- valu - ation has three components Descriptive facts + values + evaluative (what so) conclusions and action (so what?) Values get us from what so to so what See E. Jane Davidson, Evaluation Methodology Basics, The Nuts and Bolts of Sound Evaluation, Sage, 2005 Trust Company MBS Symposium, October 2013

12 Visible When the values step is transparent and explicit, then it is more likely to expect: Genuine evaluative conversations about: – what success, performance and goodness looks like – what constitutes good or excellent programme, organisational or strategy delivery and performance looks like Trust Company MBS Symposium, October 2013

13 Values in evaluation? On what basis will we decide or demonstrate to ourselves and to others that something we are investing in, or doing is high quality, valuable, worth doing, or important to pursue? And on what basis will we determine how good is good? It is values that we look to help us make these judgments. When ever we remark or comment on how good something was, or how well something was done, we are drawing on our values to make evaluative judgments about the quality and value of things. Where will we get the criteria to make the judgments we need to make? Trust Company MBS Symposium, October 2013

14 General guide for deciding how good performance is (E.J. Davidson, 2010) Performance Descriptors for Key Evaluation Questions Excellent Performance is clearly very strong or exemplary in relation to the question. Any gaps or weaknesses are not significant and are managed effectively. Very Good Performance is generally strong in relation to the question. No significant gaps or weaknesses, and less significant gaps or weaknesses are mostly managed effectively. Good Performance is reasonable in relation to the question. A few gaps or weaknesses, but none that are considered serious Adequate Performance is inconsistent in relation to the question. Some gaps or weaknesses. Meets minimum expectations/ requirements as far as can be determined. Poor Performance is unacceptably weak in relation to the question. Does not meet minimum expectations/requirements. Insufficient evidence Evidence unavailable or of insufficient quality to determine performance.

15 Many other rating options possible ExcellentHighly EffectiveSelf relianceExpert Very GoodConsolidating effectiveness WellbeingProfessionally experienced GoodDeveloping effectiveness Recognition and belief Fully competent AdequateMinimally effectiveAwakening / awareness Emerging expertise InadequateIn effectiveUntapped potentialQualified novice PoorDetrimentalIn crisisNot yet qualified Trust Company MBS Symposium, October 2013

16 Mercalli Intensity Scale (1906) for earthquake impacts

17

18 An everyday example: How valuable was the last consultation about my sons aspergers? My criteria (values) for judging the quality and value of a consultation: – The extent to which I have confidence that GPs and specialists are working collaboratively around his care – The extent to which I have a clear idea of my sons progress and future options – The extent to which Patrick is respected and included in the consultation – The extent to which GPs and specialists demonstrate an interest in Patricks wider life and context – The amount of stress involved in seeing specialists and GPs Trust Company MBS Symposium, October 2013

19 If the consultation was excellent? I would have total confidence that my GP and the specialists were working together, and had full and shared knowledge of Patricks condition and his developmental progress. I would leave the consultation with a thorough understanding of sons progress and feel confident that there were practical future options that I and my family can use and are relevant to us to support his development. I would feel that Patrick was being respected and included by everyone, that his feelings and point of view would be taken seriously Being able to see the key specialists was a straightforward experience. Trust Company MBS Symposium, October 2013

20 My criteria and the data Evaluative CriteriaThe Data Importance ranking The extent to which I have confidence GPs and specialists are working collaboratively around his care The appointment system certainly didnt indicate that there was any formal collaboration, it was clear that prior to the appointment, there had been no contact between specialists, although once together, everyone worked well 1 The extent to which I have a clear idea of my sons progress and future options The discusion was very productive and it was agreed that using the data from the recent cognitive testing, as well as other best practice, guidelines for Patricks teachers would be developed 2 The extent to which Patrick is respected and included in the consultation Patrick was engaged and talkative with the group. The GPs and specialists encouraged him to lead discussion and he reported that he enjoyed the experience 3 The extent to which GPs and specialists demonstrate an interest in Patricks wider life and context The group were very supportive of finding ways to translate clinical and psychological findings into everyday language for teachers. 4 The amount of stress involved in seeing specialists and GPs This has not changed since we have been in the system. This requires my organisation for the consultation to work effectively 5

21 What was the value / how good was the appointment ? Overall = Pretty Good CriteriaAwfulMediocreOKPretty GoodGreat The extent to which I have confidence GPs and specialists are working collaboratively around his care Extremely Important The extent to which I have a clear idea of my sons progress and future options Very important The extent to which Patrick is respected and included in the consultation Important The extent to which GPs and specialists demonstrate an interest in Patricks wider life and context Somewhat important The amount of stress involved in seeing specialists and GPs Not so important

22 Evaluative reasoning – the critical component for evaluation of strategy Intelligent, well-informed and open-minded human reasoning is what is required to go through the systematic process of reaching judgements about quality, value and importance 22 Trust Company MBS Symposium, October 2013

23 Evaluative reasoning Evidence does not speak for itself Arguments and reasoning are needed to connect evidence to judgment think slow vs think fast (Kahneman) how do you know? discover not impose 23 Trust Company MBS Symposium, October 2013

24 Evaluation linked to strategy – its all about learning Strategy - cycles of sensing, venturing, learning, and re- visioning Ongoing attention to strategy should focus on learning and adaptation Learning in real-time & adapting Evaluation + evaluative reasoning & thinking for reflection = learning and use in decision making Trust Company MBS Symposium, October 2013 What? So what? Now what?

25 What capabilities do we need? Organisational attributes Leadership that takes ownership and responsibility for learning High levels of awareness of context, and changes in the wider environment Willingness to balance development and innovation with a commitment to testing reality – being prepared to expose ideas and practice to tough feedback Inquiry, questioning and mindfulness – a willingness to explore, dig deeper, interpret data, make sense of feedback Openness to taking risks and the courage to face the possibility that some things may not be going so well, adapt and keep going Preparedness and perseverance to co-create the future, collaborate and trust Trust Company MBS Symposium, October 2013

26 Demand Capabilities Aptitudes, attitudes and skills that enable organizations to commit to and engage in evaluative thinking and practice Supply Capabilities Skills, knowledge and experience that enable organizations to produce high quality evaluative information Structural Capabilities Institutional, management, process and resource capabilities that support the production and use of evaluative information. Organisational Evaluation Capacity Evaluation Capacity and Capability Trust Company MBS Symposium, October 2013

27 Linking evaluation & strategy Preparedness to lead and commit to, and hold yourselves accountable to outcomes or key results Make measurement of outcomes thoughtful, meaningful and credible Clear articulation of dimensions of quality, value and importance are in your context Commitment to reasoned evaluative thinking and interpretation of information about what is really going on Use the results for reflection and learning - and model for others serious use of results. Trust Company MBS Symposium, October 2013

28 THANK YOU For more information: Contact Kate McKegg, Director of The Knowledge Institute Ltd www.knowledgeinstitute.co.nz (a member of the Kinnect Group www.kinnect.co.nz )www.knowledgeinstitute.co.nzwww.kinnect.co.nz Email: kate@kinnect.co.nzkate@kinnect.co.nz


Download ppt "Evaluation linked to strategy A presentation for the Trust Company MBS Symposium, October 24, 2013 Kate McKegg, Director of The Knowledge Institute & member."

Similar presentations


Ads by Google