0. Measuring success through outcomes:
A way forward
Les Hems July 2014

1. Why Outcomes Measurement is important

2. ‘For purpose’ is a better concept than ‘not-for-profit’
Limitations of existing terminology:
Not-for-profit – contrast with for-profit business - but all organisations need to make a surplus to be sustainable
Charity – dependence on grants and donations - but all organisations need a reliable source of revenue to be sustainable
Increasing preference for alternative terminology:
For-purpose – mission driven - creating social change and impact
Social enterprise – co-creation of commercial and social value
Increasing interest in alternative structures:
Co-operatives and mutuals – owned and controlled by members
Civil Society – organisations embedded in communities
Co-operative Collective Impact – real community collaboration

3. Measuring outcomes rather than Value For Money

4. For-purpose: The problems with VFM measurement
Financial measurement:
limited measure of value
2. We allocate resources
to the things we can measure
3: Stakeholders are often
left out of decision making

5. What are outcomes and impact?
Outcomes are the significant positive or negative changes for clients (customers / participants), their families, the broader community and other stakeholders.
Outcomes:
Can be short, medium and long term
Can be direct and indirect (spill over)
Can be for target beneficiaries and other stakeholders
Can be economic, social, human and environmental
Impact is the net effect of these outcomes

6. Inputs to impact via outputs and outcomes
Input: The resources you use to undertake tasks and activities – not just financial – human capital etc. Activities: The programs and tasks you undertake to generate outputs Output: Tells you an activity has taken place and is usually quantitative Intermediate outcomes: The short term change that occurs as a result of an activity Outcome: The long term change that occurs as a result of an activity – the net benefit to recipients – the intended results Impact: The long term net benefit to society

7. Why measure outcomes? External perspective
Clients / consumers
Help consumers make good choices and to achieve their goals
Funders – Government, foundations, donors and corporates
Recognised limitations of funding activities and outputs
“Investment” in prevention, early intervention and breaking the cycle
Client directed care
Payment by Results (PbR) / for Outcomes, / for Success – linked to Social Impact Bonds
Collective impact initiatives focused on outcomes
Evidence
Focus on what works – the desired outcomes being achieved
Measurement and reporting
Increasing use of approaches such as Social Return on Investment, Results Based Accountability

8. Radical / disruptive changes in complex systems
Measuring outcomes is at the heart of social change – interest in social impact has been growing and will become the norm:
Higher expectations
Ineffective responses to wicked social problems
Increasing demand
Reducing tax revenues

9. Why measure outcomes? Internal perspective
Mission and strategy
Organisations’ mission statements should focus on outcomes
Outcomes are the fundamental purpose of an organisation and therefore the focus for organisational strategy
Performance measurement
Measuring outcomes is the most important measurement of an organisation / program performance
Benchmarking
Emulating best practice – outcomes frameworks, outcomes reporting and social impact statements
High performing organisations are learning organisations
Organisational learning – the development of a framework and measurement of outcomes generate learning

13. Key concepts for measuring outcomes

14. Key concepts: Stakeholders
Stakeholder: Any actor or agency that has an effect on, or is affected by the intervention:
They contribute to or experience the change
Can be individuals, families, organisations (e.g. different tiers of government), collectives, the environment
Key elements for outcomes measurement:
Stakeholder mapping
Network analysis – interdependence between stakeholders
Stakeholder engagement
Stakeholder can tell you what’s important, what has changed and help you decide what to measure (though sometimes funders direct what to measure)
It is important to be clear that stakeholders are not necessarily just people, but could also be institutions – like the State – or groups of people – say, local residents – or something like the environment. In the case of the environment, outcomes from an intervention may be less waste generated or greenhouse gas emissions avoided and these can be assigned to the environment as a stakeholder itself or an organisation on its behalf, e.g. Department of Environment.

15. Network analysis – partnerships and collective impact

16. Key concepts: Client segmentation
Clients / customers / participants are not all the same:
Demographics
Vulnerabilities
Capabilities
Expectations and aspirations
Map client segments and their related outcomes
Many to many relationships between client segments and outcomes
How will the program / intervention respond to these different client segments and outcomes?
High, medium and low – clients & complexity of service

17. Key concepts: Materiality
Focus on the stakeholders experiencing material outcomes
If I left this outcome out of the measurement, would somebody make a different decision about the program under analysis:
Consider relevance and significance to identify materiality
Iterative process:
Make an initial determination to decide which stakeholders to include in the stakeholder engagement.
Then, consult with stakeholders to identify their perspective.
You can’t measure everything.
Stakeholders with material outcomes will be a subset of the stakeholders and outcomes you identify
Once you have listed all possible stakeholders who might affect or be affected and whether the change they experience is positive or negative, intentional or unintentional, you should perform an initial materiality check.
Materiality is a key principle in SROI. It is actually a term borrowed from accounting. Something is material if its inclusion or omission from the analysis would distort the picture to such an extent that it would influence the decision you take as a result. Essentially, this is a question of significance. We use the concept of materiality to decide which stakeholders to engage and take forward in our analysis. What we ask at this stage is: how relevant and significant is the change that each of the stakeholders on my long list is experiencing? If I don’t include this stakeholder, will it distort the picture of the organisation/project to such an extent that it could change someone’s decision-making.
Testing an outcome for relevance means considering both whether the intervention/program can deliver the outcome and whether stakeholders express a need for it, there are policies that require it or perversely block it, social norms demand it, peers already do it and have demonstrated the value for it, and/or there are financial impacts that make it desirable.
Once relevant outcomes are identified, they should be tested for significance. This essentially involves prioritising outcomes based on how much change has (or will) occur, the value of that change to stakeholders and how much is due to the program. This can be done following the stakeholder engagement but should be revisited once data is collected.
Sometimes it is fairly straightforward to determine if stakeholders experience outcomes that are not material but often there will be borderline cases. It is often necessary to consult with all potential stakeholders to determine whether they have experienced any material change.
As we can see, determining materiality is a series of judgements and decisions which should be made with as much information, consistency and transparency as possible. It is helpful to draw up a table with all potential stakeholders, the positive, negative, intentional and unintentional changes they may be experiencing and then whether they have been included or excluded from the analysis and why.

18. Key concepts: Theory of change
A depiction of how a program or intervention:
Is meant to work
What it is supposed to achieve
Identifies any cause and effect relationships between activities and outcomes
Explores the how and the why
Combines evidence and direct experience
A framework for evaluating programs, and designing and predicting the outcomes of a program
Can also be used to develop an ‘outcomes hierarchy’ for an organisation as a whole

19. Key concepts: Impact
Impact refers to the net benefit to society - the ADDED VALUE of your programs
Key concepts to consider:
The counterfactual – what would have happened otherwise – “Deadweight”
Transferring the problem to or outcome from elsewhere – “Displacement”
The proportion of the change which relates to your organisation’s activities – “Attribution” or “Contribution”
The proportion of the change which relates to working collaboratively – “Interdependence”
Deadweight - Proportion of an outcome that would have happened anyway
Calculated as a percentage
Need to compare outcomes with benchmarks
A control group that provides an indication of what would have happened in the absence of your intervention
There can also be negative deadweight where your target population would have deteriorated in an area without your intervention and now still have not improved as much as other populations but are better off than they would have been otherwise.
Attribution – how much credit can your organisation take for the outcome?
Expressed as a percentage for each outcome
Not an exact science
Applies to outcomes after deadweight
Displacement is where all or part of the value is not actually created by the activities, but is moved from elsewhere
The proportion of value that is not created but instead displaced from elsewhere must be removed from the calculation
Most relevant to crime and economic development

20. Principles for outcomes measurement
Principle 1: Involve your stakeholders to measure social value
Measure what matters from the perspective of your stakeholders – the people or organisations that have experienced change as a result of your activities.
Principle 2: Measure and understand the theory of change
Apply appropriate methods and resources to understand and them measure the change (positive and negative, intended and unintended)
Principle 3: Be accurate and credible
Added value and impact needs to take into account:
What would have happened anyway? Deadweight
How much did other organisations contribute to the change? Attribution / Interdependence
Have the issues moved somewhere else? Displacement
Could also include the following as principles:
Strike a balance between numbers and stories
Be transparent about both the investment and the value created

21. Theory of change – the foundation of outcomes measurement

22. Theory of change
A theory of change depicts how a program is intended to achieve meaningful, positive changes for clients — not simply a description of what the intended change is
Can be straightforward where there is a linear relationship between cause & effect or much more complex when the relationship between cause & effect is non-linear and there are multiple factors at play.

23. Demonstrating outcomes leads to more inputs
Feedback loop - demonstrating the outcomes – both in words and numbers – can be instrumental in generating more inputs - the evidence of what works wins tenders, attracts investors, and provides collateral for marketing and fundraising.
Can be straightforward where there is a linear relationship between cause & effect or much more complex when the relationship between cause & effect is non-linear and there are multiple factors at play.

24. Example: Community housing for disadvantaged youth
Stakeholder
Inputs
Activities
Outputs
Outcomes
Residents
Accommodation
Maintenance costs
Management costs
Staff expenses
Access to housing
Formation of groups
Number of groups formed
Number of new friends
Increased community inclusion
Increased sense of security
Improved personal health
Family members
Not applicable
Reduced number of hours family member is engaged
Increased income for family
Less time spent on care for family member
Gover-nment
Health and welfare service usage
Less usage of intensive child support services
Reduced costs to Department of Health and Human Services
Reduced cost to State medical system

25. Example: Community housing for disadvantaged youth
Intermediate outcomes
Activities
Outputs
Long-term outcomes
Increased sense of personal safety
Provision of secure & affordable housing
Increased sense of security
Access to housing
Decreased financial stress
Decreased mental stress
Referral to and provision of health, employment and other social services
More regular sleeping patterns
Improved personal health
Reduced substance misuse
Appropriate treatment accessed
Increased employment readiness
Use of employment services
Greater ability to apply to jobs
Building of inclusive communities
Increased community inclusion
Participation in group activities
Greater social engagement

26. Combines evidence and direct experience
Preparatory desk review of evidence from:
Academic research and evaluations – national & international
Population level indicators and validated measurement tools
Existing organisational programs
Direct experience of those:
Delivering the service
Receiving the service
Other stakeholders
Iterative process
Develop prototype from evidence and those delivering the service
Refine ToC through beneficiaries and other stakeholders

27. The total value created by community services
Inputs from stakeholders
Outputs & outcomes for stakeholders
The biggest challenge facing the companies we are researching is the ability to measure the social value generated from their initiatives.
Measuring the social value helps to demonstrate the impact of your initiative in creating change, but it also builds trust with stakeholders to secure licence to grow and fund future initiatives, through direct investment or partnerships.
We have developed a new approach to help companies measure the total value created from their initiatives across six capitals – human, social and relationship, environment, intellectual, manufactured and financial capital.
The total Value Creation Framework draws on the Integrated Reporting Framework as well as well known measurement frameworks such as Program Logic and Social Return On Investment.
The benefit of the Total Value Framework is that it helps companies to explicitly identify the types of social value created for each of the stakeholders which are materially affected such as funders, partners, customers, employees community and the company.
We will be further trialling the framework over the coming 12 months.
Program Logic
Social Return on Investment
Commercial Return on Social Investment

28. Recognising all inputs and value created
Financial– not just revenue but balance sheet
Manufactured– the physical assets, internal systems you use
Intellectual– the research and development, innovation and piloting you undertake
Human– the knowledge, capabilities & experience, wellbeing & health of all those who contribute– paid/unpaid
Social and Relationship– networks, trust and societal norms – relationship value – within communities (place and interest) and families – between organisations (Interdependence)
Natural– the environment

29. Measuring Change

30. Quantitative research – evidencing outcomes
Once we have identified and defined outcomes through the qualitative stakeholder engagement in the Theory of Change, we can quantify them.
Key questions:
How do we know how many people have experienced this outcome?
How do we know the extent to which this change has taken place?
By definition - need to measure change over time:
Retrospective construction – “How are you today?” and “Has this got better or worse over the last two years?”
Baseline and follow-up – embedding measurement in to an organisation

31. Some basic principles for quant work
Maximise the utility of all relevant previous work – external and internal
Build a counterfactual:
Ideally, establish a control group to assess deadweight
If not, use self reporting, ask “What would have happened if you had not received help?”
Appropriate sample size and diversity:
Be pragmatic as can be resource intensive
Use of client records
Pilot your measurement tool / instrument!
Do you need incentives?
Do you need Ethics Approval?

32. Indicators
Indicators are key component of measuring outcomes i.e. the changes that have taken place
Indicators can be Objective e.g. no. of health clinic visits or Subjective e.g. self-reporting on physical health
Intermediate outcomes achieved can be used as indicators to measure the final outcome
Lead indicators can also be used to assess likelihood of final outcomes being achieved:
Some indicators will work for multiple programs
Be pragmatic but don’t cut corners: You may not have access to data required to measure ideal indicators.
Ways of knowing that an outcome (a change) has taken place
During the stakeholder engagement, asking individuals questions like “how do you know this change has occurred?” or how would someone else know that this has happened?” will help identify indicators.

33. Evidencing outcomes – distance travelled
So far we have talked about outcomes as though they are binary – they either take place or they don’t.
In practice, most outcomes are not like that. Outcomes tend to exist on a continuum of change. So it is far more meaningful to look at how far along a 7-point well-being scale someone has moved than to ask ‘has person X experienced an improvement in well-being? Yes or No’.
Outcomes measurement that looks at movement along scale across two points in time is known as ‘distance-travelled’ outcomes measurement. This is the best type of measurement as it tells you not only whether an outcome has taken place, but how much change has occurred in that outcome.
On the slide is an example of a very good distance-travelled outcomes data collection tool. This tool is called an Outcomes Star. It was originally devised for use in the homelessness sector. Each spoke on the star represents an outcomes domain, such as ‘physical health’, ‘offending’ and managing money. You’ll see that the numbers 1 to 10 run up scale.
What happens is that at regular intervals the key worker will mark on this star where there client is on each spoke. Now what is really neat about this tool is that it is not just about someone deciding, ‘I feel like a 1 or a 2 today’, but rather that sitting behind each scale is a clear defined set of indicators that tell you what someone looks like at 1, 2, 3 etc. on the scale. This increases the reliability of the tool because it means that two service workers are more likely to classify a client in the same way.
You can see how this tool would enable you to very quickly collect information about the progress a service user is making across a number of areas. Very important, too, is that you are collecting information in a format that allows you to aggregate across service users. You can report, for example, on the average movement on each scale across a whole service user population if you are looking to understand the difference you are making.

34. Evidencing outcomes – distance travelled
Achievem ent Level
Indicators
Improved money managem ent
Very Low
I am always short of money for basic expenses
I am unable to pay any of my regularly scheduled bills
I have no idea of what my outstanding debts are
Low
I often find myself short of money for basic expenses
I often default on my regularly scheduled bills
I know that I owe money but don’t know how much or how I can pay it back
Average
I have tried using a budget but found it difficult to follow
I occasionally fall behind on my regularly scheduled bills
I am aware of the extent of my debts and want some help to clear them up
High
I am using a budget regularly but sometimes tend to lose track of my expenses
I am making some changes in my habits to be able to afford my regularly scheduled bills
I am seeking help to begin to payback my previous debts
Very High
I am able to purchase my basic requirements and stay within a budget
I never default on my regularly scheduled bills
I have been able to eliminate all my outstanding debts OR
I am following a payment plan to wipe out my outstanding debts

35. Evidencing outcomes - Example

36. Approaches for measuring outcomes

37. What’s the best approach for measuring outcomes?
There is not a ‘best’ approach
Choose the most suitable and appropriate approach based on your needs
One program, one organisation or a collective (shared measurement)
There may be a role for using multiple tools depending on your needs
Remember the principles of good outcomes measurement
Involve stakeholders
Understand what changes
Understand impact or added value

38. Social Return On Investment
A way of understanding how effectively money is spent
A way of comparing the value of change created to the investment required to create the change
SROI = [Value of change]
[Investment]
It is an enhanced form of cost-benefit analysis which considers triple-bottom-line benefits and investments (economic, social, environmental)
It measures outcomes, not outputs
It measures change that matters to stakeholders
It can be evaluative or predictive
e.g. $1: $3.60
You can analyse an organisation or intervention in two ways:
Evaluative, where the analysis is based on actual outcomes data. For example, anorganisation running an employment training programme wants to know how much social value it is creating.
Forecastive, where you calculate what you would expect to take place. Particularly useful if you are still in the planning phase as it can help you to see how much social value your intervention would generation. It might help you to make decisions between different ways of structuring the intervention, different kinds of intervention or even whether to go ahead at all.
Now the process for both types of SROI are largely the same. For instance, with a forecasted SROI you would still want to do stakeholder engagement to identify outcomes. The difference is simply in terms of the outcomes data.

39. SROI – a principle-based approach
1. Involve Stakeholders
2. Understand what changes
3. Value the things that matter
4. Only include what is material
5. Do not over claim
6. Be transparent
7. Verify the result

40. Suitable applications for an SROI
Organisational learning – finding out what works and why
In circumstances where you require a financial valuation:
Where value for money - in its broadest sense - needs to be demonstrated
Social impact bonds
Requirement of a funder
Where a common unit of measurement ($) would be useful:
Compare across projects / programs within an organisation
Different types of outcomes i.e. social, economic and environmental
Where there are multiple interventions and contributors – collective impact:
SROI provides a framework for understanding your unique contribution

41. SROI – strengths and weaknesses
Stakeholder focused process provides great insight into stakeholders and programs
Powerful as a design and evaluation tool
Provides a framework for understanding ‘added value’
Is a basket of best practice tools – can be applied in its full form or using component parts
Speaks the language of some funders/decision makers
Has wide applicability across sectors and subject matter
Overemphasis on the ratio
Heavily reliant on subjectivity and assumptions -will evolve and strengthen over time
The valuation process is often a source of discomfort/contention
It is complex and resource intensive
Often used in a snapshot or retrospective context which constrains data quality
Comparison of ratio is inevitable but not always appropriate

42. Measuring outcomes – a proposed approach
Outcomes Hierarchy for an Organisation – derived from a high level organisational Theory of Change
Select program(s) for outcomes measurement and organisational learning
SROI including program level Theory of Change
Outcomes Measurement Framework – based on Total Value Framework
Embed measurement processes in to organisational systems
Formative process evaluation – feedback learning in to programs in real time
Stories as well as numbers
Reporting – dashboards, infographics, spreadsheet models

43. Reporting outcomes

44. Case study: MLC Foundation & Lifeline Research Foundation
The MLC Community Foundation and the Lifeline Research Foundation undertook an impact study to measure the social value created by Lifeline Crisis Support Chat Service in suicide prevention in Australia.
Social Outcomes
Three main social outcomes were identified for the service users:
Reduced suicidality/self harming potential
Improved resourcefulness
Enhanced belonging
In addition, two areas of cost re-allocation were identified for public services:
Reduced use of hospital services
Reduced call outs for police and emergency services
See flyer, report and presentation

45. The Social Value Created
Total Social Value Present Value of
Social Value
Crisis Intervention
(High Risk) $2,367,059 $2,233,075
Crisis Aversion
(Medium Low Risk) $3,166,579 $2,987,339
Emergency Service $469, $443,286
Medical Service $1,639,367 $1,546,573
Total $7,642,889 $7,210,273
Total Value of Inputs $860,517
Social return on investment ratio = ($1:$x) $8.4

47. Reporting outcomes

48. Be prepared Client directed care will become the norm:
Increasing competition between providers
Greater emphasis on marketing and communication
Risk of reliance on brand instead of market intelligence
Evidence of impact - and value for money - will assist decision-making and allocation of resources
Increasing interest in social finance which requires:
A Payment by results / outcomes mechanism to trigger reward payments
A rigorous method for measuring changes in outcomes
Monetising the value of achieving outcomes – cost savings to Government and/or value of improved outcomes

49. Challenges: External Complexity
Complex problems with multiple outcomes – limitations of linear cause and effect
Soft systems & Collective Impact
Maximise use of existing evidence and knowledge
Funding systems
Most existing government administrative & funding data is based on activities and outputs
Develop own data systems in preparation for outcomes based funding
Standards
No reporting requirements for outcomes nor repository for outcomes and frameworks
Encourage self reporting and celebrate excellence

50. Challenges: Internal Human resource
Limited expertise / competency and capacity
Capacity Building and co-production
Organisation systems
Lack of outcomes related data and and reporting
Implement case management systems
Organisational change
Inertia – organisation culture - resistance to change
Case studies of success
Costs
Dollars and time
This is an investment that will yield long term value

51. Measuring outcomes to achieve success: The Way Forward
Collective Impact
Experience
From our research we have identified six mutually reinforcing foundations that enable shared value innovation.
Our research aligns with the findings of Mark Pfitzer and his team. It shows the importance of creating a shared purpose and embedding it within the strategy of the organisation, the importance of collaboration and measuring the value created from initiatives.
However, our research is also showing the importance of companies embracing a new mental model or way of thinking and the importance of leadership as key enablers of developing a culture of shared value for innovation.
We will explore each of these foundations in detail.
Biggest challenge for For- Purpose organisations
Source: Leth & Hems (2013

52. For further information
Les Hems
Director of Research and Development
Net Balance Research Institute
Erica Olesson
Service Line Lead Net Balance Research Institute & New Zealand co-ordinator

53. Stakeholder engagement: Resources
Prove and Improve
AccountAbility

54. Theory of Change: Resources
ActKnowledge’s Theory of Change community:
New Philanthropy Capital’s guide to Theory of Change:

55. Indicators and data collection: Resources
Guide to Measuring Well-being:
Outcomes Star:
National Accounts of Well-being:
The Global Value Exchange
Sample size calculator

56. References SROI Network http://www.thesroinetwork.org/
SROI Guide – on your USB
Results-Based Accountability
Logical Framework Approach, Uni of Wolverhampton Introduction – on your USB
Code of Good Impact Practice – on your USB