1. Defining and measuring outcomes
Definitions Workshop
11 Dec 2001
SSC

2. Purpose of Workshop
Objective is to approve outcome framework as basis for writing BB1 document
This includes basic definitions, etc (table)

3. Agreements Sought
Agree to accept definitions in table, modified as discussed in this WS
Approve the commencement of writing by Secretariat

4. Criteria for change Concepts first, then detail
Table is a base, we seek improvement
Majority agreement on what to change
Compatible with NZ public sector management framework
Best solution for New Zealand

5. Workshop Process 9:00-9:10 Welcome & Intro
9:10-10:00 Concepts of framework (three minutes, max)
Each agency provides comments on concepts (rather than wording)
10:00-10:15 Group decision on concepts
10:15-10:30 BREAK
10:30-11:00 Concept by concept discussion of framework wording
11:00-11:30 Seek group agreement for secretariat to commence writing chapter 1

6. CONCEPT End Outcomes State Indicators Impact Measures
Written definition of benefits from agency’s services as experienced by target group(s) in community
State Indicators
Quantitative* measures used to monitor changes in End Outcome between groups / over time
Impact Measures
Quantitative* measures of change in State Indicator caused by one or more activities of agency (or agencies)
* Includes qualitative measures where data limitations / costs prohibit more objective measurement

7. Agency Outcome Definition (formerly ‘End Outcome’)
Name / Definition
Agency Outcome Definition: Definition in measurable terms of the impacts on, or consequences for, the community of the outputs or activities of agency / ies (PFA)
Attributes
Consistent, verbal definition of changes attributable to activities of one or more agencies
Clearly identifies consequence of agency’s outputs or activities to the community
Clearly referenced to target of services
Definition is stable over time, and based on accessible source data
Only defines how performance will be measured
Limitations
Limited use in decision-making or prioritisation

8. State Indicator Name / Definition Attributes Limitations
State Indicator: Quantitative* measure of actual outcome for specified groups, based on Agency Outcome Definition
Attributes
Quantitative measure:
clearly linked to agency services
clearly linked to service recipients
segmented by e.g. population groups, regions, business units, etc
SMART/FABRIC (see NAO – March 2001, including: affordable; benefits outweigh measurement costs)
Data obtainable within acceptable timeframe
Statistically robust
Limitations
Unknown attribution to outputs or activities
of agency, due to external influences
Measurement errors

9. Impact Measure Name / Definition Attributes Limitations
Impact Measure: Quantitative* measure of the change in a State Indicator caused by one or more activities of agency (or agencies)
Attributes
Same attributes as State Indicator, plus
Compares State Indicators for ‘treated’ group with the reference group not receiving the treatment
When used to compare different interventions or treatment groups, must measure impact using a consistent method
Limitations
Model robustness
Ethical constraints on reference group(s)
Measurement errors

10. Proposed BB1 Structure Target audience(s)? Structure and flow?
Content?

11. Agreements Sought
Agree to accept definitions in table, modified as discussed in this WS
Approve the commencement of writing by Secretariat