Measures Definition Workshop Version: 1.0 Date: 26/07/10 Changes included in update: first published document Author: Candice Lequesne Notes for presenter: Use these slides to run the measures definition workshop. Additional advice and tips are available in the “Guidance for Workshop Facilitators”. If you would like to understand how this workshop fits in the benefits management process and project/programme management in general, read the “Session 3: Benefits Definition and Measurement” and the “Managing Benefits Activities in your Project or Programme” presentations and guidance in our Toolkit.

Who Should Attend Output Sought Users: Service area managers and/or Benefits Working Group representatives Project Team Senior Managers/Sponsor (sign-off) Output Sought List of measures, monitoring schedule and governance Notes for presenter: Ask the Project Manager (PM) or someone from the project team to attend the meeting to give an update on the project and answer questions. Obtain from the PM in advance of the workshop the proposed roles and responsibilities/reporting governance to be agreed. An informal chat with the service area manager(s) in advance of running the workshop to explain the aims of the day may help get better buy-in and attendance.

Workshop Outline Introductions Project update and objectives Why have you been ask to participate? What are our measures for success? (monitoring criteria and key success factors) Agreed governance and reporting schedule Who will take on the roles and responsibilities? Next steps Notes for presenter: Introduce yourself (name and role) Ask attendees to introduce themselves Get someone from the project team to give a project update. The project key benefits can be listed on a slide or written on the flipchart for reference.

Effective measurement and monitoring will: indicate the extent to which benefits are being realised give early warning of potential problems create the opportunity to adapt the benefits or changes to enable the overall objectives ensure that achieved benefits are measured, reported and communicated contribute to lessons learned Having appropriate measures in place and monitoring them regularly will allow you to track progress against targets (or the baseline if improvement levels couldn’t be predicted). Benefits monitoring and reporting should become part of the normal programme and project progress reporting process.

Financial benefits Include calculation of unit value for transparency Base calculated value on evidence (process mapping, time in motion study, case studies, …) Be careful not to overestimate Differentiate between cash release, cost avoidance and time savings In order to provide a solid appraisal of the value of the programme/project and, later, monitor progress and evaluate outcomes, you will need to define the value of each benefits and targets or success criteria. Financial benefits are any benefits which can be assigned a monetary value: e.g. cash savings, cost avoidance or time savings (in terms of salary cost saved). Their value should be calculated (or estimated if not possible) and included in the business case “economic appraisal” section. Be clear which benefits are really cash releasing and will result in unspent budget, and which maximise existing resources to deliver more productivity (e.g. time savings: will this be enough to cut staffing numbers or reduce the use of bank staff? or will it just enable more or new tasks to be delivered with the same staff contingent?).

Non-financial benefits Agree evaluation criteria with relevant stakeholders Base any estimated values on evidence (published case study, survey, …) Remember to gather baseline measures for these too Non-financial benefits will be more difficult to assess but are often important (e.g. waiting times, patient experience) and should not be ignored simply because they cannot be costed.

Examples Category Benefit Descriptions and Numbers Measure Value / Target Financial – Cash Release e.g. “reduce stationery cost by using electronic requesting for labs”  Save 1p in paper and printing cost for each request. Around 6000 lab requests submitted per week. (1p x 6000 requests) x 52 week £3120 p.a. Financial – Time Release e.g. “release lab staff time registering requests by using electronic requesting ”  Save 1 minutes per request. Around 6000 requests received per week. (6000 requests per week x 1 min) / 60 mins / shift weekly hrs = WTE released 2.7 WTE released (Band 4 = £19k p.a.= £51,300) Non-financial - Quantifiable e.g. “reduce test results turn around time”  releasing 2.7 WTE will enable 400 additional tests being run per week, which will eliminate backlogs % same day test results N/A Non-financial - Qualitative e.g. “reduce staff frustration due to need for double data entry”  staff satisfaction survey with regard to duplicate data entry % of satisfied or very satisfied For the purpose of the business case, there are only two types of benefits: financial (e.g. direct resource costs, or staff time value against average salary) and non-financial (no monetary value). It is however useful to differentiate between cash savings (including staff cuts) and time releasing or cost avoidance (not actual savings - an increase in productivity for the same cost rather than staff/budget cuts) when talking about financial benefits. For internal purpose, the non-financial benefits will also have to be measured (e.g. reduced number of audit breaches, reduced waiting time, improved patient experience (rating), …) so they can be monitored. For “qualitative” measures, criteria and a rating system (e.g. Likert scale) should be agreed as well as the measure. Financial benefits Cash release (including staff cuts) Time release (where not substantial enough to allow staff cuts) Cost avoidance Non-financial benefits Non-financial quantifiable Qualitative

Realistic and Achievable The benefits will be included in the business case and evaluated at the end of the project so keep them achievable and realistic! Benefits and targets have to reflect what can realistically be achieved or your benefits evaluation will show that benefits have not been fully realised, and users will feel they have been handed an impossible task and will not be supportive. Wherever possible, existing measures should be used (e.g. HEAT, SPSP, 18 Weeks, … performance indicators, balanced scorecards) so achieving the benefits is seen as an integral part of general performance improvement. It will also ensure that baseline figures are available and that collecting the data or recording the information will not be an extra burden. Where no measures exist, new ones should be carefully designed and agreed. The cost of capturing the extra data should be considered: if it is going to require system customisation and a disproportionate amount of effort to input, is the value of the benefit such that it is still worth measuring in that way? If it is deemed to difficult or expensive, the benefit should be given a “qualitative” measure (see slide 7). Another important aspect is whether the measure will encourage the right type of behaviour. Consider whether clinicians and admin staff will be encouraged to adopt the new process or just “tick the box” to achieve a target. For instance, would the 18 week RTT, if it didn’t take account of case complexity, encourage hospitals to provide the best care available within the 18 week window instead of the treatment which will provide the best possible outcome for the patient regardless of waiting time?

Breakout Exercise Define appropriate measures for success for each key benefit in turn: What will show there has been an improvement? What data is already collected and can be re-used? Other sources: case studies, reports, …? Is more than one measure required? Or does one measure apply to more than one benefit? Is the benefit quantifiable (cash, time, no. of cases, percentage, …)? Or will a survey/rating scale need to be created? The measures should be based on benefits (e.g. reduce discharge time to 1 hour, improve accuracy of records to 98%, save £10,000 p.a. on transport booked for DNAs/CNAs…) not on IT functionalities/deliverables (e.g. patient demographics feed to forms, lab result alert, task workflow…). For “qualitative” measures (e.g. staff motivation, patient satisfaction…), criteria and a rating system (e.g. Likert scale) should be agreed as well as the measure. Notes for presenter: Split into groups of 3 or 4 at this stage and share the benefits between these groups. This is a time consuming exercise so maximise the use of those present by giving them different benefits to devise measures for. Try to get an additional person to help facilitating, or move between the groups to give feedback on the work done and keep delegates focused. Give each group a copy of the benefits definition and a blank measure register template to capture their output. Gather all the registers at the end of the allocated time. Review them after the coffee break in one large group.

Coffee break

Reporting Review and agree: Measures definition Additional data capture activities required Dates for these and baseline measurements Reports frequency and recipients It is unlikely that we will have all the information we need to finalise this today. Some further work will be required to gather pay rates, items cost or benchmarking figures from national statistics, research or case studies. We need to ensure that measure definitions are agreed in principle and that volunteers are found to complete the work required to gather the data required to finalised each measure. One last consideration is how long after implementation can the measurements be made and what degree of realisation is expected on that date (partial or full). Remember to collect baseline data before any changes are made and to include calculations and assumptions in the documentation so how the figures shown have been calculated is clear and this can be replicated when monitoring and evaluating the benefits at a later stage. Notes for presenter: Measures definition: finalise and agree each measures with the whole group. Ensure there is no double-counting. If there is, use the measure once and cross-reference to all the benefits it applies to. What further data is required? Who will gather this? Get named individuals and agree dates for returns. Reporting template: Everyone should be using the same template for reporting to ensure consistency and ease of data collation. Re-use an existing reporting template; or use the templates in the eHealth toolkit if there isn’t anything readily available in your organisation. Reports frequency and recipients: when will each benefit be realised (date or estimate, i.e. 3 months after roll-out). How often should they be monitored? Who should receive the reports?

Governance What are the: Roles and responsibilities for Gathering data Pulling together and distributing reports Analysing and devising remedial actions Transition arrangements to ensure continued monitoring and reporting of benefits and performance after implementation Notes for presenter: Go over what governance has been set out by the Project Manager and ensure all roles can be attributed and the transition process is workable. Who is going to collect the measurements? How will the measurement be collected? (data sources) Who will oversee the collation of reports? Who will the results be reported to? Who will conduct benefits reviews? Who will be responsible for devising remedial actions? Who will be responsible for communicating about the benefits side of the project? The benefit strategy should include a robust mechanism to monitor progress and ensure that reports are distributed to the appropriate people. Those responsible for taking and coordinating corrective actions where benefits are not being met should be clearly defined. There needs to be arrangements in place to ensure that these monitoring and reporting activities continue after the project closure, which is when the benefits are likely to accrue.

Don’t re-invent the wheel Talk to your Management Information / Performance Management Teams: What are the current levels of performance and what measures are available? What improvements are being targeted in terms of Quality and Efficiency? Is there any cross over with other service improvement/18 weeks activities? What work or learning points from other projects can be used?

Before you go: In your team: Who (name) will take forward the additional data capture activities required? Who (name) will take on any/all of the reporting and monitoring responsibilities in your area? If not known, date names should be emailed by In your team: Share the proposed measures: Are they valid measures of success? Is the data available and reasonably easy to collect? Get in touch with any feedback or if you need any help! Notes for presenter: Summarise what was agreed during the workshop in terms of tasks, names and deadlines. If this was not agreed, ask delegates to identify an appropriate person in each business area to carry out the work and email this back by a given date.

Next Steps Finalise measures, templates and schedules. Use the Measures Register and Benefits Realisation Plan to capture today’s work. Determine the financial value of benefits where possible. Identify dependencies in order to plan and deliver necessary changes. Notes for presenter: The project team will now create the measures log and finalise the measure definitions, reporting governance and reports schedule from what was agreed at the workshop. This should be circulated to all those concerned in plenty of time for them to be able to set up systems to capture the data required for the baseline measures.