1. 1 Workforce Assurance Tool Training Guidance Pack - DRAFT NHS North West Friday, 11 January 2013

2. 2 Contents The need for effective workforce assurance Assurance framework overview Current assurance challenges and how the system will meet these challenges The intended benefits of the system Principles underpinning the system Key components of the system How the system can by used by different groups Expected levels of efforts and return to use the system The data that the system will have access to The main components of the tool System views, navigation and functionality Triggers, metrics and data thresholds Access to the system User exercises Key Questions and Answers Module 1 Background and Context Module 2 The system and how it will work Module 3 Intro to the Tool Module 4 FAQ’s

3. 3 Module 1 Background and Context

4. 4 The need for effective workforce assurance a.Right people, right place, right time An appropriately sized, skilled, organised and motivated workforce is critical to delivering efficient, quality & safe services. b. Managing Risk Workforce assurance in the NHS is the process of managing risk and assuring that the composition of the workforce can deliver cost effective, safe and high quality care. Regardless of reforms to the NHS that result from ‘Liberating the NHS’, the need for assurance processes that review and challenge workforce plans is likely to remain. c. Quality & Safety Who leads this process in the new system architecture is yet to be determined but it is critical providers are given the tools to review the impact of workforce changes on quality and safety for themselves – this is an important part of the SHAs’ legacy. d. Manage Change Assurance is needed due to the pressures and inherent risks associated with large scale workforce change from: Tough financial settlements – workforce expenditures represent 70% of the total NHS spend and although in recent years workforce numbers have increased inline with overall spending growth, there is now a very significant productivity challenge related to achieving the overall £20 billion national QIPP target. Increasing demands for services, and the challenges and focus resulting from the Robert Francis Inquiry into Mid- Staffordshire NHS Foundation Trust. 3 – Tool 1 – Background 2 – System4 – FAQ’s

5. 5 The need for effective workforce assurance Workforce assurance has impacts at all levels of the system: on providers to improve their workforce planning, monitoring systems and impact analysis of their increased productivity efforts. Fundamentally assessing their workforce changes and the impact on quality and safety on education commissioners to make better education commissioning decisions on service commissioners to undertake assurance with respect to quality and safety of provider delivery plans on SHAs with assurance responsibilities to exercise appropriate judgement given the system pressures. 3 – Tool 1 – Background 2 – System4 – FAQ’s

6. 6 The Workforce Assurance Framework provides a method for achieving system assurance that the planned and available workforce can deliver safe and quality services. The assurance framework 1 – Key lines of enquiry referenced in DH KLOE on Safety and Quality Assurance of Workforce Changes The Framework provides a means to identify both current workforce issues and provides an early warning system when reviewing projected workforce changes. The assurance framework also reflects the key lines of enquiry agreed by the DH Operations Board 1 The Department of Health are currently developing a national framework for workforce assurance. This document describes an approach that will support NHS organisations in implementing this framework in a consistent, transparent and highly automated manner. NB. Whilst the framework is, to a limited degree, implemented manually there are significant advantages in terms of automation and empowerment for Trusts to embedding it within a tool. 3 – Tool 1 – Background 2 – System4 – FAQ’s

7. 7 The need for a robust system to support workforce assurance Much of the data collected is currently retrospective and not forward looking Process, dashboards and metrics collected are different in different regions Individual indicators are often looked at in silos Current processes are not supported by robust statistical analysis Current processes are labour intensive as mainly manual processes Currently workforce plans go backwards and forwards between Trusts and Assurers, increasing timescales / tensions Current Challenges The tool will use trend data to project how the picture might change over time which can then be compared to actual performance Good practice adopted consistently across Trusts and regions with a collation of all existing metrics Enables consideration of the importance of interrelationship between indicators and triangulation of workforce, activity and finance Assurance processes underpinned by detailed information base and statistical analysis including detailed literature review Automation reduced manual data manipulation and handling burden Greater visibility of assurance expectations between Trusts and Assurers – a shared tool that can be also be used by individual Providers Solutions from the tool Expertise from 8 of 10 SHAs and the Department of Health have been drawn upon to support system design and to address the current limitations associated with attempting workforce assurance using static data and manual processes. The key challenges identified by this group in relation to workforce assurance and the related desired benefits of the system are detailed below: 3 – Tool 1 – Background 2 – System4 – FAQ’s

8. 8 Module 2 The system and how it will work

9. 9 Better use of technology will help achieve this During the proof of concept stage, a series of challenges that were most appropriately resolved through the use of technology were: The need to collate and manage huge amounts of both structured and unstructured data – this is currently extremely time consuming both for Trusts and SHAs; The need to display dynamic / regionalised views of this data in a consistent way; The need to compare and contrast on an equal footing across all areas of the system; The need to reduce the level of bureaucracy involved in metric analysis and performance management; The need to retain and access a clear audit trail for all analysis and action; The need to enable the assurance process and outputs to be “agile” enough to respond to changes in the operating environment; The need to support workflow and collaboration (again, with a clear audit trail); and The need to test robust workforce scenarios 3 – Tool 1 – Background 2 – System4 – FAQ’s

10. 10 A different approach to assurance What is different? A systematic approach enabling rigorous, structured evaluation Forward and historic looking analysis A statistically robust underpinning for assurance analysis (correlation analysis will help set metric thresholds and identify new areas in need of monitoring) Analysis of correlations and relationships between metrics Escalation routines to enable consideration of escalation in structured way This enables differing groups of users to: Provide a tool and legacy for trusts so that they can self-assure and pre-emptively respond to concerns of assurers Call on robust information and analysis as support for key decision making Take a more integrated approach to assurance activities (looking across activity, finance, quality and workforce) Forecast, and indentify correlations within, data to suggest expected trajectory and to see how seemingly unconnected triggers interact within an organisation Create and drive top table discussions Change the nature of assurance engagement with Trust Boards and Management Teams Justify decisions 3 – Tool 1 – Background 2 – System4 – FAQ’s

11. 11 What the tool is intended for... The Tool IS intended to provide: The workforce element that will link with wider system level assurance Triangulation between workforce, activity and finance data A national platform to workforce assurance that can be adapted as appropriate for each SHA Best practice approach that is built on existing published literature An approach that recognises that assurance criteria will change and be refined over time The Tool IS NOT intended to be: A tool to publish information A performance management tool A data collection tool; existing data is used A directorate, divisional and professional assurance tool as other resources are available for this 3 – Tool 1 – Background 2 – System4 – FAQ’s

12. 12 Underlying principles of the tool 1.Effective workforce assurance is a critical component of helping ensure the ongoing efficient and effective delivery of services across the NHS system. 2.Organisations should be supported to be rigorous in their self assurance processes, to improve outcomes and reduce the need for external assurance. 3.By using data appropriately, users can be supported to make better assurance decisions. This is a decision support tool, not a decision making tool. 4.Consistent high levels of assurance across regions. 5.Considers a cross section of demand, activity, quality, finance and service information rather than considering workforce alone. 6.Balances forward looking assessments with previous performance: i.e. Historic dashboards are a useful input but do not tell the whole story. 7.To minimise effort, Nationally available data will be used wherever possible to reduce the administrative burden. 8.Differences in data quality and availability need to be allowed for and highlighted within the system. 9.Throughout the life of the tool, the system will be actively managed so that as new national datasets become available each period they are uploaded into the tool. Fundamental Tenets Workforce Assurance in Practice Data usage 3 – Tool 1 – Background 2 – System4 – FAQ’s

13. 13 How it will be used by different audiences ProviderCommissionerSystem Assurer Key Focus Supporting trusts to be rigorous in their self assurance processes and to benchmark Supporting commissioners to discharge their assurance responsibilities Providing a framework for high quality, data driven workforce assurance and education commissioning Common Themes Consistency and transparency of assurance approach More effective use of available data Reduction in manual processes and interchange of information What you might do with the system and how you might use it Service leads and clinicians work in collaboration with workforce planners to identify potential future workforce risks, as well as pre-empting and mitigating assurance concerns Clinicians and service leads use to help quantify the links between workforce, safety and quality and benchmark against peers Potential for Executive teams and Board member use to provide key workforce risk assessment Service commissioners use to assess and consider proposed workforce plans using structured, data driven, good practice approach Support discharge of workforce assurance responsibilities Support workforce planning impact assessment Support assessment of system wide trends and failure points System not designed for Detailed operational management at ward and department level Performance management of providers Performance management of the system Advantages Trust can “see what the assurer sees” Trust can validate plans BEFORE submission Trust can recognise and pre-empt assurance concerns at an organisational level Greater insight into the strengths of individual providers Communication mechanism with provider around workforce strength Standardised and consistent view of Trusts in designated peer groups Interactive workflow to achieve assurance with full audit trail Trusts should be better prepared up front 3 – Tool 1 – Background 2 – System4 – FAQ’s

14. 14 Scenario 2: In-year reviewScenario 1: Workforce Planning Provider worked example Consults the dashboard to view overall assurance level Review A Trust works with clinicians, service managers and finance to develop a workforce plan. Before finalisation, Trust uses the tool to review the plan. This enables the Trust to pinpoint areas of risk and highlight where there are tensions between activity, finance and workforce plans. The Trust uses the comparative information, threshold and trend information from the tool to engage in a data driven conversation with clinicians and service managers as to whether alternative approaches need to be considered. In some areas the Trust concludes that despite the indication of risk their plans are in fact sound. In other areas the Trust decides to enhance their plans. The Trust submits their amended plans with a data driven narrative highlighting why they believe they are sound in perceived risk areas. The assurer receives the plans and can consider in light of the Trust’s considered risk assessment. Assess Act Submit Assure Plan Trend projections on dashboard highlight that deviations from plan for one team within the Trust could be indicative of potential quality and patient safety risks if left unchecked. Trust uses projection information to review potential risks and inform internal risk prioritisation processes. If the Trust deems that action is required, they use this information to support engagement between the team’s clinicians and service managers as to nature and extent of concerns. Appropriate action taken within that team, including development of mitigation plans and reworking forecasts as required. Identify Review Engage Act Organisations can also use this tool to review current workforce considerations and should always use the tool in conjunction with more qualitative intelligence 3 – Tool 1 – Background 2 – System4 – FAQ’s

15. 15 Commissioner / assurer worked example Scenario 1: Commissioner demand management Consults the dashboard to view overall assurance level Analyse Commissioning team decide to test the effect on workforce in Trust A from their demand management QIPP plans. Commissioners use the tool to model the activity and income changes on Trust A and understand any workforce, quality or safety issues this may cause. The Commissioner chooses a scenario which gives the least risk, as seen by the relative score from the dashboard, and discusses impacts with Trust A. Trust A uses the same scenario and discusses the dashboard score with managers and clinicians to understand the deliverability of the QIPP plan. The Commissioner and Trust A agree the scenario for demand management and the potential effect on the Trust and target decommissioning and resource toward delivering the target together. Scenario 2: Assurance during transition* Discuss Evaluate Deliver Identify Assurer chooses the organisations to be assured within the tool and opens dashboards for each of these organisations in a separate web browser window Assurer reviews comparability of potential risks across the group, including assessing trends and changing performance over time. Assurer prepares guidance on key comparable risks displayed by the dashboard and distributes to providers while also offering support and resource to help mitigate potential risks. The assurer continues to monitor month by month performance against the key risk areas and works with providers to target mitigation. Review Engage Monitor *Assurance will not just be limited to transition but transition is used as an example due to the heightened risk environment Plan 3 – Tool 1 – Background 2 – System4 – FAQ’s

16. 16 Data included in the initial build Activity: Health Episode Statistics, Quality Outcomes Framework, Unify, forward looking activity plans, Information Centre Workforce: ESR, Census, forward looking workforce plans Quality/Safety: STEIS, Unify, NHS Patient and Staff Survey, IC Complaints data base, NPSA Incident and CAS systems, HPA, Monitor Finance: FIMS, forward looking finance plans, Monitor Sector specific data that would be useful in the initial build but are not available The acute sector and national datasets provide the richest streams of data for assurance as this has been the focus of reporting and data collection for the NHS / DH in recent years Mental Health and Community services will be partially covered due to the absence of quality national data that will limit the robustness of framework (non workforce information primarily) Primary care data is not robust enough to provide for primary care workforce assurance and therefore primary care data will be looked at in relation to Provider Assurance and give a perspective on the local health economy Sector specific data that will not be included The following sectors will not be covered in the system and therefore related data will note be included: Dental practices Ophthalmology Pharmacy Private sector Social Care Data within the system 3 – Tool 1 – Background 2 – System4 – FAQ’s

17. 17 Input required from Trusts Level of Input Benefits Received The tool has been designed to require zero / minimal input from Trusts, and can operate with or without forward workforce plans. However, with these plans the value delivered from the tool with be far greater. Adding workforce plans, with summary finance and activity information Without Plans: the tool will provide triangulation, comparisons, and projection using national data sets covering workforce, quality, safety, activity and finance. With Plans: This will include comparison against plan functionality and the ability to use trend projections to test the quality and safety implications of the plan. The scenario modelling element of the tool can be done without providing visibility of the plan to assurers. Do Trusts have to input data to the tool? All data will be provided centrally with two potential exceptions: If Trusts wish to upload a workforce plan / scenario into a private part of the tool for review (i.e. Before I submit my plan, can I check if it will flag up any major areas of concern at the SHA?), then this will be possible at the users discretion; and Where a regional collection process does not exist, then approved plans may be uploaded by Trusts at their own discretion. 3 – Tool 1 – Background 2 – System4 – FAQ’s

18. 18 Module 3: part 1 Introduction to the Tool

19. 19 My Analysis My Resources My Actions Organisation Dashboard Metric Dashboard Help Workforce Assurance Portal My Flags Key Components My Qualitative Indicators My Plans My ThreadsMy Notes My Assurance Logs Supporting Materials Community Data Sources FAQ Workforce Dictionary 3 – Tool 1 – Background 2 – System4 – FAQ’s

20. 20 Purpose: This component contains the primary dashboards within the system that operate at regional, organisation and specific metric levels. Each dashboard highlights areas of potential workforce risk and provides functionality to allow users to drill into risk areas to see more detail. The dashboards also allow users to view the underlying data, and perform comparisons against peer organisations. Key features: Drill down functionality to enable users to see more detail Further information can be found on each metric, including how the metric has been calculated. The source data indicators can also be viewed. Links to enable users to jump straight to actions from the dashboards. Geographic mapping at a regional level My analysis Regional User View Organisational Dashboard Metric Dashboard 3 – Tool 1 – Background 2 – System4 – FAQ’s

21. 21 Clicking on the assurance cheese opens an Organisation summary panel The organisation summary panel shows a selection of key indicators for the trust The section at the bottom of the organisational summary panel provides quick links to the organisation dashboard and actions Regional User View 3 – Tool 1 – Background 2 – System4 – FAQ’s

22. 22 Click in any of + or – signs to expand / contract the section Summary scores are colour coded and displayed against each assurance category It will initially present the scores against each assurance group, using the navigation users can drill down into detail on specific metrics Organisational dashboard 3 – Tool 1 – Background 2 – System4 – FAQ’s

23. 23 Contribution of this metric to the Assurance Group score Opening up different assurance categories allows you to drill down to metric information. Clicking on any of the icons provides further information on the metric. The icons next to each metrics indicate: Uploaded: the values displayed have been directly uploaded from another source Calculated: the values displayed have been calculated from other information uploaded. Actual value of this metric Weighting Things to note on adding of scores: If there is only 1 metric then this will be the same as the trigger score (e.g. CIP example) If there is more than one metric in the trigger then these wont necessarily equal the trigger score due to the weighting Metrics 3 – Tool 1 – Background 2 – System4 – FAQ’s

24. 24 Indicators, Metrics and Thresholds Target Value Intervals from Target Value Metric e.g. Midwife: birth ratio An indicator could be the number of midwives, or the number of births. The data from these indicators is used to create different metrics. The metric will be assigned a risk level depending on the threshold. Some thresholds are scales where one extreme is good and the other is bad. Other thresholds are ranges where there is an ideal level and anything either side increases risk. In our example a threshold would be a scale, and less than e.g. 3 midwives per birth is ‘high risk’. Indicator (590 in total) The base level of data used to make metrics This data comes from various source data sets Metric Metrics display information about an aspect related to workforce Metrics can also be direct (taken directly from one indicator) or compound (calculated by combining more than one indicator) Threshold The threshold is the level at which metrics turn green, amber or red, indicating the risk associated with each metric The threshold is different for each metric and based on the value needed for them to be considered high risk, and turn red, or low risk, and turn green The threshold can take the form of a scale or range Example: Indicators, Metrics and Thresholds 3 – Tool 1 – Background 2 – System4 – FAQ’s

25. 25 Score Each metric within the Assurance Map is given a score indicating the level of risk (low, medium, high) These scores add up, once normalised and weighted, to generate the overall assurance group score and indication of risk Weighting Each metric is assigned a weight based on six categories: Patient safety Patient satisfaction Staff safety Quality Improvement Financial control Staff Health and Wellbeing Each category is sub weighted, and the score in each category is used to come to a final score for the metric Aggregation Scores can then be aggregated to give a score for each Trigger, Trigger Group and Assurance Group. Each Trigger and Trigger Group are also weighted Scoring Trust A Nurse to bed ratio = 0.1 Nurse to bed ratio normalised score = -0.8 Nurse to bed ratio rating = Red (Based on set threshold values) Calculation Example: Score, Weighting and Aggregation 3 – Tool 1 – Background 2 – System4 – FAQ’s

26. 26 The scoring and weighting of metrics should be seen as an overall process with distinct, time-bound, activities and linkages. WeightingAggregationAdjustmentScoring Using established and new thresholds we normalise threshold values and assign a comparable score to all metrics within the Assurance Map Each individual metric score is then weighted by a defined criteria to enable an hierarchy of importance to be constructed All weighted scores can then be subsequently adjusted to account for data coverage and confidence based on statistical analysis. The aim is to account for data quality and quantity problems and show how “reliable” the underlying data used in metrics is Scores can then be aggregated to give a score for each Trigger, Trigger Group and Assurance Group Things to note: Scores come in three forms Raw scores provide a base level indication of the perceived risk Weighted scores also indicate perceived risk, but some metrics are identified a potentially larger impact on the organisational workforce It is the weighted score, adjusted for data quality so we know which data we can rely on How the scores add up 3 – Tool 1 – Background 2 – System4 – FAQ’s

27. 27 Certain data sources are incomplete for all periods. This could be due to timing or data quality issues. To solve this problem the system will determine values it needs to fill and select from a variety of methods for inferring missing. This can be forward looking or backward looking, using a variety of methods. These include: Use last known values Leave blank Forecasting - Trends are extrapolated from the data. The type of data set will determine the model/method used to extrapolate. Forecasts for 3 and 12 months are then computed Pro-rating – Use linear interpolation to estimate the data between two points. This may include using a planned future value compared to an actual, allocating the difference equally across the time periods Relationships - The data will be examined to reveal relationships between metrics. Relationships derived from this process will then be fed in to forecasting models as inputs Custom calculations (Not used currently) – Use a custom specified mode, calculation or alternative data source to replace the data Data source provided with gaps Select data where the value is “null” based on comparisons with the expected data set (e.g. Open periods, actuality) Data that requires inferring for the future. Based on the Indicator Type, chose which approach to use Use previous value Use forecast Etc... Package is called and data is inferred / delivered Use replacement System infers and returns data Data that requires inferring for the past. CHOOSE METHOD: Inference Mechanisms 3 – Tool 1 – Background 2 – System4 – FAQ’s

28. 28 Purpose: Whereas the My Analysis section primarily allows you to view information within the system, the My Actions component allows you to undertake six different types of actions within the system Key Actions: Logs: regional assurers can automate their process of monthly log recording Threads: records communications and exchange between Trusts and regional assurers can be recorded and tracked Quantitative Indicators: can be recorded, viewed and edited Notes: user specific notes can be recorded which can only be viewed by the author Flags: alerts and flags can be viewed and overridden. Plan data: trust plans can be uploaded into the system Reports: reports can be viewed and searched for my organisation, date and scoring method Things to note: Not all action types are available to all users. Which actions you have access to will depend on the user profile you have. Within the majority of the My Analysis components there are links to the actions that you may wish to take at any given time. My Actions 3 – Tool 1 – Background 2 – System4 – FAQ’s

29. 29 Purpose of My Resources: The component provides access to a range of help resources for the system including: FAQs Help files Training Guides The section also provide access to the community forums / discussion groups Purpose of Administration: This component provides the SHA with the capability to control security access to information in the portal based on organisational preferences and reporting agreements It also enables the SHA to manage the standardised list of workforce dimensions so that these can be updated should the need arise As a user, you will not be able to view the Administer component of the portal My Resources and Administration 3 – Tool 1 – Background 2 – System4 – FAQ’s

30. 30 Plan Data Considerations: Uses year one data within the tool to provide the most robust and detailed information The 4 planning years are projections and therefore do not have the same degree of reliability Given the lead in time for education and training data that is four year ahead is needed to support education commissioning Understanding any quality and safety issues associated with the plans is important to ensure the most robust intelligence Workforce assurance tool needs to use the most accurate plans available and recognise that provider owned plans are often more complete for activity data as it reflects multi- commissioners activity levels 3 – Tool 1 – Background 2 – System4 – FAQ’s

31. 31 Freedom of Information (FOI) Requests Considerations: The majority of the information used in the tool is already publicly available and therefore will remain eligible for FOI requests The planning data that will be within the tool is already held by SHAs and the DH. Access to the planning data will still only be available to the individual organisation, PCT Commissioning Cluster and SHA. The use of the data within the tool will be under the same FOI conditions as if requested from the SHA/ DH at present. The tool will not be used to publish data but will used to support workforce assurance which will include improving the quality and safety of workforce plans. Most SHAs already use Integrated Performance Dashboards which review plans and performance against plan, the only difference with this tool is the length of the plans reviewed. 3 – Tool 1 – Background 2 – System4 – FAQ’s

32. 32 Foundation Trusts data Considerations: The range of data included in the tool for Foundation Trusts will be different to that of Non- Foundation Trusts. This issue is much wider than the workforce assurance tool and therefore it is proposed that the information available is used. The tool is not intended to collect additional information or intelligence and the project team will be reviewing the national discussions around quality, safety and Foundation Trusts to ensure it keeps up to date with any developments and continues to use the best available information. 3 – Tool 1 – Background 2 – System4 – FAQ’s

33. 33 Accessing the portal Users access the portal by navigating to the Portal homepage in Internet Explorer Users will be prompted to enter a username. Navigate to the relevant URL:  Live Environment: https://wfa.westmidlands.nhs.ukhttps://wfa.westmidlands.nhs.uk  Log in using the username and password provided  Note that users are signed out of the portal after 30 minutes of inactivity Things to note: An NHS N3 Connection is required to Access the portal The portal operates over HTTPS to increase security If problems occur the first ports of call are the Help sections in the My Resources tab of the tool, Training and pre-read material. For further issues, speak to your local IT team or the Helpdesk – information on this is available in the about section of the portal (My Resources) Helpdesk: Telephone: 0207 303 8777 Email: wfp@deloitte.co.uk 3 – Tool 1 – Background 2 – System4 – FAQ’s

34. 34 Module 3: part 2 User Exercises To be completed

35. 35 Module 4 FAQ’s

36. 36 Questions: are Foundation Trusts included? Will this be of benefit to Foundation Trusts? Whilst having a differing status within the NHS, Foundation Trusts face comparable workforce pressures and a need to assure the appropriateness of their workforce. It is envisaged that FTs will recognise the value that this tool can bring for a nominal investment and use it as a check and balance within their planning processes. Will this be of benefit to Foundation Trusts? At a system level, the assurance responsibility remains regardless of FT status, and proposals suggest that if anything this responsibility will be strengthened rather than reduced as a result of any changes under 'Liberating the NHS' During their remaining period, there is an expectation that SHAs and PCT Commissioners / Clusters will 'hold the ring' on system assurance for all NHS services. Do FTs have different data sets? National data sources and public domain information will be utilised to minimise the information requirement on all NHS bodies. Recognising that differing sets of data will be available for FTs, the system will use the information available to indicate risk. Obviously as the quantity and quality of information decreases so will the ability to effectively highlight risk areas and this will be clearly identified in the tool. 3 – Tool 1 – Background 2 – System4 – FAQ’s

37. 37 Questions: is Primary Care included? Primary care obviously has a large part to play in the assurance of a health system but it also marks the start of a large proportion of patient journeys. This importance is reflected within the assurance model by how we seek to measure referrals and supply of primary care workforce to try and judge the impact on the secondary sector. Why are you not using QOF as a metric? The lack of real variability in QOF scores makes using this data of limited benefit. How are you linking primary care to the secondary care sector? We are looking to link GP practices to secondary providers by mapping practices to commissioners and then identifying which providers these commissioners are “lead” commissioners for. Why are you concentrating only on GP practice workforce and referrals? We think these metrics should give a proxy for supply and coverage in the primary care workforce that can be matched with referral patterns, and subsequent secondary activity, to approximate demand emanating from primary care. We will look for correlations between these data points that can be used in system assurance. 3 – Tool 1 – Background 2 – System4 – FAQ’s

38. 38 Questions: what is the assurance map The assurance map is a mechanism for considering potential workforce risks in a structured, logical and data driven manner. The map is hierarchical consisting of a number of categories and levels. Trigger groups Triggers Assurance Categories Metrics Assurance categories: There are four assurance categories that group together trigger groups that show: 1.Assurance of plan (quality of the plan) 2.Assurance against plan (variance analysis) 3.Organisational workforce profile (general organisational indicators) 4.Qualitative Indicators (not metric based but a prompt for areas to look for in an assurance process) Trigger groups: a collection of similar assurance triggers e.g. workforce or finance Triggers: a collection of similar metrics / indicators that can be seen as a natural group e.g. Staff Turnover or Cost Improvement Metrics: individual, national, local and bespoke metrics on workforce, quality, safety, finance, activity, etc. High level Low level 3 – Tool 1 – Background 2 – System4 – FAQ’s

39. 39 Questions: how were the triggers decided The short-list of triggers were decided through consultation with regional reference groups / stakeholders within SHAs. We received a number of datasets from each SHA which had varying levels of detail, quality and confidence in the validity. It is recognised that metrics will be refined and further developed over time as more data becomes available, as reporting requirements change or further research shows that there are other factors or metrics that should be added or removed. NB. The triggers will be refined and added to as time goes on, and will be subject to ongoing clinical input and validation. Triggers Trigger group Metrics Refined metrics list on: Availability of data; Links to workforce, quality, safety or planning; and Links to organisational health outcomes Ran a metrics workshop with key SHA representatives to agree core set of triggers/metrics Asked SHAs to review first cut of metrics and comment (using regional reference groups) Looked at national recommendations and best practice Looked at data available 3 – Tool 1 – Background 2 – System4 – FAQ’s

40. 40 Questions: how granular is the information? The initial metric set for the build will focus on overall organisation-level performance with some speciality/ward/condition level measurement. Where possible, within the constraints of the available data the statistical analysis will look for correlations at a more granular level. Where relationships are identified more granular metrics will be added to the tool. Activity The majority of metrics will measure organisation level performance i.e. referral to treatment times, bed occupancy) A smaller sub-set of metrics will look at speciality/condition level activity and performance i.e. LOS for select procedures, maternity performance Finance FIMS data will provide organisation level financial performance details i.e. total pay bill, clinical to non- clinical staff pay, % pay bill spent on overtime Non FTs only Workforce Workforce will look at staffing and overall workforce performance at the staff group level (Doctors, Nurses, Midwifes, Health Visitors) Quality/Safety Quality and safety measures will exist largely at the organisation-level given the availability of data i.e. Number of SUIs, NPSA patient incidents As more granular data becomes available it may be added to the system 3 – Tool 1 – Background 2 – System4 – FAQ’s

41. 41 Questions: how are data quality issues managed? Using statistical analysis to provide a data confidence and coverage score will ensure that the accuracy of information is incorporated into the assurance framework and system outputs. Creating a tool that users adopt will drive up data quality. Data Quality Data Use Confidence An automatically calculated measure of “data quality” and “predictability” of a data set Coverage An automatically calculated measure of “completeness” of the data underlying a metric score. This can be over time or within peer group Definitions Historic experience suggests that as assurers, commissioners and trusts realise the benefits of the system, the data will cleanse and the quality of national and plan data that is monitored will increase. 3 – Tool 1 – Background 2 – System4 – FAQ’s

42. 42 Data is given a score representing confidence in the metrics which have been used to calculate it. Confidence is driven by the coverage and regularity of the data set. Each score is then weighted based upon its relative importance, for example recentness of gaps might be more significant than history length. The worst case scenario score for each indicator, used in a metric, is then taken. Scores are also time-dependent due to the history score and will be based on a rolling 3 year period. Factors that determine data quality are: DescriptionRationaleMeasurement methodView History How long the data has been collected for. It has been assumed that data which has been collected for 3 years will be more reliable Calculate the time between today and the time of the earliest observation. Use this to score the data. By source Granularity Is the source data supplied on a monthly, quarterly or yearly basis More granular data enables more detailed information Scores allocated for monthly, quarterly and yearly dataBy source Recentness of gaps Whether the gaps in the data are recent or further in the past More recent gaps will have a more significant impact upon the confidence in the data collection. The past 3 years of months are allocated scores from 1-36 This is multiplied by 1 for any months that have missing data. By indicator Number of gaps How many missing observations are there out of the expected data set. The more missing data, the poorer the data quality Comparing actual observations with expected, assigning a score of 1 for each missing observation By indicator Gap length Continuous sections of missing observations, e.g. Consecutive months. A large gap size indicates serious concerns with the data quality. The first gap encountered is scored at 1. A continuous gap from this period is then scored 2 and so on, resetting to 1 if there is an observation in between By Indicator Frequency How often the gaps occurA high frequency of gaps means that there are systematic problems with the data as opposed to surprise events Score 1 for the first gap encountered and do not score continuous gaps from here. By Indicator Pattern / regularity Whether gaps in the data occur at the same time of the year Gaps map occur for valid business reasons which makes them less concerning than ad hoc gaps in the data Assign 1 if there is an observation in a month. Subtract this from the previous year’s score. Do this for each year. Add the positive absolute scores together. By Indicator Questions: what is the process of adjustment for data quality? 3 – Tool 1 – Background 2 – System4 – FAQ’s

43. 43 Questions: who can see what information? When accessing the system users will be working with data for which they already have access (either because it is within the public domain, is their own data, or because it is information used in current assurance and/or performance management arrangements). The benefit of the system will not be enhanced access to data, rather enhanced access to data that has been coordinated, triangulated and presented in a way that optimises the workforce assurance process. Providers Commissioners System Assurers 1.Plans / scenarios can be “reviewed” in private by the Trust 2.Benchmarking by Trusts will be anonymous – you will see your position and the distribution, but not who the other data points relate to. 3.Ideally, the clinical value of the tool will encourage organisations to increase the role of Directors of Nursing, Medical Directors and Clinicians in decisions relating to workforce planning. Commissioners will be able to view data related to the providers from whom they commission. Assurers will have a complete view of the data for those organisations they are responsible for assuring and not those of other regions/areas (nb. this may be flexed dependent on changes to the NCB regions in future) 3 – Tool 1 – Background 2 – System4 – FAQ’s

44. 44 Questions: why is data analytics important? What additional capabilities will be enabled by data analytics? The ability to use forecasting to help identify potential issues before they occur, and enabling a behavioural change in assurance – which should become mostly proactive rather than mostly reactive; The ability to challenge accepted wisdom by supplementing expert opinion with robust statistical analysis; The ability to accurately analyse trends and cumulative workforce changes to provide a more rounded picture of the potential impact these changes may have on quality and patient safety; The ability to produce more sophisticated workforce metrics, and partly avoid the “yes, but it’s more complicated than that” factor; The ability to present logical data-driven assumptions around workforce performance benchmarks and expectations; and The ability to provide some level of “self-audit” to planners and provide feedback and challenge on their assertions around workforce plans and forecasts. 3 – Tool 1 – Background 2 – System4 – FAQ’s

45. 45 Questions: what happens to the tool after the SHAs? Workforce assurance will take place in the proposed new structure for the NHS. With the challenges facing the NHS, There is a strong rationale for this being done by an overall assurer. This is, of course, to be decided and could be the remit of the National Commissioning Board, Health Education England or Local NHS Education and Training Boards The tool will provide a robust way of assuring the workforce for commissioners and providers alike. The tool will provide Commissioning groups/Commissioning Clusters or GP Consortia confidence in current workforce performance and workforce plans, in relation to quality, safety and patient experience. It would also simplify negotiations around workforce plans and assurance which would condense timescales and potential tensions. 3 – Tool 1 – Background 2 – System4 – FAQ’s