1. NICE Evidence Standards Framework for Digital Health Technologies

2. Evidence for Effectiveness and Economic Impact - Why
Digital health technologies, both those that interact with NHS data systems and those that stand alone, continue to feature in NHS planning assumptions, but:
Manufacturers are not clear what evidence they need to produce
Commissioners are not clear what evidence they should be looking for
The NHS is slow to adopt valuable innovations
NICE is working with NHSE, PHE, MEDCITY and Digital Health London to develop a trusted and respected set of standards on what evidence to produce for different types of digital health technologies;
allowing innovators to understand the level of evidence they need to produce making evidence generation plans faster and more cost-effective for them,
allowing the NHS to commission, deploy and scale clinically and cost-effective digital health tools that meet demand.

3. Standards for digital health technologies (DHTs)
Cross-government work on a ‘code of conduct’ for the digital health market place based on an emerging set of standards covering at least 6 broad categories.
Different ‘standards’ categories are being covered by different organisations and projects.
This project will focus on how to produce evidence of clinical effectiveness and effectiveness in terms of behaviour change. This will also cover value for money / economic impact.
How to demonstrate safety will be covered by the ‘regulatory standards’ circle.
Interoperability Standards
PRSB
NHS D
InterOp
International – FIHR/HL7
Evidence for Effectiveness
Standards
NICE, NHSE, PHE, MEDCITY, DHL
Technical / Design Standards
NHS D / Code 4 Health
Industry
Information Governance + Privacy
NHS D / E
Industry
Commissioners
BSI, ICO, HRA
Standards & Interoperability
DHSC
Regulatory
Standards
(safety)
CQC
GPhC
MHRA
BSI
Commissioning Standards
NHS E
PHE
Commissioners

4. Initial code of conduct for data-driven technologies (DHSC, September 2018)
10 principles
1 Define the user
2 Define the value proposition
3 Be fair, transparent and accountable about what data you are using
4 Use data that is proportionate to the identified user need (data minimisation principle of GDPR)
5 Make use of open standards
6 Be transparent to the limitations of the data used and algorithms deployed
7 Make security integral to the design
8 Define the commercial strategy
9 Show evidence of effectiveness for the intended use
10 Show what type of algorithm you are building, the evidence base for choosing that algorithm, how you plan to monitor its performance on an ongoing basis and how you are validating performance of the algorithm.
5 commitments
1 Simplifying the regulatory and funding landscape
2 Creating an environment that enables experimentation
3 Encouraging the system to adopt innovation
4 Improving interoperability and openness
5 Listening to our users

5. Evidence standards for DHTs – principles and progress to date
Separate consideration of (clinical) effectiveness and economic impact
Informed by the needs and experience of innovators and evaluators
Levels of evidence for different types of technologies
Signpost relevant research/HTA educational resources e.g. study design, critical appraisal, assessing quality
Understandable, usable, subject to regular review, based on existing work
EXCLUDED
Not about evaluation of DHTs
Excludes safety, regulatory, technical, commissioning, etc.
DHTs out of remit: ‘True AI’.
Progress
Started June 2018
BETA launch December for comment and feedback

6. The evidence standards framework in outline
Standards for clinical effectiveness evidence
- based on functional classification
Standards for economic impact evidence
(Educational/signposting resources/case studies)
MC - new slide

7. Evidence for effectiveness (EfE) for intended use

8. Approach: a risk-based evidence framework
Classify DHT using functional classification to identify relevant evidence tier
Use contextual questions to identify relevant evidence level within each tier
….?
..... ?
….. ?
a b
Evidence for effectiveness tables describe evidence standards

9. Risk-based functional classification
Increasing evidence requirement

10. Contextual questions identify evidence level within a tier
How serious could the consequences be to the user, if the DHT failed to perform as described?
Are the intended users of this DHT considered to be in a potentially vulnerable group such as children or at- risk adults?
Is the DHT intended to be used with clinical support?
Does the DHT include machine learning algorithms or artificial intelligence?
Is the economic impact or organisational risk of the DHT expected to be very high?

11. Tier 1 evidence standards
System services
DHIs with no measurable patient outcomes but which provide services to the health and social care system
Tier 1 evidence standards
Tier 1 evidence of effectiveness
Evidence categories
Minimum evidence standard
Best practice standard
Credibility with UK H&SC experts
Be able to demonstrate that the DHT has a plausible mode of action that is viewed as useful and relevant by professional experts or expert groups in the relevant field.
Show that relevant clinical or social care professionals working within the UK health and social care system have been involved in the design, development or testing of the DHT
OR for DHTs that were not developed in the UK:
Show that relevant clinical or social care professionals working within the UK health and social care system have been involved in the sign-off of the DHT, indicating their informed approval of the DHT
Published/publically available report documenting the role of relevant UK health or social care experts in the design, development, testing or sign- off of the DHT
Relevance to current care pathways in the UK H&SC system
Evidence to demonstrate that the DHT has been successfully trialled in the UK health and social care system, showing that it is relevant to current care pathways and service provision in the UK.
Also evidence that the DHT is able to provide care to the scale required by the expected user population. This may include servers used by the DHT being capable of managing the expected number of users.
Evidence to demonstrate successful implementation of the DHT in the UK H&SC system

12. Tier 1 evidence of effectiveness continued
Acceptability with users
Be able to demonstrate that representatives from relevant user groups were involved in the design, development or testing of the DHT
Be able to demonstrate that representatives from relevant user groups were involved in the design, development or testing of the DHT and demonstrate data to show that users are satisfied with the DHT.
Equalities considerations
Where feasible, the DHT should:
Contribute to challenging health inequalities in the UK health and social care system, or improving access to care among hard-to-reach populations.
Contribute to promoting equality, eliminating unlawful discrimination and fostering good relations between people with protected characteristics (as described in the 2010 Equalities Act) and others.
Evidence to demonstrate the use of the DHT in hard-to-reach populations
(If relevant) Accurate and reliable measurement
Data based on people’s use of the DHT showing that:
measurements done by the DHT are:
accurate
reproducible
relevant to the range of values expected in the target population
the DHT is able to detect clinically relevant changes or responses
n/a
Technical data showing that numerical, text, audio, image-based, graphic-based or video information is:
not changed during the transmission process
is not biased by the data ‘value’ expected from the target patient population
Quantitative data based on people’s use of the DHT, showing that numerical, text, audio, image-based, graphic-based or video information is
not changed during the transmission process and

13. Tier 2 evidence standards
Inform
Provides information to citizens, patients or clinicians.
Tier 2 evidence for effectiveness
Minimum evidence standard
Best practice standard
Reliable information content
Be able to demonstrate that the health information provided by the DHT is:
valid (aligned to best available sources e.g. NICE guidance, relevant professional organisations or recognised UK patient organisations, and appropriate for the target population)
accurate
up to date
Reviewed and updated by relevant experts at defined intervals, such as annually.
Sufficiently comprehensive
Evidence of endorsement, accreditation or recommendation by NICE, NHSE, relevant professional body or recognised UK patient organisation OR
Evidence that the information content has been validated though independent accreditation/certification such as The Information Standard or HONcode certification
Ongoing data collection to show usage of the DHT in line with the value proposition
Commitment to:
Ongoing data collection to show usage of the DHT in the target population
Report this data in a clear and useful format to commissioners
Public reporting of:
Report this data in a clear and useful format to commissioners
Simple monitoring
Includes general health monitoring using fitness wearables and simple symptom diaries
Communicate
Allows people, patients or clinicians to communicate

14. Tier 2 evidence for effectiveness continued
Ongoing data collection to show value of the DHT in line with the value proposition
Commitment to
Ongoing data collection to show user outcomes (using non-patient identifiable information) to show ongoing value
Report this data in a clear and useful format to commissioners
Public reporting of:
Ongoing data collection to show user outcomes (using non-patient identifiable information) to show ongoing value.
Quality and safeguarding for communication platforms
Demonstrate that appropriate safeguarding measures are in place around peer-support and other communication functions within the platform:
Describe who has access to the platform, including clinical and social care professionals, carers, charity or 3rd sector providers, commercial organisations or other users.
Describe what the roles of these people are within the platform, and describe why these people/groups are suitable and qualified to have access.
Describe any measures in place to ensure safety in peer-to-peer communication, for example through user agreements or moderation.
 n/a

15. Tier 3a evidence standards
Preventative behaviour change
Address public health issues: smoking, eating, alcohol. sexual health, sleeping and exercise
Tier 3a evidence for effectiveness
Evidence categories
Minimum evidence standard
Best practice standard
Demonstrating effectiveness
High quality observational studies demonstrating relevant outcomes, such as:
Relevant user outcomes, (behavioural or condition related) such as reduction in smoking or improvement in condition management.
evidence of positive behaviour change
User satisfaction
High quality comparative studies which may be observational (such as cohort) or experimental (such as trials), showing improvements in relevant outcomes, such as:
Patient reported outcomes (preferably using validated tools) including symptom severity or quality of life
Other clinical measures of disease severity or disability
Healthy behaviours
Physiological measures
User satisfaction and engagement
Health and social care resource use, such as admissions or appointments.
The comparator should be a care option that is reflective of the current care pathway, such as a commonly used active intervention.
Use of appropriate behaviour change techniques
Be able to demonstrate that the techniques used in the DHT are:
consistent with recognised behaviour change theory
recommended practice (aligned to guidance from NICE or relevant professional organisations)
are appropriate for the target population
Published qualitative or quantitative evidence showing that the techniques used in the DHT are:
based on published and recognised effective behaviour change techniques
Self-manage
Allows people to self-manage a specified condition. May include behaviour change techniques

16. Tier 3b evidence standards
Treat
Provides treatment
Guides treatment
Tier 3b evidence for effectiveness
Minimum evidence standard
Best practice standard
Demonstrating effectiveness
High quality comparative studies which may be observational (such as cohort) or experimental (such as trials), showing improvements in relevant outcomes, such as:
Patient reported outcomes including symptom severity or quality of life
Other clinical measures of disease severity or disability
Healthy behaviours
Physiological measures
User satisfaction and engagement
The comparator should be a care option that is reflective of the current care pathway, such as a commonly used active intervention.
High quality randomised intervention study in the UK health and social care system, comparing the DHT to a relevant comparator and demonstrating a positive clinical outcomes in the target population, using validated condition-specific outcome measures.
Active monitoring
Tracking patient location, using wearable to measure, record and/or transmit data about a specified condition.
Calculate
A calculator that impacts on treatment, diagnosis or care
Diagnose
Diagnoses a specified condition
Guides diagnoses

17. Evidence for economic impact

18. Approach for economic evidence standards
Typically it is not expected there will be published economic studies for DHTs
The economic standards describe a set of information the developer will need to collect to facilitate an economic analysis so that the economic impact of the DHT can be assessed
A budget-impact model template and other educational sources will be made available alongside the standards

19. Economic impact standards
Key economic information
Economic analysis reporting
Appropriate economic analysis
Economic analysis outputs

20. 1a. NHS care pathway mapping
Existing care pathway mapping
Provide a clear description of the existing intervention and its setting as part of the current care pathway.
Be able to demonstrate that:
The care pathway has been mapped out in a detailed and stepwise approach (e.g. a flow chart) so that the relevant impact of adopting the DHT can be assessed.
It may or not be the gold standard as proposed by national guidelines.
If there are more than one commonly used care pathways then more than one comparator can be used in the economic analysis
The pathway is validated as accurate representation of current care by relevant clinical and/or social care professionals working within the UK H&SC system.

21. 1a NHS care pathway mapping contd.
Proposed new care pathway mapping with DHT
Provide a clear description of the incorporation of the proposed new DHT intervention into the care pathway
Be able to demonstrate that:
the infrastructure and service level changes, if any, which would be required to implement the proposed pathway
the proposed new care pathway with the DHT is a likely representation of what is feasible and how it will be successfully implemented locally by evidencing approval and support from relevant local clinical and/or social care professionals working within the relevant field UK H&SC system.

22. 1b Population calculation for DHT
Key parameter for budget impact analysis
User population – includes size of population and estimation of uptake
May also need to consider size of sub-groups if relevant
Expected sources of data
the user population size is based on
robust epidemiological data of incidence or prevalence of the relevant health problem,
available NHS data or expert estimates.
Estimation of uptake is based on relevant pilot data if available or other usage data from the developer.
All data could be validated be clinical and/or social care professionals working within the UK H&SC system.

23. 1c Parameters for economic analysis
Economic evidence standards
intervention-related parameters
Health and other outcomes from use of the DHT should be evidenced as specified in the effectiveness standards.
More robust data may be required for high cost interventions.
Cost parameters
Cost data related to H&SC system use should be valued using costs relevant to the decision maker e.g. NHS reference costs or national tariffs. Any methods used for adjusting estimated unit costs to the year of reported costs and rationale for these.
Any methods used for converting costs into a common currency base and the exchange rate and rationale for these.
Resource use parameters
Accurate and comprehensive itemisation of resources deployed in the existing care pathway and proposed new care pathway is expected. This data could be based on a pilot study or on information obtained from relevant clinical and/or social care professionals or other appropriate sources.
Utilities (if applicable )
As in NICE Technology Appraisal reference case the EQ-5D is the preferred measure of health-related quality of life in adults. If it is not appropriate the rationale for using another standard measure must be provided.

24. 2. Appropriate economic analysis
Financial risk likely to depend on a number of factors including
Maturity of the clinical effectiveness evidence
Total cost (£) to the UK health and care system per year for the estimated user population for the length of the proposed contract. This cost includes the cost of the DHT, implementation, operation and maintenance costs.
Potential budget impact of the technology
Level of certainty around the economic estimates
Financial risk determined by the payer
Type of economic analysis recommended
Low economic impact
Budget impact analysis and cost consequence analysis
High economic impact
Budget impact analysis and cost utility analysis

25. 3a. Economic analysis reporting standards
Analysis component
Economic evidence standard
Economic perspective
The perspective of the decision maker or payer e.g. UK Health and Social Care System perspective or societal perspective.
Time Horizon
For the economic analysis, it should be long enough to capture all costs and health outcomes to be accounted for.
Discounting
Discounting can be applied to costs and savings that occur after the initial year using standard UK Treasury recommendations.
Sensitivity analysis
Appropriate sensitivity analysis of input parameters and assumptions in the economic analysis to reflect uncertainty.
Equity analysis
Include sub-group analysis if there is good clinical data to show impacts differ by different demographic factors.
Additional analytical methods
Describe any analytical methods involved in the economic analysis e.g. methods for dealing with skewed, missing, or censored data; extrapolation methods;.
Critique of the analysis
Present the strengths and weaknesses of the economic analysis and its generalisability to the local context.

26. Economic impact standards outputs
Low economic impact (determined by the payer )
High economic impact
( determined by the payer)
Cost consequence analysis
Cost utility analysis
Estimated costs and benefits
Sensitivity analysis results
Estimated ICER ( cost/ QALY)
Budget impact analysis
Estimated annual budget impact value for years 1-5

27. Appropriate economic analysis
Economic impact standards overview
Value proposition
Economic information
Appropriate economic analysis
Economic analysis reporting standards
Budget impact analysis
Cost consequence analysis
Cost utility analysis
Economic analysis output