1. © Nuffield Trust Evaluation methods – where can predictive risk models help? Adam Steventon Nuffield Trust 8 July 2013

2. © Nuffield Trust The problem with observational studies Eligible patients All patientsIntervention patients n54,990556 % aged 85+21.646.2 Prior emergency admissions0.51.4 Number chronic conditions1.01.1 Predictive risk score22.333.6 Intervention patients Source: Steventon et al (2012)

3. © Nuffield Trust Solutions, 1) before-after study

4. © Nuffield Trust Solutions, 2) regression adjustment Y = f(age, number of chronic conditions, prior emergency admissions, intervention status)

5. © Nuffield Trust Eligible patients Intervention patients Matched controls All patientsMatched controls Intervention patients n54,990556 % aged 85+21.646.2 Prior emergency admissions 0.51.4 Number chronic conditions 1.01.31.1 Predictive risk score22.333.533.6 Solutions, 3) Matched controls Source: Steventon et al (2012)

6. © Nuffield Trust How to select matched controls Propensity score (Rosenbaum and Rubin 1983) - Predictive risk of receiving the intervention Prognostic score (Hansen 2008) - Predictive risk of experiencing the outcome (e.g. emergency hospitalisation), in the absence of the intervention Genetic matching (Sekhon and Grieve 2012) - computer-intensive search algorithm

7. © Nuffield Trust Advantages / disadvantages Disadvantage – only allows for observed variables But Matching as ‘data pre-processing’ – reduces dependence of estimated intervention effects on regression model specification Intuitive? Good for routine monitoring – once controls found, data can be updated

8. © Nuffield Trust Overcoming regression to the mean using a control group Start of intervention

9. © Nuffield Trust Overcoming regression to the mean using a control group Start of intervention

10. © Nuffield Trust Overcoming regression to the mean using a control group Start of intervention

11. © Nuffield Trust Overcoming regression to the mean using a control group Start of intervention

12. © Nuffield Trust Solutions, 4) regression discontinuity Winning the next election Fraction of votes awarded to Democrats in the previous election Source: Lee and Lemieux (2009)

13. © Nuffield Trust What is being done at the moment? Telehealth studies in Pubmed, 2006-2012 DescriptiveBefore after Dose response ControlledAll Number of studies 32411644 Median number of patients in telemonitored group (range) 45 (40 to 851) 35 (7 to 17,025) 246 102 (19 to 1,767)* 45 (7 to 17,025)* Endpoints Mortality2-136 Hospital use (or costs)36-1221 Clinical (e.g. HbA1c)-17-421 Patient reported outcomes (e.g. quality of life) 110-314 Source: Steventon, Krief and Grieve (work in progress)

14. © Nuffield Trust References Lee DS, Lemieux T. Regression discontinuity designs in economics. 2009. Available from: http://www.nber.org/papers/w14723.pdf?new_window=1 http://www.nber.org/papers/w14723.pdf?new_window=1 Sekhon JS, Grieve RD. A matching method for improving covariate balance in cost-effectiveness analyses. Health economics 2012;21:695–714. Rosenbaum P, Rubin D. The central role of the propensity score in observational studies for causal effects. Biometrika 1983;70:41–55. Hansen BB. The prognostic analogue of the propensity score. Biometrika 2008;95:481–8. Steventon A, Bardsley M, Billings J, Georghiou T, Lewis GH. The role of matched controls in building an evidence base for hospital- avoidance schemes: a retrospective evaluation. Health services research 2012;47:1679–98.

15. © Nuffield Trust 8 July 2013 www.nuffieldtrust.org.uk Sign-up for our newsletter: www.nuffieldtrust.org.uk/newsletter Follow us on Twitter: http://twitter.com/NuffieldTrust © Nuffield Trust adam.steventon@nuffieldtrust.org.uk