Presentation on theme: "Alastair Fischer: Centre for Public Health, NICE"— Presentation transcript:
1Alastair Fischer: Centre for Public Health, NICE agenda item 3Alastair Fischer: Centre for Public Health, NICEModelling in UK Health Policy Decision Making.Newton Institute, Cambridge, June 2014
2Running Order Objective function Before NICE After NICE Social value judgementsThe 4 pillars of the evaluation processAbout Public HealthHow Public Health is not Technology AppraisalA decision theory approach versus a hypothesis-testing approachUse of ‘what-if?’ or threshold analysisNew issues: cost effectiveness and infectious disease modelling
3Objective FunctionIf the objective is to maximise the health gain from interventions, there needs to be a generic expression for healthThe QALY: quality-adjusted life year. 1 QALY= 1 year at full healthPragmaticRelatively simpleIt is measured using a simple questionnaire. The UK uses the simplest – the EQ-5D5 dimensions (reasonably orthogonal), 3 states for each dimension‘Utility’ (measuring quality of life) from EQ-5D is from 1(full health) to 0 (dead) but it is possible to go below 0A QALY is the time integral of utility.Gain QALYs by either living longer or at a higher QoL.
4Problems with QALYs You name them, QALYs have them But as a quick and dirty solution to being a generic measure, they work well enoughAnd in a lot of cases, the same decision would be made whether QALYs or Life Years Gained (i.e. ignoring quality of life) were measured.From a purist (academic) viewpoint, QALYs fail, but from a pragmatic decision-making viewpoint, they are better for the decision-maker than not having them (which is the realistic alternative).
5Back to the Objective function Maximise the QALY gain from a given budgetClassic optimisation problemAll the classic theorems apply (MUX/pX = MUY/pY)This allows us to be completely efficientEr…what about equity (= fairness)?A: Well, we are prepared to trade off some efficiency to be fairer.Q: So we really should be maximising the gain in a social welfare function that consists of efficiency and equity arguments?A: YesQ: But equity is subjective, isn’t it? So how do we formulate a SWF?A: We don’t know if this will do more harm than good, because fairness depends on context. So we use the GOBSAT principle.Q: I’ve never heard of that. What is it?A: (sheepish) …Good Old Boys Sat Around A Table.And we may have multiple objectives, not just equity and efficiency. Again, done by deliberation.
6The threshold So for efficiency, we buy the cheapest QALYs first. And keep on buying the next cheapest, and the next, till we run out of money. That maximises the number of QALYs we can gain.Where we run out of money is called the threshold.But NICE only controls a relatively small part of the budget.Other parts of the NHS use time-honoured deliberation, just like most decisions made in other parts of the Civil Service, in every country of the world.So the threshold was determined essentially by trial and error in the first instance.Originally £30,000 per QALY, and then more recently in the range £20,000 to £30,000 per QALY.More recently, a more objective measure has been found that is good in theory but has many problems. It estimates the threshold to be about £15,000 per QALY.
7Value based pricing (VBP) A £15,000 per QALY threshold would mean few new drugs would be approved at their current prices. The drug companies would be affronted. The threshold is based on what the NHS could afford to pay.So another mechanism to pay higher prices for drugs than the threshold would allow has been sought. The DH has looked at VBP, which would include other benefits not connected to the NHS, such as whether a recipient of a drug could go back to work, and if so, go back sooner. The value of the additional production would be added into the ‘threshold’ for that drug.NICE has questioned this approach on the grounds that it would discriminate against the old, the young, and the unemployed and low paid workers. So equity has been invoked.
8Before NICEThere was no single place where doctors could get advice on the best ways of treating patients, including getting unbiased advice on drugs and medical devices such as pacemakers.Some PCTs would allow a particular drug to be paid for by the NHS but others would not. Postcode prescribing was rife. Drug and device companies went from hospital to hospital, GP practice to GP practice, hawking their wares. Information was inefficiently broadcast.Clinical guidelines existed, but were relatively haphazardly produced. They had no set timetable for completion, somewhat scatty searches for the best evidence, often unrepresentative committees to determine the decisions, a lack of transparency about how a decision was made and patchy consultation.
9After NICE All that changed with the advent of NICE. Draconian timelines:we know when a decision will be made, sometimes years in advance.Transparent process.Many meetings are held in public.Strict rules of engagement (methods manuals)Representative committeesIndependent adviceExtensive consultationCitizen’s Council to advise on Social Value Judgements
10The four pillars Efficiency Social Value Judgements Process (transparency, timeliness, etc)Policy Perspective (What will change a decision?)SVJs sometimes override efficiencyThe right to vote is inalienable. It cannot be traded.Additionally, other SVJs can be tradedEquity offsets efficiency to some degree.Modellers need to be aware of all these things as context to their modelling.
11About Public HealthVery many interventions in public health are cost effectiveModels can often be very simple and do not require fancy extensions to be able to determine cost effectiveness.PH committees don’t like it when the NICE PH health economist says that an intervention will be in the right direction and will be cost saving. Therefore by definition cost effective!!The committees seem to feel cheated, and hold out for numbers. Until they get numbers, they appear to fret and to worry about cost effectiveness.But put a number in front of them – however great the uncertainty – if it comes in far enough below £20,000 (such as £15,000) they take cost effectiveness for granted and move onto more interesting discussions.A caricature? Decidedly so, but as with all good caricatures, the exaggeration is based on enough grains of truth for people to recognise what is being lampooned.We use the example of a small reduction in salt consumption to illustrate the point more fully
12Small reductions in salt We already know the direction of change from our prior knowledge. We know the effect of large reductions in salt, and we also realise that the relationship will not change sign when the ‘dose’ is reduced.We also know that salt is very cheap, and that reductions in it will cause savings of future treatment costs.End of story. Cost effective. Better than that – cost saving and positive health benefit.But RCTs have been done on this topic.The effect size on blood pressure is very small at an individual levelBlood pressure is devilishly hard to measure accuratelyKeeping people to a lower-salt diet is well-nigh impossibleSo the trials were all underpowered, and showed non-significant results in the ‘right’ direction
13More about salt This prompted a Cochrane press release, 5 July 2011 A meta-analysis of the trials (n = 6,489) showed:Cardiovascular morbidity in people with normal blood pressure (longest follow-up RR 0.71, 95% CI: 0.42 to 1.20, 200 events) or raised blood pressure at baseline (end of trial RR 0.84, 95% CI: 0.57 to 1.23, 93 events)This prompted a Cochrane press release, 5 July 2011Moderate reductions in the amount of salt people eat doesn’t reduce their likelihood of dying or experiencing cardiovascular disease. This is the main conclusion from a systematic review published in the latest edition of the Cochrane Library.The study has since been withdrawn
14The currently-used paradigm We establish an effect by conducting an RCT. The hypothesis-testing approach uses a frequentist paradigm and rules out chance by means of a t-test or similar using p-values.If significance is achieved, a cost effectiveness analysis is conducted as a second stage. This uses a decision-theoretic paradigm. It establishes cost effectiveness by looking at the size of the estimated mean ICER, and does not consider the ICER variance.
15Decision theory versus frequentist hypothesis-testing Subjective probabilityPrior beliefsTo maximise, variance of effect estimate is ignoredThis is as if the decision-maker is risk-neutralMakes sense if a large number of independent projects are considered.Used routinely in the business world for maximising profitsFrequentistObjective probabilityNo prior beliefsDoes not maximise, and is rather conservativeDecision-maker is risk-averseDoes not consider other projects (maybe shouldn’t if health is concerned?)Used routinely in medical research for effectiveness.
16The effect of bias Decision theory and frequentist hypothesis-testing do not deal well with bias.The RCT avoids bias internally. A frequentist approach further avoids bias by ignoring prior beliefs. So where bias is important and can be avoided, use a frequentist approach.The underpowered RCT is very prone to publication bias. RCTs will often be underpowered in public health when the individual effect size is very small. Prior beliefs, however, will often not be subject to much bias, especially if they are firmly held. So use decision theory for effectiveness in PHAs in TA
17Technology appraisalRCT: Estimated distribution of the mean effectPosterior distribution combining prior and RCTVague prior belief about mean effectivenessPosterior distribution is influenced almost entirely by the RCTIf the prior distribution is biased, it might detract from accuracy and not enhance it.So the decision-maker may wish to remain with the red curve (RCT only) as themost accurate.
18Public health population intervention Non-vague prior belief about mean effectivenessPosterior distribution combining prior and RCTRCT: Estimated distribution of the mean effectPosterior distribution is influenced almost entirely by the prior belief curve.The RCT is mostly a distraction, and it is likely to be biased upwards because ofpublication biasSo the decision-maker may wish to ignore the red curve (RCT) because it is nothelpful for the decision-maker. Because we know only the direction of changewith the yellow curve, we must use “what if” analysis to inform cost effectiveness.To get an effect size (yellow curve doesn’t give it) we use estimated mean RCT effect as an upper bound, and do ‘what-if?’ analysis
19More on a full decision-theory approach We have published one paper on this topic (JPH Dec 2013) and have 2 more under submission, and a fourth paper is being written.NICE processes already reflect this approach, but are not given the gravitas that accords to a fully accepted paradigm like decision theory. (“We must not allow special pleading for Public Health. If the interventions do not satisfy 95% confidence intervals, we cannot accept them.” This comes from the strictest adherents to the frequentist school, but those are the voices that are the most respected.)In round figures, we believe that the new approach could lead to the same health gains at a cost about £10 billion lower than by the next-best means.
20Public GoodsThe most cost effective interventions in public health are usually some form of public good.(A “public good” is like a Freeview TV programme – once it is provided to one person, everyone else in the country with a TV set can access it without cost, and cannot be excluded from it. Programmes that are pay-per-view are ‘excludable’, so are not public goods.)LegislationSmoking ban (big health gains, low cost of initiating, subsequent cost saving)TaxationAlmost no work done to implement, improves by reducing drinking/smokingRegulationSpeed cameras, traffic lights on foodstuffs
21Public goods (2)Can’t often do RCTs in public goods, so in the past they have been difficult to model for cost effectiveness.But recommendations about public goods are telling central government what to do, and central government feels it has primacy when it comes to decision-making.NICE has to tread carefully in this area and therefore its recommendations must be oblique.Additionally, health gain is not the only consideration. A recommendation for better health could be to tax beer until a pint in a pub was £20. But this has a political dimension, too, and no politician wishing for re-election could approve such a recommendation.By making recs that are not the most cost effective, does NICE ‘crowd out’ the most cost effective interventions?
22Infectious disease modelling Time horizonMight be longer than a single lifetimeBut models only go for perhaps 20 yearsWhy? Is it because errors accumulate?Yet health gains for an individual are usually at the end of life, which might be more than 20 years away.So the transmission model stops at 20 years, and we then follow through the lives of those in the model till death? Is that the way to do it?Models are tricky to calibrateA modelling assignment cannot be done in 3 to 6 months unless modellers have already been working in the area for a couple of yearsMuch intellectual capital invested.NICE are required by law to make an executable model available to consultees if requested. This is at odds with the intellectual property rights of the model’s developers.Maybe there are better ways of ensuring model transparency
23Infectious disease modelling (2) Model transparency through calibrationModels are deemed ‘fit’ if they fit the past, and other tests of robustness.Optimising over time in an infectious disease model would appear to be the finding of an optimal time path out of the infinity of possible paths.This is an optimal control problem. Has it been modelled as such? Is it optimal to spend a huge amount today in order to keep expenditure low in future, or to spend less now and rather more later?This is a very tricky problem for governments whose time horizons might be short. NICE might not be the best vehicle for advising government on this.
24Public health and local government Public health is now the responsibility of local government.They have many other objectives, not just health.In particular, they spend a lot of money on caring for the elderly, which means additional expenditure and incentives for preventing dependency in old age.The QALY has far less meaning, and probably the only common currency for evaluation studies is money.They have far tighter budgets than central government, and thus their time horizons are shorter/their discount rates are higher.The electoral cycle is only 3 years, again raising the discount rate
25Conclusions Many rethinks need to be carried out A change to a decision-theoretic from a frequentist paradigmLearning more about transmission models and how they can best be used for determining cost effectivenessA more pragmatic approach to decision-making in line with local government priorities and capabilitiesAn educative role carried out by modellers and health economists for both local councillors and LG officialsThere’s a tension between the immediacy of treating the sick and ensuring in the longer term that the well do not become sick. That is an implied SVJ that will probably always be with us.
26Infectious disease modelling (3) In my past life as an experimental economist, I played with the idea of a vaccination programme that allowed two optimal societal vaccination schedules.At a lower level of vaccination, there was a local optimum that kept a disease under controlAt a higher level of vaccination, there was a global optimum (that by definition gave higher benefits to society than the local optimum) which eradicated the disease.In my replications of the model, the global optimum was not attained, because of free riders.