EVIDENCE BASED MEDICINE Health economics Ross Lawrenson

Critically appraising an economics paper

Critical Appraisal of an economics paper - methodology When critically appraising a paper ask yourself three questions: –Are the results valid? –What are the results? –Will the results help me in caring for my patients? Go to the economics worksheet for the complete list of questions

I. Are the results valid? In other words was this a well designed study in a relevant population. The best study design to answer an economics question is a cost benefit or a cost utility analysis. Go through the worksheet questions 1- 6 to help you decide whether you are likely to believe the results of the paper you are considering.

I. Are the results valid?

1. Was a well defined question posed in answerable form? Can you define –The population they studied (and in what context) –The intervention –The comparison group –The outcomes (the costs and effects of the services)

2. Was there evidence that the programme or intervention being studied was effective? Ideally the effectiveness of a treatment should have been established in randomised controlled trials. If you are comparing two treatments and one of the treatments is of no demonstrable benefit then ethically one should not offer this option to a patient (even if it is cheaper!) when there is an effective treatment available.

3. Were all important and relevant costs and consequences for each alternative identified? These can be divided into direct and indirect costs.

3. Were all important and relevant costs and consequences for each alternative identified? Direct costs: When we are considering the treatment of depression the direct costs include the doctors time in consulting with the patient, the costs of any medication prescribed, the costs of the rooms where the consultation occurred, the costs of the receptionist, the costs of counseling, the costs of a specialist referral, hospital admission etc.

3. Were all important and relevant costs and consequences for each alternative identified? The indirect costs: These include the patients traveling costs to get to the surgery, prescription charges, the patients time off work, the costs of relatives having to stay at home to be with a depressed patient etc.

4. Were the costs and consequences measured accurately in appropriate units (e.g. hours of doctor’s time, days off work etc.) prior to valuation? The cost of the doctors time may be measured in minutes of consultation time, medication may be cost per tablet (operating costs), whilst accommodation is a fixed or capital cost which may be an annual charge which should be divided by the number of patients seen per year. Indirect costs are often harder to estimate but may include travel costs, lost wages or an estimated cost of a relative acting as a carer.

5. Were the costs and consequences values believable? Many economic analyses will exclude fixed costs such as administrative and secretarial support, heating and lighting in the rooms used, pension, education and sick leave entitlement from a doctors salary. All these costs should be attributed in an appropriate proportion to the costs of the treatment being provided. This is not always easy!

6. Were the costs and consequences adjusted for differential timing? Generally inflation means that the cost of doing something in the future costs more than doing it today. If we want to compare the costs of a renal transplant today compared with ten years ago we have to adjust for inflation. Often analyses will take an arbitrary figure of 5 or 10% - the authors should be able to justify the % discount they have applied.

II. What are the results? Have the results been presented as a cost effectiveness analysis or a cost benefit analysis or some other measure.

Cost effectiveness Cost effectiveness measures the cost of one or more treatments or services in comparison to a single outcome. Examples of outcomes that can be compared include the cost per patient successfully treated or cost per life saved. The disadvantage of a cost effectiveness analysis is that it may not reveal other positive or negative effects of compared treatments other than those that have been recorded and which are the subject of the analysis.

Cost benefit analysis Cost benefit analysis compares two or more treatments or services, by placing a value (usually monetary) on all the accrued costs and on all the benefits. Thus, when considering a treatment, the benefits may include added years of life. This is assigned a monetary value often based on future potential earnings of the individual. It may also include the costs of continuing care or treatment. This form of economic analysis allows purchasers to compare many different treatments to help them decide which is the best buy.

Cost utility analysis Cost utility analysis accounts for all the costs of comparable treatments, but measures the benefits in a common unit (other than money). One common unit that has been used is the Quality Adjusted Life Year (QALY). A QALY combines the quantity of life gained with an adjustment for quality of life. This allows direct comparison between treatments but is less discriminatory against those with limited earning capacity than cost benefit analysis. Other measures of the benefits of medical care have been developed, for example, in 1993 the World Bank adopted a new and more sophisticated unit, "disability adjusted life years" (DALY).

1. What are the comparative costs between the two treatments or interventions? This is the difference between the competing treatments, programmes, tests etc. usually in monetary terms. You should also check whether the precision of the result is given - is it statistically different?

Did this include the incremental costs as well as the difference? If you are testing for bowel cancer with three haem-occult tests one week apart and you discover 200 cases of cancer at a cost of £20,000 then the cost per case identified is £100 per case. However if you had only used two tests you might have identified 180 cases at a cost of £14,000. Thus for spending £6,000 more you only identified 20 more cases - an incremental cost of £300 per case

2. Has a sensitivity analysis been performed? An economic analysis will usually involve an estimated cost which may vary from the true cost. Thus a hospital admission may cost £300 or £350. A sensitivity analysis would use both these values to see what difference they make to the analysis. Similarly the trial might show 30% of patients improved on treatment - what if in real life (rather than in an RCT) only 20% improve. The lower figure could also be used to see how that might affect your decision making.

III. Will the results help me in caring for my patients? In a cash limited health service cost should always be taken into account when considering which is the best treatment to offer a patient. Money spent on one patient is then not available to treat another. However the fact that one treatment is more cost effective than another does not always mean that it is the ethically correct option, especially if we are considering options such as cost per life saved where we might want to save as many lives as possible.

1. Can the results be applied to my patients care? Are the patients and the setting in which the study took place relevant to my situation. Because the way health services are provided differs so greatly from country to country the results of economic analyses are much less generalisable to other settings than are say clinical efficacy studies of a drug. Similarly a test carried out in a hospital population of patients may be much less relevant in general practice and the economic benefits very different.

2. Were all the clinically important issues considered? The problem with a cost effectiveness analysis is that it usually only looks at one outcome. There may be other outcomes of interest that need to be taken into account - a cost benefit analysis is thus better - but often not all the outcomes that you might consider are clinically relevant have been included.