Critical Reading

Critical Appraisal Definition: assessment of methodological quality If you are deciding whether a paper is worth reading – do so on the design of the methods The specific questions to be asked in the methods section are: Why was the study done? What type of study was done? Was the study design appropriate?

Why was the study done? i.e. what was the key research question/ what hypotheses were the author testing? Hypothesis is usually presented in the negative – the “null hypothesis” Studies try to disprove this lack of difference or null hypothesis.

Small Groups 15 minutes Appoint feedback person List the different types of study you have heard of Describe them – give an example Advantages & disadvantages

What type of study? Primary – these report research first hand. Experimental i.e. humans, animals; artificial and controlled surroundings. Clinical trials – intervention offered. Survey – something is measured in a group.

What type of study? Secondary – summarise and draw conclusions from primary studies. Overview –Non systematic (summary) –Systematic (rigorous and pre-defined methodology) –Meta-analyses (integration of numerical data from more than one study) Guidelines (leads to advice on behaviour) Decision analyses (to help make choices for doctor or patient) Economic analyses (i.e. is this a good use of resources?)

Specific Types of Study Randomised Controlled Trial (RCT) Population is randomly allocated to two groups One group is given a specific treatment or intervention On average the groups are identical because they are randomised and therefore any difference in the measured outcome is due to the intervention Specified follow up period and specified outcomes e.g. drug better than placebo; surgical procedure compared with sham

Randomised Controlled Trial (RCT) Advantages Allows rigorous evaluation of a single variable in a previously defined population e.g. a new drug. Prospective i.e. collect the information after you decide to do the study. Tries to disprove the null hypothesis Tries to eradicate bias because the two groups are identical. Allows for meta-analysis later.

Randomised Controlled Trial (RCT) Disadvantages Expensive and time consuming which can lead to problems including: Too few subjects Too short a time Who controls the study? End point not clinical Possibility of hidden bias: Imperfect randomisation Failure to randomise all eligible patients – who is included/excluded. Assessors not blinded.

Cohort study Two (or more) groups of people are selected on a basis of a difference in exposure to a particular agent i.e. vaccine, environmental toxin, medicine. Group followed up (usually for years) to see how many in each group develop a particular disease/outcome. e.g. Peto– 40,000 UK doctors. e.g. COCP causes breast cancer?

Case Control Study Patients with a particular disease are identified and “matched” with controls. Data is collected retrospectively either from medical records or from memory, looking for a causal agent. Looks for associations but not necessarily the same as cause. e.g.SIDS and sleeping position. Does whooping cough vaccine cause brain damage? Do overhead cables cause leukaemia?

Cross Sectional Survey A representative sample of subjects or patients are studied (interviewed, questionaired, examined) to answer a specific clinical question at a specific time. e.g. normal height of three year olds what do most GP’s think about the use of Viagra?

Case Reports Medical history of a single patient in a story form. Lots of information given which may not be seen in a trial or a survey. Often written and published fast compared to studies e.g. Thalidomide

Importance of ethics

Hierarchy of Evidence (Systematic Review and Meta-analysis) Randomised Controlled Trial Cohort Studies Case Control Studies Cross Sectional Surveys Case Reports

Was design appropriate? In general: Therapy – i.e. effect of intervention – RCT Diagnosis – ? test valid (can we trust it) or reliable (? same result if repeated) – cross sectional survey with both gold standard and new test Screening – large population, pre-symptomatic – cross sectional survey Prognosis – i.e. what happens to someone if a disease is picked up at an early stage – longitude cohort study Causation – e.g. ? possible harmful agent leads to cause – cohort or case control study - ? case report.

Assessing Methodological Quality Questions to Ask general framework specifics dependant on type of paper Logical Progression Introduction- Title - Abstract - Introduction Methods Results(Statistics!) Discussion

Seven essential questions: Introduction 1. Why was the study done? Is the study original or does it add to the literature in any way? e.g. bigger, better, larger, more rigorous Is it interesting? Is there a clear research question?

Seven essential questions: Methods 2. Who is it about? How recruited? Who included? Who excluded? Studied in “real life circumstances”? Applicable?

Seven essential questions: 3. Was it well designed? i.e. does the study make sense? What specific intervention or manoeuvre was being considered and what was it being compared to? Is what happened what the author said happened? What outcome was measured and how? i.e. length of life, quality of life, reduction in pain need to be objective.

Seven essential questions: 4. Was systematic bias avoided? i.e. was it adequately controlled for? [ Bias = anything that erroneously influences the conclusions about groups and distorts comparisons e.g. RCT – method of randomisation, assessment ? truly blind. Cohorts – population differences Case control – true diagnosis, recall (and influences) ]

Seven essential questions: 5. Was it large enough and long enough to make results credible? Size is important!

Seven essential questions: Results 6. What was found? Should be logical – simplecomplex

Seven essential questions: Discussion 7. What are the implications? For: - you - practice - patients - further work and do you agree?

Four possible outcomes from any study 1.Difference is clinically and statistically significant i.e. important and real. 2.Of clinical significance but not statistically so. ?sample size too small. 3.Statistically significance but not clinically i.e. not clinically meaningful. 4.Neither clinically nor statistically significant.

Recommended Reading Ian Crombie : The Pocket Guide to Critical Appraisal Trish Greenhalgh : How to read a paper; the basis of evidence based medicine