Jo Hunter Clinical Outreach Librarian University of Oxford Health Care Libraries Introduction to Critical Appraisal

Aims of this session Define “critical appraisal” Discuss why it is important Discover how to incorporate it in your own research/practice

Is this evidence good enough?

What is “good” evidence? We need to use evidence from research studies to support our arguments or decisions How do we recognise a good quality research study?

How can you tell whether a research study is “good quality”?

Factors that tell you about the quality of a study include… Extrinsic factors: Journal Author Institution Quoted by other authors Quoted in newspapers Intrinsic factors: Study design Number of subjects Confounders Bias (author, experimental)

Critical appraisal looks at the intrinsic factors “Critical appraisal is the process of systematically examining research evidence to assess its validity, reliability and relevance before using it to inform a decision.” (Adapted from Hill H & Spittlehouse C, EBM 2001; 3:2)

Validity, reliability, relevance 1)Validity: To what extent is the study a close representation of the “truth”? 2)Reliability: Are the results credible and repeatable? 3)Relevance: Will the results help me in my own research or practice?

Pros of critical appraisal Systematic way of assessing a paper’s validity, reliability, relevance Helps to close the gap between research and practice Encourages objective assessment Skills are not difficult – common sense & user-friendly checklists

Cons of critical appraisal Can be time-consuming initially Does not always provide the “easy” answer – may provide more questions Can be dispiriting if it highlights poor quality studies

Examples of appraisal checklists CASP for experimental studies, systematic reviewsCASP DISCERN for consumer health information (papers about treatment choicesDISCERN CriSTAL for librarianship studies (information needs analysis, user survey)CriSTAL

Getting started 1) Set up a journal club 2) Remember/devise a suitable scenario 3) Find a relevant paper 4) Choose an appropriate checklist

1) Why set up a journal club? Regular protected time for honing your “quality-detector” skills Facilitates discussing and evaluating new research and its application to your research or practice Helps to unravel complex problems

1) Five steps to journal club success 1.Identify a leader 2.Identify the goals of the club 3.Set up a regular meeting time 4.Send the scenario and paper to everyone before the meeting 5.Encourage active participation by using a checklist

2) Why bother with a scenario? Provides a practical focus Helps you to answer the applicability question Helps you to find a paper

2) Example of a scenario A 32 year old man is found in garage with engine running. He is brought to the ED by paramedics. He is awake and alert. He complains of a headache. His exam is unremarkable. Skin colour is normal. Neurologic exam is normal. What is the appropriate therapy?

2) Another example of a scenario You have been invited to join a local implementation group to look at how to improve information services to those studying ethnomusicology at the University. The group agrees that they need to find out what ethnomusicologists need, before making any improvements.

2) Another example of a scenario

3) Find a relevant paper Use key words from the scenario for database searching Browse your favourite journals Ask your librarian for help

4) Choose an appropriate checklist Identify the study type of the paper Google! Adapt a checklist that is nearly appropriate Remember: validity, reliability, applicability

Finally.. Write up a Critically Appraised Topic (CAT) for future reference Evaluate: Did you find the process helpful? If not, why not? How could it be improved?