Presentation on theme: "From sifting to synthesis"— Presentation transcript:
1 From sifting to synthesis Evidence Base Camp: February 2014Levin WhellerPractice Development TeamResearch Analysis and Information Unit1
2 This session… Recap on REAs While you were away… Peer review of sifting and the task aheadMapping research evidenceSystematic Mapping – the what and the whyAn example: Mapping Evidence on Public Order TacticsAppraising research evidenceThe Maryland Scale of Scientific MethodsSome hypothetical examplesBeyond Maryland: non-impact questionsQualitative methodsCompleting a pro formaPractical group work
4 Systematic reviews and “what works” Explicit objective/ questionStudies are selected on explicit criteriaThorough search for studiesEach study is screened according to uniform criteriaReasons for excluding studies are clearly documentedAdvantages:Systematic reviews are the most robust method for reviewing evidence because they reduce the bias in the way studies are found, included and synthesised. Transparency allows future studies to be added to the review which enables a cumulative body of sound evidence to be developed on a subject area over time. Furthermore, it should be possible for anyone else to conduct the same review and come to the same conclusions.There is a myth that systematic reviews are only suitable for experimental studies. While it is true that many of the existing systematic reviews have concentrated on large scale quantitative studies, systematic review methods can be applied to any type of question. Where quantitative studies are included it may be possible to undertake a meta-analysis – a method of analysis which combines statistically the results of a number of studies. If more qualitative data are used then a qualitative synthesis can summarise these results. Some reviews do both, and integrate a meta-analysis of data from controlled trials with a synthesis of findings from qualitative studies (Gough & Elbourne, 2002). This is also true for Rapid Evidence Assessments.Disadvantages:The strength of systematic reviews is that they aim to be as comprehensive as possible in the studies they find, include and synthesise. However, this leads to their principle disadvantage, which is that they are time consuming. They also require a team of researchers (either ‘in-house’ or contracted to carry out the work) and are therefore resource intensive (although likely to be less so than undertaking a piece of primary research).
5 Rapid Evidence Assessments Use systematic principles and the same process as a systematic review……but make compromises given available time and resourcesPragmatic and transparent approachTermsAnywhere in the article“Domestic violence” 597,000“Domestic abuse” 32,500Title only“Domestic violence“ 23,700“Domestic abuse” 1,220Adding ‘systematic review’“Domestic violence“ 7“Domestic abuse” 0
6 Some key principles Demonstrate consistency in searching/ sifting Document search and sift processProcess should be transparent and repeatableSpecify required quality of evidenceSystematic reviews only?Pre-post studies only?All ‘empirical’ papers?Be explicit/ transparent about the limitations of the approachSo what are these systematic principles? Broadly, there are four things you need to focus on.Demonstrate the consistency of your searching a sifting. Make sure that you write up and quality assure the search criteria and sift criteria before you start searching or sifting. By all means pilot the criteria on one database/ a few papers. But….Make sure you document any changes or adaptations, and how you have addressed any particular problems or issues in the searching/ sifting process. In the example we are discussing, an additional ‘’sift criteria’ needed to be added as a lot of papers on changing the way doctors treated particular illnesses were included after the sift stage.Specifying the quality of evidence required for inclusion is key. In this example, we included only reviews and or pre-post studies, this is probably the most important factor in limiting your search to manageable numbers. The quality of evidence you require is going to be linked to the question your customer has, or the kind of statements you want to be able to make as a result of the review.There will always be limitations to a review. This is absolutely nothing to be worried about or to try to hide. Be explicit and clear about the caveats and qualifications involved in the process. (Your customers should already understand some of the limitations).
7 The process (in a nutshell) Draft search termsDraft sift criteriaSift received abstractsRequest full papersRead and ‘grade’ papersWrite it up (‘synthesis’)
8 Searching Identifying potentially relevant literature Sifting Identifying truly relevant literatureSynthesis Summarising what relevant literature tells us
9 Searching Identifying potentially relevant literature
10 The Basics Boolean operators Boolean is a powerful tool and can be used to create quite complex searches. The standard Boolean operators are:ANDORNOT (or AND NOT) (advanced users only!)Boolean is most effective when used in combination with:PARENTHESES (round brackets)TRUNCATION e.g. politic*WILDCARDS e.g. randomi?eDOUBLE QUOTES for specific phrases
11 Domestic violence search Search terms((((Domestic OR Familial OR Marital OR Spousal OR Partner) Near/2 (Abus* OR violen* OR control* OR coerc*)) OR "domestic homicide" OR "marital rape" OR "intimate partner violence" OR (coercive NEAR/2 (control OR behavio*r*)))AND (risk AND (predict* OR indicat* OR factor* OR management OR identif* OR assess* OR prevent* OR likelihood OR probability))AND ("systematic review" OR "literature review*" OR trial* OR "RCT" OR experiment* OR evaluat* OR "best practice*" OR "good practice*" OR effective* OR assess* OR "What Works" OR impact* OR success*))DA/ DV c 75k papers‘What works’ c 2m papersRisk c 1m papersThese were our search terms…Run through slide…Retrieved papers themselves refer to other papers, but we did not have the time to go to original papers. This is something else that you might do in a longer systematic review, for example.Initial search identified 1,531 abstracts to sift
12 Searching as a ‘blunt tool’ Research Question:What interventions have been shown to be effective in preventing or reducing theft from the person?So how do we end up with an abstract titled…Profile on circadian blood pressure and the influencing factors in essential hypertensive patients after treatmentTheft from person synonyms include: Robbery, Mugging, Dipping, Pick pocketing, Assault with intent to rob, Snatching, Theft from person, Bag theft, Bag snatch, Personal theft…
13 Returned by theft search… Yuan Gang Qiu et al. Profile on circadian blood pressure and the influencing factors in essential hypertensive patients after treatmentNon-dippers were defined as those whose nocturnal decrease in mean systolic BP and/or mean diastolic BP was < 10% of the daytime BP. Binary logistic regression analysis was used to evaluate the correlation between circadian blood pressure profile and factors as gender, age, height, body mass index (BMI), family history of premature cardiovascular disease, women under age 65 or men under age 55, smoking habits, grade of hypertension, and strategy of antihypertensive drugs.Similarly, the incidence of non-dippers in patients of overweight (24 </= BMI < 28) and obesity (BMI >/= 28) were 3.0 and 4.8 times of those in subjects of normal weight (P = and 0.009, respectively). Compared with patients treated with long-acting calcium channel blockers (CCBs), patients treated with angiotensin-converting enzyme (ACE) inhibitors or angiotensin II receptor blockers (ARBs) alone had less prevalence of nondippers (OR = 0.139, P = 0.010).
15 Sift criteria Question Answer Action Q1. No Exclude Yes Go to Q2 Does the paper consider theft from the person or relevant synonyms?NoExcludeYesGo to Q2UnclearQ2.Does the paper consider interventions that aim to reduce/prevent theft from the person or relevant synonyms?Go to Q3Q3.Does the paper include empirical data or methods?IncludeCan’t excludeSift criteria are sometimes called inclusion criteria.Abstracts are summaries of full papers or books.The same set of questions are applied to all abstracts to attempt to remove bias/ overt ‘judgement calls’.Typically, a lot of papers (up to 95%!) identified by the search are excluded at this stage of the process.
17 Peer Review: Second sift… ‘Red’ abstracts‘Dip sample’ taken, e.g. one in 10 ‘red’ abstracts were peer reviewedCheck for ‘false-exclusions’‘Green’ and ‘Amber’ abstractsAll resifted to review against the inclusion criteriaCheck for ‘false-positives’Decide where possible on ‘Ambers’Final list of included studies sent to library…Why do we undertake a second sift?Basic quality assurance and good practice…Consistency is important in REAsTransparent about our inclusion criteria which areWith large groups sifting quickly, we are likely to see differences in interpretation, even using with same sift criteria17
18 Second sift…REAAbstractsFirst siftSecond siftDomestic Abuse2,678622411Public Order57218238Always expect to reduce the number of studies substantially through second siftingStandard process for all REAs and SRsDual coding and often used and disagreements discussed between coders to make a final decisionClarify queries/ refine criteria as a groupSpeed of EBC does not enable all of this!!!
19 Available for synthesis While you were away…AreaAbstracts returnedIncluded after 1st siftIncluded after 2nd siftAvailable for synthesisAcute Mental Health Crisis1,4081735538Barriers to career progression1,2051538246Reducing theft from the person84419258Supporting mental well-being1,2291629372Responding to prostitution9791958748A final ‘master list’ of included studies was sent to the library for each groupLibrarians get hold of papers from the library catalogue or the British Library (if required)Some studies not available, some not available in English – some were large books which we have excluded due to time constraints…Also a consideration of cost - £10 per paper we didn’t already have in the library!Leaves us with a final number ‘available for synthesis’ by you all over the next day…19
20 Synthesis Summarising what relevant literature tells us
21 Systematic Mapping… Methodology developed by EPPI-Centre Useful first step of synthesis processMapping can:Describe the nature and coverage of research in the topic areaIdentify gaps in the research literatureDescribe the design of studies and direct further review workSCIE Systematic mapping guidance - April 2009The methodology for systematic mapping was originally developed by the EPPI-CentreResponse to large quantities of evidence which does not measure impact/ is not empirical – review w/ 8000 abstracts, 150 relevant studies, only 6 w/ outcome measuresSo mapping a broad area can help direct further research in the futureTypically used in research the College does to look at how to narrow the focus of reviews, e.g. might decide to only look at experimental research, or other reviews, etc.21
22 Systematic Mapping… What does it involve: Collection of key information about each study into a pro forma/ templateAll have an example in front of you…This is from Public Order Tactics Review…22
23 Systematic Mapping…Collate pro forma results into a searchable bibliographic databaseSystematically mapping a topic enables the development of a comprehensive database of literature that includes, but is not necessarily limited to, empirical studies23
24 What is the evidence on Public Order Tactics? Country of focus 24
25 What is the evidence on Public Order Tactics? Research Method 25
26 Public Order Tactics - Mapped Overall: There is only limited research available on public order policing tactics. Only 38 papers were found to be relevant and based on empirical methods.Age: Most of the research (32/38 papers) has been published since 2000.Methods: Most papers used qualitative research methods, typically involving observations (n=19) or interviews (n=18). Only one study used experimental methods.Mapping to existing Authorised Professional Practice Tactics26
27 Public Order Tactics - Mapped Unit of study: Only five studies focussed directly on public order tactics. Instead, studies primarily focussed on an event or series of events (n=17) or the police more generally (n=10).Tactics: Of the 30 approved public order tactics, only nine were mentioned in the research evidence identified for this map. The most frequently mentioned tactics were containment (n=5), Police Liaison Teams (n=3) and CS smoke (n=3).Mapping to existing Authorised Professional Practice Tactics27
29 Maryland Scientific Methods Scale (SMS). (AKA Maryland Scale) Developed by Sherman et al (1997)Established method for judging the quality of impact studiesUseful for judging how much weight to give different studies identified in a ‘what works’/ ‘what’s effective’ REALimited use for judging qualitative studies and questions asking ‘how’ things work29
30 What does ‘good’ or ‘robust’ ‘what works’ evidence look like? Systematic Reviews(Based on level 3-5 studies)5Randomised controlled trials4Before/after measuresMultiple site comparisons3Two site comparisons2No comparison site1One-off measureStatements about‘what works’Statements about ‘what’s promising’Study designs increasingly rule out potential alternative causesShould be familiar to you all now…Study designs cannot rule out potential alternative causesStatements about possible impact
31 ‘Level 1’ Evidence“Two months after the introduction of mobile information, officers were spending 55% of their time out on patrol”Problems?Can’t rule out other explanationsTime spent on patrol before mobile info?Is 55% good/ bad/ average?No before/ after measuresNo comparison groupFictitious example!!!One off measure with no comparison site…Can’t say if mobile information makes a difference/ is worthwhile
32 ‘Level 3’ evidence% of time on patrolPrePostWestern District(receives intervention)43%50%Eastern District(no intervention)46%We still can’t be sure that improvements are down to the intervention rather than other factors“Two months after the introduction of mobile information, officers in the Western District had increased their time on the street by 7 percentage points, while officers in the Eastern District had seen an increase of 4 percentage points…”Need to say this as a statement…A good study will collect information about processes/ other factors in both areas…Before/ after measure with 2 site comparisonProblems?Still can’t rule out other explanations for changeCan’t control for differences between the two areas – different culture, different case load, management, etc.
33 RCTs - ‘gold standard’ evaluation design 2,000 officersin force300 officersin treatmentgroupOutcomesmeasuredGetmobiledataA randomsample of600 officers300 officersin controlgroupOutcomesmeasuredBusinessasusualIn here we can use the example from medicine about the use of steroids for head injuries – how it was received wisdom that these would be useful as steroids reduce swelling, but that in a randomised trial it was shown that there were higher rates of death in those given steroids. The findings were so overwhelming that use of steroids was harmful, that the trial was stopped part way through – a very rare occurrence.In terms of research methods, this is as good as it gets…Randomisation controls for differences between groups (e.g. age, gender, length of service) the ensuring differences between treatment and control groups are down to the intervention
34 ‘Level 5’ Evidence - RCTs “Two months after the start of the trial, officers with access to mobile data were spending 55% of their time out on patrol, whilst officers in the control group were spending 43% of their time out on patrol”1. Participants are randomly selected from the population and randomly assigned to each group to minimise the chance of bias2. Randomisation helps us rule out other potential explanations for differences in outcome measures3. A control group means we can compare differences in outcomes between groups where the only difference is the intervention (e.g. mobile dataFictitious example!!!Make sure to be strong in clarifying ‘in this context’ –Context leaves us with same issue as with our level 3 example – if this mobile data study is (for example) in GMP then there may be things about that specific force that might not be replicated elsewhere… so we *should* look for confirmation that the intervention works in other contexts/ locations before we roll it out everywhere… that’s why we do SYSTEMATIC REVIEWS…4. RCTs enable us to make strong statements about cause and effectWe can say that officers with access to mobile data spend more time on the street… in this context.
35 Systematic reviewsOverall, legitimacy interventions resulted in a large, significant increase in positive perceptions of police.MEASURE NUMBER OF DIFFERENT STUDIES, ACCOUNT FOR RANGE OF DIFFERENT CONTEXTS, JUDGE OVERALL EFFECTSR’s allow us to combine multiple examples of interventions across a range of contexts… and by being systematic in our reviewing, we can ID on average if something is effective generally.The bigger the sample, the more confidence we can have in its findings.So – run through…Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1.
36 What does ‘good’ or ‘robust’ ‘what works’ evidence look like? Systematic Reviews(Based on level 3-5 studies)5Randomised controlled trials4Before/after measuresMultiple site comparisons3Two site comparisons2No comparison site1One-off measureStatements about‘what works’Statements about ‘what’s promising’Study designs increasingly rule out potential alternative causesStudy designs cannot rule out potential alternative causesStatements about possible impact
37 Beyond Maryland…The Maryland scale is focussed on impact questions… and studies might answer a range of other questions…Question typeExampleNeedsWhat do people want or need?ProcessWhy/how does it work?ImplementationWhat is required to make it work?CorrelationWhat relationships are seen between phenomena?AttitudinalWhat do people think? What are their experiences?EconomicHow much does it cost and with what benefit/harm?LINK SOME OF THIS BACK INTO MOBILE DATA EXAMPLE – “if you think about introducing an intervention in your force…”All of these sorts of questions in the table **should** be part of a high quality experimentQualitative studies better at getting in-depth analysis of specific issuesQuantitative studies better at making broad generalisations and measuring impactHow ‘good’ a study is will depend upon how appropriate the methods used are for answering the research question…Qualitative studies don’t fit into the Maryland dichotomiesYou can have different quality qualitative work as well
38 Quality of Qualitative Research Frameworks to assess/ appraise the quality of qualitative research use ‘principles’ rather than hard-and-fast rulesCabinet Office e.g. in 2003 sets key principles being that qualitative research should be:• contributory in advancing wider knowledge• defensible in design• rigorous in conduct• credible in claimGuiding principles 18 questions over five pages…There are four central principles that underpin the content of the framework. All of these are based on themes that are highly recurrent in the literature and in the interviews conducted for the study. They advise that research should be:contributory in advancing wider knowledge or understanding about policy, practice, theory or a particular substantive field;• defensible in design by providing a research strategy that can address the evaluative questions posed;• rigorous in conduct through the systematic and transparent collection, analysis and interpretation of qualitative data;• credible in claim through offering well-founded and plausible arguments about the significance of the evidence generated.These principles are presented at a sufficiently high level of abstraction that they would apply to a diversity of qualitative approaches. Indeed, most of them are simply emblems of sound and logical enquiry, whatever its form or purpose.
39 What we are doing at EBC Mapping the evidence in our 5 areas… Presenting key info in chartsDrawing initial conclusions from studies in each areaSome studies will not be impact studiesPresentation of findings to NPLs and senior staff across the service…
40 Completing a pro forma 1. Basic info should be recorded Reference (title/ author)Date of publicationCountry2. Check if the study should be includedCheck the full paper against the inclusion criteriaIf the study meets the inclusion criteria, complete the rest of the pro formaIf not, exclude the study and note your reasons whyInfo requiredTry looking first in…AimsBackground/ Introduction sectionMethodsMethods sectionQuality of evidenceFindingsDiscussion/ Results sectionConclusionsResults/ Conclusions sectionTHE TABLE AIMS TO GIVE SOME SHORTCUTS SO THAT YOU DON’T HAVE TO READ THE FULL PAPER TO GET SOME OF THE KEY INFO40
41 Synthesis in practice Complete a pro forma for 2 articles 1. GMP study summary – 15 mins2. An included article for your area – 15 minsLook at the methods of the study and try to ‘score’ it on the Maryland ScaleIgnore the section on inclusion criteriaCover findings/ conclusions if you have time but the methods are your focusSome examples might not fit the scale…41
42 Synthesis in practice GMP study summary 10 mins to individually complete as much of the pro forma as possible5 mins to discuss with neighbour/ compare findingsGroup feedback42
43 GMP study summary (1/2) Training The GMP procedural justice training experiment: Wheller et al(2013)UKPolicingTest the impact of training on officer attitudes, officer behaviour and victim satisfactionTraining576Random43
44 GMP study summary (2/2)Training had a positive impact on officer attitudes, officer behaviour and victim satisfactionTraining works (in this context!)44
45 Synthesis in practice Included study from your area 10 mins to individually complete as much of the pro forma as possible5 mins to discuss with neighbour/ compare findingsGroup feedback45
46 Theft study summary (1/2) Does Heroin Prescription Reduce Crime?: Killias & Rabasa1997Policing/ HealthSwitzerlandStudy of the impact of heroin prescription programme on crime rates in SwitzerlandDrug prescription/ treatment319Non-random46
47 Theft study summary (2/2) Number of outcomes – comparison of rates of (self reported) crime pre and post intervention.Heroin prescription reduces (self reported) criminality. Police statistics also show reductions in crime.47
48 Wellbeing study summary (1/2) Mental, physical, and behavioural outcomes associated with perceived work stress in police officers: Gershon et al2009USAPolicingStudy links between perceived stress and impact of ‘coping’ on work stress and healthNone – non-impact study1,072Non- random48
49 Wellbeing study summary (2/2) Outlines a number of statistical relationships drawn from the survey, e.g. between job stressors and work stress“results underscore the need to re-evaluate police training of recruits… to ensure they get training necessary to meet the daily challenges and demands of police work”49
50 Career progression study summary (1/2) Encounters in social cyberspace: e-mentoring for professional women: Headlam-Wells et al2006UKTrainingLook at barriers for women in career progression and evaluate the effects of an e-mentoring programmeE-mentoring programme122 (89)Non-random50
51 Career progression study summary (2/2) Mentoring had a positive impact on mentee ‘employability skills’. Mentoring helped with mentee development.Suggestion that mentoring may be particularly important to women (but this study does not compare to a similar group of men…)51
52 Prostitution study summary (1/2) Attitudes regarding criminal justice responses to sex trafficking: Muftic?Bosnia and HerzegovinaPolicingExamine law enforcement attitudes to sanctioning people involved in the sex trade.None – non-impact study363Self reporting52
53 Prostitution study summary (2/2) A number of factors are associated with officer support for arrest, etc of trafficking victims, traffickers and customers – including their degree of adherence to prostitution mythsTraining for officers “need[s] to include the provision of factual information regarding prostitution and sex trafficking”53
54 MH crisis study summary (1/2) Studying Chicago Police Dept’s Crisis Intervention Team: Watson2010USAPolicingTest the impact of training on officer attitudes, officer behaviour and victim satisfactionCrisis Intervention Team216Matched sites54
55 MH crisis study summary (2/2) CIT officers directed a significantly greater proportion of subjects to services than non-CIT officers. CIT did not have an impact on arrest decisions.Challenging to conduct studies of CIT, but possible. Study suggests CIT may improve outcomes of police encounters with persons with mental illness.55
56 Available for synthesis What’s next…AreaAvailable for synthesisAcute Mental Health Crisis38Barriers to career progression46Reducing theft from the person58Supporting mental well-being72Responding to prostitution48Get synthesising!In the breakout rooms you’ll have everything you need to turn the available papers for your group into a resplendent and wonderful presentation to wow our senior guests tomorrow…Good luck!A final ‘master list’ of included studies was sent to the library for each groupLibrarians get hold of papers from the library catalogue or the British Library (if required)Some studies not available, some not available in English – some were large books which we have excluded due to time constraints…Also a consideration of cost - £10 per paper we didn’t already have in the library!Leaves us with a final number ‘available for synthesis’ by you all over the next day…56