Presentation on theme: "From sifting to synthesis Evidence Base Camp: February 2014 Levin Wheller Practice Development Team Research Analysis and Information Unit."— Presentation transcript:
From sifting to synthesis Evidence Base Camp: February 2014 Levin Wheller Practice Development Team Research Analysis and Information Unit
This session… 1.Recap on REAs 2.While you were away… Peer review of sifting and the task ahead 3.Mapping research evidence Systematic Mapping – the what and the why An example: Mapping Evidence on Public Order Tactics 4.Appraising research evidence The Maryland Scale of Scientific Methods Some hypothetical examples Beyond Maryland: non-impact questions Qualitative methods 5.Completing a pro forma Practical group work
Systematic reviews and “what works” Explicit objective/ question Studies are selected on explicit criteria Thorough search for studies Each study is screened according to uniform criteria Reasons for excluding studies are clearly documented
Rapid Evidence Assessments Use systematic principles and the same process as a systematic review… …but make compromises given available time and resources Pragmatic and transparent approach
Some key principles Demonstrate consistency in searching/ sifting Document search and sift process – Process should be transparent and repeatable Specify required quality of evidence – Systematic reviews only? – Pre-post studies only? – All ‘empirical’ papers? Be explicit/ transparent about the limitations of the approach
The process (in a nutshell) 1.Draft search terms 2.Draft sift criteria 3.Sift received abstracts 4.Request full papers 5.Read and ‘grade’ papers 6.Write it up (‘synthesis’)
Sifting Identifying truly relevant literature Searching Identifying potentially relevant literature Synthesis Summarising what relevant literature tells us
Searching Identifying potentially relevant literature
Boolean is a powerful tool and can be used to create quite complex searches. The standard Boolean operators are: AND OR NOT (or AND NOT) (advanced users only!) Boolean is most effective when used in combination with: PARENTHESES(round brackets) TRUNCATION e.g. politic* WILDCARDS e.g. randomi?e DOUBLE QUOTESfor specific phrases The Basics Boolean operators
Domestic violence search Search terms 1.((((Domestic OR Familial OR Marital OR Spousal OR Partner) Near/2 (Abus* OR violen* OR control* OR coerc*)) OR "domestic homicide" OR "marital rape" OR "intimate partner violence" OR (coercive NEAR/2 (control OR behavio*r*))) 2.AND (risk AND (predict* OR indicat* OR factor* OR management OR identif* OR assess* OR prevent* OR likelihood OR probability)) 3.AND ("systematic review" OR "literature review*" OR trial* OR "RCT" OR experiment* OR evaluat* OR "best practice*" OR "good practice*" OR effective* OR assess* OR "What Works" OR impact* OR success*)) Risk c 1m papers ‘What works’ c 2m papers DA/ DV c 75k papers Initial search identified 1,531 abstracts to sift
Searching as a ‘blunt tool’ Research Question: What interventions have been shown to be effective in preventing or reducing theft from the person? Theft from person synonyms include: Robbery, Mugging, Dipping, Pick pocketing, Assault with intent to rob, Snatching, Theft from person, Bag theft, Bag snatch, Personal theft… Profile on circadian blood pressure and the influencing factors in essential hypertensive patients after treatment So how do we end up with an abstract titled…
Returned by theft search… Yuan Gang Qiu et al. Profile on circadian blood pressure and the influencing factors in essential hypertensive patients after treatment Non-dippers were defined as those whose nocturnal decrease in mean systolic BP and/or mean diastolic BP was < 10% of the daytime BP. Binary logistic regression analysis was used to evaluate the correlation between circadian blood pressure profile and factors as gender, age, height, body mass index (BMI), family history of premature cardiovascular disease, women under age 65 or men under age 55, smoking habits, grade of hypertension, and strategy of antihypertensive drugs. Similarly, the incidence of non-dippers in patients of overweight (24 /= 28) were 3.0 and 4.8 times of those in subjects of normal weight (P = 0.003 and 0.009, respectively). Compared with patients treated with long-acting calcium channel blockers (CCBs), patients treated with angiotensin- converting enzyme (ACE) inhibitors or angiotensin II receptor blockers (ARBs) alone had less prevalence of nondippers (OR = 0.139, P = 0.010).
Sift criteria The same set of questions are applied to all abstracts to attempt to remove bias/ overt ‘judgement calls’. Typically, a lot of papers (up to 95%!) identified by the search are excluded at this stage of the process. QuestionAnswerAction Q1. Does the paper consider theft from the person or relevant synonyms? NoExclude YesGo to Q2 UnclearGo to Q2 Q2. Does the paper consider interventions that aim to reduce/prevent theft from the person or relevant synonyms? NoExclude YesGo to Q3 UnclearExclude Q3. Does the paper include empirical data or methods? NoExclude YesInclude UnclearCan’t exclude
Peer Review: Second sift… ‘Red’ abstracts – ‘Dip sample’ taken, e.g. one in 10 ‘red’ abstracts were peer reviewed – Check for ‘false-exclusions’ ‘Green’ and ‘Amber’ abstracts – All resifted to review against the inclusion criteria – Check for ‘false-positives’ – Decide where possible on ‘Ambers’ Final list of included studies sent to library…
Second sift… REAAbstractsFirst siftSecond sift Domestic Abuse2,678622411 Public Order57218238 Always expect to reduce the number of studies substantially through second sifting Standard process for all REAs and SRs Dual coding and often used and disagreements discussed between coders to make a final decision Clarify queries/ refine criteria as a group Speed of EBC does not enable all of this!!!
While you were away… AreaAbstracts returned Included after 1 st sift Included after 2 nd sift Available for synthesis Acute Mental Health Crisis 1,4081735538 Barriers to career progression 1,2051538246 Reducing theft from the person 8441928258 Supporting mental well- being 1,2291629372 Responding to prostitution 9791958748
Synthesis Summarising what relevant literature tells us
Systematic Mapping… Methodology developed by EPPI-Centre Useful first step of synthesis process Mapping can: 1.Describe the nature and coverage of research in the topic area 2.Identify gaps in the research literature 3.Describe the design of studies and direct further review work
Systematic Mapping… What does it involve: Collection of key information about each study into a pro forma/ template
Systematic Mapping… Collate pro forma results into a searchable bibliographic database
What is the evidence on Public Order Tactics? Country of focus
What is the evidence on Public Order Tactics? Research Method
Public Order Tactics - Mapped Overall: There is only limited research available on public order policing tactics. Only 38 papers were found to be relevant and based on empirical methods. Age: Most of the research (32/38 papers) has been published since 2000. Methods: Most papers used qualitative research methods, typically involving observations (n=19) or interviews (n=18). Only one study used experimental methods.
Public Order Tactics - Mapped Unit of study: Only five studies focussed directly on public order tactics. Instead, studies primarily focussed on an event or series of events (n=17) or the police more generally (n=10). Tactics: Of the 30 approved public order tactics, only nine were mentioned in the research evidence identified for this map. The most frequently mentioned tactics were containment (n=5), Police Liaison Teams (n=3) and CS smoke (n=3).
Maryland Scientific Methods Scale (SMS). (AKA Maryland Scale) Developed by Sherman et al (1997) Established method for judging the quality of impact studies Useful for judging how much weight to give different studies identified in a ‘what works’/ ‘what’s effective’ REA Limited use for judging qualitative studies and questions asking ‘how’ things work
What does ‘good’ or ‘robust’ ‘what works’ evidence look like? Systematic Reviews (Based on level 3-5 studies) 5 Randomised controlled trials 4 Before/after measures Multiple site comparisons 3 Before/after measures Two site comparisons 2 Before/after measures No comparison site 1 One-off measure No comparison site Study designs increasingly rule out potential alternative causes Statements about ‘what works’ Statements about ‘what’s promising’ Study designs cannot rule out potential alternative causes Statements about possible impact
‘Level 1’ Evidence “Two months after the introduction of mobile information, officers were spending 55% of their time out on patrol” Problems? 1.Can’t rule out other explanations 2.Time spent on patrol before mobile info? 3.Is 55% good/ bad/ average? 4.No before/ after measures 5.No comparison group Can’t say if mobile information makes a difference/ is worthwhile
‘Level 3’ evidence Problems? 1.Still can’t rule out other explanations for change 2.Can’t control for differences between the two areas – different culture, different case load, management, etc. % of time on patrolPrePost Western District (receives intervention) 43%50% Eastern District (no intervention) 46%50% We still can’t be sure that improvements are down to the intervention rather than other factors “Two months after the introduction of mobile information, officers in the Western District had increased their time on the street by 7 percentage points, while officers in the Eastern District had seen an increase of 4 percentage points…”
2,000 officers in force RCTs - ‘gold standard’ evaluation design A random sample of 600 officers 300 officers in treatment group Outcomes measured 300 officers in control group Get mobile data Outcomes measured Business as usual
‘Level 5’ Evidence - RCTs 1. Participants are randomly selected from the population and randomly assigned to each group to minimise the chance of bias “Two months after the start of the trial, officers with access to mobile data were spending 55% of their time out on patrol, whilst officers in the control group were spending 43% of their time out on patrol” 2. Randomisation helps us rule out other potential explanations for differences in outcome measures 3. A control group means we can compare differences in outcomes between groups where the only difference is the intervention (e.g. mobile data 4. RCTs enable us to make strong statements about cause and effect We can say that officers with access to mobile data spend more time on the street… in this context.
Systematic reviews Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1. Overall, legitimacy interventions resulted in a large, significant increase in positive perceptions of police.
Systematic Reviews (Based on level 3-5 studies) 5 Randomised controlled trials 4 Before/after measures Multiple site comparisons 3 Before/after measures Two site comparisons 2 Before/after measures No comparison site 1 One-off measure No comparison site Study designs increasingly rule out potential alternative causes Statements about ‘what works’ Statements about ‘what’s promising’ Study designs cannot rule out potential alternative causes Statements about possible impact What does ‘good’ or ‘robust’ ‘what works’ evidence look like?
Beyond Maryland… The Maryland scale is focussed on impact questions… and studies might answer a range of other questions… Qualitative studies don’t fit into the Maryland dichotomies – You can have different quality qualitative work as well Question typeExample NeedsWhat do people want or need? ProcessWhy/how does it work? ImplementationWhat is required to make it work? CorrelationWhat relationships are seen between phenomena? AttitudinalWhat do people think? What are their experiences? EconomicHow much does it cost and with what benefit/harm? How ‘good’ a study is will depend upon how appropriate the methods used are for answering the research question…
Quality of Qualitative Research Frameworks to assess/ appraise the quality of qualitative research use ‘principles’ rather than hard- and-fast rules Cabinet Office e.g. in 2003 sets key principles being that qualitative research should be: contributory in advancing wider knowledge defensible in design rigorous in conduct credible in claim http://www.civilservice.gov.uk/wp-content/uploads/2011/09/a_quality_framework_tcm6-38740.pdf
What we are doing at EBC Mapping the evidence in our 5 areas… Presenting key info in charts Drawing initial conclusions from studies in each area Some studies will not be impact studies Presentation of findings to NPLs and senior staff across the service…
Completing a pro forma Info requiredTry looking first in… AimsBackground/ Introduction section MethodsMethods section Quality of evidenceMethods section FindingsDiscussion/ Results section ConclusionsResults/ Conclusions section 1. Basic info should be recorded Reference (title/ author) Date of publication Country 2. Check if the study should be included Check the full paper against the inclusion criteria If the study meets the inclusion criteria, complete the rest of the pro forma If not, exclude the study and note your reasons why
Synthesis in practice Complete a pro forma for 2 articles – 1. GMP study summary – 15 mins – 2. An included article for your area – 15 mins Look at the methods of the study and try to ‘score’ it on the Maryland Scale – Ignore the section on inclusion criteria – Cover findings/ conclusions if you have time but the methods are your focus – Some examples might not fit the scale…
Synthesis in practice GMP study summary 10 mins to individually complete as much of the pro forma as possible 5 mins to discuss with neighbour/ compare findings Group feedback
GMP study summary (1/2) The GMP procedural justice training experiment: Wheller et al UK (2013) Policing Test the impact of training on officer attitudes, officer behaviour and victim satisfaction Training 576Random
GMP study summary (2/2) Training had a positive impact on officer attitudes, officer behaviour and victim satisfaction Training works (in this context!)
Synthesis in practice Included study from your area 10 mins to individually complete as much of the pro forma as possible 5 mins to discuss with neighbour/ compare findings Group feedback
Theft study summary (1/2) Does Heroin Prescription Reduce Crime?: Killias & Rabasa Switzerland 1997 Policing/ Health Study of the impact of heroin prescription programme on crime rates in Switzerland Drug prescription/ treatment 319 Non-random
Theft study summary (2/2) Number of outcomes – comparison of rates of (self reported) crime pre and post intervention. Heroin prescription reduces (self reported) criminality. Police statistics also show reductions in crime.
Wellbeing study summary (1/2) Mental, physical, and behavioural outcomes associated with perceived work stress in police officers: Gershon et al USA 2009 Policing Study links between perceived stress and impact of ‘coping’ on work stress and health None – non-impact study 1,072 Non- random
Wellbeing study summary (2/2) Outlines a number of statistical relationships drawn from the survey, e.g. between job stressors and work stress “results underscore the need to re-evaluate police training of recruits… to ensure they get training necessary to meet the daily challenges and demands of police work”
Career progression study summary (1/2) Encounters in social cyberspace: e- mentoring for professional women: Headlam-Wells et al UK 2006 Training Look at barriers for women in career progression and evaluate the effects of an e-mentoring programme E-mentoring programme 122 (89) Non-random
Career progression study summary (2/2) Mentoring had a positive impact on mentee ‘employability skills’. Mentoring helped with mentee development. Suggestion that mentoring may be particularly important to women (but this study does not compare to a similar group of men…)
Prostitution study summary (1/2) Attitudes regarding criminal justice responses to sex trafficking: Muftic Bosnia and Herzegovina ? Policing Examine law enforcement attitudes to sanctioning people involved in the sex trade. None – non-impact study 363 Self reporting
Prostitution study summary (2/2) A number of factors are associated with officer support for arrest, etc of trafficking victims, traffickers and customers – including their degree of adherence to prostitution myths Training for officers “need[s] to include the provision of factual information regarding prostitution and sex trafficking”
MH crisis study summary (1/2) Studying Chicago Police Dept’s Crisis Intervention Team: Watson USA 2010 Policing Test the impact of training on officer attitudes, officer behaviour and victim satisfaction Crisis Intervention Team 216 Matched sites
MH crisis study summary (2/2) CIT officers directed a significantly greater proportion of subjects to services than non-CIT officers. CIT did not have an impact on arrest decisions. Challenging to conduct studies of CIT, but possible. Study suggests CIT may improve outcomes of police encounters with persons with mental illness.
What’s next… AreaAvailable for synthesis Acute Mental Health Crisis38 Barriers to career progression46 Reducing theft from the person58 Supporting mental well-being72 Responding to prostitution48 Get synthesising! In the breakout rooms you’ll have everything you need to turn the available papers for your group into a resplendent and wonderful presentation to wow our senior guests tomorrow… Good luck!