1. Systematic reviews in the social sciences Sarah Whitehead, Cardiff Institute of Society and Health

2. Outline Why do systematic reviews in the social sciences? Examples of types of reviews:  Correlates of physical activity  Effectiveness of tobacco harm reduction approaches  Barriers and facilitators to tobacco harm reduction Overview of different types of evidence generated Lessons learnt

3. Why do systematic reviews in the social sciences? Chaos in the brickyard Goal of scientific research to discover explanations Research findings to be assembled like bricks to create edifices Initially bricks carefully made and only those required for the edifice produced Over time, brick making became an end in itself until the land was flooded with bricks, many of them random and useless Now we need to search through so many bricks it’s difficult to find proper ones for the task “Saddest of all, sometimes no effort was made even to maintain the distinction between a pile of bricks and a true edifice” 1 1. Forscher 1963

4. Why do systematic reviews in the social sciences? Systematic reviews allow us to make sense of the chaos in the brickyard They allow us to:  develop a comprehensive understanding of research to date  identify areas of evidence that can be translated into policy and practice  determine which areas are in need of further research

5. Examples of types of reviews

6. Example 1: correlates of physical activity Research question:  What factors are associated with physical activity participation among adolescent girls? Suitable design:  Correlates review of quantitative studies reporting a measure of physical activity and at least one potential correlate

7. Example 1: methods 4 databases searched, manual searches of journals Inclusion criteria:  Quantitative research design  Published 1999-2003  Participants in age range 10-18 years  Results available for females separately  Articles published in English language  Any type of physical activity

8. Example 1: methods (cont’d) Potential correlates classed as demographic/biological, psychological, behavioural, social/cultural, or physical environmental Variables classed as related/ not related to physical activity Strength of association noted 1 Variables only included if studied three or more times 2 Quality of physical activity measure coded 2 :  Self-report of poor or unknown reliability/validity  Self-report with acceptable reliability/validity  Acceptable objective measure 1. Cohen 1988 2. Sallis et al. 2000

9. Example 1: findings 50 published papers yielding 51 independent samples Majority of studies (63%) carried out in US Mean sample size (females only) = 1,280 (range 22-9,039) 41 (80.4%) cross-sectional in design Majority (61%) used self-report with acceptable reliability/validity

10. Example 1: findings Demographic variables:  Non-white ethnicity (-)  Age (-)  SES (+) Psychological variables:  Enjoyment (+)  Perceived competence (+)  Self-efficacy (+)  Physical self-perceptions (+) Behavioural variables:  Smoking (-)  Organised sport involvement (+) Social/cultural variables:  Family/parental support (+)  Father’s physical activity (+) No consistent trends for environmental variables

11. Example 1: conclusions Several variables related with PA across 3+ studies Many potential correlates understudied Several weak designs and measures of poor or unknown validity Most promising modifiable correlates:  Family support  Positive psychology  Organised sport

12. Example 2: effectiveness of tobacco harm reduction approaches Research questions:  How effective are tobacco harm reduction approaches with the intention of quitting (i.e. ‘cutting down to quit’ or ‘reduction to stop smoking’), with and without assistance?  How effective are long-term harm reduction approaches without the prior intention of quitting (i.e. reducing consumption without the aim of quitting), with and without assistance? Suitable design:  Two effectiveness reviews of experimental studies of tobacco harm reduction approaches, e.g. randomised controlled trials, controlled trials

13. Example 2: methods 21 databases, 28 websites, range of 'snowballing’ techniques Included populations:  People of all ages wanting to quit smoking but unable to do so abruptly / not willing to quit but wanting to reduce/ stop temporarily  Participants in cut down to quit/ reduction/ temporary abstinence interventions Included interventions:  Pharmacotherapies licensed for cutting down/ temporary abstinence  Other non-tobacco nicotine containing products, e.g. e-cigarettes  Behavioural support or counselling  Self-help

14. Example 2: methods Data extraction to capture:  Study design, setting and participants  Intervention details  Methods of analyses  Outcomes Quality appraisal to assess:  Methodological quality  External validity  Narrative synthesis and meta-analyses to summarise findings

15. Example 2: included studies and types Cut down to quit 11 studies  10 RCTs or quasi-RCTs  1 uncontrolled before & after Quality  Generally moderate Applicability  Only one UK study Cut down without quitting 45 studies  29 RCTs or quasi-RCTs  3 non RCTs  2 controlled before & after  10 uncontrolled before & after  1 secondary analysis 1 systematic review Quality  Generally moderate Applicability  5 UK studies  6 in UK-comparable countries

16. Example 2: types of evidence examined Effectiveness of pharmacotherapies in helping people cut down to quit or cut down indefinitely without the aim of quitting The effectiveness of behavioural support, counselling or self-help Whether different approaches had differential impacts on different groups

17. Example 3: barriers and facilitators to tobacco harm reduction approaches Research question:  What are the barriers and facilitators to implementing tobacco harm reduction approaches, including users’ and providers’ perspectives? Suitable design:  Mixed methods review of qualitative and quantitative views and opinions studies

18. Example 3: methods 21 databases, 28 websites, range of 'snowballing’ techniques Included populations:  Smokers interested in THR or who take part in THR interventions, or those who provide opinions or experiences regarding THR approaches  Service providers, healthcare personnel and policy makers who may deliver/ commission/ refer smokers to THR interventions Included study types:  Views studies relating to cut down to quit/ long term reduction/ temporary abstinence intervention studies previously identified  Views studies relating to tobacco harm reduction approaches in general

19. Example 3: methods Data extraction to capture:  Study design, setting and participants  Methods and analytical approach  Outcomes / key themes Quality appraisal to assess:  Methodological quality  External validity Thematic synthesis of views, experiences and perspectives Mapping of whether effectiveness studies addressed the identified barriers and facilitators to THR approaches

20. Example 3: included studies and types 41 studies  22 cross-sectional surveys  13 qualitative studies  3 mixed methods studies  2 process evaluations  1 longitudinal study Quality  Generally moderate Applicability  9 UK studies and 5 UK applicable

21. Example 3: types of evidence examined Factors that might act as barriers or facilitators to tobacco harm reduction approaches Smokers’/ their families’ and healthcare professionals' views and experiences on whether specific tobacco harm- reduction approaches are perceived to have a differential impact on particular groups Smokers’/ their families’ and healthcare professionals' views and experiences on the potential or actual unintended consequences from adopting a harm- reduction approach

22. Summary of different types of evidence that systematic reviews can generate Correlates of behaviours or outcomes Effectiveness of interventions / intervention type studies Views and opinions All can be useful in informing policy and practice and identifying evidence gaps where further research is needed

23. Lessons learnt Develop a solid search strategy Identify appropriate tools for the type of review being undertaken, for example quality assessment measures and approaches to synthesising the data; Most importantly – work with a team of experienced information specialists

24. Acknowledgement Sarah Whitehead’s role at Cardiff Institute of Society and Health is supported by the Welsh Assembly Government’s National Institute for Social Care and Health Research