Presentation on theme: "Stephen Platt Qualitative research and suicide"— Presentation transcript:
1 Use of mixed methods in the evaluation of suicide prevention strategies and interventions Stephen PlattQualitative research and suicideSeminar, Cardiff University, 2 July 2007
2 Structure of presentation Mixed methods researchSuicide in Scotland: trends and contextChoose Life strategy and action planEvaluation planSelected main findingsSelected recommendations
3 Mixed methods research Adoption of a research strategy involving more than one type of research methodMay be mix of qualitative and quantitative methods or mix of quantitative methods or mix of qualitative methodsIncreasing use of mixed methods strategies, especially combining qualitative and quantitative approachesMany reasons why this is occurring
4 Why the increasing popularity of mixed methods research? (1) Opportunity for skills enhancementBroadening methodological repertoire mitigates vs. “trained incapacities” (Reiss)Encourages thinking ‘outside the box’Cross-national research (e.g. EU) provides increased opportunities for mixed methods research
5 Why the increasing popularity of mixed methods research? (2) Fits with political currency accorded to ‘practical enquiry’ that speaks to policy/ makers and informs practice“Whole industry” (Brannen) of mixed methods research created around evidence-based policy and in policy evaluationIncreasing emphasis upon disseminationResearchers need to communicate in “double speak” (Brannen): technical/specialised language of research and popular language that can easily communicate findings/messages to ‘users’Words as important as numbers in writing up research
6 Misconceptions of ‘the other’ Quantitative attitudes qualitative:Too context specificSelection of data to fit preconceptionsUnrepresentative samples/examplesUnwarranted claimsQualitative attitudes quantitative:Overly simplisticDecontextualisedReductionist in terms of generalisationsFailing to capture subjective meaning
7 Rationales underlying choice of method(s) (1) Paradigms/philosophical assumptionsQualitative and quantitative research seen as intrinsically different (particularly with regard to philosophical traditions)But surveys not necessarily conducted on basis of positivist assumptions and qualitative researchers using participant observation often work in realist traditionMicro-level emphasises subjective interpretations. Macro-level concerned with larger patterns/ trends and seeks structural explanations. But all aim to understand individuals in society. Methods need to be congruent with this quest.
8 Rationales underlying choice of method(s) (2) PragmaticsCommonplace to argue that methods should be appropriate (and subordinate) to research QBut usually many research Qs. Some may be underpinned by realist assumptions, others by interpretevist assumptions.And practicalities of research process may change original intention anyway, with outcomes of research taking precedenceMixed methods believed/claimed to produce “better” outcomes than reliance on single method (pragmatic orientation linked to emphasis upon policy/practice application)
9 Rationales underlying choice of method(s) (3) PoliticsPolitical rationales for using mixed methodsExample: addressing condition of women in society requires use of large-scale quantitative data (structural) as well as in-depth qualitative data (personal experience/perspective) in order to understand/expose gendered inequalitiesCrucial issue is the purpose to which methods are put rather than the methods per se
10 Combining methods in research process: context of justification Stage of data analysis and interpretationData derived from different methods cannot be added together to produce unitary realityPossible outcomes when methods are combinedCorroboration (‘triangulation’) (‘same’ findings from different methods)Elaboration (one method exemplifies how findings apply in particular cases)Complementarity (findings differ but together – synergistically – they generate insights)Contradiction (findings from different methods conflict)
11 Mixed methods designs: key dimensions Logic of enquiry: inductive (aimed at discovery) or deductive (aimed at hypothesis testing)No one-to-one correspondence between one particular logic of enquiry and one type of method(If mixed method approach is warranted) the ordering of methods needs to be considered: sequential or simultaneous?How dominant is a particular method going to be (consider scarce resources)?
12 Possible permutations of mixed methods designs Examples of simultaneous designsQUAL + quanQUAN + qualQUAL + QUALQUAL + qualExamples of sequential designsQual QUANQUAL quanQuan QUALQUAN qualQual QUALQUAL QUAL
17 Scotland’s national suicide prevention strategy & action plan: Choose Life Launched in December 2002Major element of Scottish Executive’s work on health improvement and mental healthPlan being implemented in phasesBudget for phase 1 (April 2003-March 2006) was £12mAdditional £8.4m allocated for first two years of phase 2 ( )Overall aim: to reduce suicide rate in Scotland by 20% in 2013 (cf 2002)
18 Choose Life: national and local infrastructure Designated National Implementation Support Team (NIST) co-ordinates and supports national development and implementationNIST’s core functions include: awareness raising/campaigning; working with the media; development/dissemination of information and knowledge; and guiding and supporting local implementationIn each local authority Choose Life action plans have been developed by the Community Planning Partnership (CPP)National and local budgetary allocation
19 Evaluation: main objectives Assess whether sustainable infrastructure is being developed nationally and locally to support achievement of Choose Life strategyMeasure and review progress towards implementation of Choose Life milestonesExamine whether and how Choose Life is stimulating effective forms of practiceProvide detailed recommendations to guide the next phase of the action plan
20 Theory-based evaluation Theory-driven approaches are intended to address needs of programme implementersAims to articulate and test the explicit and implicit theories that shape the design and planned implementation of a programmeAttempts to determinewhether a programme has been delivered as intendedwhat aspects of the programme work, for whom and in what circumstances/contexts/settingsTheory of change: “a systematic and cumulative study of the links between activities, outcomes and contexts…” (Weiss)
21 Theory-based evaluation: the role of context Context as a powerful ingredient in programme evolution and success – not a variable to be ‘controlled for’Social programmes can’t be tested/rolled out in laboratories, so getting to grips with the interface between context and intervention is of fundamental importanceQualitative approaches can do this to a degree but rarely in a way that address concerns about programme impactA further criticism of traditional experimental approaches is that it strips programmes of their context by controlling for intervening variables. For evaluators of a qualitative and theory-based bent this is anathema.Qualitative approaches have been able to provide some answers to questions about how certain programme processes interact with their contexts and participants – for example, work looking at the circumstances in which partnership working will flourish, or the reasons that smokers engage with smoking cessation programmes but those who design policy programmes whilst they can be convinced that process and contextual information is valuable usually require some focus on outcome and effectiveness.
22 Theory-based evaluation: the process Through a collective/collaborative process, the evaluator encourages programme stakeholders to articulate at the earliest possible stage :The programme’s rationaleIts intended outcomesThe activities that will be implemented to achieve theseContextual factors and their influenceThe ensuing ‘theory of change’ can be used to improve programme planning and of make evaluation decisionsThrough a collective and collaborative process involving documentary review, stakeholder interview and groups discussion…The ensuing ‘theory of change’ can be used to improve programme planning and of make evaluation decisions about what processes, impacts and outcomes need to be captured and when.
23 Main methods Two electronic surveys of local co-ordinators Two rounds of interviews with key informants at national level (including NIST)Detailed and in-depth exploration of theories of change in 8 selected local areas (using interviews, workshops, observation, documentary analysis)Two workshops with local coordinators and national informants
24 Co-ordinator surveys Covering range of areas: (Progress towards) Local vision for changeProgress in the development of the local infrastructureResource allocation and generationExamples of innovative and effective practiceMonitoring and evaluationSustainability and mainstreamingCollection of data on suicide and deliberate self-harmReflections on national support and on local progressOpen and closed questionsRating scales to measure satisfaction with national action towards achievement of milestones and to review local implementation progress
25 Interviews with key informants Key elements of semi-structured interviews:Considering progress towards the achievement of the national milestones set out in Choose LifeUnderstanding the criteria applied at the national level to assess local implementationConsidering the quality of collaboration between the NIST and major national agencies for the achievement of Choose Life objectives.Interviews with members of NIST and sample of key national informants (e.g. ChildLine, Samaritans, SAMH, NUJ)Additional component: exploration of the NIST ‘story’ via individual interviews and joint workshop
26 Local area case studies (1) Case studies were main vehicles forexploring implementation processidentifying overall theoretical framework within which local suicide prevention teams workedRepresentative sample selected by:Geographical type (rural/remote, urban and mixed) [1º criterion]Local suicide rate, focus on priority groups, interaction between national and local levels, and approaches to coordination [2º criteria]Two fieldwork visits 6 months apart
27 Local area case studies (2) Interviews with case study informantsPurposive sampling: 4 key informants involved in decision-making process (e.g. members Choose Life partnership responsible for priority setting) and 2 representatives involved in Choose Life funded activities16 project activities selected for more detailed examination (2 per case study site)Intended to provide representative selection across Choose Life priority groups and objectivesSelection negotiated with coordinatorsInterviews conducted with project representative, usually project lead/manager
28 Local area case studies (3) Observational activities and collection of documentationObservation of Choose Life events (usually partnership meetings but also training days, evaluation days and practitioner fora)Key documents included: minutes of Choose Life partnership meetings, locality reports on progress, reports of previous needs assessmentEvaluation team aimed to understand theories of change at overall programme level as well as at project level
29 Local area case studies (4) Changes to approachWorkshops replaced individual interviews with stakeholders at second fieldwork visitProvided opportunity for joint testing of, and reflection on, local area theories of changeLocal participants encouraged to assess progress towards Choose Life objectives and milestones, and how this could be demonstratedProblems created as result of mixed levels and responsibilities of participants (e.g. strategic versus operational)
30 National workshops (1)Two workshops one year apart brought together evaluation team, NIST, local coordinators and several key stakeholders1st workshop aimed to:develop understanding of the different models of national and local actions and activities being put in placeexplore the evolving relationship between the approaches of the centre and of local areasidentify ways in which progress could be measured.
31 National workshops (2) 2nd workshop aimed to: Review progress and learning in relation to objectives of Choose LifeTest out key themes emerging from evaluationIdentify future priorities for development, support required, measures of progress and outcome, and implications for information collection and research
32 Data analysis (1) Continuous (iterative) process throughout study Evaluation database designed and used to store (and retrieve) data on all 32 local areasFindings from each element of data collection (case studies, workshops, national interviews, surveys) written up in detailed reports which were then used for comparative analysisData analysed according to predefined themes (e.g. sustainability, partnerships) …… and themes also developed from inductive analysis conducted at each phase of evaluation
33 Data analysis (2)Analytic framework developed to guide team through the evaluationThis was expanded and changed according to themes emerging from dataAnalysis primarily drew on ‘charting’ method of systematically handling complex datasets by drawing out dimensions relating to each theme across all ‘cases’
34 Evaluation: main findings Sustainable infrastructures for implementationAllocation and use of resourcesInnovative practice and use of evidenceSustainabilityDecision making processes and learning
35 Sustainable infrastructures for implementation Demonstrable progress made by NISTCPP successes … but not as evident in less ‘mature’ partnerships and in engaging with clinical servicesVarious models of local coordination developedPreference for a dedicated (full-time) coordination post …… but evaluation unable to demonstrate the superiority of this model
36 Allocation and use of resources CPPs have attracted considerable additional investment at local level and in-kind contribution …… but areas have not been equally successful in raising additional funding …… and there has been unnecessary duplication of effort at local levelChoose Life has stimulated a considerable amount of activity relating to self-harm …… but local areas have different understandings of ‘high risk’ suicidal behaviour and have adopted different responses to address the problem
37 Innovative practice and use of evidence Many examples of locally defined innovative practice …… and multiple sources of information and evidence used to inform local planning and activity …… but research rarely used systematically
38 SustainabilityNIST identified several achievements in building a sustainable infrastructure for suicide preventionAt local level, most success achieved in mainstreaming training activities
39 Decision making processes and learning Local stakeholder consultation: key approach to set implementation prioritiesShort timescale to develop first action plan was a major challengeNational support for learning has been delivered through diverse routesNIST has highlighted a strong commitment to evaluation …… but a national framework for evaluation remains to be completedIn local areas different levels of priority and attention have been attached to evaluation.
40 Recommendations: mainstreaming at national level Incorporate Choose Life objectives and priorities into other policy streams/initiativesInvolve clinical services in population-based suicide prevention activitiesInvolve national voluntary sector organisations in awareness raising and campaigningEngage in purposive innovation to test out, evaluate, learn and implement
41 Recommendations: mainstreaming at local level Using intelligence from a range of sources, as tools in planning for sustainabilityBuilding in mechanisms to track and review progress towards objectives across policy areasMore focused targeting of action is required
42 Recommendations: self-harm More consideration to be given in phase 2 to the integration of self-harm into Choose LifeThe strategy should continue to encompass high risk self-harm …… but the less ‘serious’ component of self-harm cannot be ignored
43 Recommendations: CPPThe CPP remains the most appropriate vehicle for developing strategy and overseeing delivery in relation to Choose Life at the local levelBut its limitations should be recognisedNeed to examine the necessary partnerships that have yet to be put in placePriority should be given to establishing/building on effective links with clinical & drug/alcohol servicesNIST should continue to work closely with CPPs to ensure that Choose Life budgets are fully spent on suicide prevention activities
44 Recommendations: central coordination body Some type of central coordination body will continue to be required in the immediate futureKey tasks: provide national oversight, assess and support performance and ensure accountability at local level, promote learning and effective knowledge transfer, and co-ordinate action.There should be a review of how the central coordinating function is delivered and where it is situated
45 Choose Life evaluation (first phase): research team University of EdinburghStephen Platt (RUHBC)Emma Halliday (RUHBC)Margaret Maxwell (General Practice)Scottish Development Centre for Mental HealthJoanne McLeanAllyson McCollamAmy WoodhouseLondon School of EconomicsDave McDaid (Health & Social Care)Glasgow UniversityMhairi Mackenzie (Public Health & Health Policy)Avril Blamey (Public Health & Health Policy)