Presentation on theme: "REALIST SYNTHESIS An Overview"— Presentation transcript:
1 REALIST SYNTHESIS An Overview RAY PAWSONUniversity of LeedsERSC Methods FestivalJuly 06
2 Questions for todayLessons learned - how does knowledge about policy interventions accumulate?Lessons transferred – how can past be related to present and future?Lessons applied – how should accumulated wisdom be dispensed?Lessons lived – what does it feel like to do systematic review?
3 Changing the basic metaphor - ve+veMeta-analysis – synthesis as summationThe purpose of synthesis is to ‘pool’ ‘amalgamate’, ‘combine’, ‘fuse’ the findings from all prior research on a given family programmeThis assumes a body of research consisting of a large number of controlled trials of the SAME programme (treatment) carried out across a fixed population.Not Today – why this basic assumption is futileMean effect
4 ‘Realist synthesis’ in a slide KEY METAPHORUnit of analysis is the programme theoryPrimary data is inspected for what it reveals about the programme theoryPurpose of the review is to test and refine programme theoryTECHNICAL SEQUENCEQuestion selection as ‘theory elicitation’Searching as ‘theoretical sampling’Quality appraisal by ‘theory testing potential’Data extraction as ‘confrontation of theory with evidence’Synthesis as ‘theory refinement’Dissemination as production of ‘middle-range theory’.
5 Realist Synthesis: Naming & Shaming Examples of Public Disclosure Initiatives Megan’s Law & sex offender registrationOuting prostitute’s johnsSchool league tables`Inspection and special measures for ‘failing schools’Hospital and surgeon report cardsPub-watch bans and exclusionsLocal press adverts for poll tax non-payment and council rent arrearsBeach cleanliness standards and kite markingPUBLIC DISCLOSURE initiatives. There are dozens of such programmes in US and UK at the moment The basic ‘theory’ is that one can attack under-performance by publicity, one can reduce deviance by publicity, etc. It is used in all the above fields.
6 Shaming Sanctions are Contagious Further Examples of Public Disclosure Initiatives Community Right-to-Know Act on Environmental HazardsCar crime indices and car safety reportsMandatory (public) arrest for domestic violenceRoadside hoardings naming speeding driversPosters naming streets with unlicensed TV watchingRail company SPAD ratings (signals passed at danger)EC name and shame and fame initiative on compliance with directives.AND MORE
7 Public notification – basic theory i) Identification: in which the performance or behaviour in question is observed and then classified, measured, rated, ranked, verified, etc.ii) Naming: in which information on, and the identity of, the failing or deviant party is disclosed, publicized, disseminated, notified, published, broadcast, registered etc.iii) Public sanction: in which the broader community acts on the disclosure in order to shame, reprimand, reproach, censure, control, influence, supervise the named party.iv) Recipient response: in which behavioural change follows the sanction, with the subjects being shamed, regretful, penitent, contrite, restrained, re-integrated etc.
8 NAMEDBUT WHO IS SHAMED?Poll-tax protesters named in the local newspapersSex offenders under community notificationUnder-performing schools identified in league tablesMotor manufacturers named in the Car Theft IndexExampleThe basic idea is that public disclosure will work in different ways for different subjects in different circumstances. So the research starts in exactly the same way as realist evaluation. The only difference is that one looks to a secondary analysis of existing research for the evidence. Let us look at ONE hypothesis - expect that shaming works differently according to the status of the named party.Consider 4 cases - what does the evidence say about how these four rather different parties react?In this case I want to concentrate on how one can use existing social science theories as part of the process of synthesis
9 Poll-tax protesters named in the local newspapers Poll tax non-payment – there were a great many sanctions imposed: fines, wage arrest, court appearances etc. Protesters thus ignored or even celebrated disclosure of their names in the local press. Policy abandoned.Result - ‘Naming and Proclaiming’.Public notification merely one of a long list of failed attempts.Including ‘ponding’Note on the quality of evidence here - partisan literature produced within the movement but OK!
10 Sex offenders under community notification Notified Sex Offenders are resentful or scared by public attention. One key result is non-compliance with registration and displacement to other localities. Re-offence rates remain static.Naming and skulkingNot much ‘reintegrative’ shaming going on here.
11 Under-performing schools identified in league tables Schools respond to league table positions tactically - increasing resources to marginal candidates (middle grades) and excluding/ not entering hopeless ones (lowest grades). Grades improve.Naming and SchemingAlso common reaction in hospitals
12 Motor manufacturers named in the Car Theft Index Car manufacturers – are embarrassed by adverse publicity and loss of reputation (and downturn in sales) - respond with genuine improvements in vehicle security. Car crime goes down.Finally, a group that does respond to threats to reputation. Quite good evidence that the car theft index was a significant feature in inducing Ford to change its key system.Why the differences - it sort of ‘figures’ that there might be such a pattern but what’s the explanation?
13 Is there a theory to accommodate these differences in outcome? Reference group theory.Antagonostic - hostile to threats or overturesDetached - disinterest in treats or overturesMarginal - take on the trappingsCandidates - fall into linePlace these four example directly on the typology.Merton’s Typology of aspirations to group membership of eligibles and non-eligibles
14 The theory under further refinement For N&S to ‘work’ the following configuration should be in place:the named party should be an ‘aspirational insider’the shaming mechanism should be dovetailed with other mechanism (market sanctions)the disclosure should carry intense (but controllable) media interestthe disclosed data should unambiguous both in allocating blame and in suggested remedial actionthe disclosing authority should have had exemplary watchdog credentials, which are operated benignly
15 Evidence-based Policy: A Realist Perspective … And FinallyThe book:Ray PawsonEvidence-based Policy: A Realist PerspectiveLondon: SageThe website: