Qualitative Methods as an approach to policy development and Evaluation Using Qualitative Comparative Analysis in comparative policy research

David Byrne – Durham University Presentation at: Using qualitative research to inform policy and practice – Newport 4 th April 2006

What Works? For Whom? – Cui bono? – the political actors question. How? – what implemented processes achieve any sort of outcome? – the engineers question Why? What underlying generative mechanism (s) produces any given future condition? – the scientists question.

… policy researchers, especially those concerned with social as opposed to economic policy, are often more interested in different kinds of cases and their different fates than they are in the extent of the net causal effect of a variable across a large encompassing population of observations. After all, a common goal of social policy is to make decisive interventions, not to move average levels or rates up or down by some miniscule fraction. (Rihoux and Ragin )

… an idea widely shared by social scientists – that it is possible to derive useful empirical generalizations from the examination of multiple instances of social phenomena (i.e. from comparable cases). (Ragin Fuzzy-Set Social Science )

Configuration The idea of configuration is associated with the work of Ragin but can be found earlier in Norbert Elias proposal of a figurational sociology. For us what matters is that the condition of a case at a given point in time – in complexity language its co-ordinates in multi-dimensional state space – is the product of a whole set of factors acting together in interaction.

Each logical combination of values on the independent variable is represented by one row of a truth table. Once this part of the truth table is constructed, each row is assigned an output value (a score of 1 or 0 on the dependent variable) based on the scores of the cases which share that combination of scores on the independent variables. Thus, both the different combinations of input values (independent variables) and their associated output values (the dependent variable) are summarized in a truth table. ….. each row is not a single case but a summary of all the cases with a certain combination of input values. In this respect, a truth table is like a cell from a multi-way cross- classification of several categorical independent variables. (C.C. Ragin – The Comparative Method )

mixed sixform religious comp Highdep highpcsp numberyconsist

More than one way to skin a cat A crucial implication of the idea of configuration is that a given condition may be the product of DIFFERENT combinations of causal factors. Multiple and complex causation is possible. If … we live in a world of great causal complexity, then a common pattern will be for outcomes to result from different combinations of causal conditions. (C.C.Ragin Fuzzy Set Social Science )

BUT IS IT? Configurational accounts do allow for complex causation However, they are still in some ways reductionist. They still reify components over the properties of the system as a whole Still – a step in the right direction.

For causation, the main contrast is between the conventional view of causation as a contest between individual variables to explain variation in an outcome and the diversity- oriented view that causation is both conjunctural and multiple. In the conventional view, each single causal condition, conceived as an analytically distinct variable, has an independent impact on the outcome. In the diversity-oriented view, causes combine in different and sometimes contradictory ways to produce the same outcome, revealing different paths. ( )

Fuzzy sets offer researchers an interpretive algebra, a language that is half verbal-conceptual and half mathematical-analytical …. Most theoretical arguments, as verbal formulations, deal with set-theoretic relationships, so they offer the opportunity for creating a close correspondence between theory and data analysis. (Ragin )

The Nature of Our Materials Cases about which we have both measures and descriptions The measures are Numbers. –Some of the Numbers are continuous measures. –Some of the Numbers are categorical specifications. Our descriptions take the form of texts. The texts are available as online reports

We are dealing with Cases In this kind of research we have cases which, typically are institutions or political / quasi-political entities. The cases can be identified as case nodes. We assign attributes to case nodes.

Using NVIVO with Our Cases With Measures 1.Categorical Measures can be entered directly as text attributes. 2.Continuous Measures can be banded into ordinal values and entered as numbers. With Textual Materials 1.We can read through our texts to establish new attributes in vivo. 2.We can code these new attributes as text attributes to our case nodes.

The Flexibility of NVIVO 1.We can change the range specifications of pre-existing attributes in the light of interpretive engagement with our materials. This usually means extending the rang of values. 2.We can create new attributes which emerge from our interpretive engagement with our materials.

An Example – English Secondary Schools – Pre-existing attributes Continuous Variables 1.School Performance 2.Truancy Rates 3.Size information Categorical Variables 1.School Type 2.Gender 3.Selective or not 4.Catchment area type (can be ordinal)

Data Sources Ofsted Reports – available online and can be downloaded as text documents. Ofsted reports are reports of an inspection. They include descriptive accounts of the school across a range of parameters and specification of performance in relation to objectives. There is a good deal of descriptive text which takes essentially the same form as an ethnographic report.

Procedure in NVIVO Create a set of case nodes corresponding to cases. Create or import attribute table based on existing measurements. Work through an interpretive process which establishes new attributes and varies existing attributes. Construct final attribute table. Export in.dat form.

Using SPSS Import into SPSS Recode using SPSS facilities Especially the Visual Bander for specifying ranges so as to convert continuous variables into ordinal variables. Cluster appropriately as a data reduction exercise to construct both cause conditions and effect outcomes.

Using Tosmana or QCA Input.dat file exported from SPSS Specify variable type etc. using Tosmana facilities if using ordinal variables Establish configurations Look at contradictory configurations. Extend range of causal specifications to reduce proportion of contradictory configurations. Return to materials in NVIVO to look for new causal elements.

References Rihoux, B. and Ragin, C Qualitative Comparative Analysis (QCA): State of the Art and Prospects, paper presented at : APSA 2004 Annual Meeting, Panel 47-9, Chicago, Ragin, C. (1987) The Comparative Method Berkely: University of California Press Ragin, C.C. (2000) Fuzzy-Set Social Science Chicago: Chicago University Press