Presentation is loading. Please wait.

Presentation is loading. Please wait.

Methods Festival Oxford - July 20061 Things that go wrong in comparative surveys – evidence from ESS Jaak Billiet CCT of ESS K.U. Leuven Methods Festival.

Similar presentations

Presentation on theme: "Methods Festival Oxford - July 20061 Things that go wrong in comparative surveys – evidence from ESS Jaak Billiet CCT of ESS K.U. Leuven Methods Festival."— Presentation transcript:

1 Methods Festival Oxford - July 20061 Things that go wrong in comparative surveys – evidence from ESS Jaak Billiet CCT of ESS K.U. Leuven Methods Festival June 2006 Oxford

2 Methods Festival Oxford - July 20062 outline Introduction: conceptual frame for assessing data quality in a methodologically well prepared cross-nation survey ESS Selection of two aspects: the non-response problem and measurement equivalence Evaluation of non-response bias Evaluation of equivalence of measures Conclusion: what can be done?

3 Methods Festival Oxford - July 20063 Introduction: conceptual frame attempt to combine two approaches: Total Survey Error approach (TSE) & Total Quality Management approach (TQM) (Loosveldt, Carton & Billiet, IJMR 2004) TSE: data quality is absence of variable and systematic error: often less attention to non- sampling error TQM: all components of the production process contribute to the quality of the end product

4 Methods Festival Oxford - July 20064 Introduction: conceptual frame here is focus on data-collection by means of face-to- face interviews Conceptual frame is combination of - interviewer tasks (contacting & obtain co- operation/interview in narrow sense) - result on completion (obtained sample/responses) - kind of evaluation: process/output evaluation

5 5 Conceptual frame focus is now on

6 Methods Festival Oxford - July 20066 Selection of two aspects Evaluation of non-response bias in ESS: several possibilities (1) comparing sample with population statistics (2) comparing respondents with non-respondents using additional information about nr (3) comparing co-operative respondents with reluctant respondents using information of contact forms… example 1: traces of non-response bias with (3)

7 Methods Festival Oxford - July 20067 Selection of two aspects Evaluation of measurement invariance in ESS: several possibilities - comparing distributions and looking for outliers… - looking for response sets - testing measurement models of latent variables and evaluation of levels of factorial invariance example 2: two translation problems detected

8 Methods Festival Oxford - July 20068 I. Evaluation of non-response bias Many rules and actions in ESS in order to obtain high and comparable response rates (goal = 70%) Result: response rates vary between 35% and 80% in round 1 and between 43% and 79% in round 2 Example: Round 1 Goal not obtained however much higher response rates than in other surveys in most countries

9 9 Figure 1. Response rates (target 70%)

10 Methods Festival Oxford - July 200610 When nr-bias in comparative surveys? -none if no bias in separate countries - None if equal bias in separate countries and non response rates are equal - Well if equal bias in separate countries and non response rates very different - Well if different bias in separate countries ESS = bias is expected because of very different response rates and traces of difference in bias…

11 Methods Festival Oxford - July 200611 non-response bias… ESS not worser than other surveys and lots of effort in order to estimate nr-bias How? One example of data quality assessment in ESS = comparing co-operative and reluctant respondents Reluctant respondents = original refusals who are converted into respondents

12 Methods Festival Oxford - July 200612 Data = contact files and main data files contact form: - Info about time of each contact attempt (up to 10) - Info about every mode of each C.A. - Info about every outcome of each CA - Info about kind of non-response of each CA - Info about reason of refusal - Estimation of strength of refusal -> base for conversion and distinction between soft and hard refusals - Info about ineligibles - Info about respondent (age, gender, housing form)

13 Methods Festival Oxford - July 200613 non-response bias… method assumption: reluctant respondents are informative for final refusals (some evidence for that in waves of mail surveys…) How detect? Linking contact forms of co-operative and reluctant respondents with main data file Analysis of countries with substantive number of reluctant respondents (NE and DE more than 450 reluctant respondents, GB, AT & CH more than 115)

14 Methods Festival Oxford - July 200614 non-response bias… Find differences in scores between Co & Re on social demographics (education, age, urban environment…) Find differerences in scores between Co & Re on relevant attitudinal variables (latent mutliple indicator variables: political trust, political participation, ethnic threat…) Multiple regression with all relevant control variables and type respondent (Re/Co) as predictor and attitudinal variable as dependent var (Do we find a net effect of Co/Re on the attitude?) see figure If sign. differences = trace of bias

15 Methods Festival Oxford - July 200615 Steps in analysis of nr-bias 1. Simple regression: kind of resp ? attid. var 2. multipe regression evaluate change (1) – (2) kind of resp ? attid. var soc. demograph.

16 16 Some traces of nr-bias in attitudes most in expected direction (exception CH both cases and GB for p.p.)

17 Methods Festival Oxford - July 200617 Some traces of nr-bias About the same (but smaller diff) for interest in politics, trust in politics, but not in all countries Participation in voting: not all in same direction Differences in social demographics not all in same direction! Next step is simple and multiple regression of type of respondent on attitudes Example: Political part and ethnic threat in DE and NL Parameters in simple (model 1) and multiple (model 2) regression

18 18 Some traces of nr-bias Model 2 always smaller effect but non-significance depends of country and dept. var

19 19 Nr-bias: evaluation Correctly recording contacts is hard task and more expensive – resistance of (errors in some countries) Refusal conversion – problems of privacy regulation – not done in every country (more succesfull in more countries in R2) In the coutries that were used (5 in R1) some differences in direction Therefore, not possible to apply corrections for every country, thus not usefull in comparative analysis unless… Relies on rather weak hypothesis: reluctant ± refusals

20 20 II. Evaluation of measurement equivalence Focused on sets of indicators for latent variables = evaluation of indicators within context of construct Full equivalence (see figure and rule) - invariance of corresponding slopes (factor loadings) over all countries (metric invariance) - invariance of corresponding intercepts of indicators with latent avairalbe over all countries (scalar invariance) - invariance of error terms (residuals) weaker forms of equivalence when hypotheses on equality constraints rejected = only the pattern of relation between concept and indicators is invariant

21 21 valid measurement in comparative setting idea of causal relationship* between latent variable (LV) and four observed indicators (OV), and between measured latent variable and theoretical (intended) concept (TC) measurement validity theoretical validity? random error validity parameter (=equal for all disciplines?) e ov1 e ov2 M e LV ? TC e ov3 ov4 Method effect assumed to be zero

22 Methods Festival Oxford - July 200622 Formal notation of MG comparison The relationship beween item I j and latent trait W in group g is denoted as follows group (country) (1) latent trait intercept termslope parameter link function (e.g. OLS regression) observed indicator (j = number of indicator)

23 Methods Festival Oxford - July 200623 Kinds of equivalence construct equivalence is defined as follows : Metric invariance = equal slopes with j = 1, …, J and g,h = 1, …, G; g h (2) Regression coefficients (with observe variable as dependent var) validly comparable equality of slopes is required

24 Methods Festival Oxford - July 200624 Kinds of equivalence if one wants to compare means of latent variable and indicators additional requirement = scalar invariance equality of intercepts: with j = 1, …, J and g,h = 1, …, G; g h (3) an item is measurement invariant across groups if restrictions (2) and (3) hold

25 Methods Festival Oxford - July 200625 What if slopes/intercepts of some groups not invariant? if test of equal slopes/intercepts shows that the parameters are not invariant for some items, then several options: - exclude groups - remove items (try to know why? = our examples) - resort to partial factorial invariance (free estimation of factor loadings) - conclude that construct has different meaning and rely on weaker form of equivalence

26 Methods Festival Oxford - July 200626 An example of equivalent measurement: willingness to allow immigrants Six items (D4-D9) in 21 countries Tests by Multi Group Structural Equation Modeling Proportion Odds Model (very strict) (Welkenhuysen-Gybels & Billiet, 2002) here = only MGSEM

27 Methods Festival Oxford - July 200627 To what extent do you think [country] should allow people xxxxxx to come and live here? (4-point scale: many = 1, some = 2, a few = 3, none = 4) d4: …of a different race d5: …of the same race… d6: …from the richer countries in Europe… d7: …from the poorer countries in Europe… d8: …from the richter countries outside Europe… d9: …from the poorer countries outside Europe…

28 Methods Festival Oxford - July 200628 Model specifications

29 Methods Festival Oxford - July 200629 Test information Model modifications χ²χ² dfRMSEAp (close fit)NFI M0factorial invariance951,895080,02211 M1 free τ 1 (hu) 867,015070,02011 M2 free λ 1 (hu) 847,475060,01911 M3 free τ 1 (dk) 827,295050,01911

30 30 The quasi equivalent measurement model (negative coeff because item scores not reversed many=1…none=4) (intercepts not very different between countries)

31 Methods Festival Oxford - July 200631 Things that go wrong… « lost in translation » Good results for D4-D9 set, but not for all sets of items Two examples of problematic translation Possible to detect by MGSEM? Where? - in France « asylum items » - In DK « ethnic threat » items Problems discoverd outside standard MGSEM tests because FR and DK not in release, but checked ad hoc (skip all details)

32 32 Example 1: « …generous in judging peoples applications for refugee status » How detected? In comparison between « neighbours » of Belgium very large deviation in FR on one items weighted by gender and age; weighted by gender, age, and education * *

33 CARD 40: Some people come to this country and apply for refugee status on the grounds [1] that they fear persecution in their own Using this card, please say how much you agree or disagree with the following statements.[1] The government should be generous [2] in judging peoples applications for refugee status[2] [1][1] On the grounds: in the sense of both because and stating that [2][2] Generous: liberal. In Luxemburg, Genéve (CH), and Wallonia (BE) Le gouvernement devrait être généreux en traitant les demandes du statut de réfugié Translation of D51 in source language (English)

34 Wallonia, Switserland, Luxemburg CARTE 40: Des gens viennent en [country] et demandent un statut de réfugé car ils se sentent peresécutés dans leur propre pays. A laide de cette carte, dites-moi sil vous plaît quelle mesure vos êtes daccord ou en désaccord avec les propositions suivantes… Le gouvernement devrait être généreux en traitant les demandes du statut de réfugié Luxemburg: …en accordant le statut de réfugié Switserland: …en traitant dun statut de réfugié France Certaines personnes arrivent en France et demandent le statut de réfugié parce quelles craignent des persécutions dans leur pays Equeteur montre liste 49 (tout à fait daccord etc….) Les pouvoir publics devraient se montrer plus ouverts dans lexamen de ces demandes

35 Methods Festival Oxford - July 200635 Three differences in one small statement Le gouvernement devrait être généreux en traitant les demandes du statut de réfugié Les pouvoir publics devraient se montrer plus ouverts dans lexamen de ces demandes Differences: gouvernement = the government pouvoir publics = the administration montrer plus ouvert = more acceptable than the extreme devrait être généreux «ces demandes » no direct reference to « statut de réfugié »

36 Methods Festival Oxford - July 200636 Example 2: D24 in Denmark Source questionnaire Questions D23 & D24 on serious crime and any crime D23 If people who have come to live here commit a serious crime, they should [1] be made to leave[1] D24 If people who have come to live here commit any crime, they should [2] be made to leave[2] (five point scales: completely disagree 1 ---compl agree 5) [1][1] Should in D23 and D24 have the sense of must. [2][2] Should in D23 and D24 have the sense of must.

37 Methods Festival Oxford - July 200637 Two examples: item D24 in Denmark One can expect a contrast effect (backfire, or contrast) in these items Mean approval of D23 for all countries = 79% Mean approval of D24 for alle countries = 51% How problem detected? Strange result of DK in report of EUMCR (Scheepers et al. 2004) : two items combined for all countries (PCA scores and % support reported (support of made to leave = % higher then mean (0))

38 38 unlikely figure Mean score and percentage suport on favour repatriation policies for criminal immigrants (ESS R1) (selection of countries in correct order)

39 Methods Festival Oxford - July 200639 Arguments: D23-serious crime: 77;2% of Danish want immigrants to leave = higher than AT, BE, ES, FI, FR, IE, IL, LU, NL, NO, SE about average of EU (79%) (pweight) certainly not the lowest as was reported D24: any crime: only 12.9% of the Danish want immigrants to leave = lowest of all. Much lower than average in Europe = 51% (pweight). What happened? D24 in Denmark

40 Methods Festival Oxford - July 200640 Lost in translation Contrast between D23-D24 much larger because of different translation of crime in D24 in DK D.23 Hvis mennesker, der er kommet for at bo her, begår en alvorlig forbrydelse, skal de udvises af landet D.24 Hvis mennesker, der er kommet for at bo her, begår nogen som helst form for lovovertrædelse, skal de udvises af landet Forbrydelse = crime is used in NO and SW for D23 and D24 alvorlig forbrydelse = serious crime Lovovertrædelse = any kind of law violation, associated with minor crime (violation of traffic rules included) is used in D24 in DK nogen som helst form for Lovovertrædelse = any form of minor crime

41 41 Is this detectable in tests of Factorial Invariance? YES Example of MGSEM: D51 in FR, measurement model for D49-D55 in FR, Genève, Wallonia, and Luxemburg (four groups) D49 [Country] has more than its fair share of people applying refugee status (-) D50 People applying refugee status allowed to work while cases considered (+) D51 Government should be generous judging applications for refugee status (+) D52Most refusee applicants dont fear persecution in own countries (-) D53Refusee applicants kept in detention centres while cases considered (-) D54Financial support to refugee applicants while cases considered (+) D55Granted refugees should be entitled to bring close family members (+) Set of items is balanced (4 pos, 3 neg) = a MGSEM model with a substatial factor and a style factor (aquiescence) (see Billiet & McClendon, SEM 2000) (see model)

42 42 The MGSEM measurement model

43 Methods Festival Oxford - July 200643 Detection of problematic item Aim: detect problematic item very early in the test procedure for finding adequate model Problematic item nr 3 (D51) in France directly detected in model with Asylum + Style (method effect ) item 5 for LU needs also inspection (to be done)

44 Methods Festival Oxford - July 200644 Equivalent measurement: evaluation In spite of many efforts for correct translation still some problems Translation problems are detectable by - comparing distributions (find strange outliers) - MGSEM tests ( -parameter not invariant, to detect early in stage of test) - in discussion with native speakers, translators… Other sources of in-equivalence…

Download ppt "Methods Festival Oxford - July 20061 Things that go wrong in comparative surveys – evidence from ESS Jaak Billiet CCT of ESS K.U. Leuven Methods Festival."

Similar presentations

Ads by Google