1. Cross-Cultural Survey Guidelines and Quality Monitoring Beth-Ellen Pennell 2009 International Total Survey Error Workshop (ITSEW 2009) Tällberg, Sweden Survey Research Operations Survey Research Center Institute for Social Research

2. Unique Challenges – Locating and engaging respondents 2 ITSEW 2009 Beth-Ellen Pennell

3. Literacy 3 ITSEW 2009 Beth-Ellen Pennell

4. Gatekeepers/Privacy From the Institute of Social Research’s Population and Ecology Laboratory in Nepal 4 ITSEW 2009 Beth-Ellen Pennell

5. *From the International Telecommunications Union (http://www.itu.int/ITU-D/ict/statistics/ict/index.html)http://www.itu.int/ITU-D/ict/statistics/ict/index.html Infrastructure 5 ITSEW 2009 Beth-Ellen Pennell

6. Unique Data Collection Challenges Other: – Research traditions – Languages – Seasonal – Political – Religious – Geographical 6 ITSEW 2009 Beth-Ellen Pennell

7. Standardization issues Concentrate on key design aspects Decide when to be rigid and when to be flexible Experience is that adherence to standards and regulations must be checked ISO 20252

8. What is special about quality in an international setting? Procedural equivalence is important Concepts must have a uniform meaning Scientific and administrative challenge Risk management differs Financial and methodological resources differ National pride is at stake Conflicts of interest

9. Examples of Quality Assurance Central planning and support organization Deep bench of senior experts Up-to-date translation procedure Pretesting of questions and questionnaires Interviewer training Probability sampling design Call scheduling algorithm Formulas for calculating base weights Documentation system User communication channels A set of operational specifications

10. Deviations from specifications Examples from 1995 IALS Average interviewer workload varied between 6 and 30 Two countries chose sampling control instead of 100% keystroke validation One country did not calculate the base weights correctly One country informed respondents that the survey was just a pretest

11. Evaluations and peer reviews Recently ESS Examples of recommendations: – Develop quantitative indicators for all process steps – Standardize contact forms – Set bounds on effective sample size – Improve capacity building – Expand the user base

12. Some thoughts For some countries it is a challenge to reach minimum standard Process stability is difficult to obtain in decentralized survey environments Survey organizations must be aware of the meaning of specifications and the effects of certain methodological choices Reasons for deviations must be checked Vigorous monitoring and performance checks necessary

13. 13 ITSEW 2009 Beth-Ellen Pennell

14. Cross-Cultural Survey Guidelines Contributors and Reviewers Contributors: Kirsten Alcser, UM-SRC Ipek Bilgen, UNL Ashley Bowers, UM-SRC Rachel Caspar, RTI Judi Clemens, UM-SRC Peter Granda, UM-ICPSR Sue Ellen Hansen, UM-SRC Janet Harkness, UNL Frost Hubbard, UM-SRC Rachel Levenstein, UM-SRC Christina Lien, UM-SRC Zeina Mneimneh, UM-SRC Rachel Orlowski, UM-SRC Beth-Ellen Pennell, UM-SRC Emilia Peytcheva, UM-SRC Ana Villar, UNL Graphic design assistance Larry LaFerte, UM-SRC Ruth Shamraj, UM-ICPSR Formatting and copy-editing Gail Arnold, UM-SRC Shaw Hubbard - independent consultant Programming and website maintenance Tricia Blanchard, UM-SRC Reviewers: Dorothee Behr, Gesis Bill Blyth, TNS Europe Pam Campanelli, independent consultant Somnath Chatterji, WHO Rory Fitzgerald, European Social Survey Steve Heeringa, UM-SRC Tim Johnson, University of Illinois, Chicago Achim Koch, Gesis Frauke Kreuter, University of Maryland Paul Kussumaul, European Social Survey Kristin Miller, National Center for Health Statistics Peter Mohler, University of Mannheim Meinhard Moschner, Gesis José L. Padilla, University of Granada Alisú Schoua-Glusberg, Research Support Services Eleanor Singer, UM-SRC Tom W. Smith, NORC Jare Struwig, Human Sciences Research Council Rick Valliant, University of Maryland Gordon Willis, National Institutes of Health Christine Wilson, Heriot-Watt University Christof Wolf, Gesis 14 ITSEW 2009 Beth-Ellen Pennell

15. Goal To develop and promote internationally recognized guidelines that highlight best practice for the conduct of comparative survey research across cultures and countries Initiative of Comparative Survey Design and Implementation (CSDI); 2005 annual meeting 15 ITSEW 2009 Beth-Ellen Pennell

16. Guidelines Initiative Initiative in response to: Increasing number and scope of cross-cultural surveys over past decade Desire to increase operational equivalence and survey quality through harmonization – Within and across “units” (e.g. countries) – Across waves of panel study Lack of published materials on implementation Balance standardization versus localization 16 ITSEW 2009 Beth-Ellen Pennell

17. Target Audience Researchers and survey practitioners planning or engaged in cross-cultural or cross-national research – Basic to advanced information – References – Suggested further reading 17 ITSEW 2009 Beth-Ellen Pennell

18. Process Developed over two and a half years Weekly meeting of core staff Each guideline underwent iterative, internal reviews Sent to selected external reviewers with expertise in topic area Published last summer Revised last fall 18 ITSEW 2009 Beth-Ellen Pennell

19. Guideline Topics I.Study structure II.Tenders, bids, and contracts III.Ethical considerations IV.Sample design V. Questionnaire design (in development) VI.Translation VII. Adaptation (in development) VIII.Survey instrument design IX.Pretesting X.Interviewer recruitment and training XI.Data collection XII.Harmonization of data XIII.Data processing XIV.Dissemination XV.Assessing quality 19 ITSEW 2009 Beth-Ellen Pennell

20. Components of Guidelines Introduction to topic area Goal of guideline Guideline Rationale Procedural steps Lessons learned Glossary References Further suggested reading 20 ITSEW 2009 Beth-Ellen Pennell

21. Format of Guidelines “Drill-down” approach – Increasing level of detail Links available to – Glossary – References – Other modules – External information 21 ITSEW 2009 Beth-Ellen Pennell

22. 22

23. Quality Framework Cost Burden Design Constraints Professionalism Comparability Cost Burden Design Constraints Professionalism Coherence Cost Burden Design Constraints Professionalism Interpretability Cost Burden Design Constraints Professionalism Accessibility Cost Burden Design Constraints Professionalism Timeliness Cost Burden Design Constraints Professionalism Accuracy Cost Burden Design Constraints Professionalism Relevance Fitness for Use Burden Professionalism Design Constraints Cost HighLow 23 ITSEW 2009 Beth-Ellen Pennell

24. Quality Framework Cost Burden Design Constraints Professionalism Adjustment Error Cost Burden Design Constraints Professionalism Nonresponse Error Cost Burden Design Constraints Professionalism Sampling Error Cost Burden Design Constraints Professionalism Coverage Error Cost Burden Design Constraints Professionalism Processing Error Cost Burden Design Constraints Professionalism Measurement Error Cost Burden Design Constraints Professionalism Construct Validity Accuracy Total Survey Error 24 ITSEW 2009 Beth-Ellen Pennell

25. Quality Framework 25 ITSEW 2009 Beth-Ellen Pennell

26. Summary Website has had~105,214 hits since published in June, 2008 ~288 hits per day Evolving and dynamic: feedback and comments welcome Provides framework for quality control monitoring 26 ITSEW 2009 Beth-Ellen Pennell

27. Thank you. http://ccsg.isr.umich.edu/ 27 ITSEW 2009 Beth-Ellen Pennell