Presentation on theme: "Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany."— Presentation transcript:
Asking Survey Questions in Many Tongues Paper presented at the ESS Launch Conference Brussels, November 25-26, 2003 Janet A Harkness, ZUMA Germany
Overview English as survey lingua franca The ESS translation context ESS requirements and procedures Common survey practice and where the problems lie The ESS and lessons learned
English as survey lingua franca We can all speak English Source questionnaires often in English Texts anchored to language Language-driven challenges (two way)
The ESS Context: Facts and Figures Round One fielded in 22 countries Fielded in languages, depending on how you count All languages used as first by 5% and more 5 countries fielded in more than one language (12 languages) Most often shared languages (to date): French Italian German. 9 countries shared one or more languages
The WP Context: Resources & Researchers ESF EC and Partners CCT Scientific Advisory Board Participating countries Translation Workpackage Translation Expert Panel Dr. Hans Hönig, Dr. Paul Kussmaul, Beth-Ellen Pennell, MA, Dr. Alisú Schoua-Glusberg
Key Workpackage Component TRAPD A framework for ESS translation and assessment to deal with the complex requirements
Translation framework TRAPD Translation, Review, Adjudication, Pre-testing and Documentation 5 interrelated processes with protocols involved in producing a final translated version TRAPD combines translation with assessment
Key Activities for Round One Develop framework for ESS translation and assessment: TRAPD Overview report on TRAPD Guidelines for NCs on translation and assessment procedures, team members roles. Strategies for languages shared across countries Documentation framework and guidelines
Additional practical measures Basic guidelines on organising translation and assessment, selecting & training translators Documentation template for problems, divergences, sharing harmonisation Source questionnaire annotated to help translators Consultation hotline
Ongoing work--Round One into Round Two Probe interviews with (external) translators on the source questionnaire Analysis of (external) translators appraisals of selected questions from ESS translations Comparisons of selected translations for shared languages
In Sum Funds, Guidelines, Specifications, Requirements, Illustrations and Examples Templates, Annotations, Hot-line, Presentations and Reports Ongoing: cognitive-based work on the source questionnaires and translations
Isnt this an unusual amount of effort? Arent questionnaires easy to translate?
Questionnaires at first glance Easy questions, simple wording Response categories often one or two words Simple and repeated instructions Repetition of lexical and structural elements throughout Similar or identical questions in other surveys And the work team often knows English
Questionnaires at second glance Questionnaires are complex texts They look easier than they are Good questions not easy to construct Measurement-in-text poses special problems Cross-disciplinary requirements (e.g., measurement, translation) Interdisciplinary expertise needed A key problem: resulting challenges under- estimated
A Complex Text Type Text-with-blanks genre Often largely generic Texts within the text Low Context- jump from one topic to another Cross-national aims encourage general formulations Computer applications can technically complex source formats
One example of not-so-easy features Measurement properties lead to surveyspeak and scalespeak Surveyspeak and Scalespeak are textual traces of measurement (Harkness 1996)
Surveyspeak Q:All things considered, how satisfied are you with your life as a whole nowadays ? (ESS, B13) Q:Hows life?
Q:When did you last visit a dentist? Would you say that you are //very certain, somewhat certain, neither certain nor uncertain, somewhat uncertain or very uncertain// about the date you just gave? A: Conversational Follow-up: Uh-Huh Really ? Are you sure?
Scalespeak -- response categories How important or unimportant is ---- Extremely Important Somewhat important Neither important/or unimportant Somewhat unimportant Extremely unimportant
Response scales are difficult to translate Differences across languages/ cultures in quantification structures & potential negation saliency of concept (satisfaction, justness) disclosure norms response styles numeracy and related notions scaling norms (1-6; 1-10?)
Known Measurement Effects vs. Allow/ Not allow Allow/ Forbid
So what about differences such as: vs. Agree/ Not agree Agree/ Disagree Agree/ Reject or
Translations may not be able to match English scale properties German: Agree / Not agree/ Reject English : Agree/ Not agree / Disagree
Despite these complexities views on translation translators assessment and testing are often pre-theoretical and procedures followed not grounded in research
Not by chance that the ESS work package focused on Survey translation process procedures Specifications, briefing, & training materials for translators and for those hiring translators Survey translation evaluation procedures Providing access to know-how to implement translation components
These are what the survey community in general still lacks Thus the ESS effort and activities meet real needs It is a necessary amount of effort
In addition No design costs, only translation and assessment Core questionnaire designed for replication Even ESS translations (in a deluxe version) are a very small part of total national survey costs.
Lessons learned from: Contact with national teams Documentation from national teams Discussions within the Expert Panel and CCT Cognitive interviews with external translators Related work in other projects (eg SHARE, EUYOUPART, ISSP)
Some lessons learned 1.Procedures and guidelines crucial Team work Iterative translation and review Sharing and harmonisation 2.More hands-on input would help more On the job training Training opportunities and materials
Lessons learned continued 3. Pretesting –TRApD –more needed on source questionnaires 4.Version documentation is essential – but the Ugly Duckling of the work package. Need to help it have its proper place in a user-friendly form 5.Version harmonization raises scheduling issues. No harmonization discussion raises comparability issues 6.More intensive cross-cultural input needed Questionnaire content Questionnaire formulations [Raises planning, timing and procedural questions]
Thank you very much for your time and attention