1. The CSES Questionnaire Design Process
David Howell
University of Michigan
Jessica Fortin
GESIS – Leibniz Institute for the Social Sciences
European Survey Research Association (ESRA)
Lausanne, Switzerland
July, 2011

2. CSES Questionnaire Over 60 countries participate
Common module of survey questions
new theme every five years
some questions the same
some different according to new theme
to be run “as is”
Administrative, demographic and voting variables to national standards

3. Questionnaire Design Process

4. Questionnaire Design Process
A lengthy development process
Over a three year period,
Iterative
As transparent as possible
Collaborative, with broad input

5. Planning Committee Formation
A Planning Committee, comprised of, selected by, and informed by collaborators, designs and oversees each Module.
Public call for nominations
Nominating Committee recommends members
Plenary Session confirms list and suggests additional members
Result = 20 members from 15 countries

6. Theme Development
Every five years a new module is developed to specifically address a big question in science.
Multiple public calls for theme proposals
Task Force collects proposals
Task Force presents and evaluates proposals for Plenary Session and Planning Committee
Theme(s) tentatively decided upon before questionnaire design begins

7. Questionnaire Development
After a theme is arrived at, attention turns to developing the questionnaire for it.
Subcommittees formed to pursue the new theme(s) and improvements to past questions
Subcommittees return to proposers for revisions to their original proposals
Draft questionnaire produced
Planning Committee discussion, revision
Repeat as necessary

8. Pretesting
The draft questionnaire is pretested cross-nationally in different contexts.
Partial pretests in Brazil, Germany, UK
Full pretests in Canada, Ireland
Reports received from pretest countries

9. Questionnaire Finalization
After having received extensive feedback, the questionnaire can now be finalized.
Subcommittees formed to address sections outside of the new theme(s)
A Plenary Session provides final feedback
Review of pretest results, subcommittees
Planning Committee makes final changes
Final questionnaire disseminated

10. Challenges

11. Design Challenges 10-15 minutes is not much questionnaire time
… but, pressure to reduce collaborator burden
competing with other content on surveys
competing with other comparative studies
Existing, validated questions preferred
Priority to cross-national questions
must be applicable in most all countries
must work well across most all countries
must be feasible in a variety of study designs

12. Information Challenges
A lot of people are involved
Managing and tracking lots of input
Reconciling many opinions and approaches
Consensus not always possible
Making decisions with imperfect information
More testing in more contexts would always be better, but relies on volunteers who...
Have an election coming up (optimally)
Have a funded survey with available space
Can produce reports in a timely fashion
Eventually decisions must be finalized

13. Collective Challenges
A lot of countries are involved
Cross-national and cross-cultural applicability and equivalence
Accommodating the variety of institutional arrangements in the many countries
Difficult to predict or account for all situations, even with broad cross-national participation in the process

14. Where to go from here

15. Where to go from here A formal post-process evaluation
We’ve done this a number of times now
What worked well, what didn’t
Consider improvements
Formalize and document process for next time
Some possibilities:
More extensive pretesting
Allow more time for pretesting
Test in more cultures and contexts
Possible use of focus groups

16. Where to go from here More focus on cross-national/cultural issues
... concept equivalence
... collaborator/interview instructions
... translation issues
More methodological working groups
Learning from the approaches of other cross-national studies

17. Thank you for your time!
To learn more about CSES, or to download data:
...or your questions to: