1. Quantitative methods
Questionnaire Design

2. Research Stages Stage 1: Research aims Stage 2: Literature
Stage 3: Research design
Stage 4: Instrumentation
Stage 5: Piloting
Stage 6: Data collection
Stage 7: Data cleaning and Data analysis
Stage 8: Research report

3. What is a questionnaire?
A questionnaire is a structured instrument for collecting primary data on populations of interest in applied or theory-based research.
A well-designed questionnaire facilitates the respondents to provide complete and accurate information.
A questionnaire is a structured instrument for collecting primary data on populations of interest in applied or theory-based research. A questionnaire is an important instrument of research, a tool for data collection. It is a series of predetermined written or verbal questions for which the respondent provides answers. A questionnaire is not some sort of official form, nor is it a set of questions which have been casually jotted down without much thought. Its function is measurement. (Oppenheim, 1992). A well-designed questionnaire facilitates the respondents to provide complete and accurate information.
Questionnaires are designed to serve a variety of purposes: aids in screening or selecting employees, students, trainees and recruits; providers of patient information in health-related setting that contribute to diagnostic and treatment decisions; and as survey instruments for obtaining data on populations of interest. (Aiken, 1997)

4. Main types of questionnaires
Mail/Online questionnaires
Structured interview schedules
The term 'questionnaire' has been used in different ways. Some practitioners refer to this term as self‑administered and postal/online questionnaires. Others would include interview schedules (administered face‑to-face or by telephone) under the general term of 'questionnaires'. In a different way the word 'questionnaire' is sometimes used to distinguish a set of questions, including perhaps some open‑ended ones, from more rigidly constructed scales or tests. There is, of course, some overlap between these techniques. For example, the problems of item wording or phrasing, and of ordering the questions in a particular sequence, are common to them all. In the present context we shall therefore use the term 'questionnaire' fairly loosely to cover postal questionnaires, group‑ or self‑administered questionnaires and structured interview schedules (including telephone interviews). (Oppenheim, 1992)

5. Methods of administering
Interview: face to face or telephone;
Mail or other distribution method
Computer-based
There are three main methods of administering a questionnaire. These can be interview: face to face or telephone; mail or other distribution method, and computer-based form of data collection. Each has advantages and disadvantages, and our final choice will depend on its appropriateness to our purpose and to the means at our disposal. These methods will somewhat impact on the content and format of the questionnaire.

6. Interview/telephone Sampling implication Minimise interviewer effect
Time of interview, sample selection
Minimise interviewer effect
Standardise interview schedule, scripted
Example:Year_After_9-11.doc

7. Mail or Distribution Sampling implication Not supervised
Clarity of questions
Complexity versus simplicity
Room for comments, problems with the questionnaire

8. Online or computer administered
Sampling implication
Not supervised
Clarity of questions
Can incorporate complex pathways of questionnaire items
If answer is yes, go to Q5. If answer is no, go to Q7, etc
Room for comments, problems with the questionnaire
Missing responses checked

9. Steps
Establish a table of specifications, panel and revise if necessary
Write the questions
Determine the general question content needed to obtain each of the desired information
Determine the form of response for each of the questions
Choose the exact question wording.
Panel and revise the questions if necessary
Prepare the questionnaire layout for printing
Arrange the questions into an effective sequence.
Specify the physical characteristics of the questionnaire (paper type, number of questions per page, etc.)
Panel and revise the questions and the whole questionnaire
Pre-testing and Pilot the questionnaire. Analyse and revise the questions and the whole questionnaire if needed

10. Establish a table of specifications
Steps
Identify the program objectives for which the questionnaire is being developed
Operationalise the objectives
Identify the population to be addressed
Identify the methods of administration
Establish the link between research questions, information needed, source of information and methods of data collection
Decide on how to measure each variable
Establish a table of specifications

11. What are the general program objectives or research questions?
Identify the program objectives for which the questionnaire is being developed
What are the general program objectives or research questions?
What are the specific research questions?
What are hypotheses? (draw a diagram)
A first step in constructing questionnaire involves establishing clear objectives of the survey or program for that the questionnaire is used. Some key questions are
What is the questionnaire trying to find out?
What sort of information is it required to elicit?
Answers to these questions will help ascertain what is needed in the questionnaire, and what is extraneous to it. This is a ruthlessly selective process: only relevant questions must be included, and the researcher must desist from including questions that do not explicitly promote the realisation of the survey's objectives, irrespective of their intrinsic interest.
This process is one of continuous clarification, and involves moving from a general expression of interest to clearly defined and measurable entities. One approach to this is the procedure of operationalism. This approach seeks to remove the ambiguity in concepts by specifying the operations by which they are to be measured.
To do any (quantitative) research we must be able to measure the concepts we wish to study. This entails having to specify the meaning of particular concepts precisely and to or to understand the dimensions of the concepts to develop sound measuring procedures which will stand for them.

12. Example PISA Contextual Questionnaire
PISA_Questionnaire_TechnicalReport.pdf

13. PISA Research Themes Table 3.1 Examples:
Student engagement with mathematics
Mathematics self-efficacy
Mathematics self-concept
Mathematics anxiety
Interest in and enjoyment of mathematics
Instrumental motivation to learn mathematics
Study time in mathematics

14. Example Report: Student engagement in schools
StudentEngagementOECDWillms.pdf

15. Determine specific information needed
List dimensions, variables that you should measure
List specific information you hope to collect

16. Establishing the link between information needed, source of information and methods of data collection
Information needed / variables to be measured
Source of information
Methods of data collection
Once the terms or concepts or variables are specified, the table that presents a link between information needed, and from whom and whore information can be obtained, and how to obtain this information (structured interview or mail questionnaire…)

17. Identify the population to be addressed
Source of information
Who is appropriate to provide the necessary information
Characteristics of the target population

18. Identify the methods of administration
Think in advance how the questionnaire should be administered?

19. How to measure each variable
Decide if the variable is directly observable or latent
Is the variable measured by one item?
Is the variable a composite of a number of items (indicators)?

20. Example: Student engagement in schools (see Willms report)
Define student engagement.
Student engagement is measured by two components:
Sense of belonging
Participation
What are indicators of Sense of belonging and Participation?

21. How to measure each variable
Latent variable (Construct)
Is the variable measured through a set of indicators?
If yes, what are the possible dimensions and/or indicators
How is each indicator measured?
Produce a table showing how to measure each indicator

22. Not directly observable
Questionnaire design
Latent Variables and Indicators 1 2
Latent Variable 3
Indicators 4 5
Examples of Latent Variables and Items:
Asthma severity: frequency of shortness of breaths;
Anorexia nervosa: degree of , from mild dieting to refusal of food.
Status of teachers]
Proficiency in mathematics
Language ability
Latent variables are some defined “notion” that is a continuum from “less” to “more”. That is, we can talk about measurements of such variables.
Some variables that are not good examples of latent variables: date of birth. ‘Age’ could be, but usually not latent. 6
Not directly observable
Directly observable

23. Questionnaire design (Barrett, 2002) Construct development
Step 1: Define a meaning for your construct. It will be of narrow focus, capable of sustaining precise measurement.
Step 2: Develop appropriate items for this construct.
Step 3: Test the hypothesis that the items do indeed imply the meaning of the construct as defined.
Step 4: Revise the items
(Barrett, 2002)

24. How to develop indicators (items)
Draft the first items
Panel the indicators
1. Draft the first items
Measures developed and validated in previous research can be used.
Unstructured interview, group focus interview or observation may be helpful in developing scale addressing special group
2. Panel the indicators:
Experts in the field or popple who are familiar or related to the topis can provide good comments
Informants from the group to be surveyed can provide useful clues about the quality of questions

25. Example: How to measure Sense of belonging – draft items
A. I feel like an outsider (or left out of things)
B. I make friends easily
C. I feel like I belong
D. I feel awkward and out of place
E. Other students seem to like me
F. I feel lonely
G. I do not want to go to school
H. I often feel bored.

26. Example: How to measure participation
Measured by the frequency of absence, class-skipping and late arrival at school during the two weeks prior to the PISA 2000 survey.

27. Panel and Review the table of specifications for the program/ project
Do the variables cover all the information needed for the program?
Do the indicators cover all the dimensions of the variable measured?
If more than one questionnaire is to be designed, a panel and review of the table of specifications for each questionnaire should be done

28. Data analysis to check the appropriateness of the indicators
Example on student engagement in schools: (p64, Willms)
A factor analysis of the responses found two factors
one that is based on the first six items and describes whether students feel accepted and included by their classmates,
The second is based primarily on the last two items and describes whether students like school and find it interesting.
The analysis also revealed that the six belonging items contributed almost equally to the first factor.
Therefore, the measure of sense of belonging used in this report is based on a Rasch scaling of the first six items

29. Reliability and Validity

30. Reliability and Validity
High Validity Reliable
Low Reliability High Reliability but low validity
(A) (B) (C)
What are the relationships between reliability and validity?
“Reliability and validity are usually complementary concepts, but in some special situations, they conflict with each other. Sometimes as validity increases, reliability is difficult to obtain and vice versa. This occurs when the construct has a highly abstract and not easily observable definition. Thus, there is a strain between the true essence of the highly abstract construct and measuring it in a concrete manner”. (Neuman, 1997)
If a scale is reliable will it be valid? Not necessary.
If a scale is valid will it be reliable? Yes.

31. Validity
Validity is the ability of an instrument to measure what is designed to measure (Smith, 1991)
Validity refers to the extent to which an empirical measure adequately reflects the real meaning of the concept under consideration (Babbie, 1990: 33)
It is difficult to establish validity of an attitudinal scale.
Who decides that an instrument is measuring what is supposed to measure? The person who has designed the study and experts in the field.
How can it be establish that an instrument is measuring what it is supposed to measure? There are two approaches to establish the validity of a research instrument: Logic and statical.
Establishing a logical link between the questions and the objectives through justification of each question is both simple and difficult. It is simple in the sense that you may find it easy to see a link your self, and difficult because your justification may lack the backing of experts. Therefore it is necessary to get opinions and comments form experts in the field.

32. Example: Student engagement in schools (p.18, Willms)
Participation is measured by the frequency of absence, class-skipping and late arrival at school during the two weeks prior to the PISA 2000 survey.
There are two issues concerning the validity of the participation measure.
One issue is that the measure of participation could be more extensive. It was measured in this study with a rather narrow focus on student absenteeism.
The second issue pertains to how participation is measured. A number of students may have missed school because of illness or for other legitimate reasons.

33. Reliability - 1
Reliability is concerned with how much error is included in the evidence.
If there is no error in the measurement, the same measurement should be consistent over time and context.
The reliability of a measure refers to the consistency of measurement for repeated measurements of the same phenomenon. (Willms, p65)

34. Reliability - 2 Internal consistency reliability
Description
Concerned with how well the items act together to elicit a consistent type of response. Often referred to as Coefficient a
Limitations
Requires statistical procedures to estimate reliability. Does not capture sources of error such as variation over time . Assumes all items tap into one single dimension.
Usages
Important to establish when designing a scale

35. Reliability example Willms, p65.
The measures of sense of belonging and participation are highly reliable at the country level: the reliability coefficients are 0.99 for both sense of belonging and participation.

36. Writing questions

37. Questionnaire design Design the items
Issues to be considered for each item
What information do I want to get?
Is this factual or non-factual?
How to ask?
What kind of responses do I want to get? /How do I want the respondents to answer my questions (format)
How will I code this item?
Will I include the coding in the item format?
Once a Table of Specification is established, the next step is to write the questions. These are issues to be considered when writing questions.

38. Questionnaire design Measurement Types Nominal Ordinal Interval
Nominal type of measurement entails the classification of individuals in terms of a concept or the characteristics being scaled, without there being any implication of gradation or distance between the groups.
Ordinal type of variables classifies individuals into groups but the groups can be ordered in terms of more or less of the concept in the question. Ordinal scale ranks individuals along the continuum of the concept being measured but carries no implication of distance between scale positions. The step form position 1 to position 2 may be different or may be the same as from position 2 to position 3…
Interval type of scale has equal unit of measurement. There fore it is possible to interpret both the order and the distance between categories or groups. Measuring age or income is an example of this type of measurement.
The highest level of measurement is a ratio type of scale which has the properties of an interval scale together with a fix origin or zero point; With a ratio scale we can compare both differences I scores and the relative magnitude of scores;

39. Questionnaire design Question types
To facilitate question writing, it is important to know types of questions. There are two ways of classification of questions:
Classification by response format
Classification by types of information
Factual. Non-factual (e.g., attitude)
To facilitate question writing, it is important to know types of questions. There are two ways of classification of questions:
Classification by response format
Classification by types of information

40. Questionnaire design Closed questions Open-ended questions
Question types - classification by response format
Closed questions
Open-ended questions
A closed question is one in which the respondents are offered a choice of alternative replies. They may be asked to tick or underline their chosen answer(s) in a written questionnaire, or the alternatives may be read aloud or shown to them on a prompt card or a slide. Questions of this kind may offer simple alternatives such as Yes and No, or the names of five political parties in an election; or they can offer something more complex, such as a choice of ways of keeping order in a classroom or a choice of motives for smoking cigarettes.
Open or free‑response questions are not followed by any kind of choice, and the answers have to be recorded in full. In the case of a written questionnaire, the amount of space or the number of lines provided for the answer will partly determine the length and fullness of the responses we obtain. Inevitably, some of this richness is lost when the answers are classified later, but it is useful to report a few such answers in full in the final report to give the reader some of the flavour of the replies. Statistical tabulations are important and must remain our first aim, but they make dull reading. Oppenheim, A.N (1992) Questionnaire design, interviewing, and attitude measurement: New York .(Page )
Both open-ended and close–ended questions may be included on a questionnaire.
The rules for writing good test items also apply to the construction of items to be included on questionnaires.
To defend your choice of response modes, the followings should be considered:
Who are the respondents? Are there any characteristics of the respondents that will inform or influence the choice of response modes? What are frequently used response modes for similar topics. How this inform your choice?  Why is it close but not open What are the possible choices? What are the advantages of using the selected over other types Do you use a combination of   negative or positive or just one of the two? Why?  

41. Open response types Questionnaire design Explain why you left school?
Open-ended questions
Open response types
Explain why you left school?
What were your reasons for leaving school?

42. Questionnaire design Open-ended questions - advantages
People can express their exact opinions and feelings
Do not limit the range of possible answers
Potentially produce responses which draw attentions to an unanticipated situation or outcome when constructing the questionnaire
Useful for testing hypotheses about ideas or awareness

43. Difficult and time consuming to analyse
Questionnaire design
Open-ended questions - Disadvantages
Difficult and time consuming to answer (require much effort from respondents)
Difficult and time consuming to analyse

44. Questionnaire design
Closed questions
Alternative answers are provided and respondents are asked to choose from a list of provided answers

45. Multiple choice questions Ranking Scales Scaling questions
Questionnaire design
Types of closed questions
Checklists
Two-way questions
Multiple choice questions
Ranking Scales
Scaling questions

46. Is used to verify the presence or absence of some phenomenon
Questionnaire design
Checklists
Is used to verify the presence or absence of some phenomenon

47. Which of these materials did you use?
Questionnaire design
Checklists
Example
Which of these materials did you use?
Which of these activities did you engage in?
Which of these are the steps of conducting the project?

48. Contains all the relevant options
Questionnaire design
What is a good checklist?
Contains all the relevant options
It is helpful to provide the option “other” for respondents to fill in at the end of a checklist.

49. Questionnaire design Two-way questions Measure a dichotomous variables
Respondents are asked to choose one from two alternatives:
Yes/No;
Agree/disagree;
For/Against;
Good/Bad;
Like/Dislike;
Approve/Disapprove;
Two way questions tend to reduce the issues, opinions in very simple terms and force respondents to choose one from two choices

50. Questionnaire design
MCQ questions
MCQ is useful when there are several possible responses and you want to ensure that the respondents is aware of all the possibilities.
Alternatives in MCQ should be mutually exclusive categories.

51. Questionnaire design
Ranking Scales
This format gives you an indication of how a respondent ranks a number of things.
It is useful when there are a limited number of things you would like to have ranked.

52. Questions with ratings on a latent scale
Questionnaire design
Scaling questions
Questions with ratings on a latent scale
The basic of attitude measurement is that there are underlying dimensions along which individual attitudes can be ranged.
That is a person can be assigned a numerical score to indicate his position on a dimension of interest. This will be done through combining the numerical score of all items of a scale interested.

53. More questions can be asked in a given length of time
Questionnaire design
Advantages of closed questions
Compared to open questions, this type of questions is quicker and easier to answer
More questions can be asked in a given length of time
Can deal with a large number of respondents
Low cost
Make group comparison easy
Avoid interviewer training

54. Loss of spontaneous responses
Questionnaire design
Disadvantages of closed questions
Loss of spontaneous responses
May introduce bias by forcing respondents to choose between given alternatives
May irritate respondents
Relatively difficult to design

55. Non-factual questions (Attitudes, stereotypes, beliefs, awareness)
Questionnaire design
Types of questions - Classification by types of information
Factual questions
Non-factual questions (Attitudes, stereotypes, beliefs, awareness)
Most questionnaires will employ a combination of questions asking about factual information and questions inquiring respondent attitudes.

56. Relatively easy to design
Questionnaire design
Factual questions
Can be verified
Single variable
Relatively easy to design

57. Relatively difficult to design
Questionnaire design
Non-factual questions
Difficult to verify
Latent variable
Relatively difficult to design

58. Questionnaire design
Issues to consider when writing the factual questions
Do the respondents have the necessary information to answer the question?–Knowledge, memory.
Will the respondents provide the information willingly? – Sensitive issues.

59. Questionnaire design Question Wordings Use Simple Words
“the catalogue system is too difficult for most readers to master “ vs
“I can never find the books I want” (more direct, more appealing)
Avoid acronyms, abbreviations, jargon and technical terms
Avoid ambiguous words or the words with many meanings
Have you ever assessed your colleagues’ teaching?
Avoid leading questions
You haven’t skipped any lessons in this semester, have you?

60. Avoid double-barrelled questions
Questionnaire design
Question wordings
Avoid double-barrelled questions
Do you buy weekly and monthly magazines / newspapers?
Avoid implicit assumptions
When did you last borrow a video tape?
Did your sibling’s decision to leave school influence your decision to leave school?
Don’t overtax the respondents’ memory.

61. Avoid proverbs or well-known sayings
Questionnaire design
Question Wordings
Avoid proverbs or well-known sayings
Avoid loaded words (heavily value laden terms)
Do you think union bosses should be allowed so much power?
Attitude statements are good if the respondents recognise the statements which force them to think

62. Questionnaire design Selection of types of questions
The number of respondents
The amount and types of information needed
The characteristics of respondents (knowledge, age, culture, religions)
The amount of time you have to process and interpret the data
Your knowledge of the issues (the extent to which you can anticipate the range of possible answers).
Your methods of data analysis

63. Preparing questionnaire layout Panelling, pre-testing and piloting

64. Allocating sufficient space for answers
Questionnaire design
Spacing
Allocating sufficient space for answers
Space requirements should be considered for :
Open-ended questions
Scaling questions
Coding
The amount of space you allocate to open-ended questions is important because it will determine both the length and the quality of the response. Only piloting the questionnaire will indicate the best space requirement.
In setting out questions that contain scales or indexes ensure that each is easily related to its corresponding question or statement.

65. Questionnaire design
Instructions
General
Section
Question

66. Questionnaire design General Instruction
Reason(s) for the questionnaire
A statement about anonymity
The sample design - to indicate how the respondent was chosen
How to return the questionnaire - if it is mailed
A contact person
What will happen to the results
Thanks

67. Questionnaire design Question instructions How to answer the questions
Make sure that the instructions and the questions correspond

68. Questionnaire design Order of the questions Very important
There is no correct order

69. Questionnaire design Suggestions
Begin with easy and non-threatening questions
Do not begin with open-ended questions
Arrange questions from general to specific
Group questions into sections or topics
Use filter questions to ensure that the respondents are answering relevant questions
Attitude statements are suggested to be arranged in more or less random order
Keep the questionnaire as short as possible

70. Consistency of questionnaire layout
Questionnaire design
Consistency of questionnaire layout
Try to use similar format for questions
Distinguish different instruction levels
A very important feature of questionnaire layout is consistency. Use similar formats for questions. Do not change the formats from section to section or question to question. If you begin using numbers which must be circled for close-ended questions, do not suddenly change to boxes which must be tickled. It may cause confusion.
Similarly, make sure you distinguish different instruction levels with similar font, point, size…

71. Questionnaire design Panelling and Reviewing
Relevance of questions to the topic (check against the table of specification)
Wording (Instructions, questions)
Layout
Spacing
Instructions
Order
Consistency

72. Questionnaire design Pre-testing Test questionnaire
Do the respondents understand the questions?
Are there any difficulties?
Are there any sensitive questions?
Is the question order appropriate
Does the researcher understand the respondent's response

73. Questionnaire design Who will be involved in Pre-testing
Very small sample of the population targeted

74. Questionnaire design How to conduct Pre-testing
Step 1: Brief the respondents about the questionnaire
Step 2: Researchers record the respondents’ process of completing the questionnaire:
through observation, video recording or audio recording to find out signs of difficulties or distractions and timing

75. Questionnaire design How to conduct Pre-testing
Step 3: Debrief the respondents about the questions in the questionnaire
Any difficulties? Why?
What are the easy questions? Why?
Any suggestions for improvement?
Step 4: Revise the questions if needed

76. Questionnaire design Pilot Test the whole process Questionnaire
Methods of administration and collecting the questionnaires
Response rate/missing data
Item analysis
Data analysis

77. Questionnaire design Analysis
Look at frequency of options in each question
Too many “uncertain”, “don’t know” responses, too many skipped or omitted items are bad signs in a pilot study.
Reliability of scale constructed
Decision of removing or replacing items of scales (for measuring latent variables)

78. Questionnaire design
Revise and prepare the final version