1. Goodness of Measures Reliability A matter of whether a particular technique, applied repeatedly to the same object, yields the same result each time. How stable and consistent is the measuring instrument? Validity The extent to which an empirical measure adequately reflects the real meaning of the concept under consideration. Are we measuring the right thing?

2. Reliability Stability refers to the ability of a measure to maintain consistency over time, despite uncontrollable testing conditions or the state of the respondents themselves Internal consistency indicates how well the items ‘ hang together as a set ’ and can independently measure the same concept, so respondents attach the same overall meaning to each of the items

3. Forms of validity Face validity That quality of an indicator that makes it seem a reasonable measure of a variable. Criterion related validity The degree to which a measure relates to some external criterion. For example, the validity of the VCE tests is shown in their ability to predict the college success of students.

4. Forms of validity (cont.) Construct validity The degree to which a measure relates to other variables as expected within a system of theoretical relationships. Content validity Refers to how much a measure covers the range of meanings included within a concept.

5. The Nature of Questionnaire Surveys (Veal) Data collected via a printed questionnaire or interview schedule Usually based on a sample selected from a population (see Chapter 11) Reliance on information from individuals – self-reported data Often large samples require computer-aided analysis

6. Merits of the Questionnaire Survey Method Quantified data for decision making Ideal for providing quantified information for organisations which need quantified information for decision-making. Transparency Provide a 'transparent' set of research procedures. Procedures clear for all to see. Data can be re-analysed for alternative interpretation.

7. Merits of the Questionnaire Survey Method (continued) Succinct presentation Quantification can provide complex information in a succinct form. Comparability Longitudinal and annual surveys enable the study of change over time. Capturing complexity An effective means of gathering a wide range of complex information on individuals or organisations.

8. Why do people answer questionnaires? Social exchange theory (Blau, 1964, Dillman 1978) Likelihood of response is a function of: Rewards: positive regard, verbal appreciation, helping, group values, interest, social validation Costs: subordination, embarrassment, inconvenience, complexity, privacy Trust: tokens of appreciation, sponsorship, importance, other relationships

9. When to use questionnaires GOOD FOR Descriptive research Explanatory research BUT NOT FOR Exploratory research Consider MULTIMETHOD approaches

10. Types of questionnaire (Veal, Figure 8.2) Self-administered Mail surveys - questionnaires sent and returned by mail; E-surveys - questionnaires sent and returned by email/Internet; Captive group surveys - members of groups are surveyed Delivery and collection Organisation surveys - members of an organisation/ organisations are surveyed Interviewer-administered Household surveys – respondents selected on the basis of where they live and interviewed in their home Street/quota/intercept surveys - respondents selected by stopping in street, shopping malls, etc Telephone surveys - interviews are conducted by telephone Customer/visitor on-site intercept surveys – users/customers surveyed on site

11. Household surveys Advantages Rapport and motivation for resp. Clarification possible Read non-verbal cues Visual aids possible Richer data CAPI available Disadvantages Take personal time Costs are higher Confidentiality concerns Interviewer training vital Interviewer biases possible

12. Telephone Interviews Less costly and faster than mail or personal Wide geographical area Higher anonymity than personal Can be completed using CATI - Computer-aided Telephone Interviewing Automatic sampling and dialling Answers keyed directly into computer (quick, cost- effective) Respondents can terminate at any time Non-verbal cues not available Need to be kept short (AMSRS - 20 min.) Representativeness of telephone sampling lists?

13. Mail Questionnaires High anonymity Wide geographic regions Can include token gifts At respondent convenience Relatively low response rate No opportunity for clarification Follow-up procedures required Total design method (Dillman, 1978)

14. Mail survey response rates Response rate affected by: Interest of the respondent in the survey topic Length of the questionnaire Questionnaire design, presentation, perceived complexity Style, content and authorship of accompanying letter Provision of a reply-paid envelope Rewards for responding Number/timing of reminders/follow-ups – see Fig. 8.3

15. Electronic Questionnaires Easy to administer Global reach Inexpensive Fast At respondent’s convenience Require computer literacy Respondents must have computer access Must be willing to complete

16. Questionnaire design (Veal) Topics: Research problems and information requirements Types of information Open-ended and pre-coded questions Wording of questions Measuring attitudes and opinions Ordering of questions Layout Filters Introductory remarks

17. Questionnaire design process

18. Type of Questions Open-ended questions Freedom to answer the question any way one wants Could pose problems for researcher in coding the response Closed questions Alternatives that are mutually exclusive and collectively exhaustive, given Respondent can make a quick decision Easy to intepret and code

19. Open-ended and pre-coded questions

20. Wording of questions Avoid jargon Simplify wherever possible: Would the respondent understand the words in the questionnaire? Avoid ambiguity Avoid leading questions Ask only one question at a time (avoid multi-purpose questions) - Are there double-barrelled questions? Are there any recall-dependent questions? Is there any potential of social desirability?

21. Validity of questionnaire surveys Validity: the extent to which the data truly reflect what they are thought to reflect Validity may be affected by: Exaggeration or under-statement (to give good impression) Halo effects Inaccuracy of recall (recency bias) Desire to please the interviewer Order effects Some checks: Include ‘dummy’ categories in some questions– eg. books that do not exist, events that have not happened As the same question twice – in different way Repeat interviews – some time later

22. Possible sources of questions ADOPT questions used in other questionnaires Generally already tested for validity Can compare consistency of responses (i.e. reliability) ADAPT questions used in other questionnaires DESIGN your own questions

23. Forms of closed questions (SLT p.292) List (tick all that apply) Category (which one applies to you?) Ranking (relative importance) Rating (agree-disagree, good-bad, …) Quantity (how many – record a number) Grid (multiple question responses)

24. Measuring attitudes and opinions Formats a. Open-ended or direct questions b. Checklist c. Ranking d. Likert scales e. Attitude statements f. Semantic differential

25. Classification Data – Personal Information or Demographic Data Decisions on: Name, number required? (anonymity) What info required? (income, marital status, etc needed?) Age, income, etc, should a range be given? If so, what are the appropriate ranges? Eg, 40 Should personal info be at beginning or end of questionnaire? Decisions should be based on the research questions you need to answer!!

26. Ordering of questions Start with easy questions Start with 'relevant' questions Leave sensitive questions until later

27. Sequencing of Questions The Funnel Approach From general to specific questions From easy to difficult questions

28. Questionnaire layout Be aware of reader/user – interviewer or respondent? Special care with mail survey questionnaires Compactness (eg. single page) = ease of handling Two-column layout often helps

29. Filtering

30. Introductory remarks Eg. ‘Hallo, my name is _____ : we are conducting a survey of _______: would you mind answering a few questions? Different content/formats for different survey types Interviewer-completion: include in interviewer instructions – additional information available if required Respondent-completion – printed on questionnaire Mail survey: this is dealt with in covering letter

31. Introduction to respondents Consider social exhange – increase benefits and trust, reduce costs identity of researcher and purpose of survey confidentiality & anonymity distribution and/or publication of results courteous note, thanks Aim is to provide opportunity for INFORMED CONSENT Options: covering letter, introductory statement

32. Instructions Define a desired navigational path for each page Create visual navigational guides to assist respondents to adhere to navigational path Develop additional visual guides for disruption where redirection is required (Dillman, 2000: 96)

33. Pre-testing Questionnaires Face validity Content validity Pilot study With sample of respondents from target population Ideal research design Factor analysis, reliability analysis