1. Lecture 4 - Survey design Sampling Sample size/precision Data collection issues Sources of bias

2. Why do surveys? Information on particular population –prevalence of a disease –behaviour, knowledge, attitude Planning of services Collect information on data not routinely available: –e.g., mental health status, health behaviours Repeat surveys to monitor trends (serial cross- sectional studies)

3. Bias and precision of the survey estimates Bias: –selection bias relates to sample selection –information bias relates to information collected Precision –relates to sample size

4. Reasons to sample Reduce cost Increase accuracy and quality of data collected

5. Definitions Sampling unit –person or group (e.g., household) Sampling frame –list of sampling units in the population censuses electoral lists telephone lists are institutional populations excluded (e.g., prisons, nursing homes)

6. Target and study population Target population: –population for generalization of results Study population: –population for collection of data –may be total target population or a sample

7. Types of sample Non-representative –convenience –volunteers Representative –simple random –systematic –cluster –multistage

8. Simple random sample Each sampling unit in the population has equal probability of being included Sampling with replacement: –each unit placed back in pool Sampling without replacement (usual method): –each unit selected is kept out of pool

9. Simple random sample (cont’d) Methods: –manual –tables of random numbers –computer-generated random numbers

10. Systematic sample Select every nth individual from a list –can use existing numbers –e.g., patient appointments, medical records Advantages: –Does not require complete sampling frame –Simple to carry out Disadvantages: –May be unsuitable for cyclic or ordered data (e.g., every 5th patient when only 5/day)

11. Stratified sampling Separate sample selected from different strata of population Requires separate sampling frame for each stratum Useful if there are small but important subgroups of the population (e.g., very old, very young, institutionalized, sick)

12. Cluster sampling Sampling frame comprises groups (households, villages, schools) Step 1: Simple random sample of groups Step2: All individuals in each group included in survey Advantages: –enumeration of population not needed –more efficient use of resources

13. Multistage sampling Larger units sampled in first stage, smaller units later e.g.: –stage 1 - sample of towns –stage 2 - sample of city blocks or census tracts –stage 3 - sample of households

14. Sampling for “hidden populations ” Homosexual men: –gay bars, newspapers Injection drug users: –convenience sample (e.g., treatment facilities) –snowball sampling (through networks) Capture-recapture methods –identify biases of sampling method

15. Planning a survey Define target population Select method of sampling –sampling unit, sampling frame, etc Calculate sample size Define survey data collection methods Non-respondents –number of attempts to reach –different days, times

16. Sample size estimations Requirements: –level of precision (width of confidence interval) –expected variability (estimated from previous studies, pilot study, or literature)

17. Design of questionnaires List study variables Collect existing questions and instruments Adapt and/or develop new questions Format questionaire Pre-testing (timing, responses, clarity, etc.) Revise, determine priorities, shorten

18. Question wording: clarity Use concrete rather than abstract terms, e.g., –During a typical week, how many hours do you spend doing vigorous exercise? –Not: How much exercise do you get? Avoid jargon, technical terms, slang Avoid double-negatives (Do you disagree that doctors should not make house calls?) Use active vs passive voice (Has a doctor ever told you vs Have you ever been told by a doctor?)

19. Question wording: clarity –Break long sentences into short ones (20 word or fewer) –Use good grammar but use informal style –Avoid hypothetical questions –Evaluate reading level (normally not more than 8th grade)

20. Question wording: neutrality Do not suggest desirable response, e.g.: –Not: do you ever drink alcohol? –Better: how often do you drink alcohol? Give permission to give undesirable response e.g.: –Sometimes people forget to take medications their doctor prescribes. Do you ever forget (or how often do you forget) to take your medications?

21. Question wording Introduce attitude questions, e.g.: –People have different opinions about their medical care. We are interested in your opinion. Avoid double-barreled questions –How much coffee or tea do you drink each day? Avoid assumptions –How much help do you get from your family?

22. Response wording Make them short Use as few options as possible Consider different types of non-response: –refuse –don’t know –no opinion –not applicable –omission by subject or interviewer

23. Response wording Make sure responses are mutually exclusive (or give instructions to “check all that apply”) Consider use of response card for multiple questions with same set of responses

24. Organization of questionnaire Group questions by subject matter Introduce each group with short descriptive statement (e.g., now I am going to ask you some questions about your use of health services) Begin with more emotionally neutral questions More sensitive questions (e.g., income, sexual function) near end of questionnaire

25. Organization of questionnaire interviewer-administered: repeat time frame fairly frequently self-administered: repeat time frame at top of each page or each set of questions, e.g.: During the past year, how many times have you: –Visited a doctor? –Been a patient in an emergency department? –Been admitted to hospital?

26. Organization of questionnaires Group questions with similar response scale Format skip patterns –screener questions –branching questions Time frame –group questions that ask about same time frame –“usual” behavior vs specified time period –assist respondent with milestones to help define reference time frame

27. Questionnaire mode Face-to-face Telephone Mail Other: –diaries Mixed mode

28. Face-to-face interviews: advantages reduce items with no response easier for older, less educated, lack of fluency in language some formats easier to administer: –skip patterns to avoid irrelevant questions –open-ended questions - can probe for more complete response

29. Face-to-face interviews: disadvantages cost time effort (interviewer training, evaluation of inter-rater reliability) interviewer biases differences in sociodemographic characteristics of interviewer and subject

30. Telephone interviews: advantages less expensive than face-to-face reduce items with non-response some formats easier to administer: –skip patterns to avoid irrelevant questions –open-ended questions - can probe for more complete response large, representative samples can be organized from one office avoids bias associated with appearance of interviewer

31. Telephone interviews: disadvantages misses households without telephone misses those with unlisted ‘phone numbers bias when calls made during day multiple calls may be needed perceived as intrusive by some difficult to administer items with multiple response options

32. Mailed questionnaires: advantages least expensive can be coordinated from one office social desirability minimized inconsistent results on completeness of reporting (e.g., for # MD visits)

33. Mailed questionnaires: disadvantages relatively low response rates –multiple mailings, cover letter, letterhead, advance warning, token of appreciation, SSAE difficult to get information on non-respondents –differences between early and late responders items may be omitted: 5-10% may be unusable cannot control order of questions postal strikes

34. Analysis of surveys Missing data –exclude –imputation: e.g., based on characteristics of respondents –sensitivity of estimate to method of imputation Weighting of estimates –for stratified samples

35. Analysis of surveys (cont’d) Crude estimates, confidence intervals –Continuous data: Mean, median, quartile –Categorical data: proportion –Confidence intervals to describe precision

36. Bias and precision of the survey estimates Bias: –selection bias relates to sample selection –information bias relates to information collected Precision –relates to sample size

37. Selection bias in surveys Does the final analysis sample represent the original target population? Sources of bias: –sampling method –non-response –missing data

38. Information bias in surveys Bias in measurement of outcomes Sources of information bias: –non-validated measurement instrument –unblinded or poorly trained data collectors –response set –etc.