So Now You Want to Do a Survey... Words of Advice, Words of Caution Jay P. Paul, PhD UCSF Center for AIDS Prevention Studies Words of Advice, Words of Caution Jay P. Paul, PhD UCSF Center for AIDS Prevention Studies

Key Questions To Consider When Asking Questions  Is this item understandable to the participant?  Is this item something the participant can answer?  Is this item something the participant will be willing to answer honestly?  How important is this item compared to others I have to avoid respondent burden?  Is this item understandable to the participant?  Is this item something the participant can answer?  Is this item something the participant will be willing to answer honestly?  How important is this item compared to others I have to avoid respondent burden?

Areas to Consider in Designing Your Survey  What type of survey methods are out there and what can they do for me?  What are the things to consider in maximizing the quality of data with respect to questions I ask?  How do I consider using existing scales and putting it all together?  What type of survey methods are out there and what can they do for me?  What are the things to consider in maximizing the quality of data with respect to questions I ask?  How do I consider using existing scales and putting it all together?

Modes of Survey Delivery

Survey Method Options  IAQ  SAQ  Including Diary Methods, Mail Surveys  CASI/CAPI (and A-CASI)  CATI  T-ACASI  Internet  IAQ  SAQ  Including Diary Methods, Mail Surveys  CASI/CAPI (and A-CASI)  CATI  T-ACASI  Internet

Interviewer-Administered Questionnaire (IAQ)  Most flexible and responsive to respondent re: comprehension of question, selection of response options.  Able to handle skip patterns that could confuse the respondent in a self-administered questionnaire.  Human element, rapport & responsiveness also can introduce problems in lack of standardization of delivery, self-presentation.  Most flexible and responsive to respondent re: comprehension of question, selection of response options.  Able to handle skip patterns that could confuse the respondent in a self-administered questionnaire.  Human element, rapport & responsiveness also can introduce problems in lack of standardization of delivery, self-presentation.

Self-Administered Questionnaire (SAQ)  Allows for greater privacy than face-to-face interviewing  Tends to be associated with higher rates of reporting of sensitive behaviors  Literacy requirements for respondent  Cannot be adjusted to be responsive to needs of individual participant  Cannot handle complex “skip patterns”  Low-cost survey delivery option  Allows for greater privacy than face-to-face interviewing  Tends to be associated with higher rates of reporting of sensitive behaviors  Literacy requirements for respondent  Cannot be adjusted to be responsive to needs of individual participant  Cannot handle complex “skip patterns”  Low-cost survey delivery option

Diary Methods  Provides optimal potential for recording of accurate rates of target behaviors  Recall  Exploration of variations over time  Allows for question design to examine co-occurrence of key behaviors (e.g., sexual behavior and drug use)  Burden of consistent completion of information means that data may be missing in possibly non- random pattern, or completed at a later time than designed.  NOTE: Can be useful to design for Web-based implementation.  Provides optimal potential for recording of accurate rates of target behaviors  Recall  Exploration of variations over time  Allows for question design to examine co-occurrence of key behaviors (e.g., sexual behavior and drug use)  Burden of consistent completion of information means that data may be missing in possibly non- random pattern, or completed at a later time than designed.  NOTE: Can be useful to design for Web-based implementation.

Mail Surveys  Many of considerations of SAQ are applicable -- with format even more crucial.  Many respondents report liking the flexibility of completing surveys at their own pace.  Need for clear and easy format to maintain respondent interest and ease of completion.  Critical to have contact information and clear procedures to follow up on respondents to ensure adequate response rates.  Many of considerations of SAQ are applicable -- with format even more crucial.  Many respondents report liking the flexibility of completing surveys at their own pace.  Need for clear and easy format to maintain respondent interest and ease of completion.  Critical to have contact information and clear procedures to follow up on respondents to ensure adequate response rates.

Computer-Assisted Self-Interview /Personal Interview (CASI/CAPI)  Survey is delivered via computer (can be laptops for maximal flexibility), with respondent keying in responses.  Usual mode is “Audio-CASI” or A-CASI, which involves respondent listening to audio recording of survey as they see items on- screen.  For special PCs with touch screens, can avoid keyboard and use fingers to select choice by touching on-screen.  Survey is delivered via computer (can be laptops for maximal flexibility), with respondent keying in responses.  Usual mode is “Audio-CASI” or A-CASI, which involves respondent listening to audio recording of survey as they see items on- screen.  For special PCs with touch screens, can avoid keyboard and use fingers to select choice by touching on-screen.

CASI/A-CASI Advantages  Privacy  Branching in complex questionnaires  Automated consistency checks  Automated range checks  Automated adaptation of question wording based upon prior responses  Audio-CASI allows for standardized verbal delivery of all questions and does not require respondent literacy  Privacy  Branching in complex questionnaires  Automated consistency checks  Automated range checks  Automated adaptation of question wording based upon prior responses  Audio-CASI allows for standardized verbal delivery of all questions and does not require respondent literacy

Computer-Assisted Telephone Interview (CATI)  List samples of “live” residential phone lines are called to screen households and determine eligibility of members.  Once consent is obtained, interviewer uses a computer to guide them through asking survey questions, providing prompts, entering data, etc. as in A-CASI.  Interviewer also has some flexibility in dealing with respondent comprehension/literacy level.  List samples of “live” residential phone lines are called to screen households and determine eligibility of members.  Once consent is obtained, interviewer uses a computer to guide them through asking survey questions, providing prompts, entering data, etc. as in A-CASI.  Interviewer also has some flexibility in dealing with respondent comprehension/literacy level.

CATI Advantages  Opportunity to access representative sample (costly with eligibility requirements that are less common).  Some advantages of live interviewer, but also anonymity of phone call.  Has capacities of computer-assisted technology with respect to branching, adaptation of wording, consistency/range checks, etc.  Opportunity to access representative sample (costly with eligibility requirements that are less common).  Some advantages of live interviewer, but also anonymity of phone call.  Has capacities of computer-assisted technology with respect to branching, adaptation of wording, consistency/range checks, etc.

Telephone Audio Computer- Assisted Self-Interview (T-ACASI)  First a telephone interviewer contacts a household, screens for an eligible respondent, and secures consent.  Phone call then transferred to automated system where computer-controlled, pre- recorded questions are read aloud.  Respondent provides answers by pressing keys on a touch-tone phone.  At end, respondent can be returned to live person to close out interview.  First a telephone interviewer contacts a household, screens for an eligible respondent, and secures consent.  Phone call then transferred to automated system where computer-controlled, pre- recorded questions are read aloud.  Respondent provides answers by pressing keys on a touch-tone phone.  At end, respondent can be returned to live person to close out interview.

Internet Surveys  Access to large sample at relatively low cost in relatively short time frame.  May provide access to difficult-to-reach populations.  As with other computer-assisted methods, may result in higher rates of reporting of sensitive information, and provides consistency and validity checks.  Rates of internet access are growing (if still disparities exist in different populations).  Participation rates of target population hard to gauge  Dropout rates  Direct data entry into database facilitates cleaning and analysis.  Access to large sample at relatively low cost in relatively short time frame.  May provide access to difficult-to-reach populations.  As with other computer-assisted methods, may result in higher rates of reporting of sensitive information, and provides consistency and validity checks.  Rates of internet access are growing (if still disparities exist in different populations).  Participation rates of target population hard to gauge  Dropout rates  Direct data entry into database facilitates cleaning and analysis.

Summary: Survey Delivery Considerations  Mode of survey delivery clearly influence:  Respondent comfort and likelihood of self- disclosure (due to level of privacy)  Respondent ease of navigating through the survey if there are any skip patterns  Time and effort expended per respondent  The audience you can reach with your survey  Quality of data, data entry and analysis  While important, how you actually ask the questions is just as critical -- our next focus.  Mode of survey delivery clearly influence:  Respondent comfort and likelihood of self- disclosure (due to level of privacy)  Respondent ease of navigating through the survey if there are any skip patterns  Time and effort expended per respondent  The audience you can reach with your survey  Quality of data, data entry and analysis  While important, how you actually ask the questions is just as critical -- our next focus.

Survey Construction

Developing the Survey: Question Organization  Topics should be generally organized from least to most intimate  Initial questions are crucial -- respondent must get “hooked” by at least some questions which are felt to be personally salient  Initial questions must also require low effort (cognitive demands) to answer  Thus, if possible, avoid demographic questions to start (especially income!)  Topics should be generally organized from least to most intimate  Initial questions are crucial -- respondent must get “hooked” by at least some questions which are felt to be personally salient  Initial questions must also require low effort (cognitive demands) to answer  Thus, if possible, avoid demographic questions to start (especially income!)

Developing the Survey: Question Organization  A survey -- no matter how delivered -- is a type of conversation.  Topics should follow an orderly sequence, with common questions grouped together.  This flow allows transitions to more intimate matters without jarring or acute awareness.  Transitional phrases can help in shifting set.  All this also provides respondent with notion that there is a rationale to questions asked.  A survey -- no matter how delivered -- is a type of conversation.  Topics should follow an orderly sequence, with common questions grouped together.  This flow allows transitions to more intimate matters without jarring or acute awareness.  Transitional phrases can help in shifting set.  All this also provides respondent with notion that there is a rationale to questions asked.

Question Ordering  Be careful with filter questions: if not worded carefully, respondents will learn that a “no” response means avoiding subsequent questions.  Be careful about summary attitudinal questions: more specific queries to begin will influence summary responses by forcing respondent to think about their overall beliefs more carefully.  Be careful with filter questions: if not worded carefully, respondents will learn that a “no” response means avoiding subsequent questions.  Be careful about summary attitudinal questions: more specific queries to begin will influence summary responses by forcing respondent to think about their overall beliefs more carefully.

Question Ordering  Having similar response choices reduces cognitive burden on respondent; grouping items with such options together is thus useful. It can also be useful to have the same number of options to reduce complexity (e.g., always using a four-point or five-point scale).  However, arrange types of questions to provide some variety and reduce response set.  Avoid asking unnecessary questions!!  Having similar response choices reduces cognitive burden on respondent; grouping items with such options together is thus useful. It can also be useful to have the same number of options to reduce complexity (e.g., always using a four-point or five-point scale).  However, arrange types of questions to provide some variety and reduce response set.  Avoid asking unnecessary questions!!

Question Wording/Construction

Comprehension/Interpretation  Not just question of literacy, but influenced by cognitive burden and attention required  Wording choices  Length and complexity of survey items  Ordering of clauses  Consistency of phrasing  Avoid “not’s” or items whose meaning will be lost by inattention to a single word  Phrasing also influences response distribution along a Likert-type agree-disagree scale  Not just question of literacy, but influenced by cognitive burden and attention required  Wording choices  Length and complexity of survey items  Ordering of clauses  Consistency of phrasing  Avoid “not’s” or items whose meaning will be lost by inattention to a single word  Phrasing also influences response distribution along a Likert-type agree-disagree scale

Cultural Considerations  Individuals can come up with very different understandings of survey items, based upon their personal experiences.  The meanings that respondents bring to the survey experience can be strongly influenced by culture, acculturation and other background variables -- not just education and economic status.  These differences are highlighted in the vast literature that exists on different response patterns of different ethnic groups to the same standardized scales. (For a review of some issues, see Pasick, Stewart, Bird & D’Onofrio, 2001.)  Individuals can come up with very different understandings of survey items, based upon their personal experiences.  The meanings that respondents bring to the survey experience can be strongly influenced by culture, acculturation and other background variables -- not just education and economic status.  These differences are highlighted in the vast literature that exists on different response patterns of different ethnic groups to the same standardized scales. (For a review of some issues, see Pasick, Stewart, Bird & D’Onofrio, 2001.)

Key Considerations  Examine all survey items with eye for their appropriateness for target population.  Be aware of pre-existing measures so as to not have to reinvent the wheel.  When using pre-existing scales, review in same way for clarity & ease of response.  Get feedback on item wording and construction from as many colleagues as possible.  Pre-test survey, using methods first described by Cannell et al. (1989).  Examine all survey items with eye for their appropriateness for target population.  Be aware of pre-existing measures so as to not have to reinvent the wheel.  When using pre-existing scales, review in same way for clarity & ease of response.  Get feedback on item wording and construction from as many colleagues as possible.  Pre-test survey, using methods first described by Cannell et al. (1989). Cannell C, Oksenberg I, Kalton G, Bischoping K & Fowler F. (1989). New Techniques in Pretesting Survey Questions (NCHSR #HS 05616). Survey Research Center, University of Michigan.

Cognitive Burden & Recall Demand  Recall accuracy is influenced by a variety of factors, including:  salience/vividness of behavior,  time frame,  complexity of response information necessary (ever, counts, close-ended categories, etc.),  frequency and consistency of behavior,  effort respondent is willing to exert,  cueing techniques and prompts  Recall accuracy is influenced by a variety of factors, including:  salience/vividness of behavior,  time frame,  complexity of response information necessary (ever, counts, close-ended categories, etc.),  frequency and consistency of behavior,  effort respondent is willing to exert,  cueing techniques and prompts

Time Frame - Sexual/Drug Use Behaviors  3 month time frame generally considered the outside range of accurate reporting; some suggest limiting to 1 month.  Longer time frame may be possible if asking if a behavior ever happened.  One important timeframe consideration is dealing with infrequent behaviors -- what do you want to capture with respect to either behavior events or possible behavior change (if longitudinal study)?  3 month time frame generally considered the outside range of accurate reporting; some suggest limiting to 1 month.  Longer time frame may be possible if asking if a behavior ever happened.  One important timeframe consideration is dealing with infrequent behaviors -- what do you want to capture with respect to either behavior events or possible behavior change (if longitudinal study)?

Sensitivity  Item sensitivity not consistent across populations (e.g., sexual behavior among MSM vs. general population, drug use behavior among heavy drug abusers/IDUs, discussing sexuality among different ethnic groups).  Rationale for sensitive questions helps response rates.  Asking if “ever” occurred first, prior to asking about (for example) the last month, may be less charged for respondent.  Importance of neutral wording of question and (if IAQ) careful training of interviewer  Item sensitivity not consistent across populations (e.g., sexual behavior among MSM vs. general population, drug use behavior among heavy drug abusers/IDUs, discussing sexuality among different ethnic groups).  Rationale for sensitive questions helps response rates.  Asking if “ever” occurred first, prior to asking about (for example) the last month, may be less charged for respondent.  Importance of neutral wording of question and (if IAQ) careful training of interviewer

In Closing...  Selecting your methods of data collection must be carefully considered based upon target population, study questions and design, and costs.  No matter what the method of data collection, your instrument is critical.  Survey design is a combination of art and science, with the “art” strongly reliant upon interviewing skills.  The science of survey design currently has a wealth of research literature to examine and guide you.  Selecting your methods of data collection must be carefully considered based upon target population, study questions and design, and costs.  No matter what the method of data collection, your instrument is critical.  Survey design is a combination of art and science, with the “art” strongly reliant upon interviewing skills.  The science of survey design currently has a wealth of research literature to examine and guide you.