Introduction to Survey Research

Introduction to Survey Research
Pathways to Careers in Clinical and Translational Research (PACCTR) Curriculum Core

Introduction to Survey Research
Harry Truman displays a copy of the Chicago Daily Tribune newspaper that erroneously reported the election of Thomas Dewey in Truman's narrow victory embarrassed pollsters, members of his own party, and the press who had predicted a Dewey landslide.

Overview of Sections One: Basics of Survey Research
Two: Measurement Levels and Types of Data Three: Elements of Item Construction Four: Assessing Reliability and Validity Five: Determining the Coding System Six: Deciding How to Enter and Store the Data Seven: Data Entry, Data Checking and Quality Control Resources

Basics of Survey Research
Section One Basics of Survey Research

Section 1 Basics of Survey Research
Objectives Describe what surveys are used to measure Give examples of how surveys can assess change Define "needs", "assets", "behavior", "opinions", "attitudes", "beliefs" Key Terms: Survey, questionnaire, descriptive research, causal explanation, prediction, evaluation, single sample, successive samples, panel study

What do surveys measure?
Behavior, attitudes and beliefs Repeated observations. Example: if a researcher were interested in exercise patterns in obese patients, we could follow them every time they perform any physical activity to determine frequency, duration and intensity. Similarly, if we were interested in dietary habits, we could sit at their table in the morning and record what they ate. This is not feasible and there are many behaviors that cannot be directly observed such as attitudes, beliefs and opinions.

What do surveys of people measure?
So, what are researchers to do?

Do a Survey (Questionnaires)
Surveys are a systematic way of asking people to volunteer information about their attitudes, behaviors, opinions and beliefs. The success of survey research rests on how closely the answers that people give to survey questions matches reality that is, how people really think and act. The first problem that a survey researcher has to tackle is how to design the survey so that it gets the right information. Is this survey necessary? Is the purpose of the survey to evaluate people or programs? Can the data be obtained by other means? What level of detail is required? The second problem is how accurate does the survey have to be? Is this a one-time survey or can the researcher repeat the survey on different occasions and in different settings? How will the results be used? How easy is it to do the survey?

Why do a survey? The survey is an appropriate means of gathering information under three conditions: when the goals of the research call for quantitative and qualitative data, when the information sought is specific and familiar to the respondents and; the researcher has prior knowledge of the responses likely to emerge.

Use of Surveys by study design
Descriptive research Describe phenomena and summarize them. The goal of using surveys for descriptive research is to get a precise measurement of specific concepts i.e. depression, quality of life. Causal explanation Measure associations i.e. chemotherapy and quality of life. The data from surveys can provide a causal explanation to phenomena such as why teens become pregnant or why teens do drugs. Evaluation Efficacy of a program Prediction Predict future events

Basic Survey Designs Cross-sectional surveys: Longitudinal Surveys:
Data collected at one point in time selected to represent a larger population Longitudinal Surveys: Trend: Surveys of sample population at different time points Cohort: Study of sample population each time data are collected but samples studied maybe different Panel: Data collection at various time points with the same sample of respondents

Measurement Levels and Types of Data
Section Two Measurement Levels and Types of Data

Section Two Measurement Levels and Types of Data
OBJECTIVES Define and give examples of levels of measurement Define and give examples of scales Contrast the types of measurement scales Compare and contrast interval versus ordinal scales Key Terms Categorical, ordinal, scale, attitudes, opinions, beliefs, behavior, attributes

Selecting the types of data to collect
The type of data you collect and how you collect them affects how you can use the data. Therefore, before you start data collection you must decide what will be measured and how it will be measured.

Types of data Categorical data: numbers or words are used to group things Examples: gender, race, religion, food group, or place of residence. Ordinal data: When the numbers are used to order a list of things The ranking of football or basketball teams is done using ordinal numbers. A list of things to do would also be ordinal data. Interval data. Scale or interval data would be things like height, weight, age.

Types of data collected
There are six basic types of data that you might collect: Attitudes Opinions Beliefs Behavior Attributes (demographic characteristics) Preferences

Basic types of survey questions
The way a question or statement is worded and the response options offered determine the nature of the data received. Types of survey questions include: Open-ended response Closed response Semantic differential scales Agreement and rating scales Ranking scales Checklists

Open-ended question Respondent writes response in own words Considerations for using open-ended questions: Need to enter data by hand Develop a coding scheme for responses Content analysis? Frequently used in exploratory studies to facilitate better understanding of a concept Suggestion: it’s a good idea to always include an open-ended question giving the respondent the opportunity to add any additional comments.

Open-ended question Advantages: Allows the respondent to answer the question with few limitations Report more information than with discrete answers Disadvantage: Need qualitative methods or coding system to analyze the responses Require subjective judgements Example: What habits increase a person’s risk for being overweight? Describe the pain you experience with walking?

Closed response: These are the "multiple-choice" variety where a person has to choose among several possible answers. There are two types of closed response questions: ordered answer choices represent points along a continuum. Pain on a scale of 0 (none) to 10 (worst pain ever) unordered answer choices with each choice is an independent answer. Examples: ethnicity and marital status

Closed response: Advantages: Quicker and easier to answer Easier to tabulate and analyze List of possible responses helps participant understand the meaning of the question Suitable to multi-item scales designed to provide a single score Disadvantages: Do not allow participants to express their own answers Set of answers may not be exhaustive Must be clear about selection of items, one or as manay as applicable

Closed response: When question allows more than one answer it does not force respondent to consider each answer- Not ideal. Better to use yes no and multiple items Example: Which of the following do you believe increases your risk for stroke? Yes No Don’t know Smoking Overweight Stress Drinking alcohol Hulley et al., (2007) Designing Clinical Research, 3rd Edition pg. 242

Partially open response Require specific, short answers that do not encourage free expression. are a compromise between closed response and open response forms. provide an “Other” category where a person can provide additional information. Example: blank spaces provided for the questions on racial background and persons living with you.

Semantic differential questions: use a five to seven-point rating scale with each end of the scale having an adjective or phrase. These adjectives, called bipolar adjectives, are direct opposites. Semantic differential scales have three common factors: an Evaluative factor covering such dimensions as good-bad, pleasant-unpleasant and positive-negative; a Potency factor representing the dimensions of strong-weak, hard-soft and heavy-light, and an Activity factor with such scales as fast-slow, active-passive and excitable-calm.

Agreement scales End points are identified by adjectives or phrases all of the steps may have an adjective or phrase associated with it Example: a five point scale with steps labeled Strongly Agree, Agree, Neutral, Disagree, and Strongly Disagree.

Modes of Survey Administration
Personal (face-to-face) Telephone Mail Web Combination of methods

How do you decide on the mode of data collection?
Population + Characteristics Of The Sample Types of Questions Question Topic Response Rate $$ Cost $$ Time

Modes of Survey Administration Personal (face-to-face)
ADVANTAGES: Generally yields highest cooperation and lowest refusal rates Allows for longer, more complex interviews High response quality Takes advantage of interviewer presence Multi-method data collection DISADVANTAGES: Most costly mode of administration Longer data collection period Interviewer concerns

Modes of Survey Administration Telephone
ADVANTAGES: Less expensive than personal interviews RDD samples of general population Shorter data collection period than personal interviews Interviewer administration (vs. mail) Better control and supervision of interviewers (vs. personal) Better response rate than mail for list samples DISADVANTAGES: Biased against households without telephones, unlisted numbers Nonresponse Questionnaire constraints Difficult to administer questionnaires on sensitive or complex topics

Modes of Survey Administration Mail
ADVANTAGES: Generally lowest cost Can be administered by smaller team of people (no field staff) Access to otherwise difficult to locate, busy populations Respondents can look up information or consult with others DISADVANTAGES: Most difficult to obtain cooperation No interviewer involved in collection of data Need good sample More likely to need an incentive for respondents Slower data collection period than telephone

Comparison of Modes of Survey Administration
Variable Mail Phone F/F Cost Cheapest Moderate Costly Speed Fast Slow Response rate Low to moderate High Sampling need Address Telephone number Burden on respondent Low Control participation Of others Unknown Length of Questionnaire Short Long Sensitive questions Best Poor Lengthy answer choices Good Open-ended responses Complexity of Questionnaire Possibility of interviewer bias None

Modes of Survey Administration WEB SURVEYS
ADVANTAGES: Lower cost (no paper, postage, mailing, data entry costs) Can reach international populations Time required for implementation reduced Complex skip patterns can be programmed Sample size can be greater DISADVANTAGES: Approximately 40% of homes own a computer; 30% have home Representative samples difficult - cannot generate random samples of general population Differences in capabilities of people's computers and software for accessing Web surveys Different ISPs/line speeds limits extent of graphics that can be used

Elements of Item Construction
Section Three Elements of Item Construction

Section Three Elements of Item Construction
OBJECTIVES Identify types of bias in questionnaires Identify and state how to correct common wording problems Define types of response sets Key Terms Social desirability, acquiescence, bias, midpoint, operational definition, response set

How Bias affects response?
Need we say more?

Writing questions Good Questions Are clear and use simple language
Are concise Are specific Are possible to answer Are relevant to the respondent Do not use negatives Avoid bias terms Have only one part (not two parted question)

Writing questions Common Wording Problems
Writing questions for a particular survey means doing them for a particular population, a particular purpose and for placement next to other questions in the survey. Words that are too difficult for some to understand may be perfectly acceptable for others. A question that is fairly vague may satisfy the objectives of one study but not the ones of another. So that every respondent will understand a question, it is important to keep the reading level at or below the average reading level of the population. Complex words may be replaced by simpler ones or ones more easily understood. If you are giving a survey to a particular group, you would want to use words that are common to the group.

Response Options Should reflect concepts you are trying to measure, and fit with the wording of the question Avoid simple “yes” or “no” answers and attempt to measure intensity if possible Mutually exclusive (select only one answer) Exhaustive (all possible answers are listed, including other or not applicable or don’t know)

Question Order Be attentive:
Initial questions affect answers to subsequent ones Start with easy, salient, non-threatening questions near the end Cluster questions addressing the same topic or concept together. Avoid redundancy

Questionnaire Format As short as possible
Visually attractive and nicely reproduced Readable (consider font size) Uncluttered Broken into logical sections if possible Clear skip patterns for contingency questions Clear spaces for respondents to mark answers Boxes Parentheses (X) Or numbers to circle 1…2…3 Avoid lines to put check___ Amount of space provided for open-ended questions will determine amount respondent will write.

Questionnaire Format Example
The following questions are about Instructions from the Medical outcomes study Pain measures the pain or pains you experienced in the past 4 weeks. If you had more than one pain, answer the questions by describing your feelings of pain in general. 1. How much bodily pain have you generally had during the past 4 weeks? (Circle one) None……………………………………………….1 Very mild……………………………………….2 Mild………………………………………………….3 Moderate……………………………………….4 Severe…………………………………………….5 Very Severe………………………………….6

Questionnaire Format Helpful Hint…
Get consultation on questions and response options from statisticians and data management people.

Instructions to Respondents
General instructions should be provided at the beginning of the self-administered survey Brief explanation includes: Purpose Significance of the answers How to answer the questions Stress: CONFIDENTIALITY OF RESPONSES Provide specific instructions to questions as needed

Instructions to Respondents Example
The following questions are about Instructions from the Medical outcomes study Pain measures the pain or pains you experienced in the past 4 weeks. If you had more than one pain, answer the questions by describing your feelings of pain in general.

Pre-test (pilot) Be sure to pilot test the survey instrument before the actual administration, especially if it is a new instrument that you constructed or has not been used previously in your population of interest. Helpful Hint: Also good idea to test your sample design, data collection methods, data processing and analysis if possible.

Pre-test (pilot) Use a similar population or identical population you will be measuring. Testing: Question clarity Failure to answer? Multiple answers? “Other” answers and how to handle them in data entry Qualified answers Questionnaire format Instructions? Flow? Layout? Variance in responses Consistency in responses among respondents

Pre-test (pilot) Revise instrument as needed Repeat pre-test
Ideally with some of the original pre-test respondents and new respondents

Data collection Hints Mail survey should include: Survey instrument
Return envelope Self-addressed and stamped

Data collection Hints Self-administered survey should include:
Cover letter On letterhead Signed by PI or most significant person possible Brief explanation of Purpose How results will be used Why respondent was selected Why answers are important Emphasize confidentiality (if applicable) Provide contact information for questions Describe/explain any incentives for participation or consequences or not for non-participation

Monitor survey returns
Open surveys as they are returned Assign unique ID number (if not already assigned) Stamp date on survey Track number returns daily Used to inform timing of follow-up reminders

Monitor survey returns
Optimal response occurs with 2 reminders Timing Based on returns (1-2 weeks after original) Materials (mail survey) Reminder card to have new survey sent or Send entire packet again

Assessing Reliability and Validity
Section Four Assessing Reliability and Validity

Section four Assessing Reliability and Validity
OBJECTIVES Compare and contrast types of reliability Compare and contrast types of validity Key Terms Face validity, content validity, concurrent validity, discriminant validity, construct validity, test-retest reliability, internal consistency, stability.

Reliability and validity
In order for information to be useful, it has to be: consistent, dependable, accurate and, most of all, true. Too often, we are presented with information that fails on one or more of these criteria. In research, these criteria are represented by the concepts of reliability and validity.

Reliability and validity
Expect to obtain the same information time after time. Assessed by correlation coefficient The concept of reliability can be applied to sampling If we repeatedly draw random samples of equal size from a population, we can expect to get the same sample values each time (plus or minus a certain amount due to sampling error). Validity: Measures the concept intended to measure Instrument is presented or used in the way for which it was intended An IQ test is valid only if it is used to measure intelligence it is not valid if it used to assign individuals to groups. A psychological test that is a valid measure of anxiety is not a valid measure of stress.

Types of validity Face validity Content validity Concurrent validity
Information collected appears to be what was expected. (Face value) A question that asked Do you smoke? would appear to have face validity as a measure of smoking behavior. Content validity A question adequately reflects the underlying behavior or body of knowledge. Content validity is established by having a panel of experts evaluate and agree on the relevance of the test items. Concurrent validity One instrument or question is comparable to another that has been shown to validly measure the same content or construct. Concurrent validity is established by correlating one question with another that has previously been validated.

Types of validity Discriminant validity
A question or survey that is able to discriminate between group differences. Example Depressions scales have discriminate validity if individuals who are depressed score differently from those who are not clinically depressed. Predictive validity A question can be used to predict behavior. Example: Can you walk 5 blocks? Construct validity A construct is a theoretical dimension like self-esteem that is measured by having several questions that all relate to how people view themselves. Self-esteem does not exist by itself but is represented by how people respond to these questions. In this example, construct validity measures the extent to which these responses can be called, self-esteem.

Types of reliability Test-retest reliability Internal consistency
Obtained by administering the same test on two or more successive occasions and then correlating the scores. Statistic that reflects reliability is correlation coefficient, higher is better. Internal consistency Obtained by correlating the scores on several questions that pertain to the same content to the sum total of the scores. The average item-total correlation is a measure of how consistently people respond to related items on a test. For example: if a math test had several sets of items that required people to multiply two and three digit numbers, you would expect persons who could not correctly multiply two digit numbers to be unable to multiply three digit numbers. Likewise, you would expect persons who can multiply three digit numbers to be able to multiply two-digit numbers. These expectations should be consistent from person to person and would be borne out by high item-total correlations of the items.

Types of reliability Stability
How much variation exists in scores upon repeated administrations of the instrument. Stable measures will reproduce the same score on repeated administrations of the instrument. This concept is similar to test-retest except that in test-retest situations there is no assumption that the absolute value of each persons test score will stay the same. Considerations: Time between administration, if too close may be remembering answers Learning effect on repeated administrations

Determining the Coding System
Section Five Determining the Coding System

Section Five Determining the Coding System
OBJECTIVES Develop a coding scheme that reduces a data to quantifiable terms. Compare and contrast different types of coding schemes List coding requirements of various computer programs Key Terms coding, alphanumeric, missing values, fixed field, free field, recoding, reconversion

The fine art of coding Coding:
The process of converting information into a quantifiable format (usually numerical) so that a systematic analysis of the information can be done. The coding process is made easier by precoding responses and enabling circled or checked numbers to be entered directly into a data set. The codebook is like a foreign language dictionary in that it translates English responses into numerical or categorical values. The coding system used should model or be easily convertible to the format required by the computers software. If the information is categorical in nature, then, the choice of whether to represent a data value as a number or a label depends upon the analyses that will be done. If you are only going to measure frequency counts and percentages, then either a number or label is acceptable.

The fine art of coding Statistical analyses requires numbers.
Labels cannot be manipulated mathematically. Qualitative data requires content analysis data elements might be entered as words and phrases instead of numbers. Missing values When data cant be collected on one or more variables from one or more source. Statistical computer programs deal with missing values in different ways. Generally, they give you the option of identifying what values are to be considered Missing, or allowing the computer to assume that zeroes or blank spaces represent missing values. Should you assign a number to represent missing values? Yes, if you intend on either analyzing the missing values or replacing them with other values. You must identify or define your missing values before doing any analyses.

Data organization When data is entered into a computer, it will generally be organized in a spreadsheet orientation where the rows represent different cases and the columns represent different variables. If you leave spaces blank instead of typing in a value, you may not know if the space represents a missing value or a missing data entry. By entering values in every space provided, you will know exactly what data has or has not been collected.

Section Six Deciding How to Enter and Store the Data

Section Six Creating the database structure
OBJECTIVES Define the elements of a database Describe the steps in creating a database Describe how to use a form to enter data Key Terms database, table, records, fields

Creating the database structure
Generally, the first column of a data table will have an unique ID number that will identify the survey. This is to allow the researcher to go back and check the data on the survey with the data entered on the computer. The next columns of the table will usually contain the data from the survey in the order of the questions on the survey.

Creating the database structure
To make it easier to identify and manipulate the data, each column is given a unique short name that refers to the question number or subject on the survey. If the questions on a survey are numbered, then an easy way to label the columns would be Q1, Q2, etc

Section Seven Data Entry, Data Checking and Quality Control

Section Seven: Data Entry, Data Checking and Quality Control
OBJECTIVES Define the variables of interest in the study Demonstrate procedures for developing a questionnaire Define the major methods of data collection in quantitative studies and qualitative studies Demonstrate how to determine if responses are random and representative Key Terms variable, coding scheme, tabulation, missing data, non-response bias

Data entry and quality control
Data entry is the process of transferring or transcribing the information from coding sheets, field notes, questionnaires and surveys onto a computer. The process of data entry can be facilitated by use of machine-readable sheets or by image scanners and optical character recognition software manual data entry is not required. Important to check the accuracy of the data Analyses hinges on the accuracy of the data, good quality control procedures are needed for error checking and entry consistency.

Data entry and quality control
There are computer programs that facilitate the quality control function of data entry directly by checking for acceptable values at the time they are entered. Example: if you recording ages of adults in a study, values lower than 18 would be errors. Manual checking of the data may or may not spot the error. The data entry program would immediately reject any entered value that does not fall within the accepted range or is not of the proper form.

Data coding problems Missing data handling
Prepare for nonresponse as treating it as one of the categories (example, a No response). Other situations, you might want to assign the average value to retain the rest of the data for analyses that require it. Individual item analysis, drop the missing data adding constants dont change the overall results. For statistical purposes, to keep as much of the original sample size as possible.

Data coding problems Another solution: Regression
can be used to estimate missing data using answers to other responses. The problem may be avoided by having more than one question dealing with a topic. Not everything you ask will be of equal importance some items you may afford to lose data. The main problem with nonresponse is that you dont know if someone skipped a question or decided not to respond. Including a non-response category will ensure that people respond. Remind them both at the beginning of the questionnaire and at the end to answer ALL times. Dont give people a way not to respond give them the opportunity to respond. You can identify respondents who essentially are non-participants: people who mark the middle response for all items. Only slightly better than nonresponders. Do you keep them in the data bank? No, since their responses will skew the data

Avoiding bias from non-respondents
Its important to know whether those who did not respond differ greatly from those who responded. Relatively few refusals provide the theoretical potential for introducing considerable error into estimates of the sample characteristics. The extent of the differences between respondents and nonrespondents can seldom be determined. Indirect methods may be used.

Avoiding bias from non-respondents
Mail questionnaires are also susceptible to other factors that affect response rate. Interest in the subject of the questionnaire may affect ones answers and also determine if the questionnaire will be completed and returned. Another consideration is the ability of the respondent to provide written responses. People with less education are likely to be underrepresented, partly because of lower educational attainment, but also because of more difficulties with their seeing and writing capabilities.

Web Resources: http://gsociology.icaap.org/methods/surveys.htm

Resources http://gsociology.icaap.org/methods/surveys.htm
Hulley, Cummings, Browner, Grady, Newman (2007). Designing Clinical Research, 3rd Edition. Philadelphia. Lippincott, Williams & Wilkins. ESSENTIALS OF SURVEY RESEARCH AND ANALYSIS A WORKBOOK FOR COMMUNITY RESEARCHERS © 1998 Developed by Dr. Ronald Jay Polland for the Adolescent Pregnancy Prevention Grant, Duval County Health Department. Babbie, Earl (2002), The Basics of Social Research 2nd ed. Wadsworth Thomson Learning: CA. Dillman, D.A. (2000), Mail and Internet Surveys 2nd ed. Wiley: NY. Fink, Arlene and Jacqueline Kosecoff (1998), How to Conduct Surveys. Sage: CA. Fowler, Floyd J. (1993), Survey Research Methods. Sage: CA. Fowler, Floyd J. (1995), Improving Survey Questions. Sage: CA. Sudman, Seymour and Bradburn, Norman (1982), A Practical Guide to Questionnaire Design. Jossey-Bass: San Francisco. Tourangeau, R., and Smith, T.W. (1996), “Asking Senstive Questions: The Impact of Data Collection Mode, Question Format, and Question Context, “ Public Opinion Quarterly, 60: Also see American Association of Public Opinion Research, “Best Practices for Survey and Public Opinion Research.”

Resources http://gsociology.icaap.org/methods/surveys.htm
Aday, L. A. (1996). Designing and Conducting Health Surveys, 2nd ed. San Francisco: Jossey-Bass. Biemer, P., Groves, R., Lyberg, L., Mathiowetz, N., & Sudman, S. (eds.) (1991). Measurement Errors in Surveys. New York: Wiley. Dillman, D. (1978). Mail and Telephone Surveys: The Total Design Method. New York: Wiley. Dillman, D. (2000). Mail and Internet Surveys: The Tailored Design Method. New York: Wiley & Sons. Fink, A., & Kosecoff, J. (1985). How to Conduct Surveys: A Step-by-step Guide. Beverly Hills, CA: Sage, 1985. Fowler, F. J., Jr. Survey Research Methods, 2nd ed. Newbury Park, CA: Sage, 1993. Groves, R. (1989). Survey Errors and Survey Costs. New York: Wiley, 1989. Groves, R., Biemer, P., Lyberg, L., Massey, J., Nicholls, W., II, & Waksberg, J. (eds.) (1988). Telephone Survey Methodology. New York: Wiley.

Resources Lavrakas, P. J. (1993). Telephone Survey Methods: Sampling, Selection, and Supervision. Newbury Park, CA: Sage. Lessler, J. T., & Kalsbeek, W. D. (1992). Nonsampling Error in Surveys. New York: Wiley. Lyberg, L., Biemer, P., Collins, M., deLeeuw, E., Dippo, C., Schwarz, N., & Trewin, D. (eds.) (1997). Survey Measurement and Process Quality. New York: Wiley. Marín, G,. & Marín, B. V. (1991). Research with Hispanic Populations. Newbury Park, CA: Sage. Turner, C. F., & Martin, E. (eds.) (1984). Surveying Subjective Phenomena (2 volumes). New York: Russell Sage. Journals: Public Opinion Quarterly and Journal of Official Statistics

PACCTR* Curriculum Core
Roberta Oka RN, ANP, DNSc, School of Nursing George Sawaya MD, School of Medicine Rebecca Jackson MD, School of Medicine Susan Hyde DDS, MPH, PhD, School of Dentistry Jennifer Cocohoba PharmD, School of Pharmacy Joel Palefsky MD School of Medicine * Pathways to Careers in Clinical and Translational Research

Introduction to Survey Research

Similar presentations

Presentation on theme: "Introduction to Survey Research"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Introduction to Survey Research

Similar presentations

Presentation on theme: "Introduction to Survey Research"— Presentation transcript:

Similar presentations

About project

Feedback