For Example…...The impact of different example types on frequency estimates Presented at the 2008 AAPOR annual conference in New Orleans, LA

Steps in Survey Question Responding (IR2) Interpret the question. Interpret the question. Retrieve relevant information from memory. Retrieve relevant information from memory. Integrate relevant information. Integrate relevant information. Respond in a manner allowed by the question format. Respond in a manner allowed by the question format.

Unique Demands of Survey Questions About the Frequency of Events Interpret - Understand the universe of events that fall within the category. Interpret - Understand the universe of events that fall within the category. Recall – Remember all appropriate events that occurred within the specified time frame. Recall – Remember all appropriate events that occurred within the specified time frame. Integrate – Sum the number of recalled events. Integrate – Sum the number of recalled events. Respond – Select an allowable response that most closely matches the sum. Respond – Select an allowable response that most closely matches the sum.

Potential Errors and Biases for Questions about Frequency Interpretation Interpretation Mismatches between encoding and question terms Mismatches between encoding and question terms Recall Recall Retrieval failures Retrieval failures Satisficing Satisficing Integration Integration Reliance on heuristics Reliance on heuristics

Why Include Examples in Questions about Frequency? Examples help define the universe of appropriate events. Examples help define the universe of appropriate events. Examples stimulate recall of events that might otherwise be overlooked. Examples stimulate recall of events that might otherwise be overlooked.

Examples…of Survey Frequency Questions with Examples National Election Study (2004) - Aside from attending services, in the PAST SIX MONTHS have you been an active member at your place of worship -- I mean, have you done things like serve on committees, give time for special projects, or help organize meetings? National Election Study (2004) - Aside from attending services, in the PAST SIX MONTHS have you been an active member at your place of worship -- I mean, have you done things like serve on committees, give time for special projects, or help organize meetings? National Election Study (2000) - How many times in the last week have you watched morning news programs such as "Today," "Good Morning America,“ or "The Early Show"? National Election Study (2000) - How many times in the last week have you watched morning news programs such as "Today," "Good Morning America,“ or "The Early Show"? National Crime Victimization Survey (1999) - On average, during the last 6 months, that is, since _____ _____, 19__, how often have you gone shopping? For example, at drug, clothing, grocery, hardware, and convenience stores? National Crime Victimization Survey (1999) - On average, during the last 6 months, that is, since _____ _____, 19__, how often have you gone shopping? For example, at drug, clothing, grocery, hardware, and convenience stores?

Example Types Object examples - References to people, places, or things included in the universe of appropriate events Object examples - References to people, places, or things included in the universe of appropriate events On average, during the last 6 months, that is, since _____ _____, 19__, how often have you gone shopping? For example at drug, clothing, grocery, hardware, and convenience stores? On average, during the last 6 months, that is, since _____ _____, 19__, how often have you gone shopping? For example at drug, clothing, grocery, hardware, and convenience stores? Behavior examples - References to activities included in the universe of appropriate events Behavior examples - References to activities included in the universe of appropriate events During the last 6 months, how many times per week, on average, did you visit a store to pick up or buy something, to look at a product, or to talk to a sales person? During the last 6 months, how many times per week, on average, did you visit a store to pick up or buy something, to look at a product, or to talk to a sales person?

Hypotheses Examples lead to increased estimates of frequencies. Examples lead to increased estimates of frequencies. Object examples bias reported frequencies in favor of exemplified objects. Object examples bias reported frequencies in favor of exemplified objects. Behavior examples are less likely to generate biased frequency estimates than object examples. Behavior examples are less likely to generate biased frequency estimates than object examples.

Research Design Sample - N=1124 survey panel respondents Sample - N=1124 survey panel respondents Online consumer survey content Online consumer survey content Estimated average weekly shopping trips Estimated average weekly shopping trips Attitudes toward 32 specific retailers (Net Promoter, liking, satisfaction) Attitudes toward 32 specific retailers (Net Promoter, liking, satisfaction) Estimated shopping trips to 15 store types Estimated shopping trips to 15 store types Conditions Conditions No examples (n=282) - During the last 6 months, how many times per week, on average, did you visit a store to go shopping? No examples (n=282) - During the last 6 months, how many times per week, on average, did you visit a store to go shopping? Hardware object examples (n=297) - During the last 6 months, how many times per week, on average, did you visit a store to go shopping? For example, drug stores, clothing stores, grocery stores, hardware stores, and convenience stores? Hardware object examples (n=297) - During the last 6 months, how many times per week, on average, did you visit a store to go shopping? For example, drug stores, clothing stores, grocery stores, hardware stores, and convenience stores? Pet store object examples (n=276) - During the last 6 months, how many times per week, on average, did you visit a store to go shopping? For example, drug stores, clothing stores, grocery stores, pet supply stores, and convenience stores? Pet store object examples (n=276) - During the last 6 months, how many times per week, on average, did you visit a store to go shopping? For example, drug stores, clothing stores, grocery stores, pet supply stores, and convenience stores? Behavior examples (n=269) - During the last 6 months, how many times per week, on average, did you visit a store to pick up or buy something, to look at a product, or to talk to a sales person? Behavior examples (n=269) - During the last 6 months, how many times per week, on average, did you visit a store to pick up or buy something, to look at a product, or to talk to a sales person?

Mean Estimated Weekly Shopping Trips (EWST-4)* Pet Supply examples > No examples (p No examples (p<.05) Pet Supply examples > Behavior examples (p Behavior examples (p<.07) *To minimize impact of outliers, analyses are based on the fourth root of reported frequency.

Store Types Exemplified Store Types Non-Exemplified Store Types Drug Office supply Clothing Sporting goods GroceryBook Hardware Hobby or crafts Pet supply Department ConvenienceElectronics Recorded music Jewelry

Effects of Visits to Store Types † on Estimated Weekly Shopping Trips Standardized regression coefficients for store types predicting EWST -4. Exemplified Non-exemplified Exemplified Non-exemplified No examples.20** -.07 Pet supply examples.16*.00 Hardware examples.22**.08 Behavior examples * + p <.10 * p <.05 ** p <.01 † Visits equals the 4th root of the sum of visits to exemplified or non-exemplified store types.

Summary of Findings Object examples led to larger frequency estimates. Object examples led to larger frequency estimates. The larger reported frequencies may reflect overestimates, but the “true” values are unknown. The larger reported frequencies may reflect overestimates, but the “true” values are unknown. Object examples biased event recall. Object examples biased event recall. The estimates are more heavily influenced by the exemplified objects than non-exemplified objects. The estimates are more heavily influenced by the exemplified objects than non-exemplified objects. Behavior examples did not bias event recall. Behavior examples did not bias event recall. Frequency estimates following behavior examples showed the greatest correspondence across the entire range of relevant events Frequency estimates following behavior examples showed the greatest correspondence across the entire range of relevant events