1. Issues in Improving Responsible Retailing Practices Joel W. Grube Prevention Research Center Pacific Institute for Research and Evaluation

2. Complex Factors Affect Sales Rates Clerk Store Characteristics Corporate Characteristics Context Broader Legal and Social Environment

3. Probability of at Least One Sale Given True Compliance Rate Actual Compliance Rate Probability of at least one sale

4. Number of Observations Needed to Describe Prevalence of Sale for Individual Store Observations 95% Confidence Interval

5. Why Does it Matter? The complexity and variability create uncertainty when: Evaluating program effects—How do we know what works? Evaluating program effects—How do we know what works? Monitoring store performance—How do we know when a store has improved? Monitoring store performance—How do we know when a store has improved? Comparing stores—How do we know one store is doing better or worse? Comparing stores—How do we know one store is doing better or worse?

6. Voluntary Compliance Programs Acting under the authority of Consumer Protection laws, state Attorneys General have engaged national retail chains to intensify their efforts to prevent underage sales of tobacco products Acting under the authority of Consumer Protection laws, state Attorneys General have engaged national retail chains to intensify their efforts to prevent underage sales of tobacco products National drug store, discount and convenience store/gas station chains have executed Assurances of Voluntary Compliance (“AVC”) in which they stipulate to changes in hiring, training and point-of-sales protocol National drug store, discount and convenience store/gas station chains have executed Assurances of Voluntary Compliance (“AVC”) in which they stipulate to changes in hiring, training and point-of-sales protocol

7. Voluntary Compliance Programs Evaluation shows clerks and managers have: High knowledge of laws and store policies High knowledge of laws and store policies High perceived self-efficacy High perceived self-efficacy Strong personal support for the goal of preventing underage tobacco sales Strong personal support for the goal of preventing underage tobacco salesNevertheless: Monitoring shows that clerks fail to perform age- verification protocols correctly in 20% - 30% of attempted tobacco purchases Monitoring shows that clerks fail to perform age- verification protocols correctly in 20% - 30% of attempted tobacco purchases Source: Krevor, Prottas & Fournier, 2006

8. Voluntary Compliance Programs Compliance Rates by Quarter 2004Quarter 183.7 Quarter 269.4 Quarter 369.5 Quarter 468.8 2005Quarter 172.7 Quarter 277.8 Quarter 375.4

9. Mystery Shopper Reports Mystery Shopper (“MS”) reports are being considered by Attorneys General as an additional stipulation in AVCs Mystery Shopper (“MS”) reports are being considered by Attorneys General as an additional stipulation in AVCs Mystery Shopper Reports provide detailed feedback on observed age-verification Mystery Shopper Reports provide detailed feedback on observed age-verification There is no evidence at present whether frequent MS reports improve age-verification There is no evidence at present whether frequent MS reports improve age-verification

10. Mystery Shopper Evaluation Project Brandeis Brad Krevor Stephen Fournier Jeffery Prottas PIRE Joel Grube William Ponicki

11. Mystery Shopper Evaluation Project The primary aims of the study are: Do frequent (monthly) MS reports increase correct age-verification conduct? Do frequent (monthly) MS reports increase correct age-verification conduct? Do the effects of intensive MS reporting persist? Do the effects of intensive MS reporting persist? How do managers use the reports? How do managers use the reports? Identify store, purchaser, and clerk characteristics related to age-verification conduct. Identify store, purchaser, and clerk characteristics related to age-verification conduct.Hypothesis: Increased feedback will raise the salience of tobacco age-verification at the store and district level leading to higher rates of age-verification Increased feedback will raise the salience of tobacco age-verification at the store and district level leading to higher rates of age-verification

12. Mystery Shopper Evaluation Project Randomly assign 206 CORS to treatment and control Randomly assign 206 CORS to treatment and control All CORs inspected by Mystery Shoppers twice a month for 13 months All CORs inspected by Mystery Shoppers twice a month for 13 months Treatment condition: In Months 3 –10 every 2nd inspection by the Mystery Shopper will be reported Treatment condition: In Months 3 –10 every 2nd inspection by the Mystery Shopper will be reported Control Condition: Every 6th inspection will be reported Control Condition: Every 6th inspection will be reported Telephone interviews with store and district managers in months 6 and 9 Telephone interviews with store and district managers in months 6 and 9 In months 11 - 13, the twice-monthly MS inspections will continue, but no MS reports In months 11 - 13, the twice-monthly MS inspections will continue, but no MS reports

13. Mystery Shopper Evaluation Project There are 206 CORS. For each COR there will be observations at 26 time points There are 206 CORS. For each COR there will be observations at 26 time points A total of 5,356 MS observations A total of 5,356 MS observations Sufficient observations for detecting a 5% treatment impact (75% to 80%) over the treatment period and a 10% (75% to 85%) impact within the shorter post-treatment period Sufficient observations for detecting a 5% treatment impact (75% to 80%) over the treatment period and a 10% (75% to 85%) impact within the shorter post-treatment period

14. Summary Accurately measuring individual store performance is very difficult Accurately measuring individual store performance is very difficult Measures are highly variable Measures are highly variable Performance is affected by a wide range of factors Performance is affected by a wide range of factors Adequately capturing individual store performance requires a large number of measures at different times, days, and with different clerks Adequately capturing individual store performance requires a large number of measures at different times, days, and with different clerks Performance at the aggregate level (e.g., chain, community) more easily measured Performance at the aggregate level (e.g., chain, community) more easily measured Despite difficulties, it is possible to measure performance and evaluate programs (e.g., MS) Despite difficulties, it is possible to measure performance and evaluate programs (e.g., MS)