1 Mystery Shopping SHIP Directors’ Conference June 11, 2007 Julie Leonard & Erika Melman BearingPoint, Inc.

2 Topics  Mystery Shopping –What is it? –Why Mystery Shop? –CMS and Mystery Shopping –Mystery Shopping Toolkit  Specifics of Mystery Shopping –Goals –Planning and Steps –Benefits of Mystery Shopping  Questions & Answers

3 Mystery shopping is: A beneficiary-focused self-assessment tool that allows you to view, understand, and evaluate your SHIP program from your clients’ perspective. Surveys Web site Review Reports Audits Mystery Shopping Mystery Shopping: What is it? Mystery shopping is just one piece of the puzzle in quality assurance and process improvement

4 Mystery shopping lets you see your program through the eyes of your clients

5 What Mystery Shopping is NOT… Mystery Shopping is a tool used to gauge overall program performance for the purpose of enhancing counseling services. It should NOT be used as a punitive measure to reprimand a staff member or volunteer.

6 Why Mystery Shop? Mystery shopping your program can help you:  Identify where training and other process improvement efforts are necessary  Implement a low-cost but highly effective self- assessment  Learn about your program through the data you collect  Enhance the counseling services you provide to your clients.

7 Past CMS Mystery Shopping Projects 1) The SHIP Program  Accuracy –Preventive Benefits –Part D knowledge –Basic Medicare  Access –Counselor availability and responsiveness –Call wait time –Ease of scheduling in-person appointments  Customer Service –Telephone etiquette –The manner in which information was provided

8 Past CMS Mystery Shopping Projects 2) MEDICARE  Accuracy –Appropriate use of PDPF tool –Appropriate use of standardized scripts –Basic Medicare knowledge  Access –Average wait times during special events  Customer Service –Greeting –Closing

9 Past CMS Mystery Shopping Projects 3) Medicare Prescription Drug Discount Card Program Sponsors and Medicare Prescription Drug Program Sponsors  Compliance –Marketing practices –Policy

10 How do SHIP Directors Use Mystery Shopping? SHIP Directors are interested in mystery shopping to:  Identify training needs, including refreshers  Verify message accuracy and consistency  Set performance improvement goals  Identify best practices and accolades There is always room for improvement, and this is one of many activities a Director can use to enhance their program.

11 Mystery Shopping Toolkit Purpose  To help you develop a mystery shopping activity tailored to your program Contents  How to use the mystery shopping toolkit  Background of mystery shopping  Planning and conducting mystery shopping  Templates for mystery shopping

12 Status of Mystery Shopping Toolkit In progress  Development of the toolkit contents Next steps  Test the toolkit with stakeholders in focus groups and in-depth interviews  Pilot test the toolkit with SHIP programs Your feedback is important!

13 Specifics of Mystery Shopping

14 The Goal Determine your goal before deciding on what aspects or behaviors to assess. Sample goals:  Access –Provide timely assistance to callers –Reduce wait time on telephone –Provide timely access to face to face counseling –Provide access to services for people with physical disabilities and other special needs

15 Sample Goals (Continued)  Accuracy –Provide correct information to clients –Provide all relevant information to clients –Customer Service –Provide a friendly and safe environment to clients  Referrals –Refer the client to the correct organization or agency, as appropriate and necessary

16 Sample Goals (Continued)  Compliance –Ensure client privacy before, during, and after counseling service  Publications –Provide to the client all relevant CMS and state specific publications

17 What do mystery shoppers do and how do they do it? Mystery shoppers  Pose as clients of a SHIP program  Follow a specific script (scenario)  Record their experience  Are trained to behave like typical clients Mystery shopping is done by  Telephone  In-person

18 Steps of Mystery Shopping 1.Decide what you want to assess. 2.Decide whether to use in-person or telephone mystery shopping. 3.Decide what kind of information to collect. 4.Decide whether the shopper should pose as a Medicare beneficiary, or a family member, friend, or neighbor. 5.Select shoppers.

19 Steps of Mystery Shopping (Continued) 6. Develop a scenario (and pilot test it). 7. Develop a form to record the experience. 8. Develop a mystery shopping schedule. 9. Train the shoppers. 10. Conduct the mystery shopping. 11. Analyze the information collected. 12. USE THE RESULTS.

20 Step 1: Decide What You Want to Assess Determine aspects to assess Example Goal: Provide the client with timely access to a counselor.  Objective that supports the goal  Aspect to be assessed  Information needed REMEMBER: The goal helps guide all of the other steps of the mystery shopping process.

21 Step 2: Telephone or In-Person Mystery Shopping? Choose your Mystery Shopping Method The appropriate method to use is determined by:  What you want to assess (topic)  Available resources  Time

22 Step 3: Decide What Kind of Information to Collect Determine what type of information is best Ask yourself what kind of information will help you assess your program.  In general, qualitative information is documented in words; quantitative information is documented in numbers.  The type of information that you want to collect will affect the kinds of questions that your shopper will ask the counselors.

23 Step 4: Decide on the Shopper Persona Choose a Mystery Shopper Persona The shopper can pose as:  A Medicare beneficiary  A family member  A friend or neighbor

24 Step 5: Select the Shoppers Choose a Mystery Shopper Mystery shoppers can be:  Temporary workers hired by you  Your staff or volunteers  Employees of a mystery shopping firm

25 Step 6: Develop Mystery Shopping Scenario Create a Scenario The elements of the scenario are:  Background information of the shopper  Shopper questions  Knowledge checks

26 Step 7: Develop Form Create a form to collect shopping data Purpose:  Form for the shopper to record information about the call or visit Type of Information:  Relevant data elements based on your goal Each telephone mystery shopping call or in person visit should have its own form

27 Step 8: Develop Mystery Shopping Schedule Your goals set the shopping schedule  Examples of different schedules to test access: –Normal business hours. –Periods of high call volume –During major events

28 Step 9: Train the Shoppers Tailor training to meet your goal  Possible training methods: – General training session – Walk-through of the data collection form – Role-playing

29 Step 10: Conduct the Activity Mystery Shop! Shoppers should begin conducting in-person visits and telephone mystery calls at this time.

30 Step 11: Analyze the Data Compile and Assess your Data To analyze the data:  Aggregate information from calls/visits into one file  Assess trends The number of mystery shopping calls or visits you conduct should be in proportion to your normal call or visit volume in order to apply your findings across your entire program.

31 Step 12: Use the Results Put your results to use The data that you collect from mystery shopping your program will help you meet your goal for the mystery shopping project. The data will help you improve your program.

32 Benefits of Mystery Shopping With mystery shopping, you can: 1.Get real-time, simple, and inexpensive feedback. 2.Evaluate accuracy of information provided, as well as customer service. 3.Identify strengths and weaknesses of your program. 4.Ensure that staff follow agency standards and policies. 5.Improve training.

33 Questions Thank you for your participation