1. Don’t Stop Me Now Preparing for an ageing population
Mystery Shopper Survey
Findings and Methodology

2. Contents
Part 1
Why a mystery shopper survey is valuable
Part 2
Don’t Stop Me Now methodology – What we did
Part 3
Don’t Stop Me Now findings – What we found
Part 4
Running your own mystery shopper survey - checklists

3. Mystery shopper survey
Part 1 : Why is a mystery shopper survey valuable?

4. Why is a mystery shopper survey valuable?
A Mystery Shopper Survey will enable you to:
Capture an accurate and realistic picture of service user experiences
Assess whether service users are able to easily obtain information about their local area

5. Mystery shopper surveys and age-proofing
Working with older mystery shoppers:
Is a quick and cost effective method of assessing whether local services are age-proofed and accessible to older residents
Will enable councils to benchmark their performance against the findings in Don’t Stop Me Now
Will enable councils to measure their own direction of travel by repeating the survey on a regular basis

6. Mystery shopper survey
Part 2: Methodology from Don’t Stop Me Now – What we did

7. Part 2: Methodology from Don’t Stop Me Now
Part 2 covers:
The purpose and remit of the Don’t Stop Me Now mystery shopper survey
How the Don’t Stop Me Now survey was conducted

8. Purpose and remit of the survey
The purpose of the survey was to:
Evaluate how well councils are able to direct older people to a range of local services designed to maintain their well-being and independence; and
Ascertain if councils are enabling older people to play a full and active role in the community
The focus was on:
Services and initiatives outside of social care; and
Local opportunities to keep fit, well, active and engaged

9. How the survey was conducted
The mystery shoppers
Aged 50 – 70
2 male, 2 female
All lived actively and independently with no social care needs
All made the calls from their own homes
Calls were spread over Monday – Friday and were made at different times of the day
Each used a common scenario and tested 10 – 12 councils each (49 councils in total)

10. How the survey was conducted: Making the calls
The mystery shoppers:
used a local phone directory to find the number(s) they felt were most appropriate
additionally used the internet to find numbers and information if they were comfortable doing so
asked to be transferred to the relevant person or department if they could only locate a call centre number for the council
used their own personal details (name, age, address)
prompted and probed if information was not forthcoming

11. How the survey was conducted: Scenario and script
The common scenario: “My name is x, I currently live in # and I am moving to your area in the next few months. I am # years old / recently retired and I am looking for information on #”.
The mystery shoppers were briefed to prompt and probe in all calls to assess the “best case” scenario. For example, if the council was struggling to provide information on opportunities to engage in voluntary work, the mystery shopper would provide some suggestions .e.g. “Is there a local Age Concern that you could put me in touch with?”
If the council asked for the mystery shopper’s new address or postcode the response was “I don’t have a new address yet, I am just seeking general local information at the moment”.

12. How the survey was conducted: Areas tested
In each call the shoppers asked for local information on one of the following areas:
Opportunities to keep fit and active
Opportunities to socialise
Transport
Learning
Voluntary work and employment
If a council was struggling to refer the call to the appropriate person or department then the mystery shopper would ask “Do you have a team or a person that specifically deals with issues for older people?”

13. Mystery shopper survey
Part 3: Findings from Don’t Stop Me Now – What we found in 2007

14. Survey findings – November 2007
In 82% of calls Mystery Shoppers needed to prompt and probe
It was easiest to obtain information on leisure services for older people and most difficult to obtain information on employment and volunteering opportunities
In 69% of calls shoppers were referred to a website and no information was sent in the post
56% of calls were dealt with on first contact, 35% were transferred once and the remainder transferred more than once
Most transferred calls were to adult social care

15. Mystery shoppers needed to prompt and probe in 82% of calls overall

16. Was information sent in the post?
69% were referred to the internet for more information.
31% were sent information in the post
* 71%of people over 65 have never used the internet

17. Good information provided on leisure, socialising and transport
They told me about a leisure card based on ability to pay – it gives various discounts. They produced a directory for adults over 50 and will send it out. Plenty of sports facilities for the older person – Mystery shopper
Try U3A – they do classes, nights out, pub visits, meals out - Council
Go to George Lawson hall – it’s the time of your life for the over 50s! - Council
We have a very good transport system – buses and trains - all you need to get round. We have a transport map and it will contain all the information that you need. Freedom passes are available to all over sixty - council
I’ve actually learned a lot from doing this project and speaking to all the different councils. I never realised how many possible services are out there for people my age – mystery shopper

18. Reliance on the internet and transferred calls
I felt shunted around and that there was no single place where I could get all the information – Mystery shopper
If you haven’t access to a computer you will spend hours on the phone! I asked the switchboard if they had leaflets they could send me and had no reply – mystery shopper
The lady had to ask a colleague. I felt sure I had been put through to the wrong department. I was kept on hold for several minutes….she really couldn't help me with anything else and didn't put me through to anyone else who might have been able to answer my questions – mystery shopper
The council said “Services for older people? I've never heard of this. I have been here for 2 years but never heard of that. We have social care and social welfare - go onto our website.” – Mystery Shopper

19. Mystery shopper survey
Part 4: Running your own mystery shopper survey: Methodology checklists

20. Part 4: Methodology checklists
Checklists cover:
Project management
Remit, scenarios and scripts
Areas to test
Selecting and training the mystery shoppers
Data capture
Analysis and Presentation
Using the results

21. Checklist 1 – Project Management
Appoint a project manager
Project manager will need awareness and expertise about local older people and services available
Project manager should consult other local experts to shape survey and analyse data (e.g. local analysts / local voluntary and community / older community
Agree timeframe for survey
Agree overall cost of survey

22. Checklist 2 – Remit, scenario and scripts
Agree the remit of the survey at the outset (e.g. ease of finding appropriate contact in council / ease with which information was provided / type and quality of information provided)
Agree a common scenario (e.g. I am about to move to your area / I live in this area and have recently retired)
Decide on method(s) of contact (e.g. phone calls / face to face visits / )
Decide on number and nature of contacts needed and over what period
Ensure contacts are made across a range of days and times (e.g. spread across all working days / opening hours)

23. Checklist 3 - Selecting the areas to test
Possible options are:
Opportunities to keep fit and active / leisure services
Opportunities to meet new people / socialise
Transport options in the area
Opportunities to study / learn new skills
Opportunities to get involved in volunteering
Employment options in the area
This is not an exhaustive list. The range of area tested can be expanded or contracted depending on local issues.

24. Checklist 4 - selection and training of mystery shoppers
Select shoppers with a range of ages from 50 upwards
Ensure a gender spilt
Ensure diverse representation to reflect local community
Provide a training session to brief mystery shoppers on the purpose and remit of the work
Take on board suggestions from the mystery shoppers, and adapt methods and script accordingly
Work with local voluntary and community sector partners to help select / train / brief mystery shoppers

25. Checklist 5 - Data Capture
Consult the mystery shoppers on their preferred method of data capture (e.g. written pro-forma / IT solution)
Consider recording calls to capture full conversation and to provide accurate quotes. Consult mystery shoppers on preferred methods of data recording
Ensure the method of data capture reflects each aspect of the remit of the survey (e.g. number of times call transferred / department / person transferred to / number of referrals to the intranet / offers to send written information / quality of information provided)
Allow mystery shoppers to record overall experiences / any other information on each call

26. Part 6 – Analysis and presentation of the data
Use expertise from in-house analysts
Provide quantitative data (e.g. on number of calls / number of transferred calls / number of calls that provided spontaneous information / number of calls that needed prompting and probing)
Separate data into areas tested e.g leisure / transport etc
Use quotes from calls and comments from mystery shoppers to illustrate the content and usefulness of calls
Use Audit Commission report and findings as a guide on presentation (part 3)

27. Checklist 6 - Using the results
Compare results against Audit Commission findings from November 2007 to benchmark performance (part 3)
Repeat survey on a regular basis (quarterly / bi-annual / annual) to benchmark the direction of travel of council
Report on results as part of scrutiny and performance process

28. For more information on Don’t Stop Me Now