Don’t Stop Me Now Preparing for an ageing population Mystery Shopper Survey Findings and Methodology

Contents Part 1 Why a mystery shopper survey is valuable Part 2 Don’t Stop Me Now methodology – What we did Part 3 Don’t Stop Me Now findings – What we found Part 4 Running your own mystery shopper survey - checklists

Mystery shopper survey Part 1 : Why is a mystery shopper survey valuable?

Why is a mystery shopper survey valuable? A Mystery Shopper Survey will enable you to: Capture an accurate and realistic picture of service user experiences Assess whether service users are able to easily obtain information about their local area

Mystery shopper surveys and age-proofing Working with older mystery shoppers: Is a quick and cost effective method of assessing whether local services are age-proofed and accessible to older residents Will enable councils to benchmark their performance against the findings in Don’t Stop Me Now Will enable councils to measure their own direction of travel by repeating the survey on a regular basis

Mystery shopper survey Part 2: Methodology from Don’t Stop Me Now – What we did

Part 2: Methodology from Don’t Stop Me Now Part 2 covers: The purpose and remit of the Don’t Stop Me Now mystery shopper survey How the Don’t Stop Me Now survey was conducted

Purpose and remit of the survey The purpose of the survey was to: Evaluate how well councils are able to direct older people to a range of local services designed to maintain their well-being and independence; and Ascertain if councils are enabling older people to play a full and active role in the community The focus was on: Services and initiatives outside of social care; and Local opportunities to keep fit, well, active and engaged

How the survey was conducted The mystery shoppers Aged 50 – 70 2 male, 2 female All lived actively and independently with no social care needs All made the calls from their own homes Calls were spread over Monday – Friday and were made at different times of the day Each used a common scenario and tested 10 – 12 councils each (49 councils in total)

How the survey was conducted: Making the calls The mystery shoppers: used a local phone directory to find the number(s) they felt were most appropriate additionally used the internet to find numbers and information if they were comfortable doing so asked to be transferred to the relevant person or department if they could only locate a call centre number for the council used their own personal details (name, age, address) prompted and probed if information was not forthcoming

How the survey was conducted: Scenario and script The common scenario: “My name is x, I currently live in # and I am moving to your area in the next few months. I am # years old / recently retired and I am looking for information on #”. The mystery shoppers were briefed to prompt and probe in all calls to assess the “best case” scenario. For example, if the council was struggling to provide information on opportunities to engage in voluntary work, the mystery shopper would provide some suggestions .e.g. “Is there a local Age Concern that you could put me in touch with?” If the council asked for the mystery shopper’s new address or postcode the response was “I don’t have a new address yet, I am just seeking general local information at the moment”.

How the survey was conducted: Areas tested In each call the shoppers asked for local information on one of the following areas: Opportunities to keep fit and active Opportunities to socialise Transport Learning Voluntary work and employment If a council was struggling to refer the call to the appropriate person or department then the mystery shopper would ask “Do you have a team or a person that specifically deals with issues for older people?”

Mystery shopper survey Part 3: Findings from Don’t Stop Me Now – What we found in 2007

Survey findings – November 2007 In 82% of calls Mystery Shoppers needed to prompt and probe It was easiest to obtain information on leisure services for older people and most difficult to obtain information on employment and volunteering opportunities In 69% of calls shoppers were referred to a website and no information was sent in the post 56% of calls were dealt with on first contact, 35% were transferred once and the remainder transferred more than once Most transferred calls were to adult social care

Mystery shoppers needed to prompt and probe in 82% of calls overall

Was information sent in the post? 69% were referred to the internet for more information. 31% were sent information in the post * 71%of people over 65 have never used the internet

Good information provided on leisure, socialising and transport They told me about a leisure card based on ability to pay – it gives various discounts. They produced a directory for adults over 50 and will send it out. Plenty of sports facilities for the older person – Mystery shopper Try U3A – they do classes, nights out, pub visits, meals out - Council Go to George Lawson hall – it’s the time of your life for the over 50s! - Council We have a very good transport system – buses and trains - all you need to get round. We have a transport map and it will contain all the information that you need. Freedom passes are available to all over sixty - council I’ve actually learned a lot from doing this project and speaking to all the different councils. I never realised how many possible services are out there for people my age – mystery shopper

Reliance on the internet and transferred calls I felt shunted around and that there was no single place where I could get all the information – Mystery shopper If you haven’t access to a computer you will spend hours on the phone! I asked the switchboard if they had leaflets they could send me and had no reply – mystery shopper The lady had to ask a colleague. I felt sure I had been put through to the wrong department. I was kept on hold for several minutes….she really couldn't help me with anything else and didn't put me through to anyone else who might have been able to answer my questions – mystery shopper The council said “Services for older people? I've never heard of this. I have been here for 2 years but never heard of that. We have social care and social welfare - go onto our website.” – Mystery Shopper

Mystery shopper survey Part 4: Running your own mystery shopper survey: Methodology checklists

Part 4: Methodology checklists Checklists cover: Project management Remit, scenarios and scripts Areas to test Selecting and training the mystery shoppers Data capture Analysis and Presentation Using the results

Checklist 1 – Project Management Appoint a project manager Project manager will need awareness and expertise about local older people and services available Project manager should consult other local experts to shape survey and analyse data (e.g. local analysts / local voluntary and community / older community Agree timeframe for survey Agree overall cost of survey

Checklist 2 – Remit, scenario and scripts Agree the remit of the survey at the outset (e.g. ease of finding appropriate contact in council / ease with which information was provided / type and quality of information provided) Agree a common scenario (e.g. I am about to move to your area / I live in this area and have recently retired) Decide on method(s) of contact (e.g. phone calls / face to face visits / e-mail) Decide on number and nature of contacts needed and over what period Ensure contacts are made across a range of days and times (e.g. spread across all working days / opening hours)

Checklist 3 - Selecting the areas to test Possible options are: Opportunities to keep fit and active / leisure services Opportunities to meet new people / socialise Transport options in the area Opportunities to study / learn new skills Opportunities to get involved in volunteering Employment options in the area This is not an exhaustive list. The range of area tested can be expanded or contracted depending on local issues.

Checklist 4 - selection and training of mystery shoppers Select shoppers with a range of ages from 50 upwards Ensure a gender spilt Ensure diverse representation to reflect local community Provide a training session to brief mystery shoppers on the purpose and remit of the work Take on board suggestions from the mystery shoppers, and adapt methods and script accordingly Work with local voluntary and community sector partners to help select / train / brief mystery shoppers

Checklist 5 - Data Capture Consult the mystery shoppers on their preferred method of data capture (e.g. written pro-forma / IT solution) Consider recording calls to capture full conversation and to provide accurate quotes. Consult mystery shoppers on preferred methods of data recording Ensure the method of data capture reflects each aspect of the remit of the survey (e.g. number of times call transferred / department / person transferred to / number of referrals to the intranet / offers to send written information / quality of information provided) Allow mystery shoppers to record overall experiences / any other information on each call

Part 6 – Analysis and presentation of the data Use expertise from in-house analysts Provide quantitative data (e.g. on number of calls / number of transferred calls / number of calls that provided spontaneous information / number of calls that needed prompting and probing) Separate data into areas tested e.g leisure / transport etc Use quotes from calls and comments from mystery shoppers to illustrate the content and usefulness of calls Use Audit Commission report and findings as a guide on presentation (part 3)

Checklist 6 - Using the results Compare results against Audit Commission findings from November 2007 to benchmark performance (part 3) Repeat survey on a regular basis (quarterly / bi-annual / annual) to benchmark the direction of travel of council Report on results as part of scrutiny and performance process

For more information on Don’t Stop Me Now www.audit-commission.gov.uk/nationalstudies/