# An introduction to surveys – the basics U3A Cambridge Summer term July 4 th 2014 Jill Tuffnell.

## Presentation on theme: "An introduction to surveys – the basics U3A Cambridge Summer term July 4 th 2014 Jill Tuffnell."— Presentation transcript:

An introduction to surveys – the basics U3A Cambridge Summer term July 4 th 2014 Jill Tuffnell

Scope & coverage My credentials Overview of statistical platform & key concepts: confidence interval; confidence level and sample size Survey design and methodology Response rates Examples Questions

My background Degrees in economics & statistics; operational research Work for National Coal Board, local government & public policy consultancy Headed Research Group at Cambs County Council; specialist in labour market, Census analysis and involvement in many surveys – ranging from new settlements to travellers to fire-fighters to footpaths

(Brief) overview of statistics (i) Kit of tools enabling us to measure, interpret and analyse data – quantitative & qualitative Sample surveys based on probability theory Sample size has a non-linear relationship with the population being sampled The ‘best’ surveys require every member of a population to have the same, random chance of being surveyed; important to measure response rates; also applicable to Censuses

(Brief) Overview of statistics (ii) Confidence interval: produce a result which is, for example within + or – 5%, or + or – 3% Confidence level or margin of error: measures how ‘sure’ we can be: i.e. that we are 95% certain Sample size: generally the larger the sample, the lower the confidence interval See sample size calculator

(Brief) overview of statistics (iii) Example: 5,000 in your total population Question on, e.g. involvement in volunteering Want a sample to generate a 95% confidence level within a + or – 2.5% confidence interval Requires a random sample giving 1,176 responses Actual sample numbers must account for non-response But a population of 75,000 would only require a response by 1,506; 1,000,000 population: 1,534 These sample response numbers must apply to all sub- populations of interest – e.g. women, aged over 70

Survey design Depends on ‘population’ being sampled: e.g. – Households – Population – Businesses – Footpaths Sampling frame (total ‘population’): – Electoral register (now restricted access) – Address list – Telephone book – BT list very restricted; random dialling? – Business rates; Companies House; Inter Departmental Business Register – Record of public rights of way

Main types of survey Random or quota Face to face: interviewer Postal – written questionnaire Telephone Online/email Critical to measure who responds and who does not; non-response is a major issue and can seriously impact on results Must adjust results for non-response

Face to face Randomly-chosen from total population of interest best – but ‘quota’ often used, e.g. for street surveys Collect information on key demographics such as sex, age, ethnicity, proxy for socio-economic group, such as job or tenure Can adjust results for response rates from different sectors Household-based surveys generally most robust, but expensive: must involve ‘call-back’ for non-response Problems with restricted access to electoral rolls & coverage of electoral rolls Language issues; also rarely cover children If you can afford it – the ‘Rolls Royce’ of surveys

Postal Requires comprehensive and up-to-date address list (students & private lets an issue!); no national Address List Problems with multiple-occupancy homes & institutions Issue of language important Non-response very high; how to attract? Often require second and third postings, so time frame can be long Offer incentives, such as inclusion in prize draw; promise of feed-back on results Always provide stamped, addressed envelopes Keep as short as possible Known response bias – owner-occupiers, professionals, elderly{ high; private renters, unemployed, young{ low

Telephone Most adults have a phone But there is no comprehensive sampling frame/population, even for landlines Landline inadequate – no coverage of young people who mainly use mobiles Text surveys now developing Response bias – so has to be accounted for Random dialling? Best suited to marketing – but increasingly used for short surveys, especially by Political Parties

On-line surveys Data protection may prevent access to emails Bias due to user profile – against elderly, poor & children Have to adjust for differential response rates Survey overload an increasing problem More use of incentives (e.g. entry to prize draw) Free software (e.g. Survey Monkey) suggests surveys are easy – but can be awful! Cheap to run, as data can be downloaded easily into survey analysis programs

Example – National Trust ‘visitor experience survey’ Moved from written questionnaire of visitors to on-line survey, based on a sample of visitors drawn from the barcodes of membership cards ‘zapped’ and then emailed where possible Discovered ‘drop’ in satisfaction levels But the card survey was biased – mainly offered to ‘happy’ smiling visitors! Survey restricted to existing members – rather than the wider audience of ‘one-off’ visitors and prospective members; not very helpful if you want to find out how to attract more members NT ‘VE’ targets for properties are set very high

Questionnaire design (i) Identifying what to include: initial use of ‘focus’ group(s) to bring out key issues valuable But beware recruitment of the ‘loony’ brigade! Order of options/questions important – most people lose interest/concentration towards the end; can rotate order Try to avoid giving an uneven number of choices – e.g. excellent, good, average, poor, very poor – as people tend to go for the middle option: ‘average’ Avoid ‘leading’ questions: ‘You enjoy your visits to Wimpole Hall, don’t you?’ Yes or No

Questionnaire design (ii) Many people will avoid the top & bottom choices in a range; therefore unrealistic for the NT to set targets based on ‘top’ responses alone (i.e. scores on ‘very satisfied’ alone, not counting ‘satisfied’). 7 rather than 5 options? Change order of questions over a survey as a whole Always provide respondents the opportunity to complete an ‘any other comments/views’ section No point in carrying out a survey to support a pre- determined approach – a waste of money! Keep it as short as possible & feed back results

Dealing with response bias Respondents do not fully mirror the population sampled – response bias Even the Population Census (100% of homes) has a biased response (low on young men living in shared, rented homes) If you know the true population shares you can adjust, e.g.: – Elderly home owners are over-represented – Young private renters in multiple-occupancy homes under- represented – Social renters under-represented – Unemployed under-represented – Small service businesses under-represented in business surveys

Example of interesting surveys (i) Future housing/site needs of Travellers living in parts of the East of England, 2007 for next 15 years Very expensive! Involvement of ARU and University of Buckingham specialist academics – Face to face only realistic option – Identified key Traveller with respect of the community(ies) – Recruitment and training of Travellers to carry it out – Had to be person to person because of illiteracy – Used hand-held recorders – Issues of women only interviewing female travellers and men interviewing males – Questions restricted due to community concerns

Examples of interesting surveys (ii) Cambourne new settlement in Cambs: surveyed in 2006 100% of households; known private/social split Postal, with prize draw & free post-back; two waves required to enhance social-rented sector responses (recorded questionnaires by code) Quantitative & qualitative issues covered Critical demographics on who moved in: sex, age, tenure, jobs, occupations, place of work, previous home location & tenure; compared with Census & administrative data to identify response bias Also views on the community and how it was developing; short-comings; plus points, why people moved here Feeds into planning for other new communities, such as Northstowe All new housing estates in Cambs surveyed since 2007 Results available on Cambs CC website

Cambourne survey results 2007 People attracted by Comberton Village College secondary catchment – brought in far more families with children aged 9+ than expected Identified more ‘private-renters’ than expected High levels of commuting to Papworth and Cambridge; also London commuters via St Neots Mainly professionals, with most adults working Residents liked the environment overall But were fed-up with living on a building site, with few community facilities provided as promised

Implications for U3AC surveys Likely to be a biased response from members: but we are the demographic group most likely to complete surveys! Should we be surveying eligible non-members? How do we identify and approach them? Offer incentives? Use Census 2011 data, compared with our membership list, by age/sex/ qualifications /tenure. This is generally of high quality! Use on-line surveys as most potential new members will have email