3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP 11.90 LEFT MARGIN 11.90 RIGHT MARGIN © TNS 2014 Fieldwork effort, response rate and the distribution.

Slides:

Advertisements

Similar presentations

Multiple Indicator Cluster Surveys Survey Design Workshop

Advertisements

Survey design. What is a survey?? Asking questions – questionnaires Finding out things about people Simple things – lots of people What things? What people?

The Marginal Utility of Income Richard Layard* Guy Mayraz* Steve Nickell** * CEP, London School of Economics ** Nuffield College, Oxford.

Cigarette smoking prevalence 1 Graph shows prevalence estimate and upper and lower 95% confidence intervals Base: All adults.

Correcting for Common Causes of Nonresponse and Measurement Error Andy Peytchev International Total Survey Error Workshop Stowe, June 15, 2010.

Survey Methodology Nonresponse EPID 626 Lecture 6.

Design of Experiments Lecture I

Self-Administered Surveys: Mail Survey Methods ChihChien Chen Lauren Teffeau Week 10.

Weighting Methodology for the Private Landlords Survey Robert Bucknall, ONS.

The estimation strategy of the National Household Survey (NHS) François Verret, Mike Bankier, Wesley Benjamin & Lisa Hayden Statistics Canada Presentation.

Lecture 28 Categorical variables: –Review of slides from lecture 27 (reprint of lecture 27 categorical variables slides with typos corrected) –Practice.

1 Sampling Telephone Numbers and Adults, and Interview Length, and Weighting in the California Health Survey Cell Phone Pilot Study J. Michael Brick, Westat.

The English Longitudinal Study of Ageing (ELSA) Sample design & response Shaun Scholes NatCen.

Plausible values and Plausibility Range 1. Prevalence of FSWs in some west African Countries 2 0.1% 4.3%

NLSCY – Non-response. Non-response There are various reasons why there is non-response to a survey  Some related to the survey process Timing Poor frame.

Statistical Concepts (continued) Concepts to cover or review today: –Population parameter –Sample statistics –Mean –Standard deviation –Coefficient of.

Chapter 11 Sampling Design. Chapter 11 Sampling Design.

The Excel NORMDIST Function Computes the cumulative probability to the value X Business Statistics: A First Course, 5e © 2009 Prentice-Hall, Inc

Cluster Sampling The basics. What are we trying to achieve in a survey? A sample that is representative of the larger population A sample that is representative.

Error and Sample Sizes PHC 6716 June 1, 2011 Chris McCarty.

FINAL REPORT: OUTLINE & OVERVIEW OF SURVEY ERRORS

Scot Exec Course Nov/Dec 04 Ambitious title? Confidence intervals, design effects and significance tests for surveys. How to calculate sample numbers when.

18/08/2015 Statistics Canada Statistique Canada Responsive Collection Design (RCD) for CATI Surveys and Total Survey Error (TSE) François Laflamme International.

An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of.

International Health Policy Program -Thailand Present by : Wittaya Wisutruangdaj Sopit Nasueb Alcohol control policies and alcohol consumption by youth:

Sampling. Concerns 1)Representativeness of the Sample: Does the sample accurately portray the population from which it is drawn 2)Time and Change: Was.

The effect of uncertainty on fuel poverty statistics Laura Williams, Department of Energy and Climate Change GSS Methodology Symposium, 6 th July 2011.

Nonresponse issues in ICT surveys Vasja Vehovar, Univerza v Ljubljani, FDV Bled, June 5, 2006.

Review of Chapters 1- 5 We review some important themes from the first 5 chapters 1.Introduction Statistics- Set of methods for collecting/analyzing data.

PARAMETRIC STATISTICAL INFERENCE

The 2006 National Health Interview Survey (NHIS) Paradata File: Overview And Applications Beth L. Taylor 2008 NCHS Data User’s Conference August 13 th,

Confidence Interval Proportions.

2005 Census Survey of Maricopa Household Survey Sample Design Overview Sample Size Assumptions.

1 Things That May Affect Estimates from the American Community Survey.

Scot Exec Course Nov/Dec 04 Survey design overview Gillian Raab Professor of Applied Statistics Napier University.

The Role of Over-Sampling of the Wealthy in the SCF Arthur B. Kennickell Federal Reserve Board Opinions are those of the presenter alone and do not necessarily.

1 rules of engagement no computer or no power → no lesson no SPSS → no lesson no homework done → no lesson GE 5 Tutorial 5.

Bias in Surveys MDM4U – Mathematics of Data Management.

The Challenge of Non- Response in Surveys. The Overall Response Rate The number of complete interviews divided by the number of eligible units in the.

Chapter 15 Sampling and Sample Size Winston Jackson and Norine Verberg Methods: Doing Social Research, 4e.

5-4-1 Unit 4: Sampling approaches After completing this unit you should be able to: Outline the purpose of sampling Understand key theoretical.

Chapter 6: 1 Sampling. Introduction Sampling - the process of selecting observations Often not possible to collect information from all persons or other.

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN ©TNS 2013 Are ‘better’ interviewers more successful at.

Summarizing Risk Analysis Results To quantify the risk of an output variable, 3 properties must be estimated: A measure of central tendency (e.g. µ ) A.

1 Responsive Design and Survey Management in the National Survey of Family Growth (NSFG) William D. Mosher, NCHS FCSM Statistical Policy Seminar Washington,

Confidence Intervals for a Population Mean, Standard Deviation Unknown.

Basic Business Statistics, 8e © 2002 Prentice-Hall, Inc. Chap 1-1 Inferential Statistics for Forecasting Dr. Ghada Abo-zaid Inferential Statistics for.

1 STK 4600: Statistical methods for social sciences. Survey sampling and statistical demography Surveys for households and individuals.

Sampling technique  It is a procedure where we select a group of subjects (a sample) for study from a larger group (a population)

1/53: Topic 3.1 – Models for Ordered Choices Microeconometric Modeling William Greene Stern School of Business New York University New York NY USA William.

An assessment of the robustness of weights in the Famille et Employeurs survey Nicolas Razafindratsima & Elisabeth Morand.

NCRM is funded by the Economic and Social Research Council 1 Interviewers, nonresponse bias and measurement error Patrick Sturgis University of Southampton.

6. Ordered Choice Models. Ordered Choices Ordered Discrete Outcomes E.g.: Taste test, credit rating, course grade, preference scale Underlying random.

Guillaume Osier Institut National de la Statistique et des Etudes Economiques (STATEC) Social Statistics Division Construction.

[Part 5] 1/43 Discrete Choice Modeling Ordered Choice Models Discrete Choice Modeling William Greene Stern School of Business New York University 0Introduction.

Numeracy & Quantitative Methods Laura Lake. Postal surveys or questionnaires are a form of self- completion or self-administered questionnaire. Self-completion:

TOPIC - Page 1 Representativity Indicators for Survey Quality R-indicators and fieldwork monitoring Koen Beullens & Geert Loosveldt K.U.Leuven.

Chapter 22 Inferential Data Analysis: Part 2 PowerPoint presentation developed by: Jennifer L. Bellamy & Sarah E. Bledsoe.

The impact of using the web in a mixed mode follow-up of a longitudinal birth cohort study: Evidence from the National Child Development Study Presenter:

Maintaining high response rates – is it worth the effort?

Context for the experiment?

William Greene Stern School of Business New York University

A note on questionnaire design and adminstration

Obtaining information on non-responders: a development of the basic question approach for surveys of individuals Patten Smith (Ipsos MORI) Richard Harry.

Workshop A2: Non-response Challenges in GPS Surveys

ADRCE, University of Southampton

New Techniques and Technologies for Statistics 2017 Estimation of Response Propensities and Indicators of Representative Response Using Population-Level.

Analyzing Reliability and Validity in Outcomes Assessment

5: Introduction to estimation

Presentation transcript:

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Fieldwork effort, response rate and the distribution of survey outcomes Joel Williams, TNS BMRB Patrick Sturgis & Ian Brunton-Smith, University of Southampton ESRC Research Methods Festival, July 8 th 2014

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 The value of probability samples 2 Bedrock of social research Uncertainty can be quantified via confidence intervals Confidence intervals assume all sampled cases have same response propensity If this assumption does not hold, a high response rate will limit NR bias

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Maximising the response rate 3 Need to tackle multiple reasons for not responding Make repeated visits/calls on different days/times of day ‘Reminders’ for self-completion surveys Respondent engagement strategies Incentives ‘Late’ respondents can be costly to recruit so they need to add value…

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 The increasing difficulty of fieldwork (BCS 2011) 4 (% of sampled households)

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 A formula for non response bias 5 R xp S x S p p NR bias (x) =

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 A general metric for the risk of non-response bias 6 R xp S x S p p CV(p) =

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Estimating response propensity variation (S p ) 7 Generally estimated using logistic regression models May estimate using calibration weights if sample frame is uninformative Low variation may be real or due to poor model CV(p*): partial estimate of p Single variable versions: e.g. age group

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 BCS: Response rate progression (age groups) 8 Response rate (%) BCS CV(p*) =.53

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 BCS: Response rate progression (age groups) 9 Response rate (%) BCS CV(p*) =.13

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 BCS v Taking Part (age groups) 10 Response rate (%) BCS CV(p*) =.13 TP CV(p*) =.11

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Uses simple paradata (‘visits’ per case) to assess impact of effort on a continuum from ‘barely any’ to ‘a great deal’ Assesses how impact varies by topic and question/response structure [early stages] Our analysis 11 Four major FTF surveys Representative set of variables from each (total = 393)

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Survey details 12 British Crime Survey Taking PartBritish Election Study Skills for Life PopulationE+W 16+E 16+GB 18+E TimingIssued in 2011 Interviewed in 2011 Spring Sample size46,78510,9941,8117,230 Response rate76%59%54%~57% Incentives?Stamps (U) Stamps (U) +£5 (C) £5-10 (C)£10 (C)

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Model structure (depvar: x v –x) 13 Question x S.E 1 visit Up to 2 visits Up to 3 visits Up to 5 visits Response category x=1 S.E 1 visit Up to 2 visits Up to 3 visits Up to 5 visits Response category x=2

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Early findings 14 Coding of question and response structure complete Some evidence that questions about beliefs are most sensitive to fieldwork effort …particularly if a multi-point scale is used Topic level coefficients constrained by limited set of topics in 4 surveys Impact of fieldwork effort (# of visits) not particularly substantial even with empty model…

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Estimates of bias after 1 visit to each address (n=393) 15

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Estimates of bias after 1 visit to each address (n=393) 16 Mean ‘bias’ = 1.41% w/controls = 1.14%

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Estimates of bias after 2 visits to each address (n=393) 17

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Estimates of bias after 2 visits to each address (n=393) 18 Mean ‘bias’ = 0.97% w/controls = 0.70%

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Estimates of bias after 3 visits to each address (n=393) 19

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Estimates of bias after 3 visits to each address (n=393) 20 Mean ‘bias’ = 0.71% w/controls = 0.44%

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Estimates of bias after 5 visits to each address (n=393) 21

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Estimates of bias after 5 visits to each address (n=393) 22 Mean ‘bias’ = 0.39% w/controls = 0.12%

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Points for discussion I 23 Efforts to maximise response are costly Generally lead only to marginal changes in survey estimates Modelling method can help quantify by question/topic typologies Important to remember that bias may be present even if maximised response Potential for larger bias scores for sub- groups “ “

3.14 X AXIS 6.65 BASE MARGIN 5.95 TOP MARGIN 4.52 CHART TOP LEFT MARGIN RIGHT MARGIN © TNS 2014 Points for discussion II 24 Is this evidence in favour of using cheaper data collection modes with lower response rates? Not straightforward to generalise these findings to other modes But possible that non-response bias is not as great as feared Is the non-statistical value of a high response rate enough to warrant the expenditure? “ “