1. Meth odolo gy and Statis tics | Meta - Analy sis of the Effec tiven ess of Incen tives in Orga nizati onal Surv eys | 1 Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys Vanessa Torres van Grinsven & Joop Hox October 2014 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

2. Introduction Incentives are widely used in individual or household surveys Proven method to improve response rates in these surveys But they are typically not used in official (business ) surveys Systematic synthesis of experimental research on the effectiveness of incentives in organizational surveys is still lacking Meta-analysis of the effect sizes of incentives in experimental research. October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 2 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

3. Short introduction on statistical meta-analysis Effect size (ES) makes statistical meta-analysis possible It is the dependent variable Represents the magnitude and direction of the relationship of interest Must be able to calculate a standard error for that type of ES –The standard error is needed to calculate the ES weights, called inverse variance weights –Because all meta-analytic analyses are weighted taking into account sample size. More information: Lipsey & Wilson (2001) In this case the effect size is the difference in response rate between the treatment groups and the corresponding control groups October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 3 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

4. Data collection Search strategy: –Search made in numerous databases: e.g. Web of Science, WorldCat, EBSCO, Picarta, PsycInfo, SIAM, ESOMAR (the research papers database), AAPOR (Survey Practice database). –Search words: incentive, survey, experiment Eligibility criteria: –Experiment with split ballot –Appropiate control group –RR1 reported or calculable –Sample size and group sizes are reported –Organizational survey –Incentives October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 4 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

5. Organizational survey defined as survey on: i.A commercial population: “Commercial populations are defined as those composed of commercial, industrial, administrative and/or business respondents who receive a questionnaire at their place of employment” Pressley & Tullar, 1977, p. 108. ii.Non-profit organizations iii.Other professionals who receive the questionnaire at their place of employment and about job-related issues. Incentive: a prepaid or promised monetary or non- monetary incentive. October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 5 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

6. Coding of study characteristics Study characteristics were coded to understand if they had an effect on the effect size Coding scheme based on open coding and coding schedules by De Leeuw et al. (2007) and Hox & De Leeuw (1994) Also effect size information was coded (the response rates) Coded study features were: year of publication, country, research organization, sample frame, population type, survey topic, survey method, data type, industry, respondent type, voluntary or mandatory survey, type of incentive, and value of the incentive. Additionally (incentive): prepaid vs promised, monetary vs nonmonetary, type of monetary incentive, type of nonmonetary incentive. October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 6 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

7. Data analysis (1) 34 studies with a total of 68 experimental conditions. Combined sample size of 343.500 respondents including the control groups. The response rates are expressed as proportions, which are then transformed into Z-values using the standard normal distribution. The standardized effect size d is then given by: October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 7 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

8. Data analysis (2) And the standard error of d is given by: Because a larger standard error corresponds to a less precise effect size value, the optimal weights are computed as the inverse of the squared standard error value – hence inverse variance weight (Hedges 1982; Hedges & Olkin, 1985): October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 8 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

9. Data analysis (3) Inverse variance weighted meta-analysis techniques with a random effects model with restricted maximum likelihood estimation (REML) The distribution of effect sizes was heterogeneous Meta-regression and the meta-analytic analogue of ANOVA were used for the moderator analysis SPSS Wilson’s meta-analysis macros (http://mason.gmu.edu/~dwilsonb/ma.html )http://mason.gmu.edu/~dwilsonb/ma.html More information Lipsey & Wilson (2001) October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 9

10. Publication bias A relevant issur in meta-analysis is publication bias It refers to the possibility that relevant studies might not have been published, but disappeared in the file-drawer Publication bias was assessed by inspecting a funnel plot and calculating the fail-safe N. Fail-safe N: the significant effect of the incentive on the response rate would be swiped away if we assume the existence of 426 unpublished studies all reporting no effect highly implausible October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 10 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

11. Publication bias: Funnel plot of the correlation between effect size and sample size October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 11 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

12. Results Sending or promising an incentive does have a small but significant effect on the response rate Mean effect size d =.25 (p<.0001) In terms of Cohen’s (1988) effect size definitions this is a small effect Of all the coded study features only research organization had a significant effect on the effect size (p=.04) Commercial organizations: d =.36 Universities: d =.26 Governmental institutions: d =.10 October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 12 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

13. Value of the incentive: Effect size (d) related to the amount of the monetary incentive October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 13 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

14. Results The largest effect can be expected in commercial surveys; the smallest in surveys by government-related institutions, like NSOs. If we include the response rate as a predictor, the significant differences between organization types disappears completely; while the regression coefficient for the response rate is negative and highly significant Thus: research organizations that already are obtaining high response rates, find it difficult to increase them further We conclude that incentives can be used as a general tool to increase response in organizational surveys with different survey features October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 14 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

15. Discussion What seems important thus is not the factual value (as in an economic exchange) But the psychological value: Reciprocity theory (Gouldner, 1960): a symbolic gift that needs to be reciprocated: A recipient may feel obligated to reciprocate by completing the questionnaire Cost/benefit theory (e.g. Singer, 2011): The incentive is seen as a symbolic benefit that counteracts the perceived costs of completing and returning the questionnaire October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 15 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics

16. References Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum De Leeuw, E., Callegaro, M., Hox, J., Korendijk, E. & Lensveldt-Mulders G. (2007). The Influence of Advance Letters on Response in Telephone Surveys: A Meta-Analysis. Public Opinion Quarterly, 71(3), 413-443. Gouldner, A. W. (1960). The norm of reciprocity: A preliminary statement. American Sociological Review, 25, 161-178. Hedges, L.V. (1982). Estimating effect size form a series of independent experiments. Psychological Bulletin, 92, 490-499. Hedges, L.V. and Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, Florida: Academic Press. Hox, J.J., & De Leeuw, E.D. (1994). A comparison of nonresponse in mail, telephone, and face-to-face surveys. Applying multilevel modeling to meta-analysis. Quality and Quantity, 28, 329-344. Lipsey, M.W. & Wilson, D.B. (2001). Practical Meta-Analysis. Sage Publications. Pressley, M.M. & Tullar, W.L. (1977). A Factor Interactive Investigation of Mail Survey Response Rates from a Commercial Population. Journal of Marketing Research, 14(1), 108- 111. Singer, E. (2011). Toward a Benefit-Cost Theory of Survey Participation: Evidence, Further Tests, and Implications. Journal of Official Statistics, 27, 379-392. October 2014 Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys | 16 Faculty of Social and Behavioural Sciences Department of Methodology and Statistics