1. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Task force on improvement of response rates and minimisation of respondent load Assessing and minimising the impact of non-response on survey estimates Richard McKenzie OECD

2. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Terms of reference –How to asses the impact (bias) that non-response can have on survey estimates. Work on this issue should take into account both the impact of the overall level of non-response (e.g. 10% vs 50%) and non-response by type of business or consumer (e.g. large businesses vs small) and type of variable / question. –Develop methods to minimise the impact (bias) of non- response (e.g. imputation methods, estimation methodologies).

3. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Task force work to address the issue 3 main activities: Internet research to find relevant documents on the topic Provision of documents by task force members (from their work or work they were aware of) Specific questionnaire to a sample of institutes to collect information relevant to the terms of reference

4. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Main issues explored Defining and observing non-response rates Weighting methodologies used and their relationship to the treatment of non-response Literature review to assess the likely impact on non- response on survey estimates Conclusions and recommendations for future work

5. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Defining non-response rates OECD (2003)

6. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 InstituteSurvey & Response rate Confederation of British IndustryIndustry 45%, Retail 22%, Services 20% South Africa: Bureau for Econ Research 40-45% across retail / wholesale, manufacturing, motor trade and contractors (building related). Canada: Statistics BureauManufacturing 55% Finland: Statistics BureauConsumer Survey 74% Austria: WIFOManufacturing 30%, Construction 30%, Services 34%. France: INSEEIndustry 81%, Retail 82% (5,000), Services 76%, Construction 84% Netherlands: Statistics OfficeIndustry 90% Slovenia: Statistics OfficeConsumer Survey 68%, Industry 91%, Construction 92%, Services 92%, Retail 82% Japan: Central BankWhole economy: TANKAN survey 95%. Germany: IFOManufacturing 90%, Construction 70% Slovak Republic: Statistics OfficeIndustry 75%, Construction 85%, Retail 60%, Services 63% (500) Statistics NorwayIndustry 85% ChinaWhole economy 95%

7. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Issues affecting response rates Authority data collected under Survey resources available Timeliness requirements (limits follow ups) –Regardless of the reasons, representativeness of the responding sample is the key issue

8. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Weighting methodology Weighting (and estimation) methodology and its relationship to sampling error and non-response bias is a major issue for quantitative business surveys –Doesn’t appear to have been given the same attention in qualitative business surveys Key role of qualitative business surveys is to give leading indicators of likely movements in macro- economic (qualitative) variables

9. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Weighting methodology: OECD guidelines Business weight should be composed of: Sampling probability / fraction Business size (usually employment or turnover) Above weights are used to form survey estimates at the Branch / group level, which should then be weighted together to form aggregates (e.g. total manufacturing)

10. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Importance of the sampling fraction Sampling fraction may be difficult to use as: Poor quality (or non existent) survey frame Infrequent sample updates Business selection methods (e.g. quota sampling, recruitment to panel) Therefore it may be considered unobtainable or irrelevant?

11. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Importance of the sampling fraction If small and large business are included in the same group / cell estimate, weighted only by business size then the resulting estimates are biased and not representative of the population. –Must estimate a weight for each (group) of business in the sample which accounts for other similar size business in the population that they are representing –Otherwise you are better off excluding small businesses from the scope of the survey (cutoff sampling) and saving resources.

12. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Treatment of non-response: OECD (2003) If non–response can be expected to be systematic, in the sense that units which have had or are expecting an especially good or bad development are also an unduly large part of non–response, then special measures need to be taken in order to avoid bias. One possible approach is to construct a separate “non–response stratum”, and take a repeat sub–sample from this stratum for which further strong efforts are made to collect data. This information can then be used to make separate estimates for this “non–response stratum”.

13. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 The ‘missing at random’ assumption European Commission – harmonised guidelines Non responding businesses are ignored in applying survey weights and forming survey estimates Assumes the average (weighted) distribution of answers from responding businesses is representative of non-responding businesses –Is this a reasonable assumption? It was used by all institutes surveyed for BTS Consumer surveys can use population estimates from census’s for post stratification.

14. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Is the ‘missing at random’ assumption reasonable? Appears to have been very little work done to explore this issue for business tendency surveys –Wang (2004) attempted this for Statistics Norway, but the study was based on only one cycle of the survey and involved a number of simplifying assumptions –Nearest neighbour imputation based on employment size showed some evidence of difference in response pattern between size of unit – although impact on estimates was small.

15. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Is the ‘missing at random’ assumption reasonable? Pellisier (2005) looked at the impact of response patterns on the Retail Business Confidence Indicator in the South African Retail Trade survey –3 groups of respondents: active (respond more than 75% of the time); less active (30 – 75%); occasional (< 30%) –Some evidence of differing response patterns, more details on this study in the next presentation!

16. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Is the ‘missing at random’ assumption reasonable? INSEE: ‘constant sample’ imputation method “As enterprises do not systematically answer each monthly survey, or may submit responses after the cutoff for publication, only taking into account answers from businesses responding to a particularly monthly cycle of a survey can lead to a false diagnosis of changes in the business climate if this is due only to a change in structure of the respondents” –Constant sample method explicitly imputes for both complete and item non-response based on the historical response pattern of the business

17. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Non-response bias for consumer surveys An extensive study by Curtin et al.(2000) found: –Some small difference in Index of Consumer Sentiment (ICS) between easier and harder to interview respondents, but there was a negligible impact on the overall index if harder to interview respondents are excluded –Difference in the ICS between easier and harder to contact respondents was constant over time – implying there is no non-response bias for time series (i.e. estimates of change) Conclusion was that ICSs ability to predict future changes in economic conditions is unlikely to be affected by non-response bias

18. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Conclusions and recommendations Non-response bias appears to be well contained in consumer surveys, provided institutes make full use of census data to post-stratify / calibrate survey weights to ensure the responding sample is representative of the population. –The methodologies of Statistics Finland and Statistics Slovenia reviewed in this study are cited as good practice references

19. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Conclusions and recommendations If aggregation to the branch or cell level combines businesses chosen with different probabilities or in different ways (e.g. large and small businesses), then an estimate of sampling fraction must be a factor in the weighting process – otherwise survey estimates will be biased. –If institutes are not doing this, then small businesses are most likely unrepresented in the survey estimates to such an extent that their inclusion in the survey is irrelevant

20. Joint OECD / European Commission workshop on international development in business and consumer tendency surveys Nov 14-15 2005 Conclusions and recommendations The limited research performed so far suggests that the missing at random assumption for treating non- response in business tendency surveys may not hold, therefore: –Institutes are encouraged to undertake more research on this issue, particularly through making estimates for businesses with different response behaviours (e.g. regular and irregular respondents) –Institutes are encouraged to experiment in applying the constant sample imputation methodology developed by INSEE, which may be effective in reducing the impact of non-response bias.