1. JOINT EUROPEAN COMMISSION – OECD WORKSHOP ON INTERNATIONAL DEVELOPMENT OF BUSINESS AND CONSUMER TENDENCY SURVEY Task Force on Improvement of Response Rates and Minimisation of Respondent Load Part 1 – Relationship between response rates and data collection methods Bianca Maria Martelli, Institute for Studies and Economic Analysis -ISAE – Rome, Italy BRUSSELS, 14-15 November 2005 Per aggiungere alla diapositiva il logo della società: Scegliere Immagine dal menu Inserisci Individuare il file con il logo della società Scegliere OK Per ridimensionare il logo: Fare clic su un punto qualsiasi del logo. Attorno all'oggetto verranno visualizzati i quadratini di ridimensionamento. Per ridimensionare l'oggetto, utilizzare i quadratini. Per conservare le proporzioni dell'oggetto da ridimensionare, trascinare i quadratini di ridimensionamento tenendo premuto MAIUSC.

2. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 2 1. DATA COLLECTION METHODS which is the most effective for: Improving response rates Reducing burden Impact on costs Sectoral effects Size effects 2. COMMUNICATION METHODS most effective and cost efficient for: Compulsory surveys Voluntary participation TERMS OF REFERENCE

3. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 3 ACKNOLEDGEMENTS The present research has been realised with the substantial contributions of the members of the EC/OECD task force on RR who supported me with extremely useful material, references, information, personal experience and suggestions.

4. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 4 WHY IS THE RESPONSE RATE IMPORTANT? Response Rate (RR) one of the most important factors to ensure a high quality of the BTS results (Etter, 2002) Non response rate (NR) is one of the components of non-sampling error that may affect the survey and the most efficient strategy for reducing measurement errors is to eliminate possible causes of such errors during the whole survey design stage (OECD 2003)

5. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 5 WHY FIRMS/CONSUMER RESPOND? Conceptual framework for COS/BTS Survey participation (proposed by Willimack 2002, reported by Petroni 2004): Businesses weight response burden against business goals when considering the decision to participate in a survey. Both burden and goals may be impacted by: –factors relating to the external environment, the business and the respondent, which are not under control of the survey organisation –survey design features, which are under the survey organisation control

6. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 6 FACTORS INFLUENCING RESPONSE RATES Subject Kind of Survey Questionnaire Reporting, Sampling, Response Unit Target Universe Frame Sampling design RR Data Collection Mode

7. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 7 WHAT IS (NON) RESPONSE RATE? CONFUSION ON THE TOPIC : people deals with RR/NR, but not explain what really apply NEED OF CLEAR RR/NR DEFINITION The response rate is the number of complete/incomplete interviews with reporting units divided by the number of eligible reporting units in the sample (AAPOR 2004) SEVERAL KIND OF RR/NR UNIT / ITEM NR

8. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 8 KINDS OF NON RESPONSE RATES OECD DEFINITIONS (2003) (1/3): where n’ = number of enterprises which did not submit useable information n = number of enterprises in the survey

9. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 9 KINDS OF NON RESPONSE RATES OECD DEFINITIONS (2003)(2/3): In case of unequal sampling fraction but uniform weights a proper measure is the following: where is the sampling fraction of the i th unit

10. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 10 KINDS OF NON RESPONSE RATES OECD DEFINITIONS (2003) (3/3): In the most general case, of unequal sampling fraction and reporting units with different weights: where w i is the size weight of the i th unit

11. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 11 KINDS OF (NON) RESPONSE RATES OTHER LITERATURE ON RR/NR DEFINITIONS (AAPOR 2004) COOP= Co operation rate Proportion of all cases interviewed of all eligible units ever contacted REF = Refusal Rate Proportion of all cases in which a respondent refuses to do an interview, or breaks-off an interview of all potentially eligible cases CON = Contact Rate Proportion of all cases in which a respondent refuses to do an interview, or breaks-off an interview of all potentially eligible cases

12. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 12 KINDS OF (NON) RESPONSE RATES OTHER LITERATURE ON RR/NR DEFINITIONS OVERALL COMPLETION RATE (Lozar Manfreda, 2003) % of responses among all sent invitations or exposed to invitation/eligible, including partial and complete respondents The author assumes that most often researchers when dealing with RR actually refer to it (similar to OECD NR1) FULL COMPLETION RATE Considers only the % of complete (unit) respondents DROP-OUT RATE Referring only to item non response CLICK-OUT RATE For Web surveys (% of accessing the Web questionnaire among all invited)

13. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 13 RECOMMENDATION ON (NON) RESPONSE RATES Clearly specify which kind of response rate is applied According to the specific work purposes of this paper: OECD NR1 (or OVERALL COMPLETION RATE) to check the robustness of the data collection process. is useful in the first steps of the survey design to check the robustness of the data collection process. FULL COMPLETION RATE, DROP OUT RATE, CLICK TROUGH are also useful for analysing the DCM

14. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 14 DATA COLLECTION MODES TECHNIQUES FACE TO FACE –Historically one of the first applied (since 1912) –Mainly for COS –High quality, High costs CAPI/CASI Computer Assisted Personal Intervieving/Computer Asssisted Self- Interviewing –“Computerised” version of face-to face MAIL –The first historically applied (1788 as reported by De Leeuw, 2005) – Implies the self-administration of questionnaire – Visual approach – “Low” costs, relatively low RR

15. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 15 DATA COLLECTION MODES TECHNIQUES FAX – Similar to MAIL – Widely used for BTS TELEPHONE –Useful as supporting measure –“Aural” (instead of “visual”)contact: risk of “loss of context” –Primacy/recency effects (for COS) –Primary role of interviewer CATI (Computer Aided Telephone Interview) –Widely used for COS, less for BTS (Italy, CB) –Allows to quickly collect information –Allows immediate control of inconsistencies –Expensive

16. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 16 Focus: CATI AT ISAE for BTS minus: Costs (in comparison with postal tecnique) Possible subjective biases plus: Rapidity Reliability High response rate Less sample biases Easy substitution Since 1988 for Business (since 2002 completely) Since the beginning (1992) for Services

17. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 17 Focus: CATI AT ISAE for BTS ISAE Manufacturing and Services CATI Interviews structure OutcomeNumber of phone calls MANUFACTURING (sample size 4100 units) SERVICES (sample size 2100 units) 1- Accepted interview (explanatory material received) 40321846 2 - Accepted interview (expl. material non received) 3193 3 - Accepted first interview4636 4 - Refusal6964 5 - Unreachable129730 6 -Out of target42 Failure 7 -No response 51 75 8 -Phone busy37 9 - Automatic replayer9 10 - Wrong phone number72 11 - Already contacted1 12 - No more active7 13 – Exceeding quotas105 Total phone calls43593076 Monthly average January – June 2005

18. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 18 DATA COLLECTION MODES TECHNIQUES E - MAIL –“Visual”approach, similar to MAIL and FAX –Decreasing trend –Low cost –Confidentiality problems (not allowed at INSEE ) TOUCH-TONE Data Entry/VOICE RECOGNITION Entry –For COS –Low RR –Low cost

19. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 19 DATA COLLECTION MODES TECHNIQUES WEB/CAWI –Increasing relevance for BTS (less for COS) –Low cost –Coverage errors –Risk of loss of context in the questionnaire –Higher Unit NR/Lower Item NR (vs MAIL, Annaud, 2005, Etter) –Self selection bias –Only a part of participants (30-50%) freely choose this mode (Etter, Stangl, Kershoff)

20. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 20 DATA COLLECTION MODES PROCESS STEPS (timing of interaction) –CO OPERATION / PRE SURVEY NOTIFICATION –OVERALL RESPONSE STAGE –FULL COMPLETION –REMINDINGS –FOLLOW UPs DCM often is a MULTISTAGE PROCESS and for any step more than one mode can be applied MODES the different modes can be applied : - CONCURRENT and/or in a: - SEQUENTIAL way

21. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 21 DATA COLLECTION MODE: MIXED MODE (De Leeuw, 2005) MULTIMODE SYSTEM OPTIMIZE data collection procedures REDUCE total survey ERRORS within the available time and costs

22. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 22 MIXED MODE DATA COLLECTION FIRM REDUCES BURDEN REDUCES PANEL ATTRITION (TIME IN SAMPLE) INCREASES VOLUNTARY PARTICIPATION EFFECTS ON BURDEN, COSTS, RR INSTITUTION INCREASES RR INCREASES BURDEN INCREASES COSTS ERRORS DUE TO MODE EFFECTS

23. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 23 SECTOR AND SIZE EFFECTS SECTOR Firms specific effect is present –Differences (for both Mail and Internet) (Etter, 2002) –Differences (for both CATI and FAX), Italy …..but limited Retail, Services lower RR (but also smaller size…) SIZE Generally inverse correlation between RR and Size –medium/large firms respond more regularly (Etter, 2002) –Burden is greater for small firms (Petroni, 2003) Some exception –South Africa (Kershoff, 2005), Latvia do not notice remarkable influence

24. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 24 COMMUNICATION METHODS (CO OPERATION STAGE) Action useful in getting co operation (and reducing NR): –Tailoring contacts to the characteristics of the establishment –Contacting the “right” person (mid level management) –Overcoming the lack of awareness (of the Survey / Institution) –Building the initial relationship (personal contact advisable) –Knowledge of survey characteristics –Survey pre notice

25. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 25 COMMUNICATION METHODS: INTERVIEWERS (CO OPERATION STAGE) It is strongly advisable: Use of interviewers –in the initial step, even if the ongoing DCM is different, increases RR (BLS Survey) Training of interviewers –Mainly for BTS, less for COS –Overcome gate keepers: difficult first contact does not necessarily mean unwillingness to participate

26. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 26 COMMUNICATION METHODS (CO OPERATION STAGE) all the above mentioned actions are essential also for effectively communicating: COMPULSION (OF RESPONSE) as it may increase the (perceived) burden of respondent (Mixed ) Effects of compulsions : –Increases the Overall Completion Rate (lowers the Unit NR) –May increase the Drop Outs (Item NR) –Positive effects in France, mainly for large firms –Positive effects in Italy (Retail)

27. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 27 COMMUNICATION METHODS (CO OPERATION STAGE) Voluntary participation most effective approaches : –Sending the business survey to named individuals and use telephone follow-up methods to encourage responses (Paxon, 1995) –Lowering the response burden (Linn 2003) –Increasing the perceived relevance of the survey to business goals (Linn, 2003) –Adopting a Mixed DCM

28. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 28 OVERALL RESPONSE STAGE : PANEL ATTRITION Maintain contacts with participant Renewing interest Using simple and plain questionnaires –Avoiding open ended and “difficult to answer questions” Question wording –accuracy, mainly when using concurrent DCMs Remindings Follow – ups

29. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 29 CONCLUSIONS AND RECOMMENDATIONS Efforts should be made in improving preliminary contacts and remindings / follow up actions All the modes have plusses and drawbacks The starting point of view should be the firm (to a lesser extent the consumer): adopting a company-centric point of view –Efforts should be made to allow surveys’ participants to choose the mode they prefer

30. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005 30 CONCLUSIONS AND RECOMMENDATIONS Although Internet (online) seems to be the mode of the future, a large part of firms still prefer the most traditional mail questionnaire and a significant part of respondents prefer fax (habit persistence) The future appears to be even more bound to a mixed data collection mode –Mixed mode data collection should be carried out both concurrently and in a multi stage way (as far as possible bound to BTS and COS timeliness)