JOINT EUROPEAN COMMISSION – OECD WORKSHOP ON INTERNATIONAL DEVELOPMENT OF BUSINESS AND CONSUMER TENDENCY SURVEY Task Force on Improvement of Response Rates and Minimisation of Respondent Load Part 1 – Relationship between response rates and data collection methods Bianca Maria Martelli, Institute for Studies and Economic Analysis -ISAE – Rome, Italy BRUSSELS, November 2005 Per aggiungere alla diapositiva il logo della società: Scegliere Immagine dal menu Inserisci Individuare il file con il logo della società Scegliere OK Per ridimensionare il logo: Fare clic su un punto qualsiasi del logo. Attorno all'oggetto verranno visualizzati i quadratini di ridimensionamento. Per ridimensionare l'oggetto, utilizzare i quadratini. Per conservare le proporzioni dell'oggetto da ridimensionare, trascinare i quadratini di ridimensionamento tenendo premuto MAIUSC.

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November DATA COLLECTION METHODS which is the most effective for: Improving response rates Reducing burden Impact on costs Sectoral effects Size effects 2. COMMUNICATION METHODS most effective and cost efficient for: Compulsory surveys Voluntary participation TERMS OF REFERENCE

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November ACKNOLEDGEMENTS The present research has been realised with the substantial contributions of the members of the EC/OECD task force on RR who supported me with extremely useful material, references, information, personal experience and suggestions.

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November WHY IS THE RESPONSE RATE IMPORTANT? Response Rate (RR) one of the most important factors to ensure a high quality of the BTS results (Etter, 2002) Non response rate (NR) is one of the components of non-sampling error that may affect the survey and the most efficient strategy for reducing measurement errors is to eliminate possible causes of such errors during the whole survey design stage (OECD 2003)

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November WHY FIRMS/CONSUMER RESPOND? Conceptual framework for COS/BTS Survey participation (proposed by Willimack 2002, reported by Petroni 2004): Businesses weight response burden against business goals when considering the decision to participate in a survey. Both burden and goals may be impacted by: –factors relating to the external environment, the business and the respondent, which are not under control of the survey organisation –survey design features, which are under the survey organisation control

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November FACTORS INFLUENCING RESPONSE RATES Subject Kind of Survey Questionnaire Reporting, Sampling, Response Unit Target Universe Frame Sampling design RR Data Collection Mode

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November WHAT IS (NON) RESPONSE RATE? CONFUSION ON THE TOPIC : people deals with RR/NR, but not explain what really apply NEED OF CLEAR RR/NR DEFINITION The response rate is the number of complete/incomplete interviews with reporting units divided by the number of eligible reporting units in the sample (AAPOR 2004) SEVERAL KIND OF RR/NR UNIT / ITEM NR

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November KINDS OF NON RESPONSE RATES OECD DEFINITIONS (2003) (1/3): where n’ = number of enterprises which did not submit useable information n = number of enterprises in the survey

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November KINDS OF NON RESPONSE RATES OECD DEFINITIONS (2003)(2/3): In case of unequal sampling fraction but uniform weights a proper measure is the following: where is the sampling fraction of the i th unit

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November KINDS OF NON RESPONSE RATES OECD DEFINITIONS (2003) (3/3): In the most general case, of unequal sampling fraction and reporting units with different weights: where w i is the size weight of the i th unit

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November KINDS OF (NON) RESPONSE RATES OTHER LITERATURE ON RR/NR DEFINITIONS (AAPOR 2004) COOP= Co operation rate Proportion of all cases interviewed of all eligible units ever contacted REF = Refusal Rate Proportion of all cases in which a respondent refuses to do an interview, or breaks-off an interview of all potentially eligible cases CON = Contact Rate Proportion of all cases in which a respondent refuses to do an interview, or breaks-off an interview of all potentially eligible cases

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November KINDS OF (NON) RESPONSE RATES OTHER LITERATURE ON RR/NR DEFINITIONS OVERALL COMPLETION RATE (Lozar Manfreda, 2003) % of responses among all sent invitations or exposed to invitation/eligible, including partial and complete respondents The author assumes that most often researchers when dealing with RR actually refer to it (similar to OECD NR1) FULL COMPLETION RATE Considers only the % of complete (unit) respondents DROP-OUT RATE Referring only to item non response CLICK-OUT RATE For Web surveys (% of accessing the Web questionnaire among all invited)

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November RECOMMENDATION ON (NON) RESPONSE RATES Clearly specify which kind of response rate is applied According to the specific work purposes of this paper: OECD NR1 (or OVERALL COMPLETION RATE) to check the robustness of the data collection process. is useful in the first steps of the survey design to check the robustness of the data collection process. FULL COMPLETION RATE, DROP OUT RATE, CLICK TROUGH are also useful for analysing the DCM

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November DATA COLLECTION MODES TECHNIQUES FACE TO FACE –Historically one of the first applied (since 1912) –Mainly for COS –High quality, High costs CAPI/CASI Computer Assisted Personal Intervieving/Computer Asssisted Self- Interviewing –“Computerised” version of face-to face MAIL –The first historically applied (1788 as reported by De Leeuw, 2005) – Implies the self-administration of questionnaire – Visual approach – “Low” costs, relatively low RR

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November DATA COLLECTION MODES TECHNIQUES FAX – Similar to MAIL – Widely used for BTS TELEPHONE –Useful as supporting measure –“Aural” (instead of “visual”)contact: risk of “loss of context” –Primacy/recency effects (for COS) –Primary role of interviewer CATI (Computer Aided Telephone Interview) –Widely used for COS, less for BTS (Italy, CB) –Allows to quickly collect information –Allows immediate control of inconsistencies –Expensive

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November Focus: CATI AT ISAE for BTS minus: Costs (in comparison with postal tecnique) Possible subjective biases plus: Rapidity Reliability High response rate Less sample biases Easy substitution Since 1988 for Business (since 2002 completely) Since the beginning (1992) for Services

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November Focus: CATI AT ISAE for BTS ISAE Manufacturing and Services CATI Interviews structure OutcomeNumber of phone calls MANUFACTURING (sample size 4100 units) SERVICES (sample size 2100 units) 1- Accepted interview (explanatory material received) Accepted interview (expl. material non received) Accepted first interview Refusal Unreachable Out of target42 Failure 7 -No response Phone busy Automatic replayer Wrong phone number Already contacted No more active7 13 – Exceeding quotas105 Total phone calls Monthly average January – June 2005

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November DATA COLLECTION MODES TECHNIQUES E - MAIL –“Visual”approach, similar to MAIL and FAX –Decreasing trend –Low cost –Confidentiality problems (not allowed at INSEE ) TOUCH-TONE Data Entry/VOICE RECOGNITION Entry –For COS –Low RR –Low cost

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November DATA COLLECTION MODES TECHNIQUES WEB/CAWI –Increasing relevance for BTS (less for COS) –Low cost –Coverage errors –Risk of loss of context in the questionnaire –Higher Unit NR/Lower Item NR (vs MAIL, Annaud, 2005, Etter) –Self selection bias –Only a part of participants (30-50%) freely choose this mode (Etter, Stangl, Kershoff)

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November DATA COLLECTION MODES PROCESS STEPS (timing of interaction) –CO OPERATION / PRE SURVEY NOTIFICATION –OVERALL RESPONSE STAGE –FULL COMPLETION –REMINDINGS –FOLLOW UPs DCM often is a MULTISTAGE PROCESS and for any step more than one mode can be applied MODES the different modes can be applied : - CONCURRENT and/or in a: - SEQUENTIAL way

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November DATA COLLECTION MODE: MIXED MODE (De Leeuw, 2005) MULTIMODE SYSTEM OPTIMIZE data collection procedures REDUCE total survey ERRORS within the available time and costs

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November MIXED MODE DATA COLLECTION FIRM REDUCES BURDEN REDUCES PANEL ATTRITION (TIME IN SAMPLE) INCREASES VOLUNTARY PARTICIPATION EFFECTS ON BURDEN, COSTS, RR INSTITUTION INCREASES RR INCREASES BURDEN INCREASES COSTS ERRORS DUE TO MODE EFFECTS

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November SECTOR AND SIZE EFFECTS SECTOR Firms specific effect is present –Differences (for both Mail and Internet) (Etter, 2002) –Differences (for both CATI and FAX), Italy …..but limited Retail, Services lower RR (but also smaller size…) SIZE Generally inverse correlation between RR and Size –medium/large firms respond more regularly (Etter, 2002) –Burden is greater for small firms (Petroni, 2003) Some exception –South Africa (Kershoff, 2005), Latvia do not notice remarkable influence

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November COMMUNICATION METHODS (CO OPERATION STAGE) Action useful in getting co operation (and reducing NR): –Tailoring contacts to the characteristics of the establishment –Contacting the “right” person (mid level management) –Overcoming the lack of awareness (of the Survey / Institution) –Building the initial relationship (personal contact advisable) –Knowledge of survey characteristics –Survey pre notice

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November COMMUNICATION METHODS: INTERVIEWERS (CO OPERATION STAGE) It is strongly advisable: Use of interviewers –in the initial step, even if the ongoing DCM is different, increases RR (BLS Survey) Training of interviewers –Mainly for BTS, less for COS –Overcome gate keepers: difficult first contact does not necessarily mean unwillingness to participate

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November COMMUNICATION METHODS (CO OPERATION STAGE) all the above mentioned actions are essential also for effectively communicating: COMPULSION (OF RESPONSE) as it may increase the (perceived) burden of respondent (Mixed ) Effects of compulsions : –Increases the Overall Completion Rate (lowers the Unit NR) –May increase the Drop Outs (Item NR) –Positive effects in France, mainly for large firms –Positive effects in Italy (Retail)

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November COMMUNICATION METHODS (CO OPERATION STAGE) Voluntary participation most effective approaches : –Sending the business survey to named individuals and use telephone follow-up methods to encourage responses (Paxon, 1995) –Lowering the response burden (Linn 2003) –Increasing the perceived relevance of the survey to business goals (Linn, 2003) –Adopting a Mixed DCM

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November OVERALL RESPONSE STAGE : PANEL ATTRITION Maintain contacts with participant Renewing interest Using simple and plain questionnaires –Avoiding open ended and “difficult to answer questions” Question wording –accuracy, mainly when using concurrent DCMs Remindings Follow – ups

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November CONCLUSIONS AND RECOMMENDATIONS Efforts should be made in improving preliminary contacts and remindings / follow up actions All the modes have plusses and drawbacks The starting point of view should be the firm (to a lesser extent the consumer): adopting a company-centric point of view –Efforts should be made to allow surveys’ participants to choose the mode they prefer

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys November CONCLUSIONS AND RECOMMENDATIONS Although Internet (online) seems to be the mode of the future, a large part of firms still prefer the most traditional mail questionnaire and a significant part of respondents prefer fax (habit persistence) The future appears to be even more bound to a mixed data collection mode –Mixed mode data collection should be carried out both concurrently and in a multi stage way (as far as possible bound to BTS and COS timeliness)