An integrated information system on survey quality: the experience of the Italian survey ‘Holidays and trips’ by Monica Perez 7th International Forum on Tourism Statistics ‘ For Usability, Comparability and Sustainability’ Stockholm, Sweden 9-11 June 2004 Italian National Statistical Institute Authors: Perez M., Head of Unit Mobility and tourism, Division for Survey on Households Dattilo B., Statistician, Unit Mobility and tourism, Division for Survey on Households Simeoni G., Statistician, Unit Methodologies and standard for quality, Division for Methodologies and Information Technologies

Table of contents The survey ‘Holidays and trips’ (H&T): its context, framework and main features The monitoring system on survey quality adopted for H&T The Information System for Survey Documentation (SIDI) SIDI quality indicators How SIDI quality indicators have been implemented in H&T Ad hoc quality indicators adopted by H&T

The H&T main features quarterlyhousehold Section C (demand side) of Directive 95/57/EUH&T is a quarterly survey carried out on a household sample in order to comply to Section C (demand side) of Directive 95/57/EU overnight staysH&T is mainly aimed at getting info on overnight stays (personal/business), domestics and outbound tourism flows, tourist patterns, etc… sampleThe sample : 14,000 »size: 14,000 households (3,500 per quarter) stratified20 6 »design: stratified by 20 regions and 6 socio-demographic typologies of municipality frameTelecom company’s subscribersThe frame is a national (sub) list of Telecom company’s subscribers (private users) Computer Assisted Telephone Interviews (CATI)The survey is carried out by Computer Assisted Telephone Interviews (CATI)

H&T monitoring needs The results can be affected by non-sampling errors  the possibility to have info on data collection - survey contents and how the survey is proceeding- provides an added value to understand the investigated phenomena Survey ‘field’ period: 20 days every 3 months  timeliness in detecting problems arising during the data collection phase Interviews carried out by a private firm  strengthen control activity on interview’s phase and monitoring interviewers’ performance

monitoring activity carried out by Istat specialized staff directly at the office of the firm in charge to carry out the interviews monitoring activity carried out by the analysis of a set of statistical indicators and reports on interviewers performance and data collection for the qualitative analysis The monitoring system on survey quality adopted for ‘Holidays and trips’ (H&T) monitoring activity carried out by control telephone calls to the interviewed households made by Istat staff monitoring activity carried out by a toll free telephone number working during the whole survey period 4 monitoring activities

daily Reporting and statistical indicators daily provided for H&T quality monitoring system 1.data collection reporting Tools Tools: statistical frequency distributions (a.v. and %) on each variable 2.call outcomes rates reporting Tools Tools: a set of statistical indicators on survey performance 3.data files on call outcomes Tools Tools: Call final outcome (CFO):Call final outcome (CFO): a database containing info related to telephone calls that have provided a final outcome Call provisional outcome (CPO):Call provisional outcome (CPO): a database containing info on each call outcome (calls with a final outcome and calls in a temporary status)

The Information System for Survey Documentation: SIDI SIDISIDI is the Istat centralised system for survey documentation and quality assessment: –Allowing comparisons on the quality of Istat surveys PurposePurpose: supporting survey managers in quality control activities –Monitor the production process and quality over time Main featuresMain features: –Process-oriented system –Integrated management system of metadata and quality indicators ArchitectureArchitecture: two subsystems –SIDI1: management subsystem developed in Oracle forms –SIDI-TOP: navigation subsystem developed in Java available on Istat intranet

SIDI contents MetadataMetadata available for each survey: –Information contents (observed phenomena, analysis units…) –Production process description (operations – e.g. data collection mode -; quality control actions – e.g. interviewers monitoring -…) Standard quality indicatorsStandard quality indicators, a set for each of the most important survey phases: –Frame Relevant for the monitoring –Data Collection of data collection phase –Data Entry –Editing & Imputation –Timeliness & Punctuality –Costs –Coherence

SIDI Standard quality indicators Main featuresMain features: –Standard formulae, calculated by the system: survey managers provide numerators and denominators –Tabular and graphical representations of the indicators provided by SIDI-TOP: Over time monitoring and geographical analysis Comparisons among different surveys Comparisons with average values Implementation for Holidays and TripsImplementation for Holidays and Trips: –First CATI survey -using the phone book as a frame- that has calculated SIDI indicators on frame and data collection adaptation of SIDI general definitions to specific survey situation

How SIDI quality indicators have been implemented in H&T SIDI units’ classification has been revised and improved to better fit H&T’s needs and those of similar surveys as well Operative definitions of SIDI quantities to calculate indicators have been provided A re-allocation of some call outcomes with reference to SIDI categories Basic quantities to calculate SIDI indicators have been identified and processed by call outcomes data files (CFO & CPO) SIDI indicators have been implemented in the daily reporting for H&T quality monitoring

IndicatorComputation FRAME Total Units Resolved Rateresolved units/ total units Frame Error Rate(out of scope units + no contacts due to frame errors) / resolved units Out of Scope Rateout of scope units / resolved units Non Existent Ratenon existent units / resolved units Change in Status Ratechanges in status / resolved units Out of target rateout of target units/ resolved units No Contacts Due to Frame Errors Rate no contacts due to frame errors / resolved units An overview of SIDI indicators

DATA COLLECTION Total Units Resolved Rateresolved units/ total units Total Nonresponse Rate(nonrespondents units + unresolved units) / (in scope units + unresolved units) Response Raterespondent units / (in scope units + unresolved units) Total Nonresponse Rate excluding No Contacts Due to Frame Errors (nonrespondents + unresolved units – no contacts due to frame errors)/(in scope units + unresolved units - no contacts due to frame errors) Total Nonresponse Rate referred to In Scope Units nonrespondents / in scope units Refusal Raterefusals / in scope units No Contacts Rateno contacts / in scope units No Contacts Rate Due to Frame Errors referred to In Scope Units no contacts due to frame errors/in scope units No Contacts Rate Due to Other Reasons no contacts due to other reasons/in scope units Other Reasons Nonresponse Rateother nonrespondents / in scope units IndicatorComputation

Response Rate. Years 2001, 2002

Refusal Rate. Years 2001, 2002

SIDI H&T Response rate by geographical breakdown

Ad hoc quality indicators adopted by H&T Additional indicators not supplied by SIDI are recommended to fully satisfy each quality monitoring needs The core set of indicators is broken down by –Interviewer –Geographical region Timing of interviews (timetable and duration by final status code) Call outcomes sequence Opinions on interview (general difficulty, difficulty by questionnaire’s section, refusal reason, info on refusal person, …) Info on specific aspects on collected data (% of travelling persons/households, missing values on travel expenses, etc.)

Closing comments Monitoring the survey is an important chance that should be taken into consideration by researchers and survey managers Having info on data collection during and after the data collection phase is recommended because it is an added value to understand the investigated phenomena Using a quality management system is a first important step to understand how much non sampling errors can affect the final survey results

Thank you for your attention !