Presentation on theme: "My presentation will be on the use of paradata… By"— Presentation transcript:
1 Using Paradata to Monitor and Improve the Collection Process in Annual Business Surveys My presentation will be on the use of paradata…BySylvie DeBlois, Statistics CanadaRose-Carline Evra, Statistics CanadaICES-III, Montreal, June 19th, 2007
2 OUTLINE Introduction Score Function Paradata Score Function Recent UpdateFuture DevelopmentsI will talk about the score function a tool that allows us to establish follow-up procedures for cases of non response during collection and recent update made to it. I will also talk about paradata use in the score function and paradata that could possibly be used. I will end my presentation by enumerating some of the future developments for the UES collection improvement.I will start… toll use during collection to monitor follow-ups..Then.. Followed by a section on …. & … finally…
3 IntroductionThe Unified Enterprise Survey (UES) is an annual economic survey on financial and characteristic variables, which has been conducted by Statistics Canada since It combines many surveys.Average collection period: February to early OctoberCollection Processing System: BlaiseMore than 48,000 questionnaires each year.For RY 1997 data…Blaise allows us to keep track of the # of attempts, type of contact with the respondent, priorities,.. Use by interwiewers…
4 UES QuestionnaireUES includes Services, Trades, Manufactures, Agriculture (aquaculture) and Transportation (couriers and taxi & limousine) surveys.A questionnaire has about 7 to 10 sections (the number of sections varies depending on the survey):Introduction (Stats Act - Confidentiality, Respondent info)RevenueExpensesEvents that may have affected business units…Comments
5 IntroductionCollection Process:Mail-out of questionnairesFollow-up in case of non-response for some units / Mail-back of questionnairesVerification of received questionnaires / EditsCoding of questionnairesImaging & Data CaptureSometimes during the collection period, follow-ups are required due to non-response. The score function is used to determine the priority of an enterprise in follow-up.
6 Introduction Collection follow-up tool: Score function (SF) Annual Survey of Manufactures (ASM) score functionNon-ASM score functionBoth score functions have their own ways of calculating scores, defining cells and priorities.This presentation will focus mainly on the Non-ASM score function.
7 Score Function Reduces collection costs yet retains data quality. Similar to the collection goal of obtaining a high weighted coverage response rate.PRIORITY 1: Extensive follow-up for the larger revenue Collection Entities (CE) in cases of non-response.PRIORITY 0: Minimum follow-up for the smaller CE’s in cases of non-response.Every collection entity (questionnaire) will be assigned a priority.The entity with a high weighted revenue… which mean they will have an extensive follow up.
8 Useful definitions Cell NAICS = YYYYY PROV = AACellSampling Unit (part of the enterprise within the cell)EstablishmentNAICS: North American IndustryClassification System (5-digitnumber)ABCNow, I will present you some useful definitionThese are general definitions more precisions will come later…Cell: combinaisons naics*prov ou naics*prov*strate according to the surveyMoins vite…DE
9 Method: Initial Scores Within each cell, calculate the score for each UES sampling unit (SU).Score = the sample weighted revenue of the SU as a percentage of the cell’s total revenue.Sample weight: UES sampling weightRevenue: Sampling RevenueExplain the difference between long and char…Char: smaller questionnaire (no revenue question)
10 Method: Initial Scores Cell:For Distributive Trades & Aquaculture:NAICS * ProvinceFor Transportation:NAICS*Prov*Stratum(Take All /Take Some)For Services:NAICS*Prov*Stratum(TA /TS)* Type of questionnaire (long / characteristic)
11 Method: Initial Scores Within each cellSort SUs by descending scoreCumulate to the survey’s target coverage threshold for the Priority=1s, and the rest are Priority=0s.
12 Method: Dynamic Scores During collection process, twice a week, we:receive updated response codes;recalculate the scores within the cell (i.e. make it dynamic) to update priorities;update priorities on Blaise, the collection tool.Operation, Research and Development Division
13 Method: Dynamic Scores As collection proceeds:Response (received or completed) questionnaires contribute to the cell thresholdNon-response questionnaires contribute nothing to the thresholdOut-of-scope are removed entirely from the cell (reduces the cell’s revenue total)In-Progress questionnaires are still being collected (include appointments)In-Progress: case still open to be classified as being resp, non-rsep or OOSI’ll talk more about the appointment later
14 During CollectionNew total weighted revenue for the CELL (exclude the OOS).Priority 1’s or 0’s received or completed contribute to reaching the CELL threshold.CELL: XXXXXXXX Total: 475,000kReceived or Completed15% reachedIn progressPriority 150% left to doThreshold= 65% (308,750k)In progressPriority 0All larger in progress units will receive a priority 1…NON-RESPONSEOOS50,000k
15 Method: Dynamic Scores Has the cell reached its threshold?If yes, stop follow-up.If no, recalculate scores using In-progress units and the remaining threshold.Some cells must close due to lack of In-Progress questionnairesSome In-progress Priority 0s may be promoted to Priority 1s.The priorities are updated within the cell by…Yes => priority 0 for all the cell units…
16 ParadataDefinition: All variables directly related to data collection processCurrently used:Response codeAppointment reason (edit – data collection)Appointment date (recently added)Currently used only by Annual Survey of Manufactures (ASM):Number of attempts, commodity revenue and shipment revenueCould possibly be used:Type of contact with the respondentPrevious year’s response codeType of reminder sent / Date / # (mail, r ,…)OthersParadata: not collected via the respondentResponse codes used to classify units…Type of contact: answering machine, secretary,…Attempts: to reduce scores
17 Score Function Recent Update Recently, a study was done on the impact of appointments on the response rate (for reference year 2003).Following our findings the “appointment date” was added as paradata into the score function.APPOINTMENT: During collection, in a telephone follow-up, an enterprise can ask for a delay to complete their questionnaire.Later you will find that an appointment means many things…
18 Appointments: The Study During the collection period, an appointment might be scheduled with the respondent.“Does the fact of having a appointment affect the response rate?”Note: When an appointment is made and it’s a priority 1 questionnaire, it remains in the SF with a priority 1 with the “still in progress status”. Therefore, no priority 0 will be put as priority 1.During collection, in a telephone follow-up, an enterprise can ask for a delay to complete their questionnaire.I would like to mention that when an enterprise ask for an appointment the questKeeps its priority in the SF. Which means that no priority 0 will be promoted to 1Toreplce a quest with an app which is pr=1We made a study to establish the impact of appointments to the response rates
19 Response Rates: app versus no app The response rate is significantly lower for the questionnaires with an appointment.RY2003 (Non-ASM surveys)Response rate= collection rates for quest using the score functionquest: exclude ASM & characteristic questionnaires and all other questionnaires not included in the score function (Not collected by Operation & Integration Division (OID))
20 Response Rates: Scheduling of the appointment The response rate is significantly lower for questionnaires when the appointment is made toward the end of the collection period.Response rate according to the time the app was schedule during collection
21 Other FactsThe longer a questionnaire stays in appointment, the greater is the probability of that questionnaire being a non-response at the end of the collection period.23.8% of the questionnaires with appointments were classified as non-respondent, because at the end of the collection period their cases were still open.
22 Appointment: Conclusion When possible, we should avoid making an appointment. Especially, at the end of the collection period.In cases of appointments, follow-up should occur soon after the appointment is made. An appointment is still a good way of improving the response rates.The treatment of the appointments in the score function should be modified. Extra “In progress” units will be promoted to priority 1 in order to compensate for possible non-response.Some unit will see their priority go from 0 to 1 to compensate for the possible non responseWhich led us to modify our treatment of…
23 Facts / FindingsA unit may not have an appointment date or may have one that is constantly changing.Many appointment dates are within a few weeks.It was decided to only consider units that have a late appointment date, and there are not many.Here are some of our findings after introducing the changes for the current collection cycle (RY2006)…Appointment = R , refax,…Why only late app are considered? Most likely to be nonrespondent..
24 Facts / Findings An appointment can mean many things. Many unexpected factors caused the changes to be less efficient than initially expected.
25 Human Errors The interviewer: Enters the wrong value for a variable (for example, appointment reason)Does not update a key variable (for example, appointment date)
26 System Problems System Failures Files not properly loaded As a result, some variables are affected, like the number of attempts.Files not properly loadedMissing values or variablesSome follow-up events occur outside of the system
27 Theoretical / Practical Appointment date is also used to set the “r ” (r of questionnaire) and fax date.Also, some appointment dates are default dates (differ from survey to survey).Appointment is also used as a reminder to the interviewer to call a respondent unavailable at the moment of the initial call.Aside from the operational issues, we have theoretical versus practical issues…These are all the difficulties that we can encounter when using paradata…
28 Future DevelopmentsEstablish what is really an appointment; do more studies on the appointments.Study more paradata to “quantify” the importance of each unit, give priority and improve the score function.Introduction of a cost function to help assign the priority and the type of follow-up.Combine the ASM score function and the Non-ASM score function.Cost function: allows to evaluate the average cost of follow-up for a unit (cost of sending a fax,…)
29 Thank You / Merci!!! Questions ??? Pour plus d’information veuillez contacter /For more information, please contact:ou / or