1. Data Management Seminar, 8-11th July 2008, Hamburg 1 Summary Common Sources of Error

2. Data Management Seminar, 8-11th July 2008, Hamburg 2 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3. Data Management Seminar, 8-11th July 2008, Hamburg 3 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

4. Data Management Seminar, 8-11th July 2008, Hamburg 4 Errors During Instrument Preparation Errors in translations (especially: careful if cooperate with other countries) The same terms in different questions translated differently Comments from translation verification not implemented National adaptations not or incorrectly documented on NAF Deletion of international questions and categories without strong reason and approval International question stem changed

5. Data Management Seminar, 8-11th July 2008, Hamburg 5 Errors During Instrument Preparation (ODC) Conversion in SurveySystem started before paper instruments are reviewed  start AFTER international verification Copy and paste to wrong text, numeral, or instruction  check paper version against preview Paper questionnaires and ODC questionnaires do not match in terms of disabled questions, variables or additional elements  verify within country before submitting to IEA DPC

6. Data Management Seminar, 8-11th July 2008, Hamburg 6 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

7. Data Management Seminar, 8-11th July 2008, Hamburg 7 Errors During (Within-school) Sampling School sampling frame incorrect / major parts of the population not included Sample not approved by the IEA DPC Replacement of ineligible schools (frame errors like closed schools) Missing selection of participation in Regional Module or on-line data collection when creating new project in WinW3S Schools did not list all eligible students and teachers or left out eligible but excluded them  follow-up

8. Data Management Seminar, 8-11th July 2008, Hamburg 8 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

9. Data Management Seminar, 8-11th July 2008, Hamburg 9 Errors During Codebook Adaptation Length of a national variable specified to small (one digit for a variable with 6 or more options) Wrong labeling/numbering of categories or dimensions when they were deleted or added Deletion of variables for questions that are not administered  set to „hide“ in WinDEM Codebooks for different data entry computers are different If ODC is used, codebooks for WinDEM do not match with ODC data (changes made after export from SurveySystem)

10. Data Management Seminar, 8-11th July 2008, Hamburg 10 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

11. Data Management Seminar, 8-11th July 2008, Hamburg 11 Errors During Administration Instruments distributed to wrong person (only to designated individuals) Sampled student/teacher replaced  do not substitute! Participation status not tracked (Tracking Forms) or monitored (On-line Monitor Report) Spare Booklet assignment not documented (Tracking Forms)

12. Data Management Seminar, 8-11th July 2008, Hamburg 12 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

13. Data Management Seminar, 8-11th July 2008, Hamburg 13 Errors During Material Receipt Receipt of materials not tracked appropriately Incomplete materials  contact school coordinator and try to resolve Incomplete materials because of low on-line participation  consider following up to schools with paper questionnaires If ODC used, “Monitor” reports not taken into account continuously

14. Data Management Seminar, 8-11th July 2008, Hamburg 14 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

15. Data Management Seminar, 8-11th July 2008, Hamburg 15 Errors during Scoring Single items are all scored by the same scorer Reliability booklets are marked by the same scorer in the instrument and in the reliability Scoring sheet Main scorers enter codes directly in reliability booklets before reliability Scoring has been done

16. Data Management Seminar, 8-11th July 2008, Hamburg 16 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

17. Data Management Seminar, 8-11th July 2008, Hamburg 17 Errors During Data Entry in WinW3S If ODC was used, questionnaire type not tracked correctly in WinW3S Participation status not entered in WinW3S Spare booklet assignment not performed in WinW3S ITLANG in NAF and WinW3S not set if more than one language was used

18. Data Management Seminar, 8-11th July 2008, Hamburg 18 Errors During Data Entry in WinDEM Not all materials entered Wrong or duplicate IDs, column shifts, invalid values (All avoided by using WinDEM)  train staff thoroughly Wrong treatment of out-of-range values  enter data “as is”! Missing values used incorrectly (use ‘9’ for any omitted response, ‘7’ for any invalid response and valid values for all interpretable responses)

19. Data Management Seminar, 8-11th July 2008, Hamburg 19 Errors During Data Entry in WinDEM Data entered on different computers not combined or combined multiple times (leading to duplicate IDs) Values for Check-All-That-Apply questions used incorrectly (use 1 for CHECKED, 2 for NOT CHECKED) Filters interpreted  enter data “as is”!

20. Data Management Seminar, 8-11th July 2008, Hamburg 20 Errors during Verification WinDEM checks not performed Checks performed but data not corrected Manipulating the data files with MS Excel (data file will be cut of after 255 variables)

21. Data Management Seminar, 8-11th July 2008, Hamburg 21 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

22. Data Management Seminar, 8-11th July 2008, Hamburg 22 Errors during Data Send-out Data submitted, but without documentation (Tracking Forms, NAF) NAF or instruments not sent for all languages used Completed Tracking Forms are printed from WinW3S  send paper copies or scans of original forms WinW3S file not exported Double-punching data missing Documentation incomplete or does not match data After data is submitted, national centers are not prepared to answer questions from the IEA DPC  Although data processing is exhausting, it’s not time for vacations, yet ☺

23. Data Management Seminar, 8-11th July 2008, Hamburg 23 Summary Data Processing at the IEA DPC

24. Data Management Seminar, 8-11th July 2008, Hamburg 24 Content Data and Documentation Submission Data Processing at the IEA DPC –Adapting the File Structure –Cleaning Steps Further Schedule Contact Addresses

25. Data Management Seminar, 8-11th July 2008, Hamburg 25 In general 3 month after last day of administration Latest possible dates –Southern Hemisphere: March 27, 2009 –Northern Hemisphere: August 28, 2009 Material to be Submitted to IEA DPC

26. Data Management Seminar, 8-11th July 2008, Hamburg 26 Set of Final Instruments National Adaptation Forms (NAF) version IV Exported WinW3S database National Codebooks Verified Data sets Reliability Scoring Data Data Submission DocumentationData Tracking FormsDouble Punching Data

27. Data Management Seminar, 8-11th July 2008, Hamburg 27 Please use the WebFTP sites to exchange materials between countries and the - ISC at ACER and/or - IEA Secretariat and/or - IEA DPC Please inform the recipients via email as soon you have uploaded any materials Further information on the IEA DPC WebFTP- Server has been provided in a separate WebFTP- Server Manual File-Exchange using the IEA DPC WebFTP-Server

28. Data Management Seminar, 8-11th July 2008, Hamburg 28 Content Data and Documentation Submission Data Processing at the IEA DPC –Adapting the File Structure –Cleaning Steps Further Schedule Contact Addresses

29. Data Management Seminar, 8-11th July 2008, Hamburg 29 Data Processing at the IEA DPC Data & Codebooks Instruments & Forms INPUT IEA DPC Country 1. Import to SAS and adapting the file structure

30. Data Management Seminar, 8-11th July 2008, Hamburg 30 Adapting the File Structure WinW3S database and WinDEM data files are merged and converted to SAS data files Unique system numbers (CSYSTEM, TSYSTEM, SSYSTEM) are added –System numbers usually correspond to the position of the record in the original WinDEM data files –If a data set is missing either in WinW3S or in WinDEM, higher system numbers will be assigned

31. Data Management Seminar, 8-11th July 2008, Hamburg 31 Structure of the Merged Data Set Variable Value IDBOOK2 IDPUNCH111 IDSCORA101...,ITPART1..2, IDSCHOOL, IDCLASS,... 3... 1001 100101 IDSTUD10010101 IDCHECK270 TOKEN013 CB2PDO11 SSYSTEM,...000001 Added during cleaning process Example: Student Achievement Data File Information: Tracking Forms Information: Student Achievement Booklets WinDEM (ISAxxxC2.DBF)

32. Data Management Seminar, 8-11th July 2008, Hamburg 32 Data Processing at the IEA DPC Country Database New Data Structure Reports, Statistics & Docu. INPUT OUTPUT Structure Check Background Cleaning IEA DPC Country ID Cleaning Linkage Cleaning Data & Codebooks Instruments & Forms 2. Cleaning Country communication will continue after data send-out!

33. Data Management Seminar, 8-11th July 2008, Hamburg 33 Content Data and Documentation Submission Data Processing at the IEA DPC –Adapting the File Structure –Cleaning Steps Further Schedule Contact Addresses

34. Data Management Seminar, 8-11th July 2008, Hamburg 34 Further Schedule Software: –July 21, 2008: WinW3S –August 18, 2008: SurveySystem –August 18, 2008: WinDEM Documentation in Survey Operations Procedures Manuals: –July 21, 2008: SOP (MS), Unit 2 (WinW3S I) –August 18, 2008: SOP (MS), Unit 3 (WinW3S II, WinDEM, SurveySystem) Material Submission for Main Survey:  March 27, 2009 (SH); August 28, 2009 (NH)

35. Data Management Seminar, 8-11th July 2008, Hamburg 35 Content Data and Documentation Submission Data Processing at the IEA DPC –Adapting the File Structure –Cleaning Steps Further Schedule Contact Addresses

36. Data Management Seminar, 8-11th July 2008, Hamburg 36 Contact Addresses tel: +61 3 9255 5555 e-mail:iccs@acer.edu.au Web: http://iccs.acer.edu.au tel: +31 20 625 3625 e-mail: b.malak@iea.nl International Study Center (ISC) IEA Secretariat

37. Data Management Seminar, 8-11th July 2008, Hamburg 37 Contact Addresses tel: +49 40 48 500 750 e-mail:sampling@iea-dpc.de IEA DPC Sampling IEA DPC tel: +49 40 48 500 611 e-mail:iccs@iea-dpc.de web-ftp: https://webftp.iea-dpc.org

38. Data Management Seminar, 8-11th July 2008, Hamburg 38 Thank you very much for your attention!