3rd NRC Meeting, 9-12 June 2008, Windsor 1 Data Management Procedures Michael Jung, IEA Data Processing and Research Center

3rd NRC Meeting, 9-12 June 2008, Windsor 2 Contents Receipt of FT Data and Documentation FT Data Processing at the IEA DPC Evaluation of FT Data Quality WinW3S - Within-School Sampling Software WinDEM – Data Entry Manager SurveySystem – On-line Data Collection Software Material Submission after Test Administration MS Data Processing at the IEA DPC

3rd NRC Meeting, 9-12 June 2008, Windsor 3 Receipt of FT Data and Documentation

3rd NRC Meeting, 9-12 June 2008, Windsor 4 Data from 32 countries altogether –16 data sets arrived on time –16 data sets arrived after January, 25th Receipt of FT Data and Documentation

3rd NRC Meeting, 9-12 June 2008, Windsor 5 Receipt of FT Data 25-Jan

3rd NRC Meeting, 9-12 June 2008, Windsor 6 Data from 32 countries altogether –16 data sets arrived on time –16 data sets arrived after January, 25th Documentation from 32 countries altogether –10 countries sent on time and complete –22 countries sent after January, 25th and complete Receipt of FT Data and Documentation

3rd NRC Meeting, 9-12 June 2008, Windsor 7 25-Jan Receipt of FT Documentation

3rd NRC Meeting, 9-12 June 2008, Windsor 8 FT Data Processing at the IEA DPC

3rd NRC Meeting, 9-12 June 2008, Windsor 9 Data Processing at the IEA DPC Data & Codebooks Instruments & Forms INPUT IEA DPC Country 1. Import to SAS and adapting the file structure

3rd NRC Meeting, 9-12 June 2008, Windsor 10 Adapting the File Structure WinW3S database and WinDEM data files have been merged, renamed and converted to SAS data files Unique system numbers (CSYSTEM, TSYSTEM, SSYSTEM) have been added National Scores and derived variables (e.g. final participation indicator) have been added National variables have been removed (will be sent to countries in a separate data file on demand)

3rd NRC Meeting, 9-12 June 2008, Windsor 11 Structure of the Merged Data Set Variable Value IDPUNCH1...,ITPART1..2, IDSCHOOL, IDCLASS, IDSTUD IDCHECK373 TOKEN013 CS2GSEX9 SSYSTEM, Information: Tracking Forms Information: Student Questionnaire Added during cleaning process Example: Student Questionnaire Data File WinDEM (ISGxxxF2.DBF)

3rd NRC Meeting, 9-12 June 2008, Windsor 12 Data Processing at the IEA DPC Country Database New Data Structure Reports, Statistics & Docu. INPUT OUTPUT Structure Check Background Cleaning IEA DPC Country ID Cleaning Linkage Cleaning Data & Codebooks Instruments & Forms 2. Cleaning

3rd NRC Meeting, 9-12 June 2008, Windsor 13 Evaluation of FT Data Quality

3rd NRC Meeting, 9-12 June 2008, Windsor 14 Evaluation of FT Data Quality General data quality in almost all countries was VERY GOOD Most common problems –Not all changes in the instruments reflected in the codebooks –Not administered variables deleted in codebooks instead of hidden –WinDEM quality control checks not performed or not all problems have been solved –Inconsistencies between Student Questionnaire files and Occupation data files –Information in WinW3S not completed and thus database not exported

3rd NRC Meeting, 9-12 June 2008, Windsor 15 WinW3S - Within-School Sampling Software

3rd NRC Meeting, 9-12 June 2008, Windsor 16 WinW3S Management tool for all ICCS within-school sampling operations Identifies the schools, classes, teachers and students that will participate in ICCS Creates a hierarchical ID system + passwords for ODC Samples classes + teachers Assigns booklets to the students Produces and prints all forms and labels

3rd NRC Meeting, 9-12 June 2008, Windsor 17 WinW3S – What's New New option for participation in Asian Module added Import of students and teachers from MS Excel files New participation code for students added: „participated, but excluded” New participation code for teachers added: „teacher on temporary leave” Assignment of reliability booklets changed: random assignment of ~100 R booklets per booklet type Assignment of different class IDs for ICCS Target Grade and Additional Grade students Listing of teachers birth information and gender optional on Teacher Listing Forms Possible to list more than 99 teachers per school Teacher and school principal questionnaire mode can be changed on individual level –ODC optional ONLY for countries that have trialed ODC in FT Participation status for on-line participants can be imported from SurveySystem Monitor

3rd NRC Meeting, 9-12 June 2008, Windsor 18 WinDEM – Data Entry Manager

3rd NRC Meeting, 9-12 June 2008, Windsor 19 WinDEM Provides mask for data entry from booklets and paper questionnaires Based on codebooks, provided by the IEA DPC Allows adaptation of codebooks Validates data during data entry Provides various data verification checks after data entry Allows for comparison of double-entered data

3rd NRC Meeting, 9-12 June 2008, Windsor 20 WinDEM – What's New New coding module for coding of occupations integrated in WinDEM –Entry of occupation data directly from WinDEM –Occupation data will be stored in an MS Access database –Text answers and occupation codes also available in table mode

3rd NRC Meeting, 9-12 June 2008, Windsor 21 IEA SurveySystem On-line Data Collection Software

3rd NRC Meeting, 9-12 June 2008, Windsor 22 ODC – What's New Export tool in SurveySystem Monitor application for participation status of teachers and school principals –data can be imported into WinW3S later on

3rd NRC Meeting, 9-12 June 2008, Windsor 23 Material Submission after Test Administration

3rd NRC Meeting, 9-12 June 2008, Windsor 24 Material to be submitted 4 WinW3S database exported for IEA DPC Sampling (MICS 1.dbf) (MICS 2.dbf) Send Material to IEA DPC Sampling 4 National Adaptation Forms (IV) (*.xls) 4 Set of final National Instruments (*.pdf) Send Material to ISC at ACER Send Material to IEA DPC

3rd NRC Meeting, 9-12 June 2008, Windsor 25 Material to be submitted Send Material to IEA DPC Sampling Send Material to ISC at ACER Send Material to IEA DPC 4 National Adaptation Forms (IV) (*.xls) 4 Set of final National Instruments (*.pdf) 4 WinW3S database exported for IEA DPC (MICS 1.mdb) (MICS 2.mdb) 4 Student and Teacher Tracking Forms (paper copies or scanned)

3rd NRC Meeting, 9-12 June 2008, Windsor 26 Material to be submitted Send Material to IEA DPC Sampling Send Material to ISC at ACER Send Material to IEA DPC 4 WinDEM data files (*.dbf) + 4 WinDEM codebooks (*.sdb) (ICGICSM2.DBF; *SDB) (ITGICSM2.DBF; *SDB) (ISGICSM2.DBF; *SDB) (ISAICSM2.DBF; *SDB) (ISRICSM2.DBF; *SDB) (ISEICSM2.DBF; *SDB) (ISLICSM2.DBF; *SDB) (ISSICSM2.DBF; *SDB) 4 Occupation data file (ISOICSM2.mdb)

3rd NRC Meeting, 9-12 June 2008, Windsor 27 Material to be submitted Data and Documents have to be submitted by 27-Mar-2009 latest for SH countries and by 29-Aug-2009 latest for NH countries  IEA DPC Sampling ISC at ACER IEA DPC

3rd NRC Meeting, 9-12 June 2008, Windsor 28 File-Exchange using the IEA DPC WebFTP-Server The IEA DPC has setup secure FTP sites to exchange materials between countries and the - ISC at ACER and/ or - IEA Secretariat and/ or - IEA DPC Can easily be accessed using a browser (e.g. Internet Explorer) Individual country logins from FT are still valid FT folders and documents have been moved to a folder “0_Old_Material_from_Field_Trial “  please do not upload any documents into that folder any more

3rd NRC Meeting, 9-12 June 2008, Windsor 29 File-Exchange using the IEA DPC WebFTP-Server MS Materials should be uploaded using a predefined folder structure 1_Instruments_and_NAF_for_Adaptation_Verification_ACER 2_Instruments_and_NAF_for_Translation_Verification_IEA_Secretariat 3_Instruments_and_NAF_for_Layout_Verification_ACER 4_Final_Instruments_and_NAF_for_ACER _and_IEA_DPC 5_SurveySystem_Definition_Files_for_IEA_DPC 6_Codebooks_and_Data_Files_for_IEA_DPC 7_Exported_WinW3S_Database_for_IEA_DPC 8_Other_Materials 9_To_Country

3rd NRC Meeting, 9-12 June 2008, Windsor 30 File-Exchange using the IEA DPC WebFTP-Server Please inform the recipients via as soon you have uploaded any materials Updated WebFTP-Server Manual will be sent to countries

3rd NRC Meeting, 9-12 June 2008, Windsor 31 MS Data Processing at the IEA DPC

3rd NRC Meeting, 9-12 June 2008, Windsor 32 Data Processing at the IEA DPC Data Processing of the MS data at the IEA DPC basically follows the same prcedures as for the FT But: Any problems and/or inconsitencies will be reported back to countries and need correction and/or approval Countries need to be available after data submission

3rd NRC Meeting, 9-12 June 2008, Windsor 33 Data Processing at the IEA DPC Country Database New Data Structure Reports, Statistics & Docu. INPUT OUTPUT Structure Check Background Cleaning IEA DPC Country ID Cleaning Linkage Cleaning Data & Codebooks Instruments & Forms Cleaning

3rd NRC Meeting, 9-12 June 2008, Windsor 34 Thank you very much for your attention.