1. Data Management Seminar, 9-12th July 2007, Hamburg 11 ICCS 2009 – Field Trial Survey Operations Overview

2. Data Management Seminar, 9-12th July 2007, Hamburg 2 Content Survey Operations Overview On-line Data Collection Support Materials Survey Timeline Next Steps Contact Information

3. Data Management Seminar, 9-12th July 2007, Hamburg 3 Survey Operations Overview

4. Data Management Seminar, 9-12th July 2007, Hamburg 4 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

5. Data Management Seminar, 9-12th July 2007, Hamburg 5 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

6. Data Management Seminar, 9-12th July 2007, Hamburg 6 Adapt Questionnaires & Document Adaptations Verification of Adaptations Layout Verification Translate Questionnaires & Verifiy within Country Correct & Assemble Questionnaires Correct Questionnaires (if necessary) Print Send Out Translation Verification

7. Data Management Seminar, 9-12th July 2007, Hamburg 7 NRCIEA Sec./ISC/DPC time Layout Verification Implement changes & assemble Instruments Translation Verification Adapt Instruments 2 weeks 4 weeks 1 weeks 3 weeks 6 weeks Start data collection16 weeks Instruments Send Out Printing & Send Out Translate Instruments Verification of Adaptations Time planning before administration

8. Data Management Seminar, 9-12th July 2007, Hamburg 8 Time planning before administration due date July 13, 2007 August 27, 2007 September 24, 2007 October 1, 2007 October 15, 2007 November 5, 2007 Start data collection NRC/ISC/IEAtime Layout Verification Implement Changes & Assemble Instruments Translation Verification Adapt &Translate Instruments 4 weeks 1 week 2 weeks 6 weeks Instruments Send Out Printing & Send Out 3 weeks November 5, 2007

9. Data Management Seminar, 9-12th July 2007, Hamburg 9 Meanwhile…

10. Data Management Seminar, 9-12th July 2007, Hamburg 10 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

11. Data Management Seminar, 9-12th July 2007, Hamburg 11 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

12. Data Management Seminar, 9-12th July 2007, Hamburg 12 Contact Schools and get within School Information Perform Within-School Sampling Prepare List of Schools and send it to DPC DPC provides list of sampled Schools Recruit and Train Test Administrators Recruit and Train Scorers Translate School Coordinator Manual Translate Test Administrator Manual

13. Data Management Seminar, 9-12th July 2007, Hamburg 13 School Contact Send sampling frame (list of schools) to DPC Sampling of schools will be done by DPC preferably List of sampled schools will be provided by DPC Contact schools, ensure participation Identify liaison person (School Coordinator) Provide School Coordinator Manual Record status for each sampled school in WinW3S Adaptation and translation of School Coordinator and Test Administrator Manual (no verification on international level, modifications to some extend possible)

14. Data Management Seminar, 9-12th July 2007, Hamburg 14 Within-School Sampling (Students) Two levels: students and teachers Identification of classes on Class Listing Forms –Class name –Number of students Random within-school sample of classes Identification of students within sampled classes on Student Listing Forms Send list of sampled students to School Coordinator on Student Tracking Form WinW3S serves as survey administration tool

15. Data Management Seminar, 9-12th July 2007, Hamburg 15 Within-School Sampling (Teachers) Identification of eligible target grade teachers on Teacher Listing Forms Within-school random sample of target grade teachers Send Teacher Tracking Form to School Coordinator

16. Data Management Seminar, 9-12th July 2007, Hamburg 16 Within-School Sampling WinW3S will support NRCs with –Storing school contact information –Listing of classes / teachers / students –Sampling of classes / teachers –Assignment of forms –Survey administration –Tracking of participation

17. Data Management Seminar, 9-12th July 2007, Hamburg 17 Assigning Questionnaires School Questionnaire Teacher –Every Teacher will receive one questionnaire Student –Booklet –Questionnaire (perception and background part in one form) –Regional Modules instrument(s)  WinW3S will assign forms and provide labels for questionnaires

18. Data Management Seminar, 9-12th July 2007, Hamburg 18 NRCSchoolstime Contacts Schools Decision to participate Sends Listing Forms List Students/Teachers Samples Teachers Sends Tracking Forms Sends Questionnaires Start data collection Name School Coordinator Samples Classes List Classes/ Provide Number of teachers Time Planning before Administration

19. Data Management Seminar, 9-12th July 2007, Hamburg 19 School ContactInstrumentstime Contact Schools Send Listing Forms Sample Teachers Send Tracking Forms Send Questionnaires Implement changes & assemble Quest. Adapt & Translate Instruments Printing Data collection Sample Classes Time Planning before Administration

20. Data Management Seminar, 9-12th July 2007, Hamburg 20 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

21. Data Management Seminar, 9-12th July 2007, Hamburg 21 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

22. Data Management Seminar, 9-12th July 2007, Hamburg 22 Monitor Test Administrators Ad hoc Support Arrange for Material Receipt Train Scorers Train Data Entry Staff

23. Data Management Seminar, 9-12th July 2007, Hamburg 23 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

24. Data Management Seminar, 9-12th July 2007, Hamburg 24 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

25. Data Management Seminar, 9-12th July 2007, Hamburg 25 Organize Material for (direct) Data Entry Organize Material for Marking and Reliability Marking Check if Information and Material is complete Get back to School if necessary Enter Participation Information

26. Data Management Seminar, 9-12th July 2007, Hamburg 26 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

27. Data Management Seminar, 9-12th July 2007, Hamburg 27 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

28. Data Management Seminar, 9-12th July 2007, Hamburg 28 Conduct and Monitor Marking Conduct and Monitor Reliability Marking Organize Material for Data Entry

29. Data Management Seminar, 9-12th July 2007, Hamburg 29 Marking All open ended items have to be scored according to Marking guide Reliability Marking for open ended items for half of the Student Achievement Booklets (Booklets 1, 2 and 3) Occupations of parents/guardians have to be scored according to ISCO 88

30. Data Management Seminar, 9-12th July 2007, Hamburg 30 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

31. Data Management Seminar, 9-12th July 2007, Hamburg 31 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

32. Data Management Seminar, 9-12th July 2007, Hamburg 32 Adapt International Codebooks to National Questionnaires Use National Adaptation Forms Use adapted Codebooks for Data Entry

33. Data Management Seminar, 9-12th July 2007, Hamburg 33 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

34. Data Management Seminar, 9-12th July 2007, Hamburg 34 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

35. Data Management Seminar, 9-12th July 2007, Hamburg 35 Complete Participation Information Verify Data Enter Data

36. Data Management Seminar, 9-12th July 2007, Hamburg 36 Data Entry Data entry software (WinDEM) and international codebooks for the FT are provided by the DPC on August 20, 2007 National staff should be trained on data entry, even if all or parts are being outsourced Computer workstations need to be set up National Centers enter and verify data from paper questionnaires

37. Data Management Seminar, 9-12th July 2007, Hamburg 37 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

38. Data Management Seminar, 9-12th July 2007, Hamburg 38 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

39. Data Management Seminar, 9-12th July 2007, Hamburg 39 Set of Final Instruments National Codebooks and Documentation (NAF) Verified Data sets Tracking Forms Reliability Coding Data

40. Data Management Seminar, 9-12th July 2007, Hamburg 40 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Marking

41. Data Management Seminar, 9-12th July 2007, Hamburg 41 On-line Data Collection

42. Data Management Seminar, 9-12th July 2007, Hamburg 42 On-line Data Collection (ODC) Used for all questionnaires that National Centers should answer –National Context Questionnaire –Survey Activities Questionnaire International Option for –School Questionnaire –Teacher Questionnaire IMPORTANT: Countries that decide to use ODC in Main Study are obliged to conduct Field Trial in a split half design Ensure fall-back to paper questionnaire

43. Data Management Seminar, 9-12th July 2007, Hamburg 43 Split half design –Teachers and school principals from schools with odd school-IDs will be automatically assigned to on-line mode –Teachers and school principals from schools with even school-IDs will be automatically assigned to paper mode Fall-back to paper questionnaire should be possible, but kept to minimum Goal: –Compare on-line and paper mode data on international level –Get estimates about comparability of data On-line Data Collection in FT

44. Data Management Seminar, 9-12th July 2007, Hamburg 44 True mixed mode design Goal: reduce response burden –Mode assignment for all questionnaires of one type possible –Ask schools for preferred mode On-line Data Collection in MS

45. Data Management Seminar, 9-12th July 2007, Hamburg 45 Codebook Adaptation Instrument Preparation Sampling and Preparing Administration Scoring & Data Entry and Verification Administration Receiving Material Data Sendout Codebook Adaptation Prepare ODC Monitor ODC

46. Data Management Seminar, 9-12th July 2007, Hamburg 46 School contactInstrumentstime Contact Schools Sample Students / Teachers Send Tracking Forms Send Questionnaires Implement changes & assemble Quest. Adapt & Translate Instruments Printing Data collection Printing Prepare ODC Send Quest./Cover Letters Send Listing Forms Sample Classes Time planning before administration

47. Data Management Seminar, 9-12th July 2007, Hamburg 47 On-line Data Collection (ODC) Based on concept of cultures All languages supported (character sets, text directions,…) Participants receive “cover letter” –Label with login information provided by WinW3S –Culture specific URL –Personal username and password –Sent to School Coordinator for distribution Participants can log in and out as often as desired Same administration period as paper questionnaires

48. Data Management Seminar, 9-12th July 2007, Hamburg 48 NRCs can monitor completion status of on-line questionnaires If necessary, NRCs instruct School Coordinators to remind teachers and school principals to complete the questionnaire NRCs enter participation status in WinW3S according to status provided by on-line monitor No data entry at National Center needed On-line Data Collection (ODC)

49. Data Management Seminar, 9-12th July 2007, Hamburg 49 Support Material

50. Data Management Seminar, 9-12th July 2007, Hamburg 50 Survey Operations Procedures Manual (FT) –Overview: Survey Operations Procedures Overview, Dates and Activities –Unit 1: School Contact and Material Preparation –Unit 2: Within School Sampling and Test Administration School Coordinator Manual Test Administrator Manual National Quality Control Monitor Manual Within School Sampling Software (WinW3S) Support Material

51. Data Management Seminar, 9-12th July 2007, Hamburg 51 Support Material Survey Operations Procedures Manual (FT) –Unit 3: On-line Data Collection, Marking and Data Entry IEA SurveySystem Designer Marking Guide Codebooks Windows Data Entry Manager (WinDEM)  Distribution of all materials and software packages through secure section of study website at http://iccs.acer.edu.au

52. Data Management Seminar, 9-12th July 2007, Hamburg 52 Survey Timeline

53. Data Management Seminar, 9-12th July 2007, Hamburg 53 Survey Timeline - Receive verified final instruments - Start printing and distribution of questionnaires - Send final instruments and NAF to IEA DPC - Set up on-line questionnaires October 15, 2007 Submit for layout verification to ACEROctober 1, 2007 Receive verified instrumentSeptember 24, 2007 Submit translated questionnaires and NAF - for international translation verification to IEA Sec. - for verification of adaptations to ACER August 27, 2007 Receive instruments & NAFJuly 13, 2007

54. Data Management Seminar, 9-12th July 2007, Hamburg 54 Survey Timeline Final deadline for submitting data to DPCJanuary 25, 2008 - Marking of open-ended items and occupations - Data entry and verification November 12 – January 25, 2008 Proposed FT administration period (subject to country conditions) November 5 – December 3, 2007 Submit on-line questionnaires definition files for verification to DPC October 22, 2007 Give feedback about FT procedures using the on-line Survey Activities Questionnaire February, 2008

55. Data Management Seminar, 9-12th July 2007, Hamburg 55 Next Steps

56. Data Management Seminar, 9-12th July 2007, Hamburg 56 Next Steps Prepare team at national center and plan for –Translation –School Contact –Test Administrators –Scorers (CCE items and Occupations) –Data Management (Data Entry and Verification) Marker Training on July 13-14, 2007 in Hamburg

57. Data Management Seminar, 9-12th July 2007, Hamburg 57 Contact Information Survey Operations / Data Management: IEA Data Processing and Research Center Falk Brese Michael Jung iccs@iea-dpc.de

58. Data Management Seminar, 9-12th July 2007, Hamburg 58 Thank you for your attention!