1. Data Management Seminar, 8-11th July 2008, Hamburg Data Entry Overview and Changes from the Field Trial

2. Data Management Seminar, 8-11th July 2008, Hamburg 2 The ICCS MS Data Entry Process Data Entry Part 1 –Entering School Data –Entering Teacher Data –Entering Student Regional Module Data Data Entry Part 2 –Entering Student Questionaire Data Data Entry Part 3 –Entering Student Achievement Data

3. Data Management Seminar, 8-11th July 2008, Hamburg 3 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

4. Data Management Seminar, 8-11th July 2008, Hamburg 4 Changes in Hierarchical ID System School level –8001, 8002, 8003... Teacher level –800101, 800102... 805201, 805202... Student level –80010101, 80010102... 80520101, 80520102... School level –1001, 1002, 1003... Teacher level –100101, 100102... 105201, 105202... Student level –10010101, 10010102... 10520101, 10520102... Field trial Main Survey

5. Data Management Seminar, 8-11th July 2008, Hamburg 5 One form of Student Questionnaire Student Questionnaire Form A Field trial Student Questionnaire Form B Student Questionnaire Form C Main Survey Student Questionnaire

6. Data Management Seminar, 8-11th July 2008, Hamburg 6 One form of Student Questionnaire Codebook has: –One form of Student Questionnaire inside –No IDQUEST variable Thus...

7. Data Management Seminar, 8-11th July 2008, Hamburg 7 Introduction of Coding Module (ISG) Field trial

8. Data Management Seminar, 8-11th July 2008, Hamburg 8 Main Survey Introduction of Coding Module (ISG)

9. Data Management Seminar, 8-11th July 2008, Hamburg 9 Advantages –Entering the Occupational Data using WinDEM –More convenient for the Data Entry Staff –Better coordination between the Puncher and the Coder –Higher Data Entry speed –Less errors Introduction of Coding Module (ISG)

10. Data Management Seminar, 8-11th July 2008, Hamburg 10 Number of Achievement Booklets Field trial Main Survey 6 booklets Reliability booklets – only 1, 2, and 3 7 booklets Reliability booklets –1 through 7

11. Data Management Seminar, 8-11th July 2008, Hamburg 11 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

12. Data Management Seminar, 8-11th July 2008, Hamburg 12 Data Documentation Data Organizing the Data Entry Process Instruments/ Forms International Codebooks WinW3S File Data Entry Rules... DE Unit Data Manager 12

13. Data Management Seminar, 8-11th July 2008, Hamburg 13 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

14. Data Management Seminar, 8-11th July 2008, Hamburg 14 Data Manager’s Responsibilities Data Entry Software Source of Information Hardware Software Adaptation of Codebooks Verification Documentation Instruments Tracking Forms Scoring Material Organize:Training in: Organize: Perform: Send Out Materials Data Entry Staff Equipment Data Files Data Manager 14

15. Data Management Seminar, 8-11th July 2008, Hamburg 15 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

16. Data Management Seminar, 8-11th July 2008, Hamburg 16 From ICCS International Study Centre http://iccs.acer.edu.au/ International codebooks - Questionnaire codebooks - Achievement and reliability codebooks Survey Operations Procedures (Unit 3) Windows Within-School Sampling Software (WinW3S) Windows Data Entry Manager (WinDEM) program IEA Survey System Necessary Material for Data Entry 16

17. Data Management Seminar, 8-11th July 2008, Hamburg 17 Important documents for data entry –Survey Operations Procedures (Unit 3) Copy relevant parts of documents for key punchers Copies of survey tracking forms should be available for key punchers Prepare Key Punchers for Data Entry 17

18. Data Management Seminar, 8-11th July 2008, Hamburg 18 Planning for Required Resources Staff for scoring and data entry depends on… –How much material? –How much time? –How much experience? –How much money? –How many persons available? Computer requirements –IBM-compatible (Pentium processor or higher) –OS: Windows XP/Vista –Office 97 or higher Ask countries that participated in previous IEA studies for their experiences! 18

19. Data Management Seminar, 8-11th July 2008, Hamburg 19 Amount of Materials Scoring: Scorer 7Achievement Booklets = ~ 3750 Student Achievement Test Booklets ~ 700 Student Achievement Test Booklets for Reliability Scoring ~ 150School Questionnaires ~ 2250Teacher Questionnaires ~3750Student Questionnaires ~ 3750Regional Module Questionnaires ~ 3750Student Achievement Test Booklets ~ 700Reliability Scoring Sheets Data Entry: Key Punchers 19

20. Data Management Seminar, 8-11th July 2008, Hamburg 20 Receiving Material Instrument Preparation Test Administration Codebook Adaptation Receiving Material Data Submission Sampling and Preparing Test Administration Scoring Data Entry and Verification 20

21. Data Management Seminar, 8-11th July 2008, Hamburg 21 Receiving Material 21

22. Data Management Seminar, 8-11th July 2008, Hamburg 22 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

23. Data Management Seminar, 8-11th July 2008, Hamburg 23 Control Receipts of Test Material Record return date on ICCS MS School Sampling Form 23 Contact schools that have not returned testing material or where the material is incomplete!

24. Data Management Seminar, 8-11th July 2008, Hamburg 24 Control Receipts of Test Material Check that all survey tracking forms have been returned and completed correctly... 24 participation status Check participation status on tracking forms! Contact schools that have not returned tracking forms or returned incomplete!

25. Data Management Seminar, 8-11th July 2008, Hamburg 25 Control Receipts of Test Material Check that all booklets/ questionnaires have been returned identification codes Verify identification codes on instruments! Contact schools that have not returned all testing instruments! 25

26. Data Management Seminar, 8-11th July 2008, Hamburg 26 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

27. Data Management Seminar, 8-11th July 2008, Hamburg 27 Organize Material for Data Entry Receiving Material 27

28. Data Management Seminar, 8-11th July 2008, Hamburg 28 Separate questionnaires and tracking forms from booklets! Sort and forward… Organize Material for Data Entry and Scoring Forms + Questionnaires to Data Entry Staff Booklets to Scorers... 28

29. Data Management Seminar, 8-11th July 2008, Hamburg Forms + Questionnaires to Data Entry Staff... Sorting Forms and Questionnaires 29

30. Data Management Seminar, 8-11th July 2008, Hamburg 30 1 0 0 1 0 1 0 1 1 0 0 1 1 0 0 1 0 1 1 0 0 1 0 1 0 1 0 1 1 0 0 1 0 1 1 0 0 1 Hierarchical ID System IDSCHOOL: 4 digits  school IDCLASS: 6 digits  school + class IDSTUD: 8 digits  school + class + student IDTEACH: 6 digits  school + teacher Unique identifier in ICG file Unique identifier in xSG, xSE/xSL, xSA, xSR file Unique identifier in ITG file 1 0 0 1 30

31. Data Management Seminar, 8-11th July 2008, Hamburg 31 1004 Arrange Instruments for Data Entry School Questionnaires Sort by IDSCHOOL... 31 100310021001

32. Data Management Seminar, 8-11th July 2008, Hamburg 32 100102 Teacher Questionnaires Sort by IDTEACH... Arrange Instruments for Data Entry 32 100101 School 1001 100202 100201 School 1002

33. Data Management Seminar, 8-11th July 2008, Hamburg 33... Sort by IDSTUD Student Questionnaires Arrange Instruments for Data Entry 33 10010103 10010102 10010101 IDSCHOOL 1001 IDCLASS 100101 10010203 10010202 10010201 IDSCHOOL 1001 IDCLASS 100102 10020103 10020102 10020101 IDSCHOOL 1002 IDCLASS 100201 10020203 10020202 10020201 IDSCHOOL 1002 IDCLASS 100202

34. Data Management Seminar, 8-11th July 2008, Hamburg 34... Sort by IDSTUD Regional Module Student Questionnaires Arrange Instruments for Data Entry 34 10010103 10010102 10010101 IDSCHOOL 1001 IDCLASS 100101 10010203 10010202 10010201 IDSCHOOL 1001 IDCLASS 100102 10020103 10020102 10020101 IDSCHOOL 1002 IDCLASS 100201 10020203 10020202 10020201 IDSCHOOL 1002 IDCLASS 100202

35. Data Management Seminar, 8-11th July 2008, Hamburg 35 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

36. Data Management Seminar, 8-11th July 2008, Hamburg 36 Organize Material for Data Entry Organize Material for Scoring Receiving Material 36

37. Data Management Seminar, 8-11th July 2008, Hamburg 37 Sorting Booklets for Scoring Two Teams for Scoring: A and B Booklets to Scorers Scoring Team AScoring Team B 37

38. Data Management Seminar, 8-11th July 2008, Hamburg 38... 1,2,...,6,7 Steps for Booklet Arrangement Step 1: Sort by Booklet Type 38

39. Data Management Seminar, 8-11th July 2008, Hamburg 39 Arrange Test Booklets for Scoring Step 1: Sort by Booklet Type Booklet 1 Booklet 1 Booklet 1 Booklet 1 Booklet 2 Booklet 2 Booklet 2 Booklet 2 Booklet 5 Booklet 5 Booklet 5 Booklet 6 Booklet 6 Booklet 6 Booklet 6 Booklet 7 ~536 ~3750 ~536...

40. Data Management Seminar, 8-11th July 2008, Hamburg 40 Step 2: Divide into Two Sets... Set A, Set B Step 1: Sort by Booklet Type Steps for Booklet Arrangement Note: From now on the example in this presentation will focus on booklet 1 only!

41. Data Management Seminar, 8-11th July 2008, Hamburg 41 Arrange Test Booklets for Scoring Step 2: Divide into Two Sets Booklet 1 Booklet 1 Booklet 1 Booklet 1 ~268 Booklet 1 Set A Booklet 1 Set A Booklet 1 Set A Booklet 1 Set A Booklet 1 Set B Booklet 1 Set B Booklet 1 Set B Booklet 1 Set B ~536 Scoring Team A Scoring Team B ~50%

42. Data Management Seminar, 8-11th July 2008, Hamburg 42 Step 3: Divide into Batches... Batch 1, Batch 2,... Step 1: Sort by Booklet Type Step 2: Divide into Two Sets Steps for Booklet Arrangement

43. Data Management Seminar, 8-11th July 2008, Hamburg 43 Arrange Test Booklets for Scoring Step 3: Divide into Batches ~268 Booklet 1R Set A Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A ~ 25 Booklet 1R Set A Booklet 1 Set A Scoring Team A ~ 25 Booklet 1R Set A Booklet 1 Set A ~ 25 Booklet 1R Set A Booklet 1 Set A ~ 25 Booklet 1R Set B Booklet 1 Set B ~ 25 Booklet 1R Set B Booklet 1 Set B ~268 Booklet 1R Set B Booklet 1 Set B Booklet 1R Set B Booklet 1 Set B Scoring Team B ~ 25 Booklet 1R Set B Booklet 1 Set B...

44. Data Management Seminar, 8-11th July 2008, Hamburg 44 Step 4: Separate “R” Booklets Step 3: Divide into Batches Step 1: Sort by Booklet Type Step 2: Divide into Two Sets Steps for Booklet Arrangement

45. Data Management Seminar, 8-11th July 2008, Hamburg 45 Arrange Test Booklets for Scoring ~ 5 Booklet 1R Set A Scoring Team A Scoring Team B ~20 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A ~25 Booklet 1R Set A Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A... Step 4: Separate “R” Booklets

46. Data Management Seminar, 8-11th July 2008, Hamburg 46 Arrange Test Booklets for Scoring Scoring Team A Booklet 1R Set A ~ 5 As Main Scorers As Reliability Scorers Scoring Team B ~20 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A ~20 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A... ~20 Booklet 1 Set A Booklet 1R Set A Booklet 2 Set A ~20 Booklet 1 Set A Booklet 1R Set A Booklet 2 Set A... Booklet 1R Set A ~ 5... Booklet 2R Set A ~5 Booklet 2R Set A ~5... Step 4: Separate “R” Booklets

47. Data Management Seminar, 8-11th July 2008, Hamburg 47 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

48. Data Management Seminar, 8-11th July 2008, Hamburg 48 Scoring Instrument Preparation Test Administration Codebook Adaptation Receiving Material Data Submission Data Entry and Verification Scoring Sampling and Preparing Test Administration

49. Data Management Seminar, 8-11th July 2008, Hamburg 49 Reliability Scoring Main Scoring

50. Data Management Seminar, 8-11th July 2008, Hamburg 50 Step 6: Reliability Scoring Step 5: Main Scoring (I) Reliability Scoring Sheet Step 7: Return “R” Booklets Step 8: Main Scoring (II) Step 4: Separate “R” Booklets Step 3: Divide into Batches Step 1: Sort by Booklet Type Step 2: Divide into Two Sets Steps for Booklet Arrangement

51. Data Management Seminar, 8-11th July 2008, Hamburg 51 Main Scoring Step 5: Main Scoring (I) 202 ~218 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A ~218 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set B Scoring Team AScoring Team B

52. Data Management Seminar, 8-11th July 2008, Hamburg 52 Reliability Scoring Step 6: Reliability Scoring 202 Scoring Team AScoring Team B Booklet 1R Set B ~50 Reliability Scoring Sheet 1R Set B ~50 Booklet 1R Set A ~50 Reliability Scoring Sheet 1R Set A ~50

53. Data Management Seminar, 8-11th July 2008, Hamburg 53 101 Booklet 1R Set B Reliability Scoring 101 10010103 974 2 Scoring Team A Reliability Scoring Sheet 1R Set B

54. Data Management Seminar, 8-11th July 2008, Hamburg 54 Return „R“ Booklets for Main Scoring Step 7: Return “R” Booklets Scoring Team A Scoring Team B Booklet 1R Set A ~50 Reliability Scoring Sheet 1R Set A ~50 ~218 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A Booklet 1R Set B ~50 Reliability Scoring Sheet 1R Set B ~50 ~218 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set B

55. Data Management Seminar, 8-11th July 2008, Hamburg 55 Main Scoring Step 8: Main Scoring (II) 202 Scoring Team AScoring Team B Booklet 1R Set B ~50 Booklet 1R Set A ~50

56. Data Management Seminar, 8-11th July 2008, Hamburg 56 Scoring Team B 101 Documenting Main Scorer 202 Booklet 1R Set B 202 1

57. Data Management Seminar, 8-11th July 2008, Hamburg 57 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

58. Data Management Seminar, 8-11th July 2008, Hamburg 58 Data Entry and Verification Data Entry and Verification Instrument Preparation Test Administration Codebook Adaptation Receiving Material Data Submission Scoring Sampling and Preparing Test Administration

59. Data Management Seminar, 8-11th July 2008, Hamburg 59 Complete Student Information Enter Background Data Enter Achievement and Reliability Data Data Entry and Verification

60. Data Management Seminar, 8-11th July 2008, Hamburg 60 Thank You for Your Attention. 60