Data Management Seminar, 8-11th July 2008, Hamburg Data Entry Overview and Changes from the Field Trial

Data Management Seminar, 8-11th July 2008, Hamburg 2 The ICCS MS Data Entry Process Data Entry Part 1 –Entering School Data –Entering Teacher Data –Entering Student Regional Module Data Data Entry Part 2 –Entering Student Questionaire Data Data Entry Part 3 –Entering Student Achievement Data

Data Management Seminar, 8-11th July 2008, Hamburg 3 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

Data Management Seminar, 8-11th July 2008, Hamburg 4 Changes in Hierarchical ID System School level –8001, 8002, Teacher level –800101, , Student level – , , School level –1001, 1002, Teacher level –100101, , Student level – , , Field trial Main Survey

Data Management Seminar, 8-11th July 2008, Hamburg 5 One form of Student Questionnaire Student Questionnaire Form A Field trial Student Questionnaire Form B Student Questionnaire Form C Main Survey Student Questionnaire

Data Management Seminar, 8-11th July 2008, Hamburg 6 One form of Student Questionnaire Codebook has: –One form of Student Questionnaire inside –No IDQUEST variable Thus...

Data Management Seminar, 8-11th July 2008, Hamburg 7 Introduction of Coding Module (ISG) Field trial

Data Management Seminar, 8-11th July 2008, Hamburg 8 Main Survey Introduction of Coding Module (ISG)

Data Management Seminar, 8-11th July 2008, Hamburg 9 Advantages –Entering the Occupational Data using WinDEM –More convenient for the Data Entry Staff –Better coordination between the Puncher and the Coder –Higher Data Entry speed –Less errors Introduction of Coding Module (ISG)

Data Management Seminar, 8-11th July 2008, Hamburg 10 Number of Achievement Booklets Field trial Main Survey 6 booklets Reliability booklets – only 1, 2, and 3 7 booklets Reliability booklets –1 through 7

Data Management Seminar, 8-11th July 2008, Hamburg 11 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

Data Management Seminar, 8-11th July 2008, Hamburg 12 Data Documentation Data Organizing the Data Entry Process Instruments/ Forms International Codebooks WinW3S File Data Entry Rules... DE Unit Data Manager 12

Data Management Seminar, 8-11th July 2008, Hamburg 13 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

Data Management Seminar, 8-11th July 2008, Hamburg 14 Data Manager’s Responsibilities Data Entry Software Source of Information Hardware Software Adaptation of Codebooks Verification Documentation Instruments Tracking Forms Scoring Material Organize:Training in: Organize: Perform: Send Out Materials Data Entry Staff Equipment Data Files Data Manager 14

Data Management Seminar, 8-11th July 2008, Hamburg 15 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

Data Management Seminar, 8-11th July 2008, Hamburg 16 From ICCS International Study Centre International codebooks - Questionnaire codebooks - Achievement and reliability codebooks Survey Operations Procedures (Unit 3) Windows Within-School Sampling Software (WinW3S) Windows Data Entry Manager (WinDEM) program IEA Survey System Necessary Material for Data Entry 16

Data Management Seminar, 8-11th July 2008, Hamburg 17 Important documents for data entry –Survey Operations Procedures (Unit 3) Copy relevant parts of documents for key punchers Copies of survey tracking forms should be available for key punchers Prepare Key Punchers for Data Entry 17

Data Management Seminar, 8-11th July 2008, Hamburg 18 Planning for Required Resources Staff for scoring and data entry depends on… –How much material? –How much time? –How much experience? –How much money? –How many persons available? Computer requirements –IBM-compatible (Pentium processor or higher) –OS: Windows XP/Vista –Office 97 or higher Ask countries that participated in previous IEA studies for their experiences! 18

Data Management Seminar, 8-11th July 2008, Hamburg 19 Amount of Materials Scoring: Scorer 7Achievement Booklets = ~ 3750 Student Achievement Test Booklets ~ 700 Student Achievement Test Booklets for Reliability Scoring ~ 150School Questionnaires ~ 2250Teacher Questionnaires ~3750Student Questionnaires ~ 3750Regional Module Questionnaires ~ 3750Student Achievement Test Booklets ~ 700Reliability Scoring Sheets Data Entry: Key Punchers 19

Data Management Seminar, 8-11th July 2008, Hamburg 20 Receiving Material Instrument Preparation Test Administration Codebook Adaptation Receiving Material Data Submission Sampling and Preparing Test Administration Scoring Data Entry and Verification 20

Data Management Seminar, 8-11th July 2008, Hamburg 21 Receiving Material 21

Data Management Seminar, 8-11th July 2008, Hamburg 22 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

Data Management Seminar, 8-11th July 2008, Hamburg 23 Control Receipts of Test Material Record return date on ICCS MS School Sampling Form 23 Contact schools that have not returned testing material or where the material is incomplete!

Data Management Seminar, 8-11th July 2008, Hamburg 24 Control Receipts of Test Material Check that all survey tracking forms have been returned and completed correctly participation status Check participation status on tracking forms! Contact schools that have not returned tracking forms or returned incomplete!

Data Management Seminar, 8-11th July 2008, Hamburg 25 Control Receipts of Test Material Check that all booklets/ questionnaires have been returned identification codes Verify identification codes on instruments! Contact schools that have not returned all testing instruments! 25

Data Management Seminar, 8-11th July 2008, Hamburg 26 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

Data Management Seminar, 8-11th July 2008, Hamburg 27 Organize Material for Data Entry Receiving Material 27

Data Management Seminar, 8-11th July 2008, Hamburg 28 Separate questionnaires and tracking forms from booklets! Sort and forward… Organize Material for Data Entry and Scoring Forms + Questionnaires to Data Entry Staff Booklets to Scorers... 28

Data Management Seminar, 8-11th July 2008, Hamburg Forms + Questionnaires to Data Entry Staff... Sorting Forms and Questionnaires 29

Data Management Seminar, 8-11th July 2008, Hamburg Hierarchical ID System IDSCHOOL: 4 digits  school IDCLASS: 6 digits  school + class IDSTUD: 8 digits  school + class + student IDTEACH: 6 digits  school + teacher Unique identifier in ICG file Unique identifier in xSG, xSE/xSL, xSA, xSR file Unique identifier in ITG file

Data Management Seminar, 8-11th July 2008, Hamburg Arrange Instruments for Data Entry School Questionnaires Sort by IDSCHOOL

Data Management Seminar, 8-11th July 2008, Hamburg Teacher Questionnaires Sort by IDTEACH... Arrange Instruments for Data Entry School School 1002

Data Management Seminar, 8-11th July 2008, Hamburg Sort by IDSTUD Student Questionnaires Arrange Instruments for Data Entry IDSCHOOL 1001 IDCLASS IDSCHOOL 1001 IDCLASS IDSCHOOL 1002 IDCLASS IDSCHOOL 1002 IDCLASS

Data Management Seminar, 8-11th July 2008, Hamburg Sort by IDSTUD Regional Module Student Questionnaires Arrange Instruments for Data Entry IDSCHOOL 1001 IDCLASS IDSCHOOL 1001 IDCLASS IDSCHOOL 1002 IDCLASS IDSCHOOL 1002 IDCLASS

Data Management Seminar, 8-11th July 2008, Hamburg 35 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

Data Management Seminar, 8-11th July 2008, Hamburg 36 Organize Material for Data Entry Organize Material for Scoring Receiving Material 36

Data Management Seminar, 8-11th July 2008, Hamburg 37 Sorting Booklets for Scoring Two Teams for Scoring: A and B Booklets to Scorers Scoring Team AScoring Team B 37

Data Management Seminar, 8-11th July 2008, Hamburg ,2,...,6,7 Steps for Booklet Arrangement Step 1: Sort by Booklet Type 38

Data Management Seminar, 8-11th July 2008, Hamburg 39 Arrange Test Booklets for Scoring Step 1: Sort by Booklet Type Booklet 1 Booklet 1 Booklet 1 Booklet 1 Booklet 2 Booklet 2 Booklet 2 Booklet 2 Booklet 5 Booklet 5 Booklet 5 Booklet 6 Booklet 6 Booklet 6 Booklet 6 Booklet 7 ~536 ~3750 ~536...

Data Management Seminar, 8-11th July 2008, Hamburg 40 Step 2: Divide into Two Sets... Set A, Set B Step 1: Sort by Booklet Type Steps for Booklet Arrangement Note: From now on the example in this presentation will focus on booklet 1 only!

Data Management Seminar, 8-11th July 2008, Hamburg 41 Arrange Test Booklets for Scoring Step 2: Divide into Two Sets Booklet 1 Booklet 1 Booklet 1 Booklet 1 ~268 Booklet 1 Set A Booklet 1 Set A Booklet 1 Set A Booklet 1 Set A Booklet 1 Set B Booklet 1 Set B Booklet 1 Set B Booklet 1 Set B ~536 Scoring Team A Scoring Team B ~50%

Data Management Seminar, 8-11th July 2008, Hamburg 42 Step 3: Divide into Batches... Batch 1, Batch 2,... Step 1: Sort by Booklet Type Step 2: Divide into Two Sets Steps for Booklet Arrangement

Data Management Seminar, 8-11th July 2008, Hamburg 43 Arrange Test Booklets for Scoring Step 3: Divide into Batches ~268 Booklet 1R Set A Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A ~ 25 Booklet 1R Set A Booklet 1 Set A Scoring Team A ~ 25 Booklet 1R Set A Booklet 1 Set A ~ 25 Booklet 1R Set A Booklet 1 Set A ~ 25 Booklet 1R Set B Booklet 1 Set B ~ 25 Booklet 1R Set B Booklet 1 Set B ~268 Booklet 1R Set B Booklet 1 Set B Booklet 1R Set B Booklet 1 Set B Scoring Team B ~ 25 Booklet 1R Set B Booklet 1 Set B...

Data Management Seminar, 8-11th July 2008, Hamburg 44 Step 4: Separate “R” Booklets Step 3: Divide into Batches Step 1: Sort by Booklet Type Step 2: Divide into Two Sets Steps for Booklet Arrangement

Data Management Seminar, 8-11th July 2008, Hamburg 45 Arrange Test Booklets for Scoring ~ 5 Booklet 1R Set A Scoring Team A Scoring Team B ~20 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A ~25 Booklet 1R Set A Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A... Step 4: Separate “R” Booklets

Data Management Seminar, 8-11th July 2008, Hamburg 46 Arrange Test Booklets for Scoring Scoring Team A Booklet 1R Set A ~ 5 As Main Scorers As Reliability Scorers Scoring Team B ~20 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A ~20 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A... ~20 Booklet 1 Set A Booklet 1R Set A Booklet 2 Set A ~20 Booklet 1 Set A Booklet 1R Set A Booklet 2 Set A... Booklet 1R Set A ~ 5... Booklet 2R Set A ~5 Booklet 2R Set A ~5... Step 4: Separate “R” Booklets

Data Management Seminar, 8-11th July 2008, Hamburg 47 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

Data Management Seminar, 8-11th July 2008, Hamburg 48 Scoring Instrument Preparation Test Administration Codebook Adaptation Receiving Material Data Submission Data Entry and Verification Scoring Sampling and Preparing Test Administration

Data Management Seminar, 8-11th July 2008, Hamburg 49 Reliability Scoring Main Scoring

Data Management Seminar, 8-11th July 2008, Hamburg 50 Step 6: Reliability Scoring Step 5: Main Scoring (I) Reliability Scoring Sheet Step 7: Return “R” Booklets Step 8: Main Scoring (II) Step 4: Separate “R” Booklets Step 3: Divide into Batches Step 1: Sort by Booklet Type Step 2: Divide into Two Sets Steps for Booklet Arrangement

Data Management Seminar, 8-11th July 2008, Hamburg 51 Main Scoring Step 5: Main Scoring (I) 202 ~218 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A ~218 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set B Scoring Team AScoring Team B

Data Management Seminar, 8-11th July 2008, Hamburg 52 Reliability Scoring Step 6: Reliability Scoring 202 Scoring Team AScoring Team B Booklet 1R Set B ~50 Reliability Scoring Sheet 1R Set B ~50 Booklet 1R Set A ~50 Reliability Scoring Sheet 1R Set A ~50

Data Management Seminar, 8-11th July 2008, Hamburg Booklet 1R Set B Reliability Scoring Scoring Team A Reliability Scoring Sheet 1R Set B

Data Management Seminar, 8-11th July 2008, Hamburg 54 Return „R“ Booklets for Main Scoring Step 7: Return “R” Booklets Scoring Team A Scoring Team B Booklet 1R Set A ~50 Reliability Scoring Sheet 1R Set A ~50 ~218 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set A Booklet 1R Set B ~50 Reliability Scoring Sheet 1R Set B ~50 ~218 Booklet 1 Set A Booklet 1R Set A Booklet 1 Set B

Data Management Seminar, 8-11th July 2008, Hamburg 55 Main Scoring Step 8: Main Scoring (II) 202 Scoring Team AScoring Team B Booklet 1R Set B ~50 Booklet 1R Set A ~50

Data Management Seminar, 8-11th July 2008, Hamburg 56 Scoring Team B 101 Documenting Main Scorer 202 Booklet 1R Set B 202 1

Data Management Seminar, 8-11th July 2008, Hamburg 57 Table of Content 1. Changes in Data Entry Process since Field Trial 2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring

Data Management Seminar, 8-11th July 2008, Hamburg 58 Data Entry and Verification Data Entry and Verification Instrument Preparation Test Administration Codebook Adaptation Receiving Material Data Submission Scoring Sampling and Preparing Test Administration

Data Management Seminar, 8-11th July 2008, Hamburg 59 Complete Student Information Enter Background Data Enter Achievement and Reliability Data Data Entry and Verification

Data Management Seminar, 8-11th July 2008, Hamburg 60 Thank You for Your Attention. 60