Data Management Seminar, 8-11th July 2008, Hamburg WinDEM- Verification Checks Part I

Data Management Seminar, 8-11th July 2008, Hamburg 2 Automatic checks Content 1. Unique ID Check 2. Validation Check 3. Column Shift Check 1. Unique ID Check 2. Column Shift Check 3. Validation Check 4. Token Check Additional checks 5. Double Punching Check 6. Scoring Reliability 7. WinW3S Linkcheck

Data Management Seminar, 8-11th July 2008, Hamburg 3 Data Entry and Verification Data Entry and Verification Instrument Preparation Test Administration Codebook Adaptation Receiving Material Data Submission Scoring Sampling and Preparing Test Administration

Data Management Seminar, 8-11th July 2008, Hamburg 4 Complete Student Information Enter Questionnaire Data Enter Achievement and Reliability Data Data Entry and Verification

Data Management Seminar, 8-11th July 2008, Hamburg 5 Data Entry and Verification Complete Student Information Enter Questionnaire Data Enter Achievement and Reliability Data

Data Management Seminar, 8-11th July 2008, Hamburg 6 1. Automatic checks: Those are performed automatically during data entry stage Within-Country Cleaning: Main Steps 2. Additional checks: Those are performed by the user after data entry

Data Management Seminar, 8-11th July 2008, Hamburg 7 WinDEM Checks – Automatic Checks 1. Unique ID Check 2. Validation Check 3. Column Shift Check

Data Management Seminar, 8-11th July 2008, Hamburg 8 WinDEM Checks – Automatic Checks Unique ID Check For each corresponding file (school, teacher and student) an unique identification code will be assigned: – School file: IDSCHOOL – Teacher file: IDTEACH – Student files: IDSTUD

Data Management Seminar, 8-11th July 2008, Hamburg 9 WinDEM Checks – Automatic Checks Unique ID Check – Example

Data Management Seminar, 8-11th July 2008, Hamburg 10 WinDEM Checks – Automatic Checks 1. Unique ID Check 2. Validation Check 3. Column Shift Check

Data Management Seminar, 8-11th July 2008, Hamburg 11 Valid ranges and valid values are defined in the codebook. If the value entered does not correspond to the predefined range or set of valid values, a warning message will appear: –For categorical variables only the predefined values are allowed. –For non-categorical variables it is possible to enter values out of range, but this needs to be confirmed. WinDEM Checks – Automatic Checks Validation Check

Data Management Seminar, 8-11th July 2008, Hamburg 12 WinDEM Checks – Automatic Checks Validation Check – Example 1

Data Management Seminar, 8-11th July 2008, Hamburg 13 WinDEM Checks – Automatic Checks Validation Check – Example 2

Data Management Seminar, 8-11th July 2008, Hamburg 14 WinDEM Checks – Automatic Checks 1. Unique ID Check 2. Validation Check 3. Column Shift Check

Data Management Seminar, 8-11th July 2008, Hamburg 15 Check variables (CHECK01, CHECK02...) are inserted after every odd page of a questionnaire or booklet. They are used to avoid column shifts: Column shifts appear when the puncher accidentally shifts to a different variable while punching. Only an empty space can be entered. WinDEM Checks – Automatic Checks Column Shift Check

Data Management Seminar, 8-11th July 2008, Hamburg 16 WinDEM Checks – Automatic Checks Column Shift Check – Example

Data Management Seminar, 8-11th July 2008, Hamburg Automatic checks: Those are performed automatically during data entry stage Within-Country Cleaning: Main Steps 2. Additional checks: Those are performed by the user after data entry

Data Management Seminar, 8-11th July 2008, Hamburg 18 WinDEM Checks – Additional Checks After data entry, you must check your data using the features from the "VERIFY" menu:

Data Management Seminar, 8-11th July 2008, Hamburg 19 WinDEM Checks – Additional Checks 1. Unique ID Check 2. Column Shift Check 3. Validation Check 4. Token Check 5. Double Punching Check 6. Scoring Reliability 7. WinW3S Linkcheck Although some of these checks have already been performed during data entry, the data have to be verified again! NOTE:

Data Management Seminar, 8-11th July 2008, Hamburg 20 Checks for duplicate IDs. After performing the check, a report will be created. WinDEM Checks – Additional Checks Unique ID

Data Management Seminar, 8-11th July 2008, Hamburg 21 WinDEM Checks – Additional Checks Unique ID Select „Unique ID Check“ from the „Verify“ Menu

Data Management Seminar, 8-11th July 2008, Hamburg 22 WinDEM Checks – Additional Checks Unique ID Select „Unique ID Check“ from the „Verify“ Menu Select the unique identifier

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID Select „Unique ID Check“ from the „Verify“ Menu Select the unique identifier Select the path where the report will be stored + save 23

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID Select „Unique ID Check“ from the „Verify“ Menu Select the unique identifier Select the path where the report will be stored + save Confirm the message 24

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID Select „Unique ID Check“ from the „Verify“ Menu Select the unique identifier Select the path where the report will be stored + save Confirm the message Select „yes“ to open the report 25

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID – Report 26

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID – Solving Suggestions Compare data in both records with the actual instrument Delete records without any valid data Delete data that does not fit the instrument Delete one of the two records, if both have the same data Helpful tools in WinDEM: Table View, Sort and Find Record functionality 27

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID – Table View Open the „View“ menu Select „Table Entry“ -> „Whole Table“ 28

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID – Table View 29

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID – Sort Select „Sort Data File“ from the „File“ menu 30

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID – Sort Select „Sort Data File“ from the „File“ menu Select the sorting variable 31

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID – Sort Select „Sort Data File“ from the „File“ menu Select the sorting variable 32

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID – Find Record Select „Find Record(s)“ from the „Edit“ menu 33

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID – Find Record Select „Find Record(s)“ from the „Edit“ menu Select the variable you want to search Select the operant (=, <...) Enter a value Click „Find“ 34

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Unique ID – Find Record Select „Find Record(s)“ from the „Edit“ menu Select the variable you want to search Select the operant (=, <...) Enter a value Click „Find“ 35

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks 1. Unique ID Check 2. Column Shift Check 3. Validation Check 4. Token Check 5. Double Punching Check 6. Scoring Reliability 7. WinW3S Linkcheck 36

Data Management Seminar, 8-11th July 2008, Hamburg Checks for column shifts: Column shifts appear when the puncher accidentally shifts to a different variable while entering the data. The result will eventually be that a check variable receives a value. Looks for check variables with a value. After performing the check, a report will be created. WinDEM Checks – Additional Checks Column Shift 37

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Column Shift Select „Column Shift Check“ from the „Verify“ menu Again, select the path where the report will be stored + save Confirm the message and open the report 38

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Column Shift – Report 39

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Column Shift – Solving Suggestions Compare data with the actual instrument Re-enter all values that do not correspond with the instrument Delete the value of the check variable 40

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks 1. Unique ID Check 2. Column Shift Check 3. Validation Check 4. Token Check 5. Double Punching Check 6. Scoring Reliability 7. WinW3S Linkcheck 41

Data Management Seminar, 8-11th July 2008, Hamburg Checks for violations by comparing with predefined ranges. When performing the check, a report will be created. WinDEM Checks – Additional Checks Validation 42

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Validation Select „Validation Check“ from the „Verify“ menu Again, select the path where the report will be stored + save Confirm the message and open the report 43

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Validation – Report 44

Data Management Seminar, 8-11th July 2008, Hamburg Compare the record with instrument Correct the data in WinDEM if necessary WinDEM Checks – Additional Checks Validation – Solving Suggestions 45

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks 1. Unique ID Check 2. Column Shift Check 3. Validation Check 4. Token Check 5. Double Punching Check 6. WinW3S Linkcheck 7. Scoring Reliability 46

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks 4. Token Check 5. Double Punching Check 6. Scoring Reliability 7. WinW3S Linkcheck The following checks are not available as automatic WinDEM checks: 47

Data Management Seminar, 8-11th July 2008, Hamburg Checks whether the following conditions are fulfilled: If a Token is “1“ or “2“, no valid data should be available. (Token 1 = instrument available but empty) (Token 2 = instrument not available or lost) If a Token is “3“, valid data should be available in the questionnaire or in the corresponding achievement booklet. (Token 3 = instrument available with reponses) When performing the check, a report will be created. WinDEM Checks – Additional Checks Token 48

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Token Select „Token Check“ from the „Verify“ menu Again, select the path where the report will be stored Confirm the message and open the report 49

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Token – Report 1 50

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Token – Report 2 51

Data Management Seminar, 8-11th July 2008, Hamburg Compare the record with instrument Delete record from WinDEM if whole instrument is empty Set Token variable to „1“ if there are no data available for the booklet Set Token to „3“ and enter/check the data if there are data available WinDEM Checks – Additional Checks Token – Solving Suggestions 52

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks 1. Unique ID Check 2. Column Shift Check 3. Validation Check 4. Token Check 5. Double Punching Check 6. Scoring Reliability 7. WinW3S Linkcheck 53

Data Management Seminar, 8-11th July 2008, Hamburg This routine checks the accuracy of your key punchers. A random sample of at least 30 units of each instrument (e.g. also 30 units per booklet type) has to be punched in by two punchers in a different data file. During the double punching check, all variables in both data files are cross-checked for all records, compared by their identification codes. A detailed report will be created when performing the check. WinDEM Checks – Additional Checks Double Punching 54

Data Management Seminar, 8-11th July 2008, Hamburg Double Punching = Double Entry of Data It is not possible to compare data files which do not contain exactly the same variables! The Double Punching procedure is not the same as the Double Scoring. The data should be entered in a separate file and not in the original file. The double punching has to be performed by different key punchers. The double punching should also be performed at the beginning of the data entry process and may result in additional training to punchers, regarding e.g. the use of missing values. WinDEM Checks – Additional Checks Double Punching 55

Data Management Seminar, 8-11th July 2008, Hamburg Puncher 111 has punched in the original data 30 student achievement booklets Stored under C:\Puncher 111 Puncher 222 punches in the same data again for the double punching check The same 30 student achievement booklets as puncher 111 Stored under C:\Puncher 222 WinDEM Checks – Additional Checks Double Punching – Example 56

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Double Punching – Example Open the data file from Puncher 222 first 57

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Double Punching – Example Open the data file from Puncher 222 first 58

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Double Punching – Example Open the data file from Puncher 222 first Select the „Double Punching Check“ from the „Verify“ menu 59

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Double Punching – Example Open the data file from Puncher 222 first Select the „Double Punching Check“ from the „Verify“ menu Select the unique identifier 60

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Double Punching – Example Open the data file from Puncher 222 first Select the „Double Punching Check“ from the „Verify“ menu Select the unique identifier Select the data file from Puncher

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Double Punching – Example Open the data file from Puncher 222 first Select the „Double Punching Check“ from the „Verify“ menu Select the unique identifier Select the data file from Puncher 111 Save report, confirm any messages and open the report 62

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Double Punching – Report 63

Data Management Seminar, 8-11th July 2008, Hamburg Margin of errors occurring in double punching should be: below 1 % for questionnaires below 0.1 % for international student achievement file and reliability file. An error rate > 1 %, or 0.1 % respectively indicates unreliable data. WinDEM Checks – Additional Checks Double Punching 64

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks 1. Unique ID Check 2. Column Shift Check 3. Validation Check 4. Token Check 5. Double Punching Check 6. Scoring Reliability 7. WinW3S Linkcheck 65

Data Management Seminar, 8-11th July 2008, Hamburg This routine checks the reliability of your scoring. Each student booklet that has randomly been selected as Reliability Booklet in WinW3S has to be scored by two scorers. Reliability Scorer scores into the Reliability Scoring Sheet. Main Scorer scores directly into the booklet A detailed report will be created when performing the check. WinDEM Checks – Additional Checks Scoring Reliability 66

Data Management Seminar, 8-11th July 2008, Hamburg Scoring Reliability Check compares open-ended items across Student Achievement and Student Reliability File. The Scoring Reliability Check should definitely be performed at the beginning of the scoring process and may result in additional training to scorers. The Scoring Reliability Check should be repeated over the scoring period. The error rate of the final data must not exceed 15%. WinDEM Checks – Additional Checks Scoring Reliability 67

Data Management Seminar, 8-11th July 2008, Hamburg Reliability Scorer 101 has scored the open-ended items into the Reliability Scoring Sheet 150 student achievement booklets Stored under C:\Scorer 101 Main Scorer 201 has scored the open-ended items into the Student Booklets The same 150 student achievement booklets as reliability scorer 101 Stored under C:\Puncher 201 WinDEM Checks – Additional Checks Scoring Reliability – Example 68

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Scoring Reliability – Example 69 Open the reliability data file ISRDEUM2 first

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Scoring Reliability – Example Open the reliability data file ISRDEUM2 first 70

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Scoring Reliability – Example Open the reliability data file ISRDEUM2 first Select the „Scoring Reliability“ from the „Verify“ menu 71

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Scoring Reliability – Example Open the reliability data file ISRDEUM2 first Select the „Scoring Reliability“ from the „Verify“ menu Select the achievment data file ISADEUM2 72

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Scoring Reliability – Example 73 Open the reliability data file ISRDEUM2 first Select the „Scoring Reliability“ from the „Verify“ menu Select the achievment data file ISADEUM2 Save report, confirm any messages and open the report

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks Scoring Reliability – Report 74

Data Management Seminar, 8-11th July 2008, Hamburg WinDEM Checks – Additional Checks 1. Unique ID Check 2. Column Shift Check 3. Validation Check 4. Token Check 5. Double Punching Check 6. Scoring Reliability 7. WinW3S Linkcheck More on this in P23! NOTE: 75

Data Management Seminar, 8-11th July 2008, Hamburg Thank You very much for Your Attention! 76

Data Management Seminar, 8-11th July 2008, Hamburg Hands-On Training

Data Management Seminar, 8-11th July 2008, Hamburg Hands-On Training 1. Open WinDEM  Options  Settings… Select the path for WinDEM files 78

Data Management Seminar, 8-11th July 2008, Hamburg 2. Delete the path use backspace / delete on the keyboard [Del (English) = Entf (German)] Hands-On Training 79

Data Management Seminar, 8-11th July 2008, Hamburg 3. Open Explorer 4. Select path: D:\Seminar Examples\5_Fake data\Work_WinDem_checks_1 5. Copy path  Ctrl + c (on your keyboard) [Ctrl (English) = Strg (German)] Hands-On Training 80

Data Management Seminar, 8-11th July 2008, Hamburg 5. Paste path  Ctrl + v (on your keyboard) [Ctrl (English) = Strg (German)] Hands-On Training 81

Data Management Seminar, 8-11th July 2008, Hamburg Hands-On Training – Solve Inconsistencies What you will need: –Fake data in... D:\Seminar Examples\5_Fake data\Work_WinDem_checks_1 –Schools 1001, 2002 –Student, Teacher and School Questionnaires, Student Achievement Booklets, Reliability Scoring Sheets (Tabs 5, 8-10) Perform the following checks: –Unique ID-, Column-, Validation- and Token- Check, Scoring Reliability Check files in the following order: –ISG, ITG, ICG, ISA, ISE Additional checks: –Double Punching Check D:\Seminar Examples\4_Work_WinDem (your own work) D:\Seminar Examples\6_Masters\WinDem_database 60 min 82