1. 3rd NRC Meeting, 9-12 June 2008, Windsor 1 ICCS 2009 Main Survey Sampling Quality Standards

2. 3rd NRC Meeting, 9-12 June 2008, Windsor 2 Overview Why do we need sampling quality standards? Sampling quality standards –For the student survey –For the teacher survey Examples of how to meet the quality standards

3. 3rd NRC Meeting, 9-12 June 2008, Windsor 3 Sampling quality standards Quality standards require low exclusion rates high participation rates of –schools –students –teachers

4. 3rd NRC Meeting, 9-12 June 2008, Windsor 4 Why sampling quality standards? Needed to avoid biased estimates e.g., excluded groups of students could perform different from non-excluded students e.g., non-responding teachers could have a different attitude towards citizenship than responding teachers Needed to ensure comparability between countries Needed to achieve quality of IEA surveys

5. 3rd NRC Meeting, 9-12 June 2008, Windsor 5 Student survey: exclusion rates Students can be excluded at three levels Excluded schools before sampling Excluded classes in sampled schools Excluded students in sampled classes Overall exclusions must not exceed 5%

6. 3rd NRC Meeting, 9-12 June 2008, Windsor 6 Student survey – participating school A school is regarded as participating if 50% of the sampled students replied. In a few countries, two ore more classes will be sampled For details, please join us in the country consultations

7. 3rd NRC Meeting, 9-12 June 2008, Windsor 7 Student survey - participation categories Three categories will be defined that describe the quality of the sample implementation Of course, the aim is 100% of participation at all levels The categories are defined as in other IEA student surveys The categories refer to both school participation rates and student participation rates

8. 3rd NRC Meeting, 9-12 June 2008, Windsor 8 Student survey participation rates – Category 1 A country will be placed in category 1 if the school participation rate is ≥ 85% before using replacement schools AND the student participation rate is ≥ 85%.

9. 3rd NRC Meeting, 9-12 June 2008, Windsor 9 Student survey participation rates – Category 2 A country will be placed in category 2 if the school participation rate is ≥ 50% before using replacement schools AND the school participation rate is ≥ 85% after using replacement schools AND the student participation rate is ≥ 85%.

10. 3rd NRC Meeting, 9-12 June 2008, Windsor 10 Base of student participation rate The base for each rate is different The school participation rate is based on all sampled schools The student participation rate is based only on the participating schools (sampled and replacement)

11. 3rd NRC Meeting, 9-12 June 2008, Windsor 11 Student survey participation rates - second chance Low response rates at one level can be compensated with high response rates at another level If a country reaches 75% combined participation rate... before using replacement schools: category 1 after using replacement schools: category 2

12. 3rd NRC Meeting, 9-12 June 2008, Windsor 12 Example for combined participation rate 80% schools participation rate before replacement 95% student participation rate within schools 0.80 x 0.95 = 0.76 0.76 ≥ 0.75  OK!

13. 3rd NRC Meeting, 9-12 June 2008, Windsor 13 Student survey participation rates – third chance These requirements can be met either with unweighted data or with weighted data. A country will be placed in category 3 if it fails to meet the requirements for category 1 and 2

14. 3rd NRC Meeting, 9-12 June 2008, Windsor 14 Teacher survey For the teacher survey, similar requirements apply Teacher participation may be harder to achieve than student participation However, there is no statistical justification to treat teacher data differently from student data

15. 3rd NRC Meeting, 9-12 June 2008, Windsor 15 Teacher survey – participating school A school is regarded as participating if at least 50% of the sampled teachers participate

16. 3rd NRC Meeting, 9-12 June 2008, Windsor 16 Teacher Survey participation rates – Category 1 A country will be placed in category 1 if the school participation rate is ≥ 85% before using replacement schools AND the teacher participation rate is ≥ 85%.

17. 3rd NRC Meeting, 9-12 June 2008, Windsor 17 Teacher Survey participation rates – Category 2 A country will be placed in category 2 if the school participation rate is ≥ 50% before using replacement schools AND the school participation rate is ≥ 85% after using replacement schools AND the teacher participation rate is ≥ 85%.

18. 3rd NRC Meeting, 9-12 June 2008, Windsor 18 Teacher survey participation rates - second chance and third chance Low response rates at one level can be compensated with high response rates at another level If a country reaches 75% combined participation rate... before using replacement schools: category 1 after using replacement schools: category 2 All rates will be calculated unweighted and weighted A country will be placed in category 3 if it fails to meet the requirements for category 1 and 2

19. 3rd NRC Meeting, 9-12 June 2008, Windsor 19 Reporting policy All data will be published! Readers of the international report should be informed about the quality of sample implementation Not meeting the requirements in one survey does not affect the reporting in the other survey International Reports: different form of display Category 1: without annotation Category 2: annotated Category 3: in a separate section of the tables

20. 3rd NRC Meeting, 9-12 June 2008, Windsor 20 How can you enhance participation? Examples of good practice Taken from the SAQ and form other surveys Some of these examples may work in your in your country

21. 3rd NRC Meeting, 9-12 June 2008, Windsor 21 (1) Gain political support Backing from the Ministry of Education –Official bulletins –On-line newsletters –MOE website –Ministry letters to schools, or even to individual teachers Support by teacher unions A steering committee underlines importance of the study

22. 3rd NRC Meeting, 9-12 June 2008, Windsor 22 (2) Make yourself seen Articles in teacher journals Presentation of some FT results as examples Flyers and brochures Put emphasis on the international character of the study

23. 3rd NRC Meeting, 9-12 June 2008, Windsor 23 (3) Dispel doubts Explanation of the purpose of the study Ensure confidentiality and data protection Teacher meetings at sampled schools

24. 3rd NRC Meeting, 9-12 June 2008, Windsor 24 (4) Provide motivation Feedback to schools Incentives to schools Incentives to teachers –Tokens –Lottery

25. 3rd NRC Meeting, 9-12 June 2008, Windsor 25 (5) Organize data collection best Early school contact Avoid testing in busy times of school year Avoid testing at the same time as other surveys Support school coordinator in his work Support of school principal Support of DPC

26. 3rd NRC Meeting, 9-12 June 2008, Windsor 26 Thank you for your attention!