1. 1 Guide to the PISA Data Analysis ManualPISA Data Analysis Manual Computation of Standard Errors for Multistage Samples

2. What is a Standard Error (SE) In PISA, as well as in IEA studies, results are based on a sample – Published statistics are therefore estimates Estimates of the means, of the standard deviations, of the regression coefficients … – The uncertainty due to the sampling process has to be quantified Standard Errors, Confidence Intervals, P Value OECD (2001). Knowledge and Skills for Life: First Results from PISA 2000. Paris: OECD.

3. Let us imagine a teacher willing to implement the mastery learning approach, as conceptualized by B.S. Bloom. Need to assess students after each lesson With 36 students and 5 lessons per day… What is a Standard Error (SE)

4. Description of the population distribution What is a Standard Error (SE) The teacher decides to randomly draw 2 student’s tests for deciding if a remediation is needed How many samples of 2 students from a population of 36 students?

5. Number of possible sample of size n from a population of size N What is a Standard Error (SE) If the thickness of a coin is 1 mm, then 1 billion of coins on the edge corresponds to 1000 km

6. What is a Standard Error (SE)

8. Graphical representation of the population mean estimate for all possible samples

9. The distribution of sampling variance on the previous slide has: – a mean of 10 – a Standard Deviation (STD) of 1.7 – The STD of a sampling distribution is denoted Standard Error (SE) What is a Standard Error (SE)

10. The sampling distribution on the mean looks like a normal distribution

11. Let us count the number of samples with a mean included between – [(10-1.96SE);(10+1.96SE)] – [(10-3.30);(10+3.30)] – [6.70;13.30] – There are: 6+28+38+52+60+70+70+70+60+52 +38+28+16=598 samples, thus 94.9 % of all possible samples With a population N(10, 5, 83), 95% of all possible samples of size 2 will have a population mean estimate included between 6.70 and 13.30 What is a Standard Error (SE)

12. Sampling distribution of the mean estimates of all possible samples of size 4

13. What is a Standard Error (SE) The distribution of sampling variance on the previous slide has: – a mean of 10 – a Standard Deviation of 1.7

14. Distribution of the scores versus distribution of the mean estimates What is a Standard Error (SE)

15. The sampling variance of the mean is inversely proportional to the sample size: – If two students are sampled, then the smallest possible mean is 5.5 and the highest possible mean is 14.5 – If four students are sampled, it ranges from 6 to 14 – If 10 students are sampled, it ranges from 7 to 13 The sampling variance is proportional to the variance: – If the score are reported on 20, with a sample of size 2, then it ranges from 5,5 to 14,5 – If the score are reported on 40 (multiplied by 2) then it ranges from 11 to 29 What is a Standard Error (SE)

16. Individual 1Individual 2Individual 3Individual 4Mean estimates Sample 1X 11 X 21 X 31 X 41 û1û1 Sample 2X 12 X 22 X 32 X 42 û2û2 Sample 3X 13 X 23 X 33 X 43 û3û3 Sample 4X 14 X 24 X 34 X 44 û4û4 Sample 5X 15 X 25 X 35 X 45 û5û5 Sample 6X 16 X 26 X 36 X 46 û6û6 Sample 7X 17 X 27 X 37 X 47 û7û7 Sample 8X 18 X 28 X 38 X 48 û8û8 Sample 9X 19 X 29 X 39 X 49 û9û9 …… Sample XX 1x X 2x X 3x X 4x ûxûx What is a Standard Error (SE)

18. As we don’t know the variance in the population, the SE for a mean of  as obtained from a sample is calculated as similarly, the SE for a percentage P is calculated as What is a Standard Error (SE)

19. Linear regression assumptions: – Homoscedasticity the variance of the error terms is constant for each value of x – Linearity the relationship between each X and Y is linear – Error Terms are normally distributed – Independence of Error Terms successive residuals are not correlated If not, the SE of regression coefficients is biased What is a Standard Error (SE)

20. With a multistage sample design, errors will be correlated

21. Multistage samples are usually implemented in International Surveys in Education: – schools (PSU=primary sampling units) – classes – students If schools/classes / students are considered as infinite populations and if units are selected according to a SRS procedures, then: Standard Errors for multistage samples

22. PISA: – 2 stage samples : schools and then students IEA: – 2 stage samples : schools and then 1 class per selected school Standard Errors for multistage samples

23. Three fictitious examples in PISA Standard Error for multistage samples If considered as a SRS or random assignment to schools

24. 24 Standard Error for multistage samples

25. 25 Variance Decomposition for Reading Literacy in PISA 2000 Standard Error for multistage samples

26. Impact of the stratification variables on sampling variance N 1, N 2 considered as constant Independent samples so COV=0

27. 27 EffectSum of Squares Degree of Freedom Mean square Gender (50F+50G)2500L-1 (1)2500 ERROR7500N-L (98)76.53 TOTAL10000N-1 (99)101.01 Standard Error for multistage samples

28. School and within school variances of the student performance in reading, intraclass correlation with and without control of the explicit stratification variables (OECD, PISA 2000 database) Country School variance Within school variance School variance under control of stratification Rho Rho under control of stratification AUT635642436240.600.13 BEL7050472434890.600.42 CHE4517590931190.430.35 CZE481242036040.530.13 DNK1819797016960.190.18 ESP147756498230.210.13 FIN99870968690.120.11 FRA418142199100.500.18 GBR2077763719900.21 GRC4995490736190.500.42 HUN6604323046380.670.59 IRL1589734914950.180.17 ISL65278845630.080.07 ITA4719402820310.540.34 Standard Error for multistage samples

29. Consequences of considering PISA samples as simple random samples – In most cases, underestimation of the sampling variance estimates Non significant effect will be reported as significant How can we measure the risk? Computation of the Type I error Standard Error for multistage samples

30. Consequences : Type I error underestimation

31. Sampling Variance Standard Error RatioRatio x Z scoreType I Error Unbiased estimate 244.90 Biased estimate 204.47 0.911.79 0.07 164.00 0.821.60 0.11 123.46 0.711.38 0.17 82.83 0.581.13 0.26 42.00 0.410.80 0.42 Standard Error for multistage samples Consequences : Type I error underestimation

32. Sampling Design Effect Standard Error for multistage samples

33. CountrySDEType ICountrySDEType I Australia5.90 0.42 Korea5.89 0.42 Austria3.10 0.27 Latvia10.16 0.54 Belgium7.31 0.47 Liechtenstein0.48 0.00 Brazil6.14 0.43 Luxembourg0.73 0.02 Canada9.79 0.53 Mexico6.69 0.45 Czech Republic3.18 0.27 Netherlands3.52 0.30 Denmark2.36 0.20 New Zealand2.40 0.21 Finland3.90 0.32 Norway2.97 0.26 France4.02 0.33 Poland7.12 0.46 Germany2.36 0.20 Portugal9.72 0.53 Greece12.04 0.57 Russian Federation13.53 0.59 Hungary8.64 0.50 Spain6.18 0.43 Iceland0.73 0.02 Sweden2.32 0.20 Ireland4.50 0.36 Switzerland10.52 0.55 Italy1.90 0.16 United Kingdom5.97 0.42 Japan19.28 0.66 United States17.29 0.64 Standard Error for multistage samples Sampling design effect in PISA 2000 Reading

34. Factors influencing the SE other than the sample size – School Variance: depending on the variable Usually high for performance Low for other variables – Efficiency of the stratification variables A stratification variable can be efficient for some variables and not for others – Population parameter estimates Standard Error for multistage samples

35. A few examples (PISA2000, Belgium) – Intraclass correlation Performance in reading: Rho=0.60 SDE=7.19 Social Background (HISEI): Rho= 0.24 SDE=3.45 Enjoyment for Reading: Rho=0.10 SDE=1.86 – Statistics Regression analyses: Reading = HISEI + GENDER – InterceptSDE= 5.50 – HISEISDE=3.78 – GENDERSDE=3.91 Logistic regression Level (0/1 Reading below or above 500) =HISEI – InterceptSDE= 2.39 – HISEISDE=2.09 Standard Error for multistage samples

36. Very few mathematical solutions for the estimation of the sampling variance for multistage samples – For mean estimates under the condition Simple Random Sample (SRS) and stratified Probability Proportional to Size (PPS) sample but with no stratification variables – No mathematical solutions for other statistics – Use of replication methodologies for the estimation of sampling variance For SRS: – Jackknife: n replications de n-1 cases – Bootstrap : an infinite number of samples of n cases randomly drawn with replacement. Standard Error for multistage samples

37. Student12345678910Mean Value1011121314151617181914.50 Replication 1011111111115.00 Replication 2101111111114.88 Replication 3110111111114.77 Replication 4111011111114.66 Replication 5111101111114.55 Replication 6111110111114.44 Replication 7111111011114.33 Replication 8111111101114.22 Replication 9111111110114.11 Replication 10111111111014.00 Replication methods for SRS Jackknife for SRS

38. Replication methods for SRS Jackknife for SRS – Estimation of the SE by replication – Estimation by using the mathematical formula

39. Replication methods for SRS

40. Student12345678910Mean Value1011121314151617181914.50 Structure 1111111111114.50 Structure 22111111110From 13.7 to 15.4 Structure 32211111100From 12.9 to 16.1 Structure511111 0000 611110 0000 7111 000000 811 0000000 91 00000000 10 000000000 From 10 to 19 Replication methods for SRS Bootstrap for SRS

41. ReplicateR1R2R3R4R5R6R7R8R9R10 School 10.001.11 School 21.110.001.11 School 31.11 0.001.11 School 41.11 0.001.11 School 51.11 0.001.11 School 61.11 0.001.11 School 71.11 0.001.11 School 81.11 0.001.11 School 91.11 0.001.11 School 101.11 0.00 Replication methods for multistage sample Jackknife for unstratified Multistage Sample Each replicate= contribution of a school

42. Pseudo- stratum SchoolR1R2R3R4R5 1121111 1201111 2310111 2412111 3511211 3611011 4711101 4811121 5911112 51011110 Replication methods for SRS Jackknife for stratified Multistage Sample Each replicate= contribution of a pseudo stratum

43. Pseudo- stratum SchoolR1R2R3R4R5R6R7R8 1122222222 1200000000 2320202020 2402020202 3522002200 3600220022 4720022002 4802200220 5922220000 51000002222 Replication methods for multistage sample Balanced Replicated Replication Each replicate= an estimate of the sampling variance

44. IDSizeFromToSAMPLED 115 1 1 220 26350 325 36600 430 61900 535 911251 640 1261650 745 1662100 850 2112601 960 2613200 1080 3214001 Total400 – Within explicit strata, with a systematic sampling procedure, schools are sequentially selected. – Pairs are formed according to the sequence School 1 with School 5 School 8 with School 10 Replication methods for multistage sample How to form the pseudo-strata, i.e. how to pair schools?

45. IEA TIMSS / PIRLS procedure IDParticipationPseudo- Stratum 1411 2111 3512 560 7812 9913 1030 11513 12614 13714 Replication methods for multistage sample PISA procedure IDParticipationPseudo- Stratum 1411 2111 3512 5602 7813 9913 10304 11514 12615 13715 How to form the pseudo-strata, i.e. how to pair schools?

46. Stratum 1Stratum 2Stratum 3Stratum 4 11111 21112 31121 41211 52111 61122 71212 82112 91221 102121 112211 121222 132122 142212 152221 162222 Replication methods for multistage sample Balanced Replicated Replication – With L pseudo-strata, there are 2 L possible combinations – If 4 strata, then 16 combinations – Same efficiency with an Hadamard Matrix of Rank 4

47. Each row is orthogonal to all other rows, i.e. the sum of the products is equal to 0. Selection of school according to this matrix Combination 11111 211 311 41 1 Replication methods for multistage sample Hadamard Matrix

48. Replication methods for multistage sample

49. Pseudo- stratum SchoolR1R2R3R4R5R6R7R8 111.5 120.5 231.50.51.50.51.50.51.50.5 24 1.50.51.50.51.50.51.5 35 0.5 1.5 0.5 36 1.5 0.5 1.5 47 0.5 1.5 0.5 1.5 480.51.5 0.5 1.5 0.5 591.5 0.5 5100.5 1.5 Replication methods for multistage sample Fays method

50. General formula BRR / Fay : each replicate is an estimate of the sampling variance C = average Same number of replicate for each country JK2 : each replicate corresponds of the pseudo-stratum to the sampling variance estimate C = sum Possibility of different number of replicates Replication methods for multistage sample

51. In the case of infinite populations, the sampling variance of the mean estimate consists of 2 components in the case of a PISA sampling design: – Between school variance – Within school variance Replication methods for multistage sample Replication methods, by removing entire schools, only integrate the uncertainty due to the selection of schools, not due to the selection of students within schools

52. Replication methods for multistage sample Let us image an educational system with no school variance and with infinite populations of students within schools

53. SchoolFullR1R2R3R4R5R6R7R8R9R10 1100 2110 3120 4130 5140 6150 7160 8170 9180 10190 Mean145.0150.0148.9147.8146.7145.6144.4143.3142.2141.1140.00 25.0015.127.722.780.31 2.787.7215.1225.00 Replication methods for multistage sample An illustration of this mathematical equality: – Estimation of the SE by JK1 replication method

54. School School mean 11002025 21101225 3120625 4130225 514025 615025 7160225 8170625 91801225 101902025 Mean145 SS8250 MS916.666667 Replication methods for multistage sample An illustration of this mathematical equality: – Estimation of the SE by the formula