HLM with Educational Large-Scale Assessment Data: Restrictions on Inferences due to Limited Sample Sizes Sabine Meinck International Association for the Evaluation of Educational Achievement (IEA), Data Processing and Research Center, Germany, Hamburg Caroline Vandenplas University of Lausanne, Switzerland

Introduction: Inferences and Sample Sizes Usually, researchers are not interested in features of the sample, but want to infer from it on population features. Remember your statistics course: When inferring from sample data on populations, You just estimate a population feature. You need to indicate the “uncertainty” or precision of your estimate. The related measure is the standard error (s.e.). Using the standard error, You can build a confidence interval (CI) around the mean as You can test for group differences. Look up you statistic books for the formula on estimating se! Or, when doing it for LSA, look in the User Guides!

Introduction: Inferences and Sample Sizes Sometimes, we cannot detect “significant” differences between groups to be compared. However: The fact that we cannot detect a difference does not mean there is none!! Based on our data, we just don’t know… How can we get more precise results? One possibility: Increase the sample size How/why does this work?

The Sample is Picturing the Population… What if a picture is our population? Has about 340,000 pixels (if I remember correctly…)

The Sample is Picturing the Population… Sample Size = 1,000 10,000 50,000 We can play with the sample size to change the precision of our picture. With increasing sample size, sampling error reduces. Measure for this preciseness: Standard error (s.e.)

Introduction: Inferences and Sample Sizes This relationship between sample size and s.e. holds for any estimated parameter. Percentages Correlation coefficients Regression coefficients Etc. It also holds for any estimated coefficient of a hierarchical model! The relationship is not linear though and can depend on many factors.

Connection to HLM What is HLM? HLM = Hierarchical Linear Modeling Nice introduction to HLM give, e.g., Snijder & Bosker (1999) Analysis method that addresses the hierarchical structure of data/populations Almost all datasets from large-scale assessments (LSA) display a hierarchical structure. E.g., students nested in classes/schools (TIMSS, PIRLS, PISA …) Effects playing out at different levels of a hierarchy can be disentangled. We can specify “fixed” and “random” effects. HLM is an enhancement of linear regression analysis… Students nested in teachers. (TIPI) Teachers nested in schools. (ICCS, ICILS) Example Random effects: If we know the slope of some parameter differs between cluster a lot, we may want to let it vary at random.

From Linear Regression to HLM Assume we are trying to predict student math achievement from SES scores. Linear regression model: 𝑦 𝑖 = 𝛽 0 + 𝛽 1 𝑥 𝑖 + ε, for i = 1, …,n where 𝑦 𝑖 = student achievement, 𝛽 0 and 𝛽 1 are unknown coefficients, 𝑥 𝑖 = student SES, ε = error term (residual variance) This model confounds effects between and within groups! Model parameters to be estimated.  All subject to sampling error! Subscript “i” denotes students in the sample.

Example: What if we have a case like this?

Example: What if we have a case like this? Linear regression, no consideration of cluster effects

Example – Individuals Belong to Clusters! Intercepts can vary (“random intercepts”) Slopes are fixed (parallel lines) Slopes can vary (“random slopes”)

From Linear Regression to HLM Extending the model allows us to disentangle the effects at different levels (= Hierarchical Linear Model) E.g., we can specify a model with one explanatory variable at level 1 and level 2, the intercept and the slope are random. Example research question: Does the influence of SES on achievement vary between schools, i.e., does SES affect students achievement in different schools in different magnitudes or even directions? We acknowledge the fact that within each school, we may have a different regression line. Residuals at both levels are assumed to follow normal distributions with zero means.

From Linear Regression to HLM Hierarchical model: The research question can be answered by measuring by U1 and its significance. Model parameters to be estimated.  All subject to sampling error! First equation = normal regression equation. Specifying β0 (second equation), we acknowledge the fact that each school has a different intercept AND we consider the average SES at school level. Specifying β1 (last equation), we acknowledge the fact that each school has a regression line with a different slope. U0, U1 & ε are all variances. Residuals at both levels are assumed to follow normal distributions with zero means.

Purpose of this Research Increasing demand to apply hierarchical linear models to educational large-scale assessment (LSA) data FAQ: “How many units do I need on different levels of hierarchy to do meaningful multilevel modeling?“ (secondary researchers, survey designers) “Meaningful“ = achieving parameter estimates with certain precision levels

Data and Methods We utilized a Monte-Carlo simulation study, mimicking typical large-scale assessment data: Selection of samples from a virtual population with 2 hierarchical levels Varied parameters: Sample sizes at both levels Intra-class correlation coefficient Selection probabilities of level 2 units Covariance distributions between levels Model complexity 288 sampling scenarios were explored, each with 6,000 replicates

Research Question What is the association between sample sizes and precision of the estimated parameters of hierarchical models under varying population conditions?

Results Precision of all explored parameters increase when sample size increases in all scenarios No rule of thumb concerning required sample sizes can be given! Rather, required sample sizes depend heavily on the parameter of interest: Sampling precision levels vary extremely for different model parameters Sample size requirements differ widely for the estimation of fixed model parameters vs. the estimation of variances Research interest in macro-level regression coefficients: Increase the number of sampled clusters Focus on variance estimates: The level at which the sample size is increased is less important In opposition to what is often suggested in the literature, no rule of thumb…

Results With the sample sizes typically employed in LSA, some parameters cannot be estimated with sufficient precision (i.e., being significantly different from zero) in relatively simple hierarchical models With such imprecise parameters, group differences are even less likely to be detected Mean of random intercepts, residual variance: about 1% to 5% Slope of random intercepts: > 100% (S.E. Bigger than the parameter)

Conclusions Be mindful when phrasing your research questions and interpreting your results: Does the data at hand actually suits the type of analysis? What are relevant group differences that you want to detect? Examine closely standard errors of the explored model parameters. Full paper with practical guidelines and detailed results can be downloaded at http://www.ierinstitute.org/dissemination-area.html (IERI Monograph Series Special Issue 1, October 2012) Mean of random intercepts, residual variance: about 1% to 5% Slope of random intercepts: > 100% (S.E. Bigger than the parameter)

Thank you for your attention! We are grateful to the National Center for Education Statistics (NCES), U.S. Department of Education, who funded the project. Contact: sabine.meinck@iea-dpc.de caroline.vandenplas@unil.ch