Multilevel Models 1 Sociology 229A Copyright © 2008 by Evan Schofer Do not copy or distribute without permission

Multilevel Data Often we wish to examine data that is “clustered” or “multilevel” in structure –Classic example: Educational research Students are nested within classes Classes are nested within schools Schools are nested within districts or US states We often refer to these as “levels” Ex: If the study is individual/class/school… Level 1 = individual level Level 2 = classroom Level 3 = school –Note: Some stats books/packages label differently!

Multilevel Data Students nested in class, school, and state Variables at each level may affect student outcomes Class School Class School California Class School Class School Oregon

Multilevel Data Simpler example: 2-level data Which can be shown as: Class 1 S1S1 S2S2 S3S3 Class 2 S1S1 S2S2 S3S3 Class 3 S1S1 S2S2 S3S3 Level 2 Level 1

Multilevel Data We are often interested in effects of variables at multiple levels Ex: Predicting student test scores Individual level: grades, SES, gender, race, etc. Class level: Teacher qualifications, class size, track School: Private vs. public, resources State: Ed policies (funding, tests), budget –And, it is useful to assess the relative importance of each level in predicting outcomes Should educational reforms target classrooms? Schools? Individual students? Which is most likely to have big consequences?

Multilevel Data Repeated measurement is also “multilevel” or “clustered” Measurement at over time (T 1, T 2, T 3 …) is nested within persons (or firms or countries) Level 1 is the measurement (at various points in time) Level 2 = the individual Person 1 T2T2 T1T1 T4T4 T3T3 T5T5 Person 2 T2T2 T1T1 T4T4 T3T3 T5T5 Person 3 T2T2 T1T1 T4T4 T3T3 T5T5 Person 4 T2T2 T1T1 T4T4 T3T3 T5T5

Multilevel Data Examples of multilevel/clustered data: Individuals from same family –Ex: Religiosity People in same country (in a cross-national survey) –Ex: Civic participation Firms from within the same industry –Ex: Firm performance Individuals measured repeatedly –Ex: Depression Workers within departments, firms, & industries –Ex: Worker efficiency –Can you think of others?

Example: Pro-environmental values Source: World Values Survey (27 countries) Let’s simply try OLS regression. reg supportenv age male dmar demp educ incomerel ses Source | SS df MS Number of obs = F( 7, 27799) = Model | Prob > F = Residual | R-squared = Adj R-squared = Total | Root MSE = supportenv | Coef. Std. Err. t P>|t| [95% Conf. Interval] age | male | dmar | demp | educ | incomerel | ses | _cons |

Aggregation If you want to focus on higher-level hypotheses (e.g., schools, not children), you can aggregate Make “school” the unit of analysis OLS regression analysis of school-level variables Individual-level variables (e.g., student achievement) can be included as school averages (aggregates) –Ex: Model average school test score as a function of school resources and average student SES Problem: Approach destroys individual-level data Also: Loss of statistical power (Tabachnick & Fidel 2007) Also: Can’t draw individual-level interpretations: ecological fallacy.

Example: Pro-environmental values Aggregation: Analyze country means (N=27). reg supportenv age male dmar demp educ incomerel ses Source | SS df MS Number of obs = F( 7, 19) = 0.91 Model | Prob > F = Residual | R-squared = Adj R-squared = Total | Root MSE = supportenv | Coef. Std. Err. t P>|t| [95% Conf. Interval] age | male | dmar | demp | educ | incomerel | ses | _cons | Note loss of statistical power – few variables are significant when N is only 27

Ecological Fallacy Issue: Data aggregation limits your ability to draw conclusions about level-1 units The “ecological fallacy” –Robinson, W.S. (1950). "Ecological Correlations and the Behavior of Individuals". American Sociological Review 15: 351–357 Among US states, immigration rate correlates positively with average literacy Does this mean that immigrants tend to be more literate than US citizens? NO: You can’t assume an individual-level correlation! –The correlation at individual level is actually negative –But: immigrants settled in states with high levels of literacy – yielding a correlation in aggregate statistics.

OLS Approaches Another option: Just use OLS regression Allows you to focus on lower-level units –No need for aggregation Ex: Just analyze individuals as the unit of analysis, ignoring clustering among schools Include independent variables measured at the individual-level and other levels Problems: 1. Violates OLS assumptions (see below) 2. OLS is too limited; can’t take advantage of richness of multilevel data –Ex: Complex variation in intercepts, slopes across groups.

Multilevel Data: Problems Issue: Multilevel data often results in violation of OLS regression assumption OLS requires an independent random sample… Students from the same class (or school) are not independent… and may have correlated error –If you don’t control for sources of correlated error, models tend to underestimate standard errors This leads to false rejection of H0 –Too many asterisks in table (Type I error) This is a serious issue, as we always want to err in the direction of conservatism… false findings = bad!

Multilevel Data: Problems Why might nested data have correlated error? –Example: Student performance on a test Students in a given classroom may share & experience common (unobserved) characteristics Ex: Maybe the classroom is too dark, causing all students to perform poorly on tests –If all those students score poorly, they fall below the regression line… and have negative error –But OLS regression requires that error be “random” –Within-class error should be random, not consistently negative –Other sources of within-class (or school) error An especially good teacher; poor school funding Other ideas?

Multilevel Data: Problems Sources of correlated error within groups –Ex: Cross-national study of homelessness People in welfare states have a common unobserved characteristic: access to generous benefits –Ex: Study of worker efficiency in workgroups Group members may influence each other (peer pressure) leading to group commonalities.

Multilevel Data: Problems When is multilevel data NOT a problem? –Answer: If you can successfully control for potential sources of correlated error Add a control to OLS model for: classroom, school, and state characteristics that would be sources of correlated error in each group Ex: Teacher quality, class size, budget, etc… But: We often can’t identify or measure all relevant sources of correlated error Thus, we need to abandon simple OLS regression and try other approaches.

Example: Pro-environmental values Source: World Values Survey (~26 countries). reg supportenv age male dmar demp educ incomerel ses Source | SS df MS Number of obs = F( 7, 27799) = Model | Prob > F = Residual | R-squared = Adj R-squared = Total | Root MSE = supportenv | Coef. Std. Err. t P>|t| [95% Conf. Interval] age | male | dmar | demp | educ | incomerel | ses | _cons |

Robust Standard Errors Strategy #1: Improve our estimates of the standard errors –Option 1: Robust Standard Errors reg y x1 x2 x3, robust The Huber / White / “Sandwich” estimator An alternative method of computing standard errors that is robust to a variety of assumption violations –Provides accurate estimates in presence of heteroskedasticity Also, robust to model misspecification –Note: Freedman’s criticism: What good are accurate SEs if coefficients are biased due to poor specification?

Example: Pro-environmental values Robust Standard Errors. reg supportenv age male dmar demp educ incomerel ses, robust Linear regression Number of obs = F( 7, 27799) = Prob > F = R-squared = Root MSE = | Robust supportenv | Coef. Std. Err. t P>|t| [95% Conf. Interval] age | male | dmar | demp | educ | incomerel | ses | _cons | Standard errors shift a tiny bit… fairly similar to OLS in this case

Robust Cluster Standard Errors Option 2: Robust cluster standard errors –A modification of robust SEs to address clustering reg y x1 x2 x3, cluster(groupid) –Note: Cluster implies robust (vs. regular SEs) It is easy to adapt robust standard errors to address clustering in data; See: – – Result: SE estimates typically increase, which is appropriate because non-independent cases aren’t providing as much information as would a sample of independent cases.

Example: Pro-environmental values Robust Cluster Standard Errors. reg supportenv age male dmar demp educ incomerel ses, cluster(country) Linear regression Number of obs = F( 7, 25) = Prob > F = R-squared = Number of clusters (country) = 26 Root MSE = | Robust supportenv | Coef. Std. Err. t P>|t| [95% Conf. Interval] age | male | dmar | demp | educ | incomerel | ses | _cons | Cluster standard errors really change the picture. Several variables lose statistical significance.

Dummy Variables Another solution to correlated error within groups/clusters: Add dummy variables Include a dummy variable for each Level-2 group, to explicitly model variance in means A simple version of a “fixed effects” model (see below) Ex: Student achievement; data from 3 classes Level 1: students; Level 2: classroom Create dummy variables for each class –Include all but one dummy variable in the model –Or include all dummies and suppress the intercept

Dummy Variables What is the consequence of adding group dummy variables? A separate intercept is estimated for each group Correlated error is absorbed into intercept –Groups won’t systematically fall above or below the regression line In fact, all “between group” variation (not just error) is absorbed into the intercept –Thus, other variables are really just looking at within group effects –This can be good or bad, depending on your goals.

Dummy Variables Note: You can create a set of dummy variables in stata as follows: xi i.classid – creates dummy variables for each unique value of the variable “classid” –Creates variables named _Iclassid_1, _Iclassid2, etc These dummies can be added to the analysis by specifying the variable: _Iclassid* Ex: reg y x1 x2 x3 _Iclassid*, nocons –“nocons” removes the constant, allowing you to use a full set of dummies. Alternately, you could drop one dummy.

Example: Pro-environmental values Dummy variable model. reg supportenv age male dmar demp educ incomerel ses _Icountry* Source | SS df MS Number of obs = F( 32, 27774) = Model | Prob > F = Residual | R-squared = Adj R-squared = Total | Root MSE = supportenv | Coef. Std. Err. t P>|t| [95% Conf. Interval] age | male | dmar | demp | educ | incomerel | ses | _Icountry_32 | _Icountry_50 | _Icountry_70 | … dummies omitted … _Icountr~891 | _cons |

Dummy Variables Benefits of the dummy variable approach It is simple –Just estimate a different intercept for each group sometimes the dummy interpretations can be of interest Weaknesses Cumbersome if you have many groups Uses up lots of degrees of freedom (not parsimonious) Makes it hard to look at other kinds of group dummies –Non-varying group variables = collinear with dummies Can be problematic if your main interest is to study effects of variables across groups –Dummies purge that variation… focus on within-group variation –If you don’t have much within group variation, there isn’t much left to analyze.

Dummy Variables Note: Dummy variables are a simple example of a “fixed effects” model (FEM) Effect of each group is modeled as a “fixed effect” rather than a random variable Also can be thought of as the “within-group” estimator –Looks purely at variation within groups –Stata can do a Fixed Effects Model without the effort of using all the dummy variables Simply request the “fixed effects” estimator in xtreg.

Fixed Effects Model (FEM) Fixed effects model: For i cases within j groups Therefore  j is a separate intercept for each group It is equivalent to solely at within-group variation: X-bar-sub-j is mean of X for group j, etc Model is “within group” because all variables are centered around mean of each group.

Fixed Effects Model (FEM). xtreg supportenv age male dmar demp educ incomerel ses, i(country) fe Fixed-effects (within) regression Number of obs = Group variable (i): country Number of groups = 26 R-sq: within = Obs per group: min = 511 between = avg = overall = max = 2154 F(7,27774) = corr(u_i, Xb) = Prob > F = supportenv | Coef. Std. Err. t P>|t| [95% Conf. Interval] age | male | dmar | demp | educ | incomerel | ses | _cons | sigma_u | sigma_e | rho | (fraction of variance due to u_i) F test that all u_i=0: F(25, 27774) = Prob > F = Identical to dummy variable model!