Slide 1 Estimating Performance Below the National Level Applying Simulation Methods to TIMSS Fourth Annual IES Research Conference Dan Sherman, Ph.D. American Institutes for Research June 8, 2009

Slide 2 Current TIMSS Not Appropriate For Direct State-level Estimates  Sample is designed to produce national rather than state estimates  Small number of schools sampled in about 40 states in 2007 –234 public schools with 4 th grade scores CA has 30+ schools, FL, TX, and NY have 10+ schools –207 public schools with 8 th grade scores CA has 20+ schools, TX, NY, and MI have 10+ schools  Large variations in school means within states makes direct estimates sensitive to choice of schools

Slide 3 One Alternative to Direct Estimation for States Is to Use Regression Model Approach  Relate individual school-level mean score to variables observed for school, district, and state –Regression can take account of data structure (e.g., clustering of schools) in estimation  Use regression coefficients to create expected score for schools outside TIMSS sample  “Add up” expected scores across schools (weighted by number of students within state) –Provides estimate of mean score for state  Variance estimation is analytically complex; here handled by simulation

Slide 4 Overview of Regression Model  Model begins with Common Core of Data (CCD) variables on right hand side –Share of students by gender, race, poverty status; also school size and location indicator (i.e.,urban, rural) –Can apply these variables to all public schools in sample  Then add average 2007 NAEP state math score for grade –Helps measure whether school in high or low performing state, relative to expectation of CCD variables  Also add in relative performance of school on state math test (percent proficient in school in SD units relative to mean percent proficient in state) –Helps adjust for position of school in state compared to other schools

Slide 5 Goodness of Fit (Adjusted R 2 ) for Alternative Specifications Model Demographics Only Locality / School Size Added NAEP Score added Relative Position on State Test Added Grade 4 Math Grade 4 Science Grade 8 Math Grade 8 Science

Slide 6 Summary of Alternative Regression Specifications  Models explain significant portion of variation in TIMSS (much more than PISA)  Demographics are (collectively) most significant group of variables –Poverty status is single best predictor variable –State NAEP score related to school mean, as is (strongly) position in state on test  Key is to predict large share of variance in mean scores of sampled schools –Larger R 2 is desirable to closely track school mean estimates; otherwise will have large variance when school estimates aggregated to states

Slide 7 Validation of School-level Model with (Three) Individual States  Can validate outside the model by predicting mean scores in 2 states with larger TIMSS samples (MA and MN with approximately 50 schools per grade per state –California design part of national sample  Fit (R 2) between actual means and those predicted by model similar to overall model:  Obviously would like to validate/ examine model fit to schools in other states Grade/ Subject MassachusettsMinnesotaCalifornia Grade 4 Math Grade 4 Science Grade 8 Math Grade 8 Science

Slide 8 Illustration of Model Fit to Individual School Data – High R 2

Slide 9 Illustration of Model Fit to Individual School Data - Lower R 2

Slide 10 Approach to computing State-Level Means  Create estimate of school-level mean from TIMSS sample to create point estimate of mean for each school in CCD  Weight school estimates up to state-level mean using CCD number of students in grade  Should be unbiased in terms of expectation across repeated samples of schools

Slide 11 Sources of Variance in Estimates of State-level Means 1.Mix and characteristics in TIMSS sample used in given regression –Different sample will produce different coefficients 2.Individual school estimate has prediction error around its expectation from regression –Standard error of regression – reduces with R 2 and increases with distance from sample mean –Mean square error of regressions is about 20–25 points 3.Measurement error around mean (relatively small – about 5 points) for sampled schools

Slide 12 Estimating Variance in State-Level Means 1.Take coefficients from regression model using 2007 TIMSS sample as given 2.Predict expected mean score for all regular public schools in CCD and add sources of error (i.e., random draws from distributions) for individual schools 3.Apply TIMSS sampling methodology (or any other!) and draw sample of schools 4.Estimate regression for this sample and compute coefficients 5.Compute state means from coefficient sample 6.REPEAT procedure drawing different samples to compute state-level means 7.Summarize means and standard deviation of estimated means (i.e, SEs) by state

Slide 13 Illustration of National Results (4 th Grade Math Scores)

Slide 14 Summary of National Means and Standard Errors  Regression/ simulation provides national estimates similar to direct estimates  Can be “apportioned” to states pulling out observations for individual states Grade/ Subject Model Estimate Mean (SE) Direct Estimate Mean (SE) Grade 4 Math528 (3.4)527 (2.6) Grade 4 Science534 (1.8)536 (2.9) Grade 8 Math508 (3.5)506 (3.0) Grade 8 Science520 (3.4)517 (3.0)

Slide 15

Slide 16 Comparison of Model Results to Direct State-Level Estimates Grade/ Subject Model Estimate Mean (SE) Direct Estimate Mean (SE) State = MA Grade 4 Math557 (7.7)572 (3.5) Grade 4 Science562 (4.5)571 (4.3) Grade 8 Math549 (8.0)547 (4.6) Grade 8 Science568 (7.7)556 (4.6) State = MN Grade 4 Math555 (5.5)554 (5.9) Grade 4 Science564 (3.4)551 (6.1) Grade 8 Math549 (6.2)532 (4.2) Grade 8 Science566 (5.9)539 (4.8)

Slide 17 Some Observations on State-Level Estimates  4 th grade math presented as illustration; high correlation in state estimates across grade and subject (r = 0.90 to 0.98)  Median standard errors across states are 6.1 for 4 th grade math, 3.3 for 4 th grade science, 5.6 for 8 th grade math, and 5.4 for 8 th grade science  Large standard errors in HI, AK, and DC estimates reflect large population shares of minorities multiplied through by variance of coefficients of associated variables

Slide 18 Potential Extensions  Current estimates are regression-based only and do not directly combine with existing state-level samples –Composite estimator would weight the two by relative variance, though current sample doesn’t support these estimators well with small state samples  Can validate model against two states (MA and MN) but would be useful to have other states with sufficient samples –Could work with 1999 data that had 13 state samples  Regression / Simulation approach could be used to assess alternative sampling schemes or models, given regression-based assumptions of scores at school level –Could draw larger samples to assess precision of direct estimates and compute composite estimates –Could draw from more high-minority schools to reduce standard errors for some states

Slide 19 Conclusions  Regression model works well with current TIMSS given relatively high explanatory power of regression models  Direct estimates for most states would not be reliable/ credible with current sampling  Sample could be potentially modified to provide direct estimates for states and “borrow strength” from regression and require smaller state samples –Simulation can help evaluate sampling schemes a priori for complex estimators